arXiv · 2025 · Preprint
Wu et al. (Tsinghua University) · → Paper · Demo: ? · Code: ?
VoxRole introduces the first benchmark for speech-based role-playing conversational agents, comprising 13,335 multi-turn dialogues from 1,228 characters across 261 movies, paired with a multi-dimensional evaluation framework that assesses persona consistency across paralinguistic and semantic dimensions.
Problem
Evaluation of speech-based role-playing conversational agents (RPCAs) has lacked rigorous benchmarks. Existing spoken dialogue datasets target fundamental capabilities such as ASR quality and logical reasoning, but feature thinly sketched character profiles and fail to measure long-term persona consistency. Text-based RPCA research has advanced considerably, but the speech dimension — prosody, intonation, vocal timbre, and rhythm — has been systematically ignored despite these cues being essential to character identity. The absence of standardized evaluation resources has prevented systematic comparison of spoken dialogue models on core role-playing competencies.
Method
VoxRole is a benchmark dataset constructed via a two-stage automated pipeline that avoids manual annotation. In the first stage, movie audio is aligned with screenplays using forced word-level alignment: audio tracks are denoised with Resemble, transcribed with Whisper-large-v3, then aligned using Wav2Vec2.0 to obtain word-level timestamps. A dynamic edit-distance algorithm matches transcribed words to script dialogue, and semantic similarity via MPNet filters confirmed matches at a threshold of 0.8 to handle actor paraphrasing and improvisation. Consecutive two-speaker segments of at least three turns are retained; movies with less than 5 minutes of matched audio are excluded, yielding 261 movies with 1,228 characters and 13,335 dialogue sessions totalling 65.6 hours.
In the second stage, character personas are distilled from the aligned scripts using a two-pass LLM pipeline. Characters are represented across four dimensions: personality, linguistic style, interpersonal relationships, and acoustic characteristics. Acoustic features (mean pitch, energy, speech rate) are extracted per character and discretized into high/medium/low bins. Personality and relationships are extracted from scene-level event summaries generated by an LLM; linguistic style is inferred from direct dialogue lines. The four dimensions are synthesised into a second-person narrative prompt that serves as the role-playing instruction.
The evaluation framework is dual-mode. Metric-based evaluation uses Rouge-L, Meteor, and BertScore-F1 for text response quality, and UTMOSv2 for synthesised speech naturalness. LLM-based evaluation adds an acoustically-aware judge that transcribes the model’s spoken response with Whisper, extracts per-sentence emotion labels via Emotion2Vec, and augments the text input with pitch, energy, and speaking rate bins before judging. The judge scores responses on six dimensions: Human-Likeness, Personality Consistency, Linguistic Fidelity, Relational Coherence, Contextual Coherence, and Paralinguistic Appropriateness.
Key Results
Seven models are evaluated: five open-source (Qwen2.5-Omni 7B, GLM-4-Voice 9B, Baichuan-Audio 7B, MiniCPM 8B, Step-Audio 132B) and two proprietary (GPT-4o, Gemini-2.5-flash). GPT-4o leads across both metric-based and LLM-based evaluations, achieving Rouge-L 12.91, Meteor 18.30, and UTMOS 3.66 (Table 2), as well as an LLM-based overall score of 4.28 (Table 3). Among open-source models, Qwen2.5-Omni achieves UTMOS 3.57, nearly matching GPT-4o in speech naturalness despite being 7B parameters. Step-Audio at 132B achieves the highest BertScore F1 (84.16) but scores poorly on speech naturalness (UTMOS 2.42), a clear trade-off between semantic accuracy and synthesis quality.
Across the LLM-based dimensions, Contextual Coherence is the strongest capability for all models (GPT-4o: 4.48), while Acoustic (Paralinguistic) Appropriateness is universally the weakest (GPT-4o: 3.82, best open-source Step-Audio: 3.43). The gap between proprietary and open-source models is most pronounced in Personality Consistency and Relational Coherence — GPT-4o outperforms the best open-source model by over 15% on these dimensions. Human evaluation on 20 sampled instances yields a Pearson correlation of 0.762 between LLM judge scores and human annotator scores, supporting the validity of the automated evaluation approach. An ablation over context window length (4–10 turns) shows a non-monotonic trend: a context of 6 turns achieves the best overall score, suggesting diminishing returns from additional history.
Novelty Assessment
The primary contribution is a dataset and evaluation framework, not an architectural advance. The dataset construction pipeline itself — audio-script alignment followed by LLM-based persona distillation — is methodologically interesting and reasonably novel in combining existing tools (Whisper, Wav2Vec2.0, MPNet) into a scalable, annotation-free workflow. The acoustically-aware LLM judge that augments text with discrete acoustic feature bins is a useful extension of the standard text-only judge paradigm and addresses a genuine gap in spoken dialogue evaluation. The underlying components are individually established, but their composition for role-playing benchmarking fills a real void in the evaluation landscape. The benchmark’s coverage — 261 movies, 1,228 characters, multi-dimensional persona profiles — is substantive. The main limitation is that movie dialogue is a somewhat stylised source domain.
Field Significance
Moderate — VoxRole addresses a genuine gap: the evaluation of paralinguistic persona fidelity in spoken dialogue systems is an underserved problem, and the field has lacked a benchmark that combines rich character profiles with audio grounding. The acoustically-aware LLM evaluation paradigm that incorporates emotion labels and acoustic bins alongside transcribed text is a useful methodological contribution that other spoken dialogue benchmarks could adopt. The finding that open-source models consistently underperform on Personality and Relational Coherence relative to proprietary systems, while acoustic quality is a universal bottleneck even for the strongest models, provides useful directional evidence for where investment is needed.
Claims
- Acoustic quality (paralinguistic appropriateness) is the most consistently underperformed capability across current spoken dialogue models, even for proprietary systems, lagging well behind text generation and reasoning dimensions. (§Experiments: LLM-based Result, Table 3)
- Model parameter count does not reliably predict speech synthesis quality in spoken dialogue systems: smaller, better-trained models can match or exceed much larger counterparts on naturalness metrics. (§Experiments: Metric-based Results, Table 2)
- LLM-based evaluation augmented with discrete acoustic features achieves strong alignment with human judgment on multi-dimensional spoken role-playing tasks. (§Evaluation Framework, §Experiments: Subjective experimental results)
- There is an optimal context window length for spoken role-playing models beyond which additional conversational history degrades rather than improves performance. (§Experiments: Ablation study on context length, Table 4)
Limitations and Open Questions
Warning
The benchmark is sourced exclusively from movie dialogue, which is scripted and stylistically distinct from spontaneous conversational speech. Generalisation of the evaluation findings to natural speech interaction scenarios is not established.
Character quality validation covers only 20 characters rated by 5 annotators — the 93% satisfactory/acceptable rate is encouraging but the sample is small. The human evaluation for judge validation similarly covers only 20 dialogue instances. The benchmark is currently restricted to two-speaker conversations, limiting applicability to multi-party or group dialogue settings. Future work noted by the authors includes expanding the dataset and fine-tuning models on the role-playing task, but neither training data nor fine-tuning baselines are provided in this paper.
Wiki Connections
Concept pages most informed by this paper: spoken-language-model, evaluation-metrics, subjective-evaluation, emotion-synthesis, prosody-control.
In-corpus papers cited: 2503.04721 (FullDuplexBench, a complementary spoken dialogue benchmark), 2502.17810 (URO-Bench, another spoken dialogue evaluation benchmark).