arXiv · 2025 · Preprint
Jiang et al. (Fudan University) · → Paper · Demo: ? · Code: ✓
SpeechRole introduces a large-scale speech-to-speech dataset (111k dialogues, 98 roles) and a multidimensional benchmark for evaluating Speech Role-Playing Agents, revealing that interaction quality and role-playing fidelity in current systems are primarily determined by the underlying text LM rather than the speech modality.
Problem
Role-playing agent research has overwhelmingly focused on text, with speech-based character interaction remaining poorly resourced and unevaluated. Existing benchmarks such as RoleEval and INCHARACTER assess behavioural consistency in text, but ignore paralinguistic cues — pitch, rhythm, intonation, timbre — that are central to convincing spoken character portrayal. No large-scale speech-to-speech corpora with controlled role diversity existed for training or evaluating Speech Role-Playing Agents (SRPAs), and no automatic evaluation protocol existed that could assess spoken output without first transcribing it through ASR (and thereby discarding prosodic and emotional information in the process).
Method
SpeechRole addresses both the data and evaluation gaps through a unified framework with two components.
SpeechRole-Data is constructed via a synthetic pipeline: 98 characters sourced from television dramas, films, animations, and games have their dialogue scripts processed by GPT-4.1 to generate approximately 800 single-turn and 800 multi-turn dialogues per role. Reference voices are extracted from source audio using the Emilia framework (source separation, diarization, VAD), cleaned with DNSMOS P.835 quality filtering, and mapped to characters via Whisper transcription and LLM identity inference. A representative reference voice per role is selected by ranking clips against an intra-role speaker embedding centroid using CAM++. Role speech is then synthesised using three TTS systems — CosyVoice2, F5-TTS, and E2-TTS — each conditioned on the reference clip, yielding multiple speech variants per dialogue to reduce model-specific artefacts. User utterances are generated with a fixed voice (VolcEngine TTS). The resulting dataset covers 111k speech-to-speech dialogues, split into a 78-role training set and a 20-role out-of-domain test partition, with a held-out 392-instance test set across all 98 roles.
SpeechRole-Eval defines three evaluation dimensions: interaction ability (Instruction Adherence, Conversational Coherence), speech expressiveness (Speech Fluency, Naturalness, Prosodic Consistency, Emotion Appropriateness), and role-playing fidelity (Personality Consistency, Knowledge Consistency). The protocol is pairwise: an LLM judge (Gemini-2.5-Pro) compares a candidate response against three synthesised references drawn from different TTS backends. Scores are computed as a ratio of candidate to reference ratings, then averaged across references and instances. This multi-reference ratio design is intended to normalise for reference-specific variation. Human-LLM agreement is validated via Spearman’s ρ and Kendall’s τ against expert annotators.
Eight systems are benchmarked: four cascaded (Qwen3-8B, Llama-3.1-8B, Mistral-7B, plus an Alibaba Cloud API) and three end-to-end (GPT-4o Audio, Qwen2.5-Omni-7B, LLaMA-Omni), plus SpeechRole-Agent — a fine-tuned Qwen2.5-Omni trained on SpeechRole-Data.
Key Results
GPT-4o Audio leads on speech fluency (SF: 1.098) and naturalness (SN: 1.021) in English, but trails cascaded systems on prosody consistency (PC: 0.900 vs 0.966 for Llama-3.1-8B) and emotion appropriateness (EA: 0.816 vs 0.964). Cascaded systems built on strong text LMs — particularly Llama-3.1-8B — achieve the best overall English scores (0.957), outperforming GPT-4o Audio overall (0.921). Open-source end-to-end models (Qwen2.5-Omni, LLaMA-Omni) fall well below cascaded baselines across all dimensions, with Qwen2.5-Omni scoring 0.568 overall in English.
Fine-tuning Qwen2.5-Omni on SpeechRole-Data substantially improves the resulting SpeechRole-Agent (0.801 English overall vs 0.568 for the base model). Gains extend to unseen roles (OOD: 0.783), suggesting the benefit is not purely role memorisation. In Chinese, SpeechRole-Agent achieves 0.936 overall, competitive with cascaded systems.
Human-LLM agreement is strong: Spearman’s ρ ranges from 0.67 to 0.98 across metrics in English and 0.64 to 0.96 in Chinese, with Human-LLM Agreement rates exceeding 0.75 for most dimensions.
Note
All evaluation references are TTS-synthesised rather than natural human recordings, so benchmark scores reflect performance relative to synthetic reference speech rather than against human-quality ground truth.
Novelty Assessment
The primary contributions are the dataset and the evaluation framework. Neither is architecturally novel: the data construction pipeline is a competent application of existing tools (GPT-4.1 for dialogue generation, CosyVoice2/F5-TTS/E2-TTS for voice synthesis, Emilia for audio processing). The evaluation protocol adopts pairwise LLM-as-judge methodology, which is established practice; the multi-reference ratio scoring is a reasonable engineering refinement but not a conceptual advance. The empirical finding that interaction quality is largely driven by the text LM irrespective of cascaded vs. end-to-end architecture is interesting and practically important, though not entirely surprising given that paralinguistic modelling in end-to-end systems is still maturing. The framework’s value lies in providing a reproducible, scalable benchmark where none previously existed for spoken role-playing.
Field Significance
Moderate — SpeechRole fills a genuine gap by providing the first large-scale speech-to-speech dataset and reproducible automatic benchmark for Speech Role-Playing Agents. The finding that role-playing fidelity and interaction quality are dominated by the underlying text LM offers concrete diagnostic evidence for where the field needs to invest, rather than treating the cascaded vs. end-to-end distinction as the primary axis of progress. The benchmark itself is likely to become a standard reference point for subsequent SRPA work, which elevates the paper’s contribution beyond what the methodology alone would suggest.
Claims
- The quality of spoken role-playing in both cascaded and end-to-end systems is more strongly determined by the underlying language model’s text reasoning ability than by the speech synthesis or speech-to-speech architecture. (§5.2)
- End-to-end speech-language models exhibit larger gains from task-specific fine-tuning in expressive speech dimensions than in interaction or role fidelity dimensions, which remain text-LM bottlenecked. (§5.2)
- LLM-based pairwise evaluation of speech quality can achieve high rank correlation with human expert judgements when using multi-reference normalisation, making automated SRPA benchmarking feasible at scale. (§5.3, Table 3)
- Current open-source end-to-end speech models lag proprietary cascaded systems in prosody consistency and emotion appropriateness by margins large enough to constitute a capability gap, not merely a performance gap. (§5.2, Table 2)
Limitations and Open Questions
Warning
All reference speech in SpeechRole-Eval is TTS-synthesised rather than recorded from human actors, which means benchmark scores measure performance relative to synthetic reference quality rather than against a naturalness ceiling. This design choice limits the benchmark’s ability to surface failures specifically in human-likeness.
Coverage is limited to English and Chinese, constraining generalisability to other languages and speech communities. The evaluation judge (Gemini-2.5-Pro) is proprietary, introducing a dependency on a black-box system whose biases are not fully characterised; developing open-source alternatives is noted as future work. The benchmark does not assess long-turn character consistency, adversarial or ambiguous instruction handling, or fine-grained prosodic controllability — dimensions relevant for real deployment. SpeechRole-Agent is built by fine-tuning an existing end-to-end model rather than exploring alternative architectures or training objectives, leaving open how much of the performance gap between open-source end-to-end and cascaded systems is architectural versus a training data and scale question.