arXiv · 2025 · Preprint
Jiatong Shi et al. (Carnegie Mellon University / Anuttacon) · → Paper · Demo: ? · Code: ✓
Introduces Speech-DRAME, a framework for evaluating spoken role-play that pairs a human-annotated benchmark and a fine-tuned audio-aware judge (DRAME-Eval) with a system-level benchmark (DRAME-RoleBench) that scores the acoustic delivery, prosody, and character realism of role-play systems’ own generated speech.
Problem
Speech role-play, generating character-consistent spoken responses conditioned on a persona and scene, extends text-based role-play with prosody, emotion, and vocal delivery that text alone cannot capture. Prior evaluation pipelines mostly rely on audio-capable large language models (ALLMs) as zero-shot judges. The authors identify three specific shortcomings of this approach: it lacks sensitivity to paralinguistic cues such as intonation, pacing, and emotional dynamics; it collapses many aspects of role-play quality into coarse, undifferentiated scores; and it depends on synthetic speech references rather than grounding judgments in real human speech, limiting how well it reflects real-world role-play diversity. Existing speech-native role-play resources (e.g. SpeechRole, VStyle) are built largely from LLM-generated dialogue and TTS output with limited human validation, and rely on proprietary ALLM-as-judge pipelines rather than a purpose-trained, human-aligned evaluator.
Method
Speech-DRAME formalizes speech role-play as a conditional generation problem: given dialogue context, a character profile, and a scene specification, a speech foundation model (SFM) produces a spoken role-play response (a waveform or latent audio representation). A speech evaluation model (SEM) then scores that response conditioned on the same character and scene signals; the paper focuses on the single-turn setting, where a response is scored independently of the preceding dialogue, with multi-turn scores obtainable by aggregating single-turn scores across a dialogue.
The framework separates two evaluation strategies. Archetype Evaluation is top-down: it judges role-play against broad stereotypical archetypes (e.g. “firefighter”, “ER doctor”) drawn from seven role categories, using speech collected from a range of end-to-end and cascaded SFMs prompted with audio inputs, scored on three dimensions (Audio Quality, Human-likeness, Appropriateness) after an initial semantic Content Pass filter. Realism Evaluation is bottom-up: it grounds judgments in real speech, curated from licensed media, open-source corpora (e.g. NCSSD), and newly recorded professional and amateur voice-actor performances, plus contrastive negative samples (mismatched scenes and re-synthesized-transcript negatives), scored on ten fine-grained dimensions spanning Prosodic Dynamics (pitch, rhythm, stress), Emotional Expressiveness (accuracy, intensity, transition, with progressive gating), Character Consistency (voice identity, trait embodiment), Contextual Relevance (local/global fit), and Semantic Match. Both strategies rely on standardized human annotation protocols with content screening, rubric-based perceptual scoring, and confidence judgments (§4), yielding DRAME-EvalBench: 8,280 archetype samples and 15,000 realism samples across Mandarin and English (Table 1).
DRAME-Eval, the paper’s evaluation model, is a fine-tuned SEM initialized from Qwen2Audio-7B-Instruct and adapted with LoRA. Because multiple human annotations exist per sample, training randomly samples one annotator’s label per step (a label-distribution-learning strategy), while inference decodes the full distribution over discrete 1-5 labels and takes its expectation as the final score, following decoding-based regression practice. Separate DRAME-Eval models are trained for archetype and realism evaluation. Building on the trained judge, DRAME-RoleBench benchmarks a comprehensive set of SFMs, six end-to-end models (GPT-4o-realtime, KimiAudio, GLM4-Voice, Qwen2.5Omni, Qwen3Omni, StepAudio2Mini) and ten cascaded pipelines (two text LLMs, Gemini2.5Pro and Qwen3-30B-Instruct, each combined with five TTS systems: F5-TTS, IndexTTS2, MaskGCT, Vevo1.5, CosyVoice2), scoring their role-played speech with DRAME-Eval as the automatic judge.
Key Results
On DRAME-EvalBench, DRAME-Eval substantially outperforms zero-shot and few-shot ALLM judges. For archetype evaluation, average Pearson correlation with human ratings rises from 0.480 (best zero-shot, Gemini2.5Pro) and 0.495 (best few-shot, Gemini2.5Pro) to 0.629 for DRAME-Eval, with Content Pass accuracy at 93.6% (Table 2). For realism evaluation, average correlation rises from 0.390 (best zero-shot) and 0.432 (best few-shot) to 0.625 for DRAME-Eval, with consistent gains across all ten dimensions (Table 3). On the harder real-recording realism test set, all methods degrade sharply (zero-shot proprietary models drop to ~0.04-0.08 average correlation), and DRAME-Eval’s advantage over few-shot prompting narrows (0.247 vs. 0.226 average correlation), with some dimensions (e.g. Character Identity) remaining weak (Table 4). A small-scale human study on 10% of the benchmark reports Spearman correlation of 0.706 between human judgments and DRAME-Eval for archetype evaluation, but only 0.375 for realism evaluation, which the authors attribute to realism data being constructed to separate human from synthetic speech rather than to maximize model-level discrimination.
On DRAME-RoleBench, cascaded pipelines generally outperform end-to-end systems on archetype role-play: the strongest cascaded systems (G-IndexTTS2, Q-IndexTTS2) exceed 94% content pass rate with balanced Audio Quality, Human-likeness, and Appropriateness scores, while top end-to-end systems (e.g. GLM4-Voice) still surpass 3.0 average rating and are closing the gap (Table 5). On realism role-play, the end-to-end/cascaded gap narrows considerably, with several end-to-end models (GLM4-Voice, Qwen2.5Omni) matching or exceeding the best cascaded pipelines at ~3.24 average, and most models clustering within a narrow 2.9-3.2 band across prosodic, emotional, character, and contextual dimensions (Table 6).
Novelty Assessment
The core contribution is a dataset and evaluation-methodology package rather than a new generative architecture: DRAME-Eval reuses an existing audio-LLM backbone (Qwen2Audio-7B-Instruct) with standard LoRA fine-tuning. The genuinely new elements are (i) the dual archetype/realism evaluation taxonomy grounded in sociological role theory, applied for the first time to speech role-play with matched human-annotation protocols for both strategies; (ii) a large, bilingual, human-annotated benchmark that includes real (not just TTS-synthesized) professional and amateur role-play recordings as a distinct hard test condition; and (iii) a sampling-based label-distribution training and decoding-based regression recipe for aligning an audio-LLM judge to multi-annotator perceptual ratings. The system-level benchmark (DRAME-RoleBench) is an application of this evaluation infrastructure to compare current SFMs and TTS-based cascaded pipelines, rather than a novel modeling contribution in its own right.
Field Significance
Tip
High — This paper directly evaluates the generated speech of role-play systems, judging Audio Quality, Human-likeness, and fine-grained prosodic and emotional delivery from audio input, validated by real human listening studies rather than by text-only correctness. It provides a human-validated evaluation model as well as a benchmark dataset and protocol, releasing data, model, and code, which together give researchers reusable infrastructure for training and testing speech evaluation models beyond this paper’s own results. It also documents a clear synthetic-to-real generalization gap (Table 3 vs. Table 4) that motivates future work on evaluation robustness.
Claims
- supports: A judge model fine-tuned on human-annotated perceptual ratings of generated speech can substantially outperform zero-shot and few-shot audio-LLM judges at predicting those same human ratings.
Evidence: DRAME-Eval raises average Pearson correlation with human archetype ratings from 0.480 (zero-shot Gemini2.5Pro) to 0.629, and realism ratings from 0.390 to 0.625, after LoRA fine-tuning Qwen2Audio-7B-Instruct on DRAME-EvalBench. (§5.2, Tables 2-3)
- complicates: Evaluators trained on synthetic or curated role-play speech generalize poorly to real, unscripted human performance.
Evidence: On the real-recording realism test set, all models, including the fine-tuned DRAME-Eval, drop to 0.04-0.25 average correlation with human ratings, versus 0.625 on the synthetic/curated realism test set, with the fine-tuning advantage over few-shot prompting shrinking sharply. (§5.2, Table 4)
- complicates: Coarse, stereotype-driven (archetype) evaluation and fine-grained, human-grounded (realism) evaluation of speech role-play can yield different rankings and different levels of automatic-judge reliability, so a single evaluation strategy is not sufficient to characterize role-play quality.
Evidence: Human-judgment alignment with DRAME-Eval is 0.706 Spearman correlation for archetype evaluation but only 0.375 for realism evaluation, and system rankings shift: cascaded TTS pipelines lead clearly on archetype scoring while end-to-end models match or exceed them on realism scoring. (§6, “Human alignment”; Tables 5-6)
- supports: Cascading a text LLM with a dedicated zero-shot TTS system can still outperform end-to-end spoken dialogue models on stereotype-driven role-play delivery, though the gap is narrowing.
Evidence: On DRAME-RoleBench archetype evaluation, cascaded pipelines using IndexTTS2 (G-IndexTTS2, Q-IndexTTS2) achieve the highest content pass rates (94.3%, 94.7%) and average scores, while the best end-to-end model (GLM4-Voice, 88.5% content pass) trails slightly but already exceeds 3.0 average rating. (§6, Table 5)
Limitations and Open Questions
The realism evaluator shows weak agreement with human judgment (0.375 Spearman) and poor generalization to real, unscripted recordings (0.04-0.25 average correlation), which limits how much the current DRAME-Eval realism scores can be trusted as a proxy for human perception outside the curated synthetic/negative-sample setting they were trained on.
The single-turn evaluation formulation does not directly model multi-turn dialogue quality; the paper defers a full treatment of multi-turn aggregation to an appendix and only validates single-response scoring. The human-alignment study is small-scale (10% of the benchmark), and Content Pass filtering, which screens for basic instruction compliance and audio validity before rubric scoring, is not itself validated against human judgment in the main text. Archetype role-play data is drawn from SFM outputs rather than real human archetype performances, so archetype evaluation remains partly synthetic-referenced even though realism evaluation was designed to address that gap.
Wiki Connections
- Evaluation Metrics — introduces a human-validated evaluation-model training recipe (label-distribution sampling plus decoding-based regression) as an alternative to zero-shot ALLM-as-judge scoring for speech role-play.
- Subjective Evaluation — grounds both the training data and a held-out validation study in real human rubric-based ratings of generated and real speech, rather than relying solely on automated scoring.
- Spoken Language Model — benchmarks a range of end-to-end audio LLMs (GPT-4o-realtime, KimiAudio, GLM4-Voice, Qwen2.5Omni, Qwen3Omni, StepAudio2Mini) on their spoken role-play ability.
- Speech-to-Speech — evaluates spoken role-play as an audio-in, audio-out dialogue task, comparing end-to-end systems against cascaded LLM-plus-TTS pipelines.
- Zero-Shot TTS — benchmarks five zero-shot TTS systems (F5-TTS, IndexTTS2, MaskGCT, Vevo1.5, CosyVoice2) as the speech-rendering component of cascaded role-play pipelines.
- Emotion Synthesis — scores generated role-play speech on emotional expressiveness sub-dimensions (accuracy, intensity, transition) as part of its realism evaluation rubric.
- SpeechRole — cited as the closest prior speech role-play resource, critiqued for relying on LLM-generated dialogue and ALLM-as-judge pipelines with limited human validation, which Speech-DRAME directly addresses.
- VStyle — cited alongside SpeechRole as a closely related benchmark for expressiveness and fidelity in spoken role-play/instruction-following.
- IndexTTS2 — used as the top-performing TTS component in cascaded pipelines benchmarked on DRAME-RoleBench archetype evaluation.
- F5-TTS — one of five zero-shot TTS systems compared as cascaded pipeline components on DRAME-RoleBench.
- CosyVoice 2 — one of five zero-shot TTS systems compared as cascaded pipeline components on DRAME-RoleBench.
- MaskGCT — one of five zero-shot TTS systems compared as cascaded pipeline components on DRAME-RoleBench.
- GLM-4-Voice — benchmarked as an end-to-end spoken dialogue model on both archetype and realism role-play.
- Qwen2.5-Omni — benchmarked as an end-to-end spoken dialogue model on both archetype and realism role-play.