EMNLP · 2025 · Conference
Ruiqi Yan et al. (Shanghai Jiao Tong University) · → Paper · Demo: ✗ · Code: ✓
Introduces a 40-dataset, two-track benchmark that evaluates end-to-end spoken dialogue models across understanding, reasoning, and oral conversation, combining content-quality scoring with dedicated speech-quality, speech-text alignment, and latency metrics, and validates the automatic scores against human listening judgments.
Problem
End-to-end speech-to-speech (S2S) dialogue models remove the cascaded ASR-LLM-TTS pipeline, reducing latency while improving naturalness and context understanding, and increasingly need to handle multilingual input and complex audio beyond speech (emotion, music, ambient sound). However, evaluation frameworks for these models have lagged behind their capabilities. Prior benchmarks each cover only a slice of what a spoken dialogue model needs to do: AIR-Bench and SD-Eval test audio understanding but score only text output; VoiceBench adds multi-round dialogue but likewise evaluates only textual responses; ADU-Bench adds multilingual and speech-to-speech scope but omits multi-turn conversation. None combine cognitive evaluation (reasoning, knowledge, instruction-following) with speech-specific evaluation (paralinguistic understanding, output speech quality) across both English and Chinese, and none test the generated speech itself rather than its transcribed content alone.
Method
URO-Bench evaluates spoken dialogue models along three axes summarized in its name: Understanding, Reasoning, and Oral conversation. The benchmark is organized into two difficulty tiers, each comprising 20 test sets (40 total): a basic track (ten English, ten Chinese test sets covering everyday QA, summarization, listening comprehension, and casual chat) and a pro track (eleven English, eight Chinese, and one seven-language multilingual test set covering multi-round dialogue, code-switching, speech emotion, environmental sound and music comprehension, speaker awareness, singing/recitation, and cross-lingual conversation).
Test data is constructed through a six-stage pipeline: task curation against the U/R/O taxonomy; data selection from existing textual benchmarks (e.g., TruthfulQA, OpenBookQA, GSM8K, MT-Bench-101) and LLM-generated conversational scenarios; filtering of content unsuitable for TTS (code, equations, symbols); speech synthesis using state-of-the-art TTS systems (CosyVoice, CosyVoice2, F5-TTS, and GPT-4o-Audio-Preview for emotion- and speaker-controlled subsets); an ASR-based filtering pass (Whisper-large-v3 transcription compared against source text, discarding samples above a 2% WER/CER threshold); and a final manual review focused on paralinguistic quality and scenario appropriateness. A curated 1,000-sample “miniset” (25 samples per dataset) supports fast iteration before full-scale evaluation.
Four evaluation metrics are defined per response: a task accomplish score (0-100) that, depending on the test set, is computed either by transcribing the response with an ASR model and scoring the transcript with GPT-4o mini against custom accuracy/relevance/clarity/completeness criteria, by direct word-error comparison for instruction-repetition tasks, or by scoring the response audio directly with an audio-capable judge (Gemini 2.0 Flash, GPT-4o-Audio-Preview) or emotion2vec for tasks requiring output expressiveness; UTMOS, an automatic mean-opinion-score predictor applied to the SDM’s generated speech to measure clarity, naturalness, and fluency; ASR-WER/CER between the response transcription and the response text, used as a proxy for speech-text alignment and generation artifacts (repetition, mispronunciation, long pauses); and first packet latency, measured on a subset of the AlpacaEval-derived test sets.
The paper benchmarks six open-source SDMs (Mini-Omni, Mini-Omni2, SLAM-Omni, Freeze-Omni, LLaMA-Omni, GLM-4-Voice) plus GPT-4o-Audio-Preview, alongside cascaded Whisper-large-v3 + LLM baselines using the same backbone LLMs (Qwen2, Llama-3.1, GLM-4, GPT-4o) to isolate how much capability is lost when a text-based LLM is adapted into an end-to-end speech-in speech-out system.
Key Results
On content-quality (task accomplish) scores, GLM-4-Voice is the strongest open-source SDM, outperforming the next-best open-source models by at least 10 points per dimension in the basic track, while small-backbone models (Mini-Omni, Mini-Omni2, SLAM-Omni, all built on a 0.5B LLM) average roughly 25 points. Every open-source SDM trails its own cascaded Whisper+LLM counterpart substantially in reasoning: GLM-4-Voice scores 55.46 on basic-track English reasoning versus 76.29 for Whisper + GLM-4-9B-Chat-HF using the identical backbone LLM (Table 4). GPT-4o-Audio-Preview leads all open-source SDMs but still trails Whisper + GPT-4o on most basic-track dimensions.
On speech quality (Table 5), UTMOS and ASR-WER/CER show that smaller models with less diverse output tend to score better on speech-text alignment (e.g., SLAM-Omni: 4.05% WER, UTMOS 4.45 on English) than larger models, which produce more varied but occasionally disfluent audio (GLM-4-Voice: 11.12% WER on English; Freeze-Omni: 20.88% WER). Pro-track results show open-source SDMs perform close to random guessing on environmental sound and music comprehension, extremely weak speaker-awareness (distinguishing speakers by timbre/pitch), and multilingual support limited to at most two languages, whereas GPT-4o-Audio-Preview shows a wider (though still imperfect) multilingual range and stronger non-speech audio understanding.
Novelty Assessment
The contribution is an evaluation resource, not a new model or training method. Its genuine novelty lies in being the first benchmark to jointly cover speech-to-speech format, multilingualism, multi-round dialogue, and both input and output paralinguistics in one framework (Table 1 explicitly positions this against AIR-Bench, SD-Eval, VoiceBench, and ADU-Bench, none of which cover all five dimensions), and in defining an evaluation protocol that scores the SDM’s actual generated audio (via UTMOS, audio-aware judges, and emotion2vec) rather than only its transcribed text content, which several predecessor benchmarks (SD-Eval, VoiceBench) explicitly do not do. The data-construction pipeline itself (TTS synthesis, ASR-based filtering, manual review) is a careful engineering integration of existing tools rather than a novel technique.
Field Significance
Tip
High — URO-Bench closes a concrete gap in spoken dialogue model evaluation by combining cognitive assessment with direct scoring of generated speech (via UTMOS and audio-aware judges) across two languages and difficulty tiers, and validates its automatic scoring pipeline against human listening judgments rather than assuming automatic-human agreement.
The paper’s diagnostic findings (catastrophic forgetting of backbone LLM capability after speech-interaction fine-tuning, near-random performance on non-speech audio understanding, weak paralinguistic and multilingual capability) give the field concrete, benchmarked evidence of where current end-to-end SDMs fall short relative to their text backbones and to cascaded or proprietary alternatives. Its significance is bounded by evaluating only six open-source SDMs available at the time and by relying on LLM-based judges for most content scoring, albeit with a human-validation study.
Claims
- supports: Automatic scoring of a spoken dialogue model’s response, whether by transcribing and LLM-judging the content or by scoring the response audio directly with an audio-aware judge or embedding model, correlates strongly with human listening judgments, supporting its use as a scalable proxy for human evaluation.
Evidence: Pairwise preference matching rates between GPT-4o mini scoring and human listeners exceed 70% on open-ended dialogue tasks and 90% on QA tasks, and rule-based (emotion2vec + WER) scoring for emotion generation matches human preference at 80% on average. (§4.3, Appendix B.3, Tables 10-13)
- complicates: Adapting a text-based LLM into an end-to-end speech-in speech-out dialogue model substantially degrades the reasoning and knowledge-retrieval capability the backbone LLM had in text form.
Evidence: On basic-track English reasoning, GLM-4-Voice scores 55.46 versus 76.29 for a cascaded Whisper + GLM-4-9B-Chat-HF baseline using the identical backbone LLM; other SDMs show similarly large drops relative to their backbones on datasets like MLC, Gsm8kEval, and OpenbookQA-zh. (§4.2, Table 4)
- complicates: Current end-to-end spoken dialogue models fail to leverage non-lexical acoustic information (environmental sound, music, speaker identity via timbre and pitch) even when their lexical content understanding is reasonable.
Evidence: All evaluated open-source SDMs perform close to random guessing on environmental sound and music comprehension tasks (ClothoEval-en, MuChoEval-en), and their ability to distinguish speakers by paralinguistic cues in multi-turn dialogue (SpeakerAware) is described as extremely weak; a cascaded ASR + audio-captioner + LLM baseline substantially outperforms the strongest SDM (GLM-4-Voice) on the same tasks, 64.00 vs 17.36 on ClothoEval. (§4.2 Pro Track discussion, Appendix B.2, Table 8)
- complicates: Larger spoken dialogue models with more diverse generated output trade off speech-text alignment fidelity against content quality, producing higher ASR-derived word/character error rates and lower automatic speech-quality scores than smaller, less diverse models.
Evidence: GLM-4-Voice (9B backbone) reaches 11.12% English ASR-WER versus SLAM-Omni’s (0.5B backbone) 4.05%, attributed to larger models being more prone to long pauses and repetitions in generated audio; UTMOS follows the same pattern, with GLM-4-Voice (4.15) below SLAM-Omni (4.45). (§4.2 Speech Quality and Speech-text Alignment, Table 5)
Limitations and Open Questions
Warning
Most content-quality scoring depends on an LLM judge (GPT-4o mini) applied to ASR transcriptions of the SDM’s response rather than on native audio understanding of the judge itself; while the paper validates this against human judgments (matching rates above 70-90%), residual judge bias and ASR transcription errors remain unquantified confounds for the reported task accomplish scores.
The benchmark evaluates only six open-source SDMs available at the time of construction, all released before mid-2025; newer or larger systems are untested. First packet latency could not be measured automatically for all systems due to source-code modification requirements, limiting comparability of that metric. The authors note LLM-based scoring may still exhibit bias and fluctuation, and that the high API cost of audio-capable judges (Gemini 2.0 Flash, GPT-4o-Audio-Preview) constrained the size of the singing/recitation test sets, which the paper flags as needing alternative evaluators in future work.
Wiki Connections
- Spoken Language Model — evaluates six open-source end-to-end spoken dialogue models built on LLM backbones, diagnosing how much reasoning capability each retains relative to its text-only backbone.
- Speech-to-Speech — targets genuinely end-to-end speech-in speech-out dialogue systems specifically, distinguishing them from cascaded ASR-LLM-TTS pipelines throughout its evaluation design.
- Evaluation Metrics — introduces a four-metric evaluation protocol (task accomplish score, UTMOS, ASR-WER/CER, first packet latency) spanning content quality, speech quality, speech-text alignment, and responsiveness.
- Subjective Evaluation — conducts human listening evaluations on the SDM’s generated speech responses (pairwise preference and QA correctness) specifically to validate the benchmark’s automatic scoring pipeline.
- GLM-4-Voice — evaluated as the strongest open-source SDM across most test sets, attributed to its stronger backbone LLM and larger training data.
- LLaMA-Omni — evaluated as one of six open-source SDMs, showing moderate capability but a large reasoning gap relative to its backbone.
- SLAM-Omni — evaluated as one of six open-source SDMs; its small backbone yields the best speech-text alignment (lowest ASR-WER) despite low content-quality scores.
- Freeze-Omni — evaluated as one of six open-source SDMs, used as one side of the pairwise human-evaluation comparison against GLM-4-Voice.
- Mini-Omni — evaluated as one of six open-source SDMs, among the lowest-scoring systems due to its 0.5B backbone.
- VoiceBench — a predecessor SDM benchmark whose AlpacaEval and CommonEval test sets URO-Bench directly incorporates, while extending its text-only scoring to also assess generated speech.
- AIR-Bench — a predecessor audio-understanding benchmark whose pairwise human-evaluation methodology URO-Bench adopts for its open-ended dialogue test sets.
- emotion2vec — used as one of the automatic scoring tools for expressive speech-emotion generation tasks, validated against human preference judgments.