EMNLP · 2025 · Conference
Chengqian Ma et al. (Peking University / LIGHTSPEED) · → Paper · Demo: ? · Code: ✓
Introduces a bilingual (English/Chinese) benchmark of 1,079 real-world spoken dialogue instances, grounded in a corpus-level literature review and statistical analysis, that specifically targets phonological and semantic ambiguity, omission, coreference, and multi-turn interaction, phenomena the authors show existing spoken dialogue benchmarks largely omit.
Problem
End-to-end spoken dialogue models (SDMs) that process voice input and generate voice output are proliferating (GPT-4o-Audio-Preview, Moshi, GLM-4-Voice, Qwen2.5-Omni, and others), but benchmarking of their conversational competence lags far behind text-based LLM evaluation. Prior SDM benchmarks (ADU-Bench, AIR-Bench, SUPERB, AudioBench, SD-Eval, VoiceBench) each cover only a subset of what makes spoken conversation genuinely hard: none jointly cover phonological ambiguity (heterographs, heteronyms, stress, intonation, tone), none address comprehension difficulties from coreference and omission, most use single-language (English) data even when claiming multilingual coverage (translated rather than natively collected), and none evaluate multi-turn dialogue handling. The paper argues that spoken dialogue is inherently more complex than text dialogue because ambiguity and context-dependency manifest through channels (prosody, phonetics) that text-only evaluation cannot capture, and that no existing benchmark isolates these phenomena to measure SDMs’ practical effectiveness.
Method
The authors first conduct an empirical grounding study before designing the benchmark: using CABank (English) and MagicData-RAMC (Chinese) real-world spoken dialogue corpora, they quantify the prevalence of five target phenomena (phonological ambiguity, semantic ambiguity, omission, coreference, multi-turn interaction) via corpus statistics (e.g., 99.25% of Chinese characters have tonal homophones, 41.14% of English dialogue sentences show syntactic ambiguity, subject omission occurs in 16.51% of Chinese and 2.42% of English utterances) to justify that each phenomenon is a real, frequent challenge rather than a contrived edge case.
Building on this analysis, the benchmark (named C3, informally “C-cubed”) collects real-world spoken dialogue data exhibiting each phenomenon from web sources and existing text datasets (RISAWOZ for omission/coreference in Chinese, the Winograd Schema Challenge for coreference in English, CrossWoz and DailyDialog for multi-turn interaction). Each dialogue is converted into a unified question-answer instance: a phenomenon-specific prompt is paired with the dialogue audio, and a manually produced reference answer defines the expected SDM response. To remove confounds from timbre and background noise, dialogue audio is regenerated with a TTS system (Seed-TTS) with a unified voice and no background noise; a manual quality pass replaces poorly synthesized instances (e.g., ambiguous pause and stress cues in the phonological subset) with human-voiced recordings. The dataset totals 1,079 instances (1,586 audio-text pairs after expanding multi-turn dialogues into per-turn samples), split into an ambiguity set (Cam-data: phonological + semantic) and a context-dependency set (Ccon-data: omission + coreference + multi-turn), with the latter explicitly filtered to exclude instances that would also introduce ambiguity.
Ten end-to-end SDMs are evaluated (Freeze-Omni, GLM-4-Voice, GPT-4o-Audio-Preview, Kimi-Audio, LLaMA-Omni, MooER-Omni, Moshi, Qwen2.5-Omni, Step-Audio, VITA-Audio); cascaded ASR+LLM+TTS pipelines are excluded because ASR discards the phonetic cues (intonation, stress) needed to interpret ambiguity. Moshi is excluded from the multi-turn comparison because its full-duplex interruption behavior makes it incomparable under the same protocol; LLaMA-Omni and Moshi are evaluated in English only since they lack Chinese support. Evaluation is primarily performed on the transcribed text of each SDM’s response (via Whisper for the one model, Moshi, that does not emit paired text) using an LLM-as-a-judge protocol with GPT-4o and DeepSeek-R1 as judges, whose scores are averaged; accuracy is the fraction of instances judged correct. The one exception is the phonological “Generation” subtask, where the SDM is asked to produce a corrected prosodic realization (correct pause placement, stress, intonation, or corrected pronunciation for heteronyms/tonal homophones): because these phonological features are not recoverable from the transcript, three human experts directly listen to and label each generated audio response as correct or incorrect, with majority voting resolving disagreement. A separate reliability analysis validates the LLM-judge protocol against this same human-labeling procedure applied to GPT-4o-Audio-Preview’s text responses, reporting Pearson/Spearman/Kendall correlation coefficients between LLM-judge and human-expert scores.
Key Results
Averaging GPT-4o and DeepSeek-R1 judge scores, GPT-4o-Audio-Preview leads the English subset with 55.68% overall accuracy (average across all ten SDMs: 35.15%), while Qwen2.5-Omni leads the Chinese subset with 40.08% (average: 23.33%). Accuracy on the ambiguity set (Cam-data) is substantially lower than on the context-dependency set (Ccon-data) in both languages: 12.21% vs. 31.22% in Chinese and 27.91% vs. 40.22% in English, a gap exceeding 10 percentage points in each language. Semantic ambiguity is the hardest category overall, with Chinese accuracy at only 3.97% versus 26.86% in English (Table 2). Within Ccon-data, omission is consistently the hardest phenomenon for nearly every model, and granular detection-vs-completion/resolution comparisons (Table 3) show that most SDMs detect the presence of omission or coreference far more reliably than they can complete or resolve it (e.g., overall omission detection reaches 38.65% in Chinese vs. only 3.75% for completion). Cross-linguistically, accuracy on every phenomenon except omission is higher in English than Chinese, with gaps exceeding 8 percentage points and, for some models (Freeze-Omni, GLM-4-Voice), more than doubling English performance relative to Chinese. Reliability analysis against human labels of GPT-4o-Audio-Preview responses reports Pearson/Spearman/Kendall correlations all above 0.87 (p < 0.001) in both languages for both LLM judges, supporting the validity of the LLM-based evaluation protocol.
Novelty Assessment
The contribution is a dataset and evaluation-methodology paper, not an architectural one. Its novelty rests on identifying, through an explicit literature review plus corpus statistics on CABank and MagicData-RAMC, a set of concrete gaps in prior SDM benchmarks (ambiguity coverage, coreference/omission, native bilingual data rather than translated text, multi-turn interaction) and building a targeted dataset around them, together with a manual audio-quality control step and a human-validated LLM-as-a-judge protocol. The LLM-as-a-judge methodology itself is not new (the paper explicitly follows an existing paradigm), and the ten SDMs evaluated are all pre-existing systems; the paper’s engineering contribution is the evaluation pipeline (prompt design per phenomenon, detection/resolution decomposition, dedicated human audio-listening protocol for the phonological generation subtask) rather than a new model or metric family.
Field Significance
moderate — C3 identifies and operationalizes a genuine gap in spoken dialogue model evaluation: no prior benchmark jointly probes phonological/semantic ambiguity, omission, coreference, and multi-turn interaction with natively bilingual (rather than translated) data, and no prior benchmark directly evaluates whether an SDM’s generated audio (not just its transcript) carries correct prosodic realizations. The paper’s empirical findings, that SDMs struggle far more with ambiguity than context-dependency, that pointing out a phenomenon only partially helps a model resolve it, and that performance is highly language-dependent, provide concrete, reproducible evidence for where current end-to-end SDMs fall short.
Claims
- supports: LLM-as-a-judge evaluation of a spoken dialogue model’s transcribed text response can closely approximate human judgment of correctness, provided the judgment task does not require perceiving audio-only information.
Evidence: Pearson, Spearman, and Kendall correlation coefficients between LLM-judge (GPT-4o and DeepSeek-R1) scores and human-expert labels of GPT-4o-Audio-Preview’s responses all exceed 0.87 in both the English and Chinese subsets, with p < 0.001 for every coefficient. (§4.2, Reliability Analysis; Table 4)
- complicates: Transcript-based evaluation of spoken dialogue model outputs cannot capture whether the model produces correct prosodic realizations, so assessing certain generation capabilities requires directly evaluating the generated audio rather than its text.
Evidence: For the phonological ambiguity “Generation” subtask (correcting pause placement, stress, intonation, heteronym or tonal pronunciation), the authors bypass transcription and have three human experts label each SDM’s audio output directly, “as it requires examining phonological features that cannot be captured by the transcribed text.” (§4.2, LLM-based Evaluation)
- complicates: A spoken dialogue model’s relative competence ranking is language-dependent, so single-language evaluation cannot be assumed to generalize to the model’s overall conversational competence.
Evidence: GPT-4o-Audio-Preview is the top performer on the English subset (55.68% overall accuracy) but is outperformed by Qwen2.5-Omni on the Chinese subset (40.08% vs. GPT-4o-Audio-Preview’s 29.45%); the English-Chinese accuracy gap exceeds 8 percentage points in every phenomenon category, and some models (e.g., Freeze-Omni, GLM-4-Voice) more than double their accuracy from Chinese to English. (§5.2.3; Table 2)
- complicates: Explicitly pointing out to a spoken dialogue model that a comprehension-relevant phenomenon (e.g., omission, coreference) is present in the input only partially improves the model’s ability to correctly resolve it.
Evidence: Comparing detection accuracy against completion/resolution accuracy for the same phenomena, most SDMs score much higher on detecting the presence of omission or coreference than on completing or resolving it (e.g., overall omission detection reaches 38.65% in Chinese versus 3.75% for completion, and 8.57% versus 8.98% in English), showing the benefit of an explicit hint is limited or absent for most models. (§5.2.2; Table 3)
Limitations and Open Questions
The benchmark's dialogue audio is largely TTS-synthesized (Seed-TTS, unified timbre, no background noise) rather than drawn directly from the underlying real-world recordings, with a manual pass re-recording only the instances found to have poor audio quality. This trades off ecological validity (natural variation in accent, background conditions, real speaker prosody) for controlled, confound-free stimuli, so results may not fully transfer to noisier or more naturalistic spoken input.
The authors themselves note two further limitations: the five complexity phenomena are studied only in English and Chinese, though they are not inherently limited to these languages; and human-expert audio labeling carries potential rater bias, which the paper mitigates with a three-expert majority-voting scheme rather than eliminating. Additionally, cascaded ASR+LLM+TTS SDMs are excluded from evaluation by design (on the reasonable grounds that ASR discards phonetic cues), so the benchmark’s findings characterize only the current generation of end-to-end SDMs, and Moshi’s full-duplex interruption behavior means it could not be evaluated under the same multi-turn protocol as the other nine models.
Wiki Connections
- Spoken Language Model — benchmarks ten end-to-end SDMs (audio-in, audio-out speech LMs) on conversational phenomena rather than proposing a new speech LM itself.
- Speech-to-Speech — targets the dialogue sub-paradigm of speech-to-speech systems, evaluating models that take spoken queries and must produce coherent voice responses.
- Evaluation Metrics — introduces an LLM-as-a-judge accuracy protocol for SDM responses, validated against human labels with correlation coefficients above 0.87.
- Subjective Evaluation — uses three human experts to directly label SDM audio output as correct or incorrect for the phonological generation subtask and to validate the LLM-judge protocol.
- Kimi-Audio Technical Report — one of the ten end-to-end SDMs benchmarked on the C3 dataset.
- Moshi — benchmarked SDM; excluded from the multi-turn comparison because its full-duplex interruption behavior is incomparable under the paper’s protocol.
- Qwen2.5-Omni Technical Report — benchmarked SDM; the top performer on the Chinese subset and strongest at multi-turn interaction overall.
- GLM-4-Voice — benchmarked SDM, cited as an example of the cross-linguistic (English/Chinese) capability C3 evaluates.
- Freeze-Omni — benchmarked SDM used to illustrate the large English-Chinese accuracy gap the paper reports.
- Step-Audio — benchmarked SDM evaluated across all five target phenomena.
- Seed-TTS — the TTS tool used to regenerate dialogue audio with a unified timbre and no background noise when constructing the C3 dataset.
- VoiceBench — prior SDM benchmark that C3 explicitly differentiates itself from for lacking ambiguity, coreference/omission, and multi-turn coverage.