arXiv · 2025 · Preprint
Du et al. (The Chinese University of Hong Kong, Shenzhen) · → Paper · Demo: ? · Code: ?
MTalk-Bench introduces the first benchmark for holistic evaluation of speech-to-speech LLMs in multi-turn dialogues, covering semantic, paralinguistic, and ambient sound dimensions via a dual Arena-plus-Rubrics evaluation protocol.
Problem
Evaluation of speech-to-speech models has remained fragmented: existing benchmarks either focus on single-turn tasks, cover only semantic content, or rely on text-based scoring that discards the audio modality. No unified framework assessed S2S model performance across the compound demands of real spoken dialogue, including paralinguistic cues (emotion, prosody) and ambient acoustic conditions, across multiple conversational turns. The absence of holistic, audio-grounded, multi-turn benchmarks makes it difficult to diagnose model weaknesses or track genuine progress in spoken conversational agent capability.
Method
MTalk-Bench organises evaluation around three core dimensions: Semantic Information, Paralinguistic Information, and Ambient Sound. Within each dimension, nine foundational capabilities are defined via a two-tier hierarchy, with fine-grained sub-capabilities derived from large-scale pairwise preference surveys among university participants using Bradley-Terry ranking. Nine realistic dialogue scenarios (e.g., health and medical communication, workplace communication, casual interaction) were selected by the same user-voting process.
The dataset consists of 270 validated audio samples constructed through a multi-stage pipeline. Text dialogues were generated using an LLM-human hybrid approach, with approximately 19% retained after coherence and naturalness filtering. Native English speakers recorded Type I (semantic) dialogues in a neutral tone; Type II (paralinguistic) dialogues added emotion guided by annotation; Type III (ambient sound) dialogues mixed speech with real environmental sounds from Freesound and FSD50K. Voice style variation (e.g., child or elderly speakers) was synthesised using an external voice conversion model. Three rounds of quality filtering further reduced the pool.
Evaluation uses two complementary protocols. Arena-style evaluation presents human evaluators with blind pairwise comparisons of model outputs, producing Elo ratings initialised at 1000 with elasticity coefficient K=4. Rubrics-based evaluation scores each response against 7-9 binary criteria organised in a three-level hierarchy: universal criteria (grammar, relevance), dimension-specific criteria, and LLM-generated instance-specific rubrics reviewed by human annotators. Both human evaluators (recruited via Amazon Mechanical Turk) and LLM-as-judge systems (GPT-4o Realtime, Gemini-2.5-pro, Qwen-Omni-Turbo) participate as evaluators, enabling direct comparison of their reliability.
Ten S2S models are benchmarked: closed-source systems including GPT-4o Realtime, Doubao, and Qwen-Omni-Turbo; open-source models including Step-Audio-Chat (130B), GLM-4-Voice, VITA-Audio-Plus-Vanilla, MiniCPM-o 2.6, Kimi-Audio, Moshi, and AnyGPT. Human-generated responses serve as the performance baseline.
Key Results
Under human evaluation, Step-Audio-Chat achieves the highest Arena Elo overall (1049) among open-source models, with Doubao (1037) and GPT-4o Realtime (1030) the leading closed-source systems. In Rubrics-based scoring, GPT-4o Realtime leads overall (77.38/100), followed by Doubao (73.69) and Step-Audio-Chat (71.86). No system crosses 80 points on the rubrics scale, indicating substantial room for improvement.
Performance is consistently stronger on semantic dimensions than on paralinguistic and ambient sound dimensions across all models. Models outperform the human baseline on semantic understanding but fall below it on ambient sound perception. Turn-level analysis shows most models experience a quality dip from Turn 1 to Turn 2 before partially recovering at Turn 3. This early-context-accumulation bottleneck correlates with a linear decline in content density across turns: later responses add more tokens without proportional informational gain.
Among architectural patterns, Step-Audio-Chat’s approach of transcribing prior turns into text before processing achieves the strongest multi-turn performance, outperforming models that encode the entire dialogue history as raw audio. Within open-source models, parameter count shows no reliable correlation with performance, suggesting that modality-aware design matters more than scale.
On evaluator reliability: LLM-as-judge aligns closely with human annotators when performance gaps between models are large, but agreement drops sharply for closely matched systems. LLMs exhibit statistically significant position bias (Gemini-2.5-pro: +4.74%) and length bias (+17.3% preference for longer responses, versus +12.9% for human evaluators). Critically, LLM evaluators achieve near-zero Spearman correlation with humans on non-verbal cues (ambient sounds, paralinguistics) when assessing raw audio; this correlation exceeds 0.85 when the same cues are provided as text annotations (e.g., [laughs]).
Novelty Assessment
The benchmark itself is genuinely new: no prior S2S evaluation framework combined multi-turn dialogues, three evaluation dimensions (including ambient sound), human-recorded audio inputs, and both arena-style and rubrics-based protocols. The capability taxonomy derived from user-voting via Bradley-Terry ranking is a principled methodological contribution, though the scale remains modest at 270 samples.
The dual Arena-plus-Rubrics evaluation design is well-motivated: the authors validate that the two methods produce consistent rankings (bootstrap analysis, Spearman rho > 0.95) while serving complementary purposes, relative comparison versus absolute diagnostic scoring. The finding that LLM-as-judge fails on raw audio non-verbal cues is a useful caution for practitioners building automated evaluation pipelines, and the text-annotation workaround (providing transcriptions with paralinguistic labels) is practically actionable.
The benchmark’s main limitation is scale: 270 samples is small enough that the paper’s own analysis notes only large performance gaps yield reliable distinctions. The English-only coverage and the use of Amazon Mechanical Turk for some recordings also limit ecological validity claims. Models lacking public APIs required a non-standard microphone capture method, introducing potential confounds.
Field Significance
Moderate — MTalk-Bench fills a real gap by providing the first holistic multi-turn evaluation framework for speech-to-speech systems that treats paralinguistic and ambient sound capabilities as first-class evaluation targets alongside semantic understanding. Its dual-method evaluation design and systematic documentation of LLM-as-judge biases in the audio domain provide methodological grounding that can inform future benchmark design.
Claims
-
supports: Current speech-to-speech language models exhibit a systematic capability gap between semantic understanding and non-verbal speech dimensions such as paralinguistic and ambient sound processing.
Evidence: On MTalk-Bench, all evaluated models outperform the human baseline on semantic rubric scores but underperform on paralinguistic and ambient sound dimensions; no model exceeds 80/100 overall. (§6.3.1, Table 2)
-
complicates: Multi-turn spoken dialogue systems compensate for context-accumulation bottlenecks by increasing response verbosity rather than informational density.
Evidence: Turn-level rubric scores on MTalk-Bench show a non-linear dip from Turn 1 to Turn 2, with partial recovery at Turn 3; content density declines linearly across turns while token count increases, indicating coherence is regained by spending tokens rather than by improving context modelling. (§6.3.2, Table 4)
-
supports: Task-specific modality-aware design provides greater performance gains in multi-turn spoken dialogue than parameter scaling alone.
Evidence: Step-Audio-Chat (130B, with explicit text transcription of prior turns) achieves the highest overall Arena Elo (1049) among open-source models; within the evaluated set, parameter count shows no reliable correlation with rubric or arena scores. (§6.3.3, Table 2)
-
complicates: LLM-as-judge automated evaluation of speech quality is unreliable for non-verbal audio cues when models are given raw waveforms.
Evidence: LLM evaluators achieve near-zero Spearman correlation with human evaluators on ambient sound and paralinguistic dimensions under raw-audio evaluation, while the same dimensions yield Spearman rho > 0.85 when non-verbal cues are provided as text annotations. (§7.2.2)
-
refines: Arena-style and rubrics-based evaluation methods yield consistent model rankings for speech systems, but only when performance gaps between models are sufficiently large.
Evidence: Bootstrap rubric-exclusion analysis shows Spearman rho > 0.95 alignment between Arena and Rubrics rankings; however, small ELO and rubric score differences correspond to statistically insignificant win-rate gaps, and adding comparisons does not resolve near-parity distinctions. (§7.2.1, §7.2.3, Table 2)
Limitations and Open Questions
Warning
The benchmark contains only 270 audio samples. The paper’s own analysis shows that Arena and Rubrics evaluations yield reliable distinctions only when performance gaps are large; this sample size limit is a material constraint on the benchmark’s ability to discriminate between competitive models.
The benchmark is English-only. Models lacking public APIs were evaluated via a non-standard microphone capture method, which may introduce acoustic confounds. Amazon Mechanical Turk recordings exhibit variable quality, evidenced by rejection rates above 40% in the first quality-check round. The LLM-as-judge analysis covers only three judge models (GPT-4o Realtime, Gemini-2.5-pro, Qwen-Omni-Turbo), limiting generalisation of the bias findings. Ambient sound evaluation relies on artificially mixed audio rather than genuinely recorded noisy environments, which may not reflect deployment conditions.
Wiki Connections
- Evaluation Metrics — introduces Arena-style (Elo-based pairwise) and Rubrics-based (binary criterion) evaluation protocols as a dual-method framework for assessing S2S models.
- Subjective Evaluation — employs human raters via Amazon Mechanical Turk for both Arena pairwise comparisons and Rubrics absolute scoring, with systematic quality filtering across three evaluation rounds.
- Spoken Language Model — benchmarks ten speech-to-speech LLMs (including GPT-4o Realtime, Step-Audio-Chat, GLM-4-Voice) on multi-turn spoken dialogue capabilities.
- Speech-to-Speech — designed specifically to evaluate end-to-end S2S systems that map input speech directly to output speech, covering semantic, paralinguistic, and ambient sound dimensions.