arXiv · 2025 · Preprint
Gosai & Vuong et al. (Scale AI) · → Paper · Demo: ? · Code: ✓
An open-source benchmark extending the text-based MultiChallenge framework to natural, multi-turn spoken dialogue, adding an audio-native Voice Editing axis and audio-cue-gated memory tasks, evaluated with human-authored rubrics judged by an LLM verified against human agreement, revealing that even frontier end-to-end spoken dialogue systems reach only a 54.65% pass rate.
Problem
End-to-end (E2E) spoken dialogue systems increasingly replace cascaded ASR-LLM-TTS pipelines, but existing benchmarks mostly evaluate them on synthetic speech and single-turn tasks. Text-only LLM research has already shown that strong single-turn performance does not guarantee robust multi-turn behavior, and this weakness is expected to be amplified for audio: speech is temporally continuous and represented as high-frame-rate token streams that consume context quickly, making long-horizon state tracking harder than in text. It remains unclear whether current E2E systems can handle realistic multi-turn interaction patterns, self-consistency, following evolving instructions, using audio cues over long contexts, and recovering from natural speech repairs like mid-utterance self-corrections and barge-ins, since prior multi-turn audio evaluation efforts are limited to short dialogues.
Method
Audio MultiChallenge extends the text-based MultiChallenge framework’s three axes, Inference Memory, Instruction Retention, and Self Coherence, to the audio modality, and adds a new axis specific to speech: Voice Editing, which tests robustness to mid-utterance speech repairs and backtracking (e.g., “Let’s make it four yuccas, hmm no six”), since unlike text, spoken edits are audible rather than silently backspaced before submission. Inference Memory is further extended with Audio-Cue-gated tasks that require recalling ambient sounds or paralinguistic signals (not just spoken words) from earlier turns. Data is curated through a hybrid pipeline: a multi-agent loop (a Planner Agent, a Tester Agent producing TTS-synthesized prompts, and a Target Audio LM) synthetically explores failure modes along a target axis and distills each discovered failure into a human-friendly blueprint; human contributors then use these blueprints only as strategic guidance, improvising their own real, unscripted spoken interactions with an Audio LM (GPT-4o Audio Preview or Gemini 2.5 Pro) until they trigger a verifiable failure, preserving natural disfluencies, pauses, and cross-talk. For each triggered failure, contributors author a set of atomic, binary rubric criteria targeting only the final turn and the assigned axis. Evaluation uses a fixed-context protocol: each model receives the identical pre-seeded conversation history (user turns as audio, prior assistant turns as text) and is scored only on its response to the final turn, avoiding conversation drift from sequential turn-by-turn generation. An LLM-as-a-judge grades the model’s text-stream response against the human-written rubrics (a response passes only if it satisfies every corresponding rubric), with the judge selection itself validated against human graders (Cohen’s kappa and Macro F1 across six candidate judge models).
Key Results
Across 452 conversations with 1,712 rubrics, the highest-performing evaluated model, Gemini 3 Pro Preview (Thinking), reaches only a 54.65% overall pass rate (APR); most other proprietary and nearly all open-source E2E speech-to-speech models score well below that, with no open-source model surpassing Qwen3-Omni’s 24.34%. The two new audio-native axes prove hardest: Voice Editing averages 17.99% APR and Audio-Cue Inference Memory scores 36.5% lower (relative) than semantic Inference Memory on average. A “modality gap” appears for models supporting both output types: the same models average 19.36% APR configured for text output versus 16.76% for audio output. Self Coherence degrades steadily with more cumulative user-audio context, from 33.3% APR at 0-60 seconds down to 20.0% at 3-5 minutes. An ablation replacing real human audio with ASR-then-TTS resynthesized versions of the same transcripts improves text-output-configured models’ pass rate by 7.5% relative (largest gains on Voice Editing, since resynthesis smooths over the natural hesitations that make tracking self-corrections hard), while audio-output-configured models show a small decline, suggesting their post-training may already be tuned toward real human speech input specifically. The LLM-as-a-judge setup (o4-mini as primary judge) achieves 0.873 Cohen’s kappa and 0.937 Macro F1 agreement with human graders.
Novelty Assessment
The genuinely new contribution is the Voice Editing axis, an evaluation dimension with no direct text-domain analog, since text inputs are conventionally cleaned before submission while spoken edits remain part of the audible signal the model must parse, plus the Audio-Cue-gated extension of Inference Memory to non-semantic acoustic content. The hybrid agentic-plus-human-in-the-loop curation pipeline and fixed-context, rubric-based LLM-judge evaluation protocol are adaptations of the existing text-based MultiChallenge methodology to the audio domain rather than novel evaluation techniques in themselves, but the paper’s direct validation of judge reliability against human agreement, and its TTS-vs-human-audio ablation isolating the effect of natural disfluency, are genuine methodological contributions supporting the benchmark’s credibility.
Field Significance
moderate — This is a well-constructed, thoroughly validated evaluation resource that surfaces a genuine and previously under-measured gap (natural multi-turn, audio-native interaction robustness) in E2E spoken dialogue systems, backed by real human-recorded data and human-validated LLM-judge scoring. Its relevance to speech generation specifically is indirect: the benchmark evaluates whether a system’s response content is correct given spoken multi-turn context, not the quality, naturalness, or fidelity of any speech the system generates, so it characterizes spoken dialogue understanding and state-tracking robustness rather than speech generation quality per se.
Claims
- supports: End-to-end spoken dialogue systems that perform reasonably on single-turn or synthetic-speech benchmarks reveal substantial capability gaps when evaluated on natural, multi-turn, human-recorded spoken interaction.
Evidence: Even the highest-performing frontier model (Gemini 3 Pro Preview, Thinking) achieves only a 54.65% pass rate on Audio MultiChallenge, with most open-source E2E speech-to-speech systems scoring below 25%. (§3.1, Table 2)
- supports: Configuring an audio-capable dialogue model to output speech rather than text measurably degrades its multi-turn task performance relative to the same model configured for text output, indicating a persistent modality gap in current speech-output post-training.
Evidence: Across models supporting both output modes, average pass rate drops from 19.36% under text-output configuration to 16.76% under audio-output configuration on the same benchmark tasks. (§3.1)
- supports: Current end-to-end spoken dialogue systems substantially underperform at recalling non-semantic audio information (ambient sounds, paralinguistic cues) from earlier conversation turns relative to recalling semantic spoken content, even within the same multi-turn evaluation format.
Evidence: Across evaluated models, Average Rubric Score on Audio-Cue-gated Inference Memory tasks falls 36.5% relative to semantic Inference Memory tasks on average. (§3.1)
- complicates: Natural, unscripted human speech with disfluencies and mid-utterance self-corrections is measurably harder for current E2E spoken dialogue systems to handle than synthetic TTS-resynthesized speech of the same underlying content, particularly for tasks requiring tracking of spoken self-corrections.
Evidence: Replacing real human audio with ASR-then-TTS resynthesized versions of the same transcripts improves text-output-configured models’ pass rate by 7.5% relative, with the largest gains on the Voice Editing axis. (§3.4, Table 4)
Limitations and Open Questions
The benchmark’s fixed-context evaluation protocol scores only the model’s response to a single final turn given an audited, error-free conversation history, which the authors note differs from fully sequential, turn-by-turn deployment where earlier model errors could compound. All rubric-based scoring is performed by an LLM judge rather than direct human rating for the reported leaderboard results, validated only against a human-agreement subset rather than for every scored instance. The dataset is English-only, and several duration bins used in the audio-length analysis (5-8 minutes and 8+ minutes) contain very few samples (n=4 and n=2 respectively), limiting confidence in those specific trend estimates.
Wiki Connections
- Spoken Language Model — evaluates real end-to-end audio language models across a range of architectures on their ability to consume genuine multi-turn human speech and respond appropriately.
- Speech-to-Speech — benchmarks E2E spoken dialogue systems, several of which support direct audio-to-audio response generation, on realistic conversational robustness.
- Evaluation Metrics — introduces a rubric-based, human-validated LLM-as-a-judge evaluation methodology and a novel audio-native Voice Editing evaluation axis with no direct text-domain analog.
- Qwen3-Omni — evaluated directly as one of the E2E audio language models benchmarked on Audio MultiChallenge.