ACL · 2025 · Conference
Kim et al. (Seoul National University) · → Paper · Demo: ✓ · Code: ✓
Introduces ContextDialog, a speech-to-speech benchmark for multi-turn context recall, and demonstrates that all evaluated open-source voice interaction models fail at this task, with text-only LLM counterparts outperforming them by a wide margin and RAG providing no consistent remedy.
Problem
Multi-turn voice assistants must retain and recall information from earlier dialogue turns, yet no benchmark existed to systematically measure this capability in open-source models. Closed-source systems (GPT-4o, Gemini 2.0) handle this fluently, but whether open-source speech models share the ability was unexplored. Existing benchmarks for voice interaction evaluate single-turn semantics, paralinguistic cues, or task-completion without requiring that models actually reference prior conversational history to generate a correct response.
Method
ContextDialog is constructed from MultiDialog, a spoken dialogue corpus of approximately 340 hours with at least 10 turns per conversation from 12 speakers. The benchmark generation pipeline has two stages. First, GPT-4o is prompted on dialogue transcripts to produce question-answer pairs, each targeting a single piece of information that appears exactly once in the conversation, so that correct answers cannot be inferred without access to that specific supporting utterance. A second-pass validator (o1-mini) filters pairs that are inconsistent or inferable without the supporting context. This yields four QA pairs per dialogue across two splits: test_freq (363 dialogues, 1,452 QA pairs) and test_rare (290 dialogues, 1,160 QA pairs). Second, the text QA pairs are converted to speech using Fish Speech, a speaker-adaptive TTS system, matching the timbre of each original speaker via reference audio; the sample with the lowest WER (measured by Whisper) is selected per pair.
Four open-source multi-turn voice interaction models are evaluated: GLM-4-Voice, Lyra, MiniCPM-o, and Freeze-Omni. All take speech as input and generate both text and speech outputs. Spoken responses are evaluated using an LLM-as-a-judge approach (GPT-4o-mini, 5-point scale) after ASR transcription by Whisper. To disentangle recall ability from synthesis quality, intermediate text responses are also scored separately. A retrieval-augmented generation (RAG) experiment augments generation with top-k utterances retrieved by e5-large-v2 from ASR-transcribed dialogue history.
Key Results
All four speech models score substantially below their text-only LLM backbones on ContextDialog. Across the main evaluation, overall GPT Scores range from 2.00 (Freeze-Omni) to 2.83 (Lyra) on a 5-point scale, compared to 4.03-4.10 for the equivalent text-based LLMs (Table 2). The degradation is most severe in Freeze-Omni, where the LLM backbone is kept frozen during speech training, suggesting that joint fine-tuning is necessary to preserve long-context handling.
A modality-specific recall gap is observed within speech models: all models score significantly higher on questions about the model’s own past utterances than about past user utterances (p < 0.01). This asymmetry is mechanistic: recent architectures generate text responses alongside speech, so model-side content is available both as text and audio; user utterances exist only as speech. Attention maps confirm that models allocate less attention to user-spoken supporting utterances than to model-generated ones.
RAG with a dedicated text retriever yields no consistent improvement across any model or retrieval depth (top-1 to top-3, Figure 4). Supplying the exact ground-truth supporting utterance improves scores for three of four models (Table 3), confirming that the bottleneck is retrieval fidelity rather than the augmentation mechanism itself. Top-1 retrieval recall is 0.58 (Table 5), and injecting an irrelevant utterance consistently degrades all models below baseline. A comparison against gpt-4o-mini-audio-preview (Table 7) shows the closed-source model scores 3.83-4.50 overall, far above any open-source result, even when dialogue history is provided only as concatenated audio rather than a proper multi-turn API context. Human evaluation on Amazon Mechanical Turk (180 participants) confirms the same ordering of results (Table 8).
Novelty Assessment
The paper’s contribution is primarily diagnostic. ContextDialog is the first benchmark explicitly designed so that correct answers require access to prior turns, filling a gap that prior spoken dialogue benchmarks overlook by allowing responses derivable from single-turn context. The evaluation protocol (LLM-as-a-judge on ASR transcripts, separate scoring of text and speech outputs, attention map analysis) is careful and thorough. The finding that RAG fails not because of ASR noise (ground-truth transcripts yield similar retrieval recall at top-1) but because retrieval errors compound in a model trained to generate short responses is a concrete mechanistic insight rather than a general claim. The paper does not propose a new architecture, training recipe, or mitigation, making it purely evaluative. The benchmark itself is the reusable artifact.
Field Significance
Moderate — This paper provides controlled empirical evidence that the modality gap between speech and text LLMs is most severe for long-context dialogue history, a finding with direct implications for spoken conversational agent design. It demonstrates that a widely assumed path to improving recall (retrieval augmentation) does not transfer cleanly from text NLP to voice interaction, and quantifies the gap between open-source and closed-source systems on a specific capability. ContextDialog can serve as a reproducible reference for future work targeting multi-turn spoken context modeling.
Claims
-
supports: Extending a pre-trained text LLM to process speech input substantially degrades its ability to recall and utilize long-range conversational context.
Evidence: All four evaluated open-source voice interaction models score 2.00-2.83 on ContextDialog (5-point GPT Score), compared to 4.03-4.10 for their text-only backbone LLMs; the gap is largest when the LLM backbone is frozen during speech adaptation. (§4.1, Table 2)
-
supports: Voice interaction models exhibit a modality-specific recall asymmetry, retrieving information from model-generated text responses more reliably than from user-spoken utterances.
Evidence: On ContextDialog, all four speech models score significantly higher on questions about the model’s own past utterances than past user utterances (p < 0.01), consistent with the fact that model responses are preserved as text while user turns exist only as speech; attention maps confirm systematically lower attention weight on user-spoken supporting utterances. (§4.1, Table 2, Figure 3)
-
complicates: Retrieval-augmented generation provides a practical route to improving multi-turn context recall in spoken dialogue models.
Evidence: Augmenting four open-source voice interaction models with a top-k text retriever (e5-large-v2) yields no consistent improvement and often degrades performance; top-1 retrieval recall is only 0.58 and irrelevant retrieved utterances lower all models below baseline across all prompt templates tested. (§4.2, Figure 4, Table 3, Table 5)
-
supports: Open-source voice interaction models show critically underdeveloped long-context capabilities compared to closed-source systems, even on simple factual recall tasks.
Evidence: On the same ContextDialog benchmark, gpt-4o-mini-audio-preview scores 3.83-4.50 (text-provided or audio-concatenated history), more than 1 point above the best open-source model (Lyra, 2.83), even though the closed-source API does not support a proper multi-turn spoken dialogue context. (§A.1.4, Table 7)
Limitations and Open Questions
Warning
ContextDialog is entirely synthetic: questions are generated by GPT-4o and corresponding audio is synthesized by Fish Speech. The benchmark may overfit to patterns that TTS-generated speech handles well (clear pronunciation, controlled voice), potentially underestimating recall difficulty on real user speech with natural disfluency and acoustic variability.
The evaluation covers only four open-source models evaluated in 2024-2025; other architectures (pure speech-output models such as early SpeechGPT, full-duplex models) are excluded because they either lack multi-turn support or have incompatible interaction formats. Text-based RAG is the only augmentation strategy tested; no speech retriever module suitable for open-source voice interaction currently exists, so the analysis cannot address whether speech-native retrieval would perform better. The study also does not measure latency costs of RAG, which is critical for real-time voice assistants. Benchmarks testing more complex reasoning over past context rather than direct retrieval are identified as a natural extension.
Wiki Connections
- Evaluation Metrics — introduces ContextDialog as a new speech-to-speech benchmark specifically measuring multi-turn context recall, filling a gap in existing evaluation methodology for voice interaction models.
- Subjective Evaluation — employs both LLM-as-a-judge scoring (GPT-4o-mini, 5-point scale) and human evaluation via Amazon Mechanical Turk, with strong inter-method correlation (Pearson r = 0.88) validating the automated metric.
- Speech-to-Speech — all evaluated models take speech as input and generate spoken responses, and ContextDialog is designed specifically for speech-to-speech interaction rather than cascaded ASR-LLM-TTS pipelines.
- Spoken Language Model — the benchmark and analysis target multi-turn spoken language models that process and generate both text and speech tokens, revealing a systematic recall failure mode in this model class.
- CosyVoice — used as the speech synthesis backbone in GLM-4-Voice (one of the four evaluated models), providing token-to-speech reconstruction via a CosyVoice-based module with chunk-wise autoregressive modeling.