arXiv · 2024 · Preprint
Ji et al. (Zhejiang University / Microsoft / Alibaba Group / Tencent) · → Paper · Demo: ? · Code: ✓
A comprehensive survey of spoken dialogue models that organises the field into cascaded and end-to-end paradigms, taxonomises the core enabling technologies (speech representations, training strategies, streaming and duplex systems), and identifies the benchmarks and open challenges needed to drive the field forward.
Problem
Prior to this survey, the spoken dialogue field lacked a systematic and unified treatment of its rapidly expanding landscape. The arrival of GPT-4o and Moshi in 2024 catalysed a wave of concurrent work on end-to-end spoken dialogue models, each differing substantially in representation choice, alignment strategy, architecture, and interaction design. Without a shared taxonomy, the field was difficult to survey from first principles: practitioners had no reliable framework for comparing cascaded systems (which rely on text as a central intermediary) against end-to-end ones (where the core language model directly understands and generates speech), and no consolidated view of the evaluation landscape. The survey addresses this gap by providing a structured, technology-by-technology overview from a team that includes authors of several of the primary systems surveyed.
Scope and Coverage
WavChat covers the spoken dialogue model literature from early prototypes (dGSLM, AudioGPT) through late 2024 systems including Moshi, LLaMA-Omni, Mini-Omni, Freeze-Omni, OmniFlatten, and SyncLLM, organising them along two primary axes: cascaded versus end-to-end, and the four core technology dimensions that cut across both paradigms.
The survey’s organisational framework rests on four pillars. First, speech representations, decomposed into semantic tokens (HuBERT, Whisper, WavLM, Wav2Vec 2.0) and acoustic tokens (EnCodec, SpeechTokenizer, Mimi, SNAC), examined from both input and output perspectives. Second, training paradigms, covering the five architectural approaches to speech-text modal alignment (text-output-only, chain-of-modality, interleaved tokens, parallel text-speech generation, and direct speech-to-speech), multi-stage training strategies, and the nascent application of preference optimisation to spoken language models. Third, streaming and duplex capabilities, distinguishing half-duplex turn-taking systems from full-duplex models that support simultaneous listening and speaking, and analysing the core techniques that enable real-time operation (causal convolution, causal attention, queue management). Fourth, training resources and evaluation, cataloguing datasets across four training stages and benchmarks across eleven capability dimensions including text intelligence, speech intelligence, streaming latency, and interaction capability.
Key Results
The survey’s main synthesis findings span each technology dimension. On representations, no consensus approach exists: different systems use semantic inputs with acoustic outputs (Mini-Omni), semantic on both sides (SpiritLM), or acoustic on both sides (Moshi), and each combination carries distinct trade-offs between LLM compatibility, expressiveness, and pipeline complexity. The survey identifies single-codebook codecs (WavTokenizer, Single-Codec, BigCodec) as a promising direction that could reduce the autoregressive burden compared to multi-layer RVQ approaches, but notes the area is nascent.
On training, the survey finds that nearly all current spoken dialogue systems depend on text-speech paired data for modality alignment, leading to a persistent risk of catastrophic forgetting of text capabilities. The chain-of-modality (CoM) approach (generate text first, then speech) improves quality but is incompatible with low-latency interaction, while parallel text-speech generation methods (Moshi, LLaMA-Omni, Mini-Omni) trade some response quality for substantially reduced latency. Preference optimisation has been applied to TTS (Seed-TTS, Qwen2-Audio) but remains almost entirely unexplored for spoken dialogue models.
On streaming and interaction, the survey documents that true full-duplex capability (simultaneous listen and speak with interrupt handling) has been achieved in Moshi and OmniFlatten but that most other systems only address the output side, using non-streaming encoders (Whisper-based) on the input side. The survey argues that standardised benchmarks for interaction capabilities, turn-taking evaluation, and duplex performance are absent, constituting a major gap in evaluation infrastructure.
The benchmark review (Section 6, Table 3) maps eight existing benchmarks (VoiceBench, SUPERB, AudioBench, AIR-Bench, SpokenWOZ, SD-Eval, SuperCLUE, MMAU) against eleven capability dimensions and reveals systematic gaps: no benchmark covers audio generation by spoken dialogue models, multimodal QA evaluation is underdeveloped, and interaction capability evaluation remains largely ad hoc.
Novelty Assessment
The primary contribution is taxonomic and synthetic rather than empirical. WavChat offers a more comprehensive and technically precise treatment of spoken dialogue systems than any prior survey, drawing clear distinctions that the field itself does not consistently enforce: the cascaded vs. end-to-end boundary, the semantic vs. acoustic token distinction, and the half-duplex vs. full-duplex capability gap. The five-way architectural taxonomy for modal alignment (§4.1) is a useful organising schema that does not exist in consolidated form elsewhere. The discussion sections at the end of each chapter surface actionable open questions, including the possibility of speech tokenisers that achieve text-space alignment during encoding (eliminating the need for paired data at training time), and the design of full-duplex benchmarks. The survey does not introduce new experimental results or models.
Field Significance
Tip
High — This survey provides the most comprehensive taxonomy and technical synthesis of spoken dialogue models available as of late 2024, covering a field that had grown extremely rapidly with no consolidating overview. Its framework for distinguishing cascaded from end-to-end systems, decomposing representation choices, and mapping the evaluation gap is likely to serve as a shared reference for researchers entering or navigating the SCA space. The absence of standardised interaction benchmarks, which the survey clearly documents, is an actionable gap whose articulation alone is a contribution to the field’s research agenda.
Claims
- The cascaded and end-to-end paradigms for spoken dialogue impose fundamentally different trade-offs between latency, paralinguistic fidelity, and intelligibility, and the appropriate choice depends on the target interaction scenario. (§2.2, §2.3)
- Semantic speech representations offer higher compression rates and better LLM compatibility than acoustic representations, but sacrifice expressiveness, timbre, and style fidelity, necessitating additional vocoders in pipeline-based generation. (§3.3.1, Table 1)
- Achieving genuine full-duplex spoken dialogue (simultaneous listening and speaking with interrupt handling) requires architecturally causal models throughout the full pipeline, a constraint that current systems largely satisfy only on the output side. (§5.1, §5.2)
- Speech-text modality alignment in current spoken dialogue systems relies heavily on paired data, introducing catastrophic forgetting risk and creating a structural dependency on the availability of labelled speech corpora. (§4.4.1)
- Evaluation infrastructure for spoken dialogue lags substantially behind system capabilities: interaction, streaming latency, and audio generation are either absent from or severely underrepresented in existing benchmarks. (§6.2, §6.3, Table 3)
Limitations and Open Questions
The survey was produced concurrently with the primary wave of open-source spoken dialogue models it covers (late 2024), meaning that some systems are described in early form and the field will have evolved by the time readers encounter the paper. The coverage of music and sound understanding and generation within dialogue systems is acknowledged as thin (the authors defer to an appendix), and security evaluation for spoken dialogue receives less treatment than its importance warrants. A number of prominent systems (Westlake-Omni, Hertz-dev, SpeechGPT2, Fish-Agent) lack published papers and are excluded from the timeline figure, which may leave gaps for practitioners interested in the deployed-systems landscape. The SuperCLUE benchmark, one of the more comprehensive interaction evaluations listed, is not open-source and focuses on Mandarin, limiting its utility for the broader research community.
Open questions surfaced include: whether speech tokenisers can be designed to enforce text-space alignment during encoding, eliminating the need for large paired corpora; what granularity of temporal alignment priors (sentence, word, phoneme level) is optimal for spoken dialogue training; and how preference optimisation techniques can be adapted for the joint text-speech output space.
Wiki Connections
Core systems surveyed: 2410.00037 (Moshi), 2301.02111 (VALL-E), 2305.11000 (SpeechGPT), 2402.05755 (SpiritLM), 2409.06666 (LLaMA-Omni), 2408.16725 (Mini-Omni), 2411.00774 (Freeze-Omni), 2410.17799 (OmniFlatten)
Codecs and representations: 2210.13438 (EnCodec), 2308.16692 (SpeechTokenizer), 2408.16532 (WavTokenizer)
TTS and VC systems referenced: 2407.05407 (CosyVoice), 2406.02430 (Seed-TTS), 2303.03926 (VALL-E X), 2304.09116 (NaturalSpeech 2), 2403.03100 (NaturalSpeech 3), 2306.12925 (AudioPaLM)
Evaluation: 2402.07729 (AIR-Bench), 2410.19168 (MMAU)
Concepts: spoken-language-model, neural-codec, self-supervised-speech, streaming-tts, evaluation-metrics, speech-to-speech
2501.06282 — MinMo: A Multimodal Large Language Model for Seamless Voice Interaction