arXiv · 2024 · Preprint
Cui et al. (Chinese University of Hong Kong) · → Paper · Demo: ? · Code: ?
The first comprehensive survey of Speech Language Models (SpeechLMs), providing a unified taxonomy of their architectures, training recipes, downstream applications, and evaluation methods, while systematically contrasting end-to-end speech generation with the cascade ASR + LLM + TTS approach.
Problem
The rapid proliferation of speech language model work in 2023-2024 lacked a systematic organising framework. Individual papers introduced their own terminology and component categorisations, making it difficult to compare architectures, identify gaps, or trace the field’s trajectory. The cascade pipeline (ASR + LLM + TTS) was the dominant deployed approach, but it suffers from information loss (paralinguistic features discarded at each text bottleneck), compounding latency from three sequential modules, and cumulative transcription and synthesis errors. The survey addresses the need to frame SpeechLMs as a coherent research area with common components, shared vocabulary, and open questions.
Method
The survey organises SpeechLMs around three core components: (1) the speech tokenizer, which converts continuous audio waveforms into tokens the language model can process; (2) the autoregressive language model, typically a decoder-only transformer borrowed from text LMs; and (3) a token-to-speech synthesizer (vocoder) that reconstructs waveforms from generated tokens.
Speech tokenizers are classified along a semantic-to-acoustic axis. Semantic tokenizers (HuBERT, wav2vec 2.0, W2v-BERT) excel at content fidelity but produce speech that lacks fine acoustic detail. Acoustic tokenizers (EnCodec, SoundStream) capture high-fidelity waveform reconstruction but may generate semantically inconsistent content. Mixed-objective tokenizers (SpeechTokenizer, Moshi’s Mimi) distil semantic information into the first RVQ layer while reserving later layers for acoustic refinement, aiming to capture both.
On the language model side, the survey distinguishes cold initialisation (training from scratch, as in GSLM) from continued pre-training on text LM checkpoints (as in TWIST, SPIRIT-LM), showing that text-pretrained initialisations improve convergence and speech understanding. Text-speech representation alignment is identified as a key training strategy: interleaved token sequences (text and speech tokens mixed in a single stream), multi-sequence parallel generation (simultaneous text and speech token decoding), and cross-modal supervision all appear in the literature with different latency/quality trade-offs.
Training is divided into three stages following text LM practice: pre-training on large speech corpora, instruction-tuning on task-specific speech QA datasets (many synthesised from text data using TTS), and post-alignment (RLHF or DPO) to improve semantic consistency and acoustic quality of responses.
The survey also covers the interaction paradigm dimension: from batch-mode generation to streaming (low-latency chunk-by-chunk processing) to full-duplex (simultaneous listen-and-speak), characterising the ability to handle user interruptions and background non-query speech.
A taxonomy of downstream applications spans semantic tasks (spoken dialogue, ASR, TTS, speech translation), speaker-related tasks (speaker ID, verification, diarisation, voice-conditioned generation), and paralinguistic tasks (emotion recognition, speech separation, paralinguistics-enhanced generation). Evaluation is similarly systematised across representation probing (ABX), linguistic benchmarks (sWUGGY, sBLIMP), paralinguistic evaluations (STSP), downstream task suites (SUPERB, VoiceBench, MMAU), and real-time interaction metrics.
Key Results
As a survey, the paper does not report original experimental results. Its contributions are taxonomic and analytical. Key findings from the literature reviewed include:
- HuBERT consistently outperforms CPC and wav2vec 2.0 as a semantic tokenizer in speech generation quality (GSLM).
- Text-pretrained LM checkpoints yield faster convergence and better speech understanding than cold initialisation; image-pretrained checkpoints perform worse than cold initialisation, confirming that cross-modal transfer requires semantic proximity.
- Interleaved speech-text pre-training (as in SPIRIT-LM) demonstrably improves alignment between text and speech representations, measurable through feature similarity visualisations.
- Mixed-objective tokenizers (SpeechTokenizer, Mimi) show promise in balancing semantic and acoustic trade-offs, but the area is described as early-stage.
- Post-alignment for SpeechLMs (Align-SLM, SpeechAlign) remains underexplored relative to its counterpart in text LMs.
- Full-duplex modelling is identified as an open frontier; most evaluated systems still use streaming architectures without true simultaneous bidirectional capability.
Novelty Assessment
The contribution is primarily taxonomic and conceptual rather than empirical or architectural. The survey’s value lies in its unified framework: by decomposing SpeechLMs into tokenizer-LM-vocoder triplets, classifying tokenizers along the semantic/acoustic axis, and distinguishing training stages (pre-training, instruction-tuning, post-alignment), it provides vocabulary and structure that subsequent work can reference. The taxonomy of interaction paradigms (batch, streaming, full-duplex) and the classification of evaluation methods are additional contributions. The survey does not introduce new models, datasets, or experimental results.
A reduced version was accepted at ACL 2025, confirming peer-reviewed recognition. Given the rapid pace of the field, some sections covering 2024 systems were necessarily incomplete at the time of writing (notably full-duplex systems and post-alignment methods).
Field Significance
Tip
High — This survey provides the field’s first systematic taxonomy of SpeechLMs, introducing a coherent shared vocabulary (tokenizer types, training stages, interaction paradigms) at a time when the literature was fragmented across dozens of independently framed systems. It enables cross-paper comparison and identifies the post-alignment and full-duplex interaction gaps as understudied frontiers. The ACL 2025 acceptance confirms its standing as a reference point for the spoken conversational agent research community.
Claims
- End-to-end speech generation models avoid the information loss, latency, and cumulative error introduced by cascaded ASR-LLM-TTS pipelines, but require integrating speech tokenisation and synthesis into a unified training regime. (§I, §II)
- Semantic tokenizers and acoustic tokenizers impose an inherent trade-off: semantic tokens produce coherent content but poor acoustic quality, while acoustic tokens enable high-fidelity reconstruction but risk content inaccuracies. (§III-A, §IV-A1)
- Initialising a speech LM from a text-pretrained checkpoint accelerates convergence and improves speech understanding, whereas initialisation from image-pretrained checkpoints yields worse results than random initialisation. (§IV-B1)
- Interleaving speech and text tokens during pre-training measurably improves cross-modal representation alignment compared to training on speech tokens alone. (§IV-B1)
- Post-alignment techniques (RLHF, DPO) for speech LMs remain substantially underexplored relative to their established role in text LM development, leaving semantic consistency and acoustic quality gaps in deployed systems. (§IV-B3, §VII)
Limitations and Open Questions
Warning
The survey’s arXiv version was submitted in October 2024 and the rapidly evolving SpeechLM landscape means several systems surveyed (notably Moshi, Mini-Omni, Llama-Omni) were still very recent preprints without peer-reviewed evaluation. The coverage of full-duplex systems and post-alignment techniques is acknowledged by the authors as incomplete.
The survey identifies several open questions: whether end-to-end joint training of all three components (tokenizer, LM, vocoder) outperforms separately trained pipelines; how to enable real-time speech generation with acceptable latency; the unique safety risks of SpeechLMs (toxicity, acoustic inappropriate content, speaker identity leakage) that differ from text LM safety challenges; and the potential of SpeechLMs for low-resource spoken languages where audio data is more available than text. The question of whether text alignment universally helps or degrades paralinguistic modelling (by anchoring the model too closely to textual semantics) is flagged but unresolved.
Wiki Connections
Concept pages most relevant to this survey:
- spoken-language-model — the central concept this survey taxonomises
- neural-codec — acoustic tokenizer component reviewed in depth (EnCodec, SoundStream, SpeechTokenizer, Mimi)
- self-supervised-speech — semantic tokenizers (HuBERT, wav2vec 2.0, WavLM) as the backbone of most SpeechLM pipelines
- autoregressive-codec-tts — the codec LM paradigm (VALL-E-style) is a specialised subset of the broader SpeechLM family
- speech-to-speech — SpeechLMs as the end-to-end alternative to cascade speech-to-speech systems
- evaluation-metrics — the survey’s Section VI systematises SpeechLM evaluation across representation, linguistic, paralinguistic, and downstream dimensions
In-corpus papers cited and central to the survey’s coverage:
- 2402.05755 (SPIRIT-LM) — interleaved speech-text pre-training
- 2408.16725 (Mini-Omni) — streaming SpeechLM with parallel text-speech decoding
- 2409.06666 (LLaMA-Omni) — instruction-following SpeechLM
- 2411.00774 (Freeze-Omni) — low-latency frozen-LLM spoken dialogue
- 2408.02622 (LSLM) — full-duplex listen-while-speak modelling
- 2412.10117 (CosyVoice 2) — scalable multilingual TTS within the SpeechLM context
- 2407.05407 (CosyVoice) — supervised semantic token TTS as a SpeechLM vocoder
- 2406.02430 (Seed-TTS) — high-quality speech generation surveyed under TTS capability
- 2411.13577 (WavChat) — concurrent survey of spoken dialogue models
- 2410.17196 (VoiceBench) — evaluation benchmark for SpeechLM voice assistants