ACL · 2025 · Conference
Wenqian Cui et al. (Chinese University of Hong Kong / Tencent / National University of Singapore) · → Paper · Demo: ? · Code: ?
A systematic taxonomy of Speech Language Models (SpeechLMs), covering the three-component architecture (speech tokenizer, language model, vocoder), training recipes across all three stages, evaluation benchmarks, and open challenges including safety and full-duplex interaction.
Problem
The naive “ASR + LLM + TTS” pipeline for speech interaction has three fundamental problems: paralinguistic information (pitch, timbre, prosody) is lost when speech is converted to text and back; chaining three independently-trained modules introduces cumulative error; and sequential processing adds substantial latency. End-to-end SpeechLMs — foundation models that tokenize, model, and synthesize speech directly — address these problems but had, at time of publication, no comprehensive taxonomy. The space of design choices (which tokenizer? which LM architecture? which vocoder? how to align text and speech?) was fragmented across dozens of papers with inconsistent terminology.
Method
This is a survey paper. The taxonomy organizes SpeechLMs along three structural axes.
The speech tokenizer converts waveforms into tokens used by the language model. The survey distinguishes three objective families: semantic understanding tokenizers (HuBERT, WavLM, w2v-BERT) produce tokens that align well with text but sacrifice acoustic detail; acoustic generation tokenizers (neural codecs: SoundStream, EnCodec, DAC) preserve high-fidelity audio characteristics but often produce semantically inconsistent outputs; mixed-objective tokenizers (SpeechTokenizer, Mimi) distill semantic representations into the first quantizer layer of a codec architecture to capture both. The choice between discrete and continuous token representations is also surveyed.
The language model is almost universally a decoder-only transformer, adapted for speech either by replacing the text vocabulary with a speech vocabulary or by expanding the joint vocabulary to include both text and speech tokens. Joint modeling enables transfer from pretrained TextLMs, which the survey shows consistently improves convergence and downstream performance (TWIST, AudioPaLM). The survey distinguishes cold initialization from continued pretraining, and single-sequence interleaving (SPIRIT-LM) from multi-sequence parallel generation (Mini-Omni, Moshi).
The vocoder reconstructs waveforms from generated tokens. GAN-based vocoders (HiFi-GAN, BigVGAN) dominate in practice due to their computational efficiency and real-time factor orders of magnitude better than autoregressive or diffusion alternatives. The survey documents two synthesis pipelines: direct synthesis (acoustic tokens → HiFi-GAN) and input-enhanced synthesis (semantic tokens → flow-matching intermediate → HiFi-GAN, as in CosyVoice).
Training is covered across three stages: large-scale pretraining (with or without TextLM initialization), instruction tuning (cross-modal fine-tuning, chain-of-modality tuning), and post-alignment (DPO for semantic consistency in AlignSLM; token distribution alignment in SpeechAlign). The survey identifies post-alignment as substantially underexplored compared to the TextLM literature.
The interaction paradigm section distinguishes traditional (full input → full output) from real-time/full-duplex modeling. Full-duplex systems that support simultaneous bidirectional communication — including user interruption and simultaneous response — are surveyed as an emerging frontier, with dGSLM, Parrot, Moshi, and LSLM as representative systems. Evaluation is catalogued across representation, linguistic, paralinguistic, generation quality/diversity, real-time interaction, and downstream task dimensions. VoxEval is highlighted as the only benchmark requiring speech output rather than text output, closing a gap in full end-to-end SpeechLM assessment.
Key Results
As a survey, this paper reports no original experimental results. The synthesised empirical picture includes: HuBERT outperforms CPC and wav2vec 2.0 as a semantic tokenizer (GSLM); continued pretraining from TextLMs improves convergence and task performance over cold initialization (TWIST); interleaving text and speech tokens during pretraining improves cross-modal performance (SPIRIT-LM); larger TextLM checkpoints and more data continue to improve SpeechLMs (AudioPaLM, GLM-4-Voice with 13M hours of synthetic data). Across vocoder comparisons, GAN-based vocoders achieve real-time factors 70–3,500x faster than diffusion models at competitive perceptual quality, explaining their prevalence in production SpeechLMs. The semantic–acoustic token trade-off is formalized: semantic tokens excel at content modeling (lower WER) but fail at acoustic fidelity (lower SPK-SIM), while acoustic tokens invert this profile; mixed tokens offer partial balance at the cost of design complexity.
Novelty Assessment
The contribution is organizational and synthesizing rather than empirical. The three-component taxonomy (tokenizer, LM, vocoder) with sub-classification of tokenizer objectives is the paper’s principal intellectual contribution, and it is genuinely clarifying for the field. The coverage of full-duplex modeling and post-alignment is timely; both were underrepresented in prior SpeechLM overviews. The survey does not introduce new methods, metrics, or benchmarks. The VoxEval benchmark is cited (by the same first author group) as the only speech-output evaluation tool, which is a self-citation that slightly inflates its prominence. At 160 references across roughly two years of rapid system development, some coverage decisions are selective and some systems (especially those from late 2024) receive only brief treatment. The companion GitHub repository (Awesome-SpeechLM-Survey) adds ongoing curation value beyond the static paper.
Field Significance
Moderate — This survey provides the field’s most systematic taxonomy of SpeechLMs at a moment when the design space was expanding rapidly and terminology was inconsistent across communities (NLP, speech, and dialogue systems). Its value is primarily organizational: the tokenizer objective classification and the full-duplex interaction taxonomy give subsequent work a shared vocabulary. As with most surveys, it is most useful as a map for entering the area rather than as evidence for specific design choices.
Claims
- GAN-based vocoders occupy the dominant position in production speech generation systems because their computational efficiency advantage over autoregressive and diffusion alternatives is orders of magnitude, at acceptable perceptual quality cost. (§3.3.1, §F.3, Table 8)
- Continued pretraining from a text language model checkpoint improves SpeechLM convergence and downstream task performance compared to random initialization. (§4.2.1)
- Semantic tokenizers and acoustic tokenizers represent complementary capability profiles — strong semantic content fidelity versus strong acoustic reconstruction fidelity — and no single tokenizer type dominates both dimensions. (§3.1, §F.2, Table 6)
- Post-alignment of SpeechLMs via preference optimization addresses qualitatively different failure modes (semantic inconsistency, token distribution mismatch) than post-alignment of text LLMs. (§4.2.3)
- Full-duplex spoken interaction — simultaneous bidirectional speech with interruption support — requires joint modeling of both speaker streams and remains an open research challenge. (§4.3)
Limitations and Open Questions
Coverage necessarily lags the field: systems published after mid-2024 receive limited treatment, and the overall corpus skews heavily toward English and Mandarin. The safety section identifies toxicity and speaker privacy risks but does not analyze them quantitatively. End-to-end training that backpropagates gradients from vocoder output to tokenizer input is flagged as a potentially high-value research direction but remains unexplored. The question of whether incorporating text modality fundamentally benefits or constrains speech intelligence is left open.
Wiki Connections
This survey is the primary entry point for the spoken-language-model concept and provides context for neural-codec (SoundStream, EnCodec, SpeechTokenizer, Mimi), self-supervised-speech (HuBERT, WavLM), evaluation-metrics (VoxEval, VoiceBench, Full-Duplex-Bench), rlhf-speech (AlignSLM, SpeechAlign), and autoregressive-codec-tts (GSLM through Moshi lineage). In-corpus paper: Full-Duplex-Bench.