On The Landscape of Spoken Language Models: A Comprehensive Survey

arXiv · 2025 · Preprint

Arora et al. (CMU, Johns Hopkins, EHESS, NTU, et al.) · → Paper · Demo: ? · Code: ?

A unifying survey of spoken language models that proposes a shared formulation spanning pure speech LMs, speech+text LMs, and speech-aware text LMs, and maps architecture, training, and evaluation choices across the field.

Problem

The spoken language model (SLM) field had grown rapidly but without standardised terminology or a unifying conceptual framework. Models of similar capability were defined differently across research groups, evaluated on disjoint benchmarks, and placed in incompatible taxonomies depending on whether the authors came from speech processing, NLP, or multimodal AI backgrounds. Prior surveys either addressed audio broadly (including music and environmental sound) without sufficient depth on SLMs, or focused narrowly on speech-aware text LMs while omitting pure speech LMs and speech+text LMs. The result was a fragmented literature in which it was difficult to place any new model in context or to compare design choices systematically.

Method

The survey organises the SLM literature around a unified architectural formulation: a model that takes speech or text (or both) as input, passes speech through a speech encoder and a modality adapter, and feeds the resulting representations into an autoregressive sequence model, which can output speech, text, or both. This formulation subsumes three major model classes: (1) pure speech LMs, which model p(speech) via next-token prediction on discrete speech units; (2) speech+text LMs, which model the joint distribution p(text, speech) and produce both modalities; and (3) speech-aware text LMs, which model p(text | speech, text) by combining a pre-trained text LLM with a speech encoder and modality adapter.

Within this framework the survey examines four principal component choices. For the speech encoder, it distinguishes continuous features (mel filterbanks, SSL representations from wav2vec 2.0/HuBERT/WavLM, supervised encoders like Whisper, codec encoders) from discrete token types (phonetic tokens derived from quantised SSL features versus audio codec tokens from RVQ-based codecs). For the modality adapter, it reviews linear projections, strided CNNs, CTC-based compressors, Q-Former cross-attention adapters, and AlignFormer hybrids. For the sequence model, it categorises token generation strategies into four hierarchical patterns: coarse-to-fine sequential generation (AudioLM, VALL-E), interleaved coarse-and-fine generation (SpiRit-LM), temporal-plus-depth generation (Moshi, UniAudio), and delay-pattern generation (pGSLM). For speech+text hybrid decoding it identifies four alignment strategies: fixed-padding after text (Mini-Omni), fixed inter-token padding (LLaMA-Omni), dynamic padding (Moshi), and interleaved sequences (SpiRit-LM, GLM-4-Voice).

Training is mapped onto pre-training and post-training phases. Pre-training strategies include generative (pure speech or joint speech+text next-token prediction), conditional (Whisper-style ASR/ST initialisation), and implicit or explicit modality alignment. Post-training covers task-specific multi-task fine-tuning, instruction tuning (with LLM-generated SQA data, TTS-synthesised instruction data, and compositional instructions), and chat-specific training for duplex dialogue. Additional techniques such as parameter-efficient fine-tuning (LoRA, frozen encoders), progressive fine-tuning, pre-training data replay, and preference optimisation (RLHF, DPO) are also catalogued.

Evaluation is grouped into four categories: (1) likelihood-based metrics for pre-trained pure SLMs (sWUGGY, sBLIMP, StoryCloze, ProsAudit, SALMon); (2) generative metrics including intrinsic speech quality measures and task-based benchmarks (VoiceBench, Dynamic-SUPERB Phase 2, AIR-Bench, AudioBench, MMAU); (3) interactivity benchmarks for full-duplex dialogue (Full-Duplex-Bench, Game-Time Benchmark, latency tables); and (4) trustworthiness evaluations for hallucination, toxicity, bias, and deepfake vulnerability. A separate section covers duplex dialogue architectures: dual-channel models (dGSLM, Moshi) and time-multiplexing models with fixed (OmniFlatten, SCoT) or model-controlled switching.

Key Results

This is a survey paper; it reports no experimental results of its own. The key synthesised findings across the surveyed literature include:

• Cascaded ASR+LLM+TTS systems consistently outperform end-to-end SLMs on semantic and content-heavy benchmarks, but underperform on paralinguistic and speaker-related tasks, a split observed across Dynamic-SUPERB Phase 2, MMAU, and VoiceBench.
• VoiceBench and MMAU scores are negatively correlated across models, revealing that content-oriented QA ability and broader audio reasoning represent distinct capabilities not jointly optimised by current training pipelines.
• Among pure SLMs, models that achieve the best content modelling scores (StoryCloze, sWUGGY) are not necessarily the best on acoustic detail metrics (SALMon), indicating that semantic quality and acoustic fidelity are partially decoupled objectives.
• Reported first-packet latency for full-duplex SLMs ranges from 200 ms (Moshi, official claim) to 5,588 ms (SpeechGPT), a 28x spread that reflects fundamentally different architectural choices rather than just hardware differences (Table 3).
• Scaling studies for SLMs are almost entirely absent; the sole investigation (Cuervo and Marxer 2024) covers only pure speech LMs, leaving scaling behaviour for speech+text LMs and speech-aware text LMs unknown.

Novelty Assessment

The contribution is conceptual rather than empirical or architectural. The value lies in imposing a common formulation across a literature that has accumulated under incompatible terminologies. The unified encoder–adapter–sequence-model–decoder abstraction in Section 2 is the most reusable intellectual contribution: it lets a reader place any new SLM paper precisely in the taxonomy without relying on the authors’ own self-description. The survey also identifies gaps that were implicit in the literature but had not been stated as clearly before: the lack of scaling studies for non-pure SLMs, the negative correlation between content-focused and reasoning-focused benchmarks, and the absence of standardised full-duplex evaluation. The authors’ treatment of trustworthiness (hallucination, bias, deepfakes) is more systematic than in any prior SLM survey. The paper does not introduce new architectures, training procedures, or experimental results of its own.

Field Significance

Tip

High — this survey provides the most comprehensive unifying framework for spoken language models published as of early 2025. Its four-category taxonomy (pure speech LM, speech+text LM, speech-aware text LM, plus a general architectural formulation) can serve as a reference structure for researchers situating new work in the field. The explicit identification of evaluation inconsistencies, the latency comparison table, and the outline of open challenges are practically useful for anyone entering or navigating the SLM research space.

Claims

• Models that excel at content-level spoken language understanding tend to underperform on tasks requiring reasoning about acoustic and paralinguistic details, indicating that these capabilities require separate optimisation. (§7.2, Table discussion; VoiceBench vs. MMAU comparison)
• Cascaded speech pipelines and end-to-end spoken language models have complementary strengths: cascades are stronger on semantic tasks, while end-to-end models better handle speaker and paralinguistic information. (§7.2 “Comparisons of cascaded vs. end-to-end SLMs”)
• Jointly training a speech encoder with an LLM backbone improves instruction following and paralinguistic understanding but introduces risk of catastrophic forgetting of pre-trained text capabilities. (§7.2, §4.3)
• Full-duplex spoken dialogue requires architectural support beyond turn-taking assumptions, and current approaches (dual-channel and time-multiplexing) each involve significant trade-offs in latency, naturalness, and modelling complexity. (§6)
• The tokenisation choice (phonetic tokens versus audio codec tokens) determines the trade-off between linguistic coherence and speaker or acoustic fidelity in spoken language model outputs. (§3.1.2 “Comparison of token types”)

Limitations and Open Questions

Warning

The survey is primarily a snapshot of English-centric, high-resource SLM research. Multilingual, low-resource, and accessibility-focused SLMs receive minimal coverage, and the authors acknowledge that SLM research has largely not addressed dialects, accents, or speech-related medical conditions.

Beyond that scope limitation, several structural gaps are identified:

• Scaling behaviour for speech+text LMs and speech-aware text LMs is entirely unknown; existing scaling studies apply only to pure speech LMs.
• Most SLM evaluations use non-overlapping benchmarks, making direct performance comparisons across model families impossible.
• Very few SLMs are fully open-source (code, weights, and data), which prevents controlled ablations of design choices.
• The representation of spoken output in SLMs is unsettled: discrete tokens, continuous flows, and hybrid approaches have not been systematically compared under controlled conditions.
• Safety and trustworthiness evaluation for SLMs is nascent; non-verbal toxicity and speaker-type bias remain largely unstudied.

Wiki Connections

• spoken-language-model — primary concept taxonomised; the survey’s encoder–adapter–sequence-model formulation unifies pure speech LMs, speech+text LMs, and speech-aware text LMs
• neural-codec — audio codec tokens (§3.1.2) reviewed as one of the two principal speech token types
• self-supervised-speech — phonetic tokens from HuBERT/WavLM (§3.1.1, §3.1.2) reviewed as the semantic token paradigm
• speech-to-speech — duplex dialogue and speech+text LM output covered in §5.2 and §6
• evaluation-metrics — §7.1 and §7.2 systematise likelihood-based, generative, interactivity, and trustworthiness evaluation for SLMs
• 2410.00037 (Moshi) — full-duplex dual-channel SLM; central to the duplex taxonomy
• 2407.05407 (CosyVoice) and 2407.04051 (FunAudioLLM) — surveyed as speech+text LM examples
• 2410.17196 (VoiceBench) — generative benchmark covered in §7.2; negative correlation with MMAU identified
• 2406.14294 (DASB) — evaluation benchmark surveyed
• 2409.05377 (BigCodec) — audio codec token system reviewed
• 2402.13236 (Codec-SUPERB) — codec evaluation benchmark surveyed
• 2408.16725 (Mini-Omni) and 2410.11190 (Mini-Omni2) — speech-aware text LM examples surveyed
• 2412.02612 (GLM-4-Voice) — speech+text LM example surveyed
• 2411.13577 (WavChat) | 2502.06490 (Discrete Speech Tokens Review) | 2503.05085 | 2509.03940 | 2509.06926 | 2509.08753 — related surveys and concurrent work