ICML · 2025 · Conference (Oral)
Park et al. · → Paper · Demo: ✓ · Code: ✗
SpeechSSM introduces the first textless spoken language model family to generate coherent speech at multi-minute timescales by replacing Transformer sequence modeling with a hybrid state-space architecture that maintains constant memory during decoding.
Problem
Spoken language models struggle to generate plausible, coherent speech beyond tens of seconds. Three compounding failures drive this ceiling: the high temporal resolution of speech tokens (10 or more per 1-2 text-equivalent tokens) forces models to maintain coherence over sequences far longer than text LMs face; standard Transformer architectures have quadratic prompt-time cost and linear per-step decoding cost, making multi-minute generation computationally infeasible; and Transformers trained on short sequences degrade sharply when generating beyond their training length. Prior efforts to improve coherence — via text-joint training or interleaved text-speech modeling — sidestep rather than solve this problem and require text at generation time. Generating directly from and to speech over many minutes, without text intermediates, remains unaddressed.
Method
SpeechSSM is a family of hybrid state-space spoken language models at 2B and 9B parameters. The semantic stage uses Griffin (De et al., 2024), which interleaves gated linear recurrent units (LRUs) with local sliding-window multi-query attention in a fixed two-recurrent, one-attention pattern. Recurrent layers maintain a constant-size hidden state, giving O(1) memory per decoding step, while local attention efficiently captures recent context. To support length extrapolation beyond training lengths, all explicit position embeddings are removed from the SSM layers, following the NoPE approach (Kazemnejad et al., 2023), preventing the model from anchoring to absolute positions seen during training.
The semantic tokenizer is USM-v2, a speaker-invariant speech tokenizer trained with masked language modeling on untranscribed audio (with auxiliary ASR supervision), producing 25 Hz discrete units from a 32k vocabulary. Both SpeechSSM-2B and -9B are initialized from RecurrentGemma-2B/9B-IT text LM weights, with text embeddings replaced by freshly initialized audio token embeddings. Models train on the LibriLight unlab60k split (~60k hours) with sequences up to 4 minutes using a standard language modeling objective.
A separate SoundStorm model non-autoregressively converts windows of semantic tokens into SoundStream acoustic tokens, conditioned on a 3-second speaker prompt. This two-stage design deliberately isolates speaker identity from the semantic LM: SoundStorm handles voice characteristics, leaving the semantic LM free to model content coherence without spending capacity on speaker representation.
Windowed tokenization and synthesis enable processing of arbitrarily long speech within bounded memory. Overlapping fixed-length windows (30 seconds, 4-second overlaps) are tokenized and synthesized independently, with boundary artifacts minimized by a symmetric overlap adjustment. A key engineering detail: to prevent implicit end-of-sequence signals arising from the non-causal USM-v2 tokenizer at window boundaries, the final window is padded with speech from the start of the example rather than silence, ensuring final-window tokens appear identical to mid-sequence tokens.
Key Results
On short-form generation (7s continuations of 3s prompts, LibriSpeech test-clean), SpeechSSM-2B achieves N-MOS of 3.87, outperforming GSLM (2.23), TWIST-7B (3.24), and Spirit LM Expressive (3.00), and slightly edging the matched SpeechTransformer-2B baseline (3.74). SpeechSSM-2B also achieves the lowest transcript perplexity (5.76) despite other systems being larger (7B) or jointly text-trained. Speaker similarity (0.79) is highest among all systems, attributed to the speaker-prompted acoustic stage (Table 2).
The long-form results are the paper’s centerpiece. On 4-minute continuations of 10-second prompts (LibriSpeech-Long test-clean), SpeechSSM-2B maintains consistent N-MOS scores of 4.12-4.16 across all time strata, matching ground-truth naturalness. In contrast, TWIST-1.3B drops from 1.96 to 1.36 MOS across time bins, and Spirit LM Expressive degrades acoustically over time. SpeechSSM-2B wins 33.9% of side-by-side comparisons against its matched Transformer counterpart. No model achieves any wins against LibriSpeech-Long ground truth in side-by-sides, placing all current systems clearly below human-level long-form speech (Table 3).
At 16-minute generation, SpeechSSM-9B achieves transcript perplexity of 3.39 and wins 68.3% of comparisons versus SpeechSSM-2B, while the Transformer baseline degrades sharply (PPL rises to 5.91, win rate 24.9%). Semantic coherence over length (SCL) confirms SpeechSSM maintains coherence past its 4-minute training boundary, while the Transformer fails at 16 minutes despite matching at shorter lengths (Table 4, Figure 5). Inference throughput exceeds 120x that of the comparable Transformer at 16,400-token sequences, with a real-time factor under 0.2x on a TPU v5e.
Novelty Assessment
The architectural contribution is genuine and domain-first: applying hybrid SSMs to spoken language modeling is new, and the combination of constant-memory decoding, training-length extrapolation, and long-form coherence stability has not been demonstrated in textless speech generation before. The Griffin architecture itself is borrowed from RecurrentGemma, but adapting it to speech requires non-trivial decisions: windowed tokenization, boundary artifact suppression, implicit EOS avoidance, and speaker-isolated two-stage synthesis. The SpeechTransformer-2B controlled baseline is an important methodological choice that helps isolate architecture from confounds.
The evaluation contribution is also substantive. The paper provides direct evidence that existing spoken LM metrics (sWUGGY, sBLiMP, transcript PPL, short-form holistic MOS) are unreliable or saturating, and proposes a more discriminative framework combining reference-based embedding similarity (SBERT, Gecko) with LLM-as-judge side-by-sides and time-stratified naturalness MOS. LibriSpeech-Long is a well-motivated benchmark derived from existing public-domain data that fills a clear gap.
The main caveat is that cross-system comparisons are partially confounded by differences in training data scale, tokenizer vocabulary, and text-joint training, even with the matched baseline. The semantic coherence advantage may partly reflect USM-v2’s strong speaker invariance rather than the SSM architecture alone. Model weights are not released.
Field Significance
Tip
High — SpeechSSM is the first spoken language model to demonstrate coherent, naturalistic speech generation at multi-minute timescales without text intermediates, establishing that SSM architectures address a practical ceiling that standard Transformers cannot overcome at inference time. The LibriSpeech-Long benchmark and accompanying evaluation methodology provide the field with tools to measure long-form generation quality, an area where prior metrics were demonstrably unreliable.
Claims
-
supports: Linear-time state-space sequence models enable stable long-form speech generation that Transformer spoken LMs cannot sustain at comparable scale.
Evidence: SpeechSSM-2B maintains N-MOS of 4.12-4.16 across all time strata up to 4+ minutes and stable semantic coherence in 16-minute generation, while all Transformer-based baselines and windowed extensions of prior spoken LMs degrade rapidly beyond their training lengths. (§7.1, §7.2, Tables 3-4)
-
supports: Separating speaker conditioning into a dedicated acoustic synthesis stage preserves voice identity more effectively in long-form speech generation than encoding speaker information in semantic tokens.
Evidence: SpeechSSM achieves SpkrSim of 0.85 in long-form evaluation (vs. 0.33-0.41 for GSLM and TWIST), attributed to SoundStorm’s speaker-prompted acoustic stage handling identity independently of the semantic LM. (§6.1, §7.1, Tables 2-3)
-
complicates: Standard evaluation metrics for spoken language models are unreliable or saturating for modern systems, particularly in long-form settings.
Evidence: sWUGGY scores negatively correlate with generation quality at large token vocabularies (32k vs. hundreds); short-form holistic MOS approaches ground truth at 7 seconds while qualitative failures persist; transcript perplexity favors repetitive outputs at default temperatures. Reference-based embedding metrics and LLM-as-judge side-by-sides prove more discriminative. (§6.1, §4)
-
complicates: Long-form speech generation remains far below human quality even for state-of-the-art spoken language models.
Evidence: No model achieves any wins against LibriSpeech-Long ground truth in 4-minute side-by-side comparisons judged by an LLM on fluency, coherence, logicality, and interestingness, demonstrating a large remaining quality gap despite strong short-form performance. (§7.1, Table 3)
Limitations and Open Questions
Warning
Model weights are not released and inference requires TPU infrastructure; independent reproducibility of the long-form results is not currently possible.
Comparison fairness across baselines is imperfect: systems differ in training data scale, semantic tokenizer choice, token vocabulary size, and whether text was seen during training. The authors partially address this with the matched SpeechTransformer-2B baseline, but cannot control all confounds across the full set of comparisons.
The semantic coherence advantage may partly reflect USM-v2’s speaker invariance (reducing token entropy from vocal variation) rather than the SSM architecture per se. SpeechSSM-X, the extemporaneous variant trained on 216k hours of monologue speech, is evaluated qualitatively only without quantitative benchmarks. LibriSpeech-Long covers read audiobook speech; generalization to spontaneous, multi-speaker, or conversational long-form speech is not evaluated.
Wiki Connections
- Spoken Language Model — introduces the first SSM-based spoken LM, enabling coherent generation at durations an order of magnitude beyond what existing Transformer spoken LMs can sustain
- Neural Audio Codec — uses SoundStream as the acoustic codec in the two-stage semantic-to-acoustic pipeline, with SoundStorm performing non-autoregressive acoustic token generation
- Self-Supervised Speech — depends on USM-v2 as the core semantic tokenizer, a model trained with masked language modeling on untranscribed audio, providing the speaker-invariant discrete representations the speech LM learns over
- Evaluation Metrics — introduces the LibriSpeech-Long benchmark and proposes LLM-as-judge side-by-sides and time-stratified naturalness MOS as more discriminative alternatives to saturated spoken LM evaluation metrics