arXiv · 2025 · Preprint
Morteza Rohanian et al. (University of Zurich) · → Paper · Demo: ✓ · Code: ?
Shapes a speech language model’s token embeddings and training objectives, without touching the tokenizer or architecture, to make a 0.7B-parameter model more acoustically consistent than 7B-parameter baselines.
Problem
Speech language models that predict discrete codec tokens autoregressively often fail to hold acoustic properties (speaker identity, gender, room, background, sentiment) stable across a generated utterance, especially under timing variation or noisy context. Prior attempts to fix this typically add architectural machinery: multi-stage decoders, adapter modules, supervised control heads, or expressive control tokens layered onto the codec/LM pipeline. This raises system complexity and makes it hard to tell whether a gain in stability came from the added structure or from the underlying language model itself. The paper asks whether acoustic consistency can instead be controlled purely through how the LM’s embeddings and training objectives are designed, holding the tokenizer, codec, and decoding path fixed.
Method
The system, called CAST, treats a frozen neural audio codec (WavTokenizer, single-stream scalar/vector quantization, 4096 codes at 24kHz) as a black box and performs all adaptation on the language-model side. A decoder-only transformer (Gemma 3 1B backbone) is extended with one new token per codec index plus [Text]/[Speech] delimiters, so speech and text share a single vocabulary and next-token prediction objective; the BPE core and codec are never modified.
Three design choices constitute CAST. First, semantic-distilled initialization: rather than starting audio-token embeddings from scratch, each is initialized from a frozen HuBERT encoder’s per-code centroid (projected through a small linear map), and a stop-gradient alignment loss during training keeps the LM’s hidden states close to these SSL features. Second, consistency training: sequences are thinned at variable rates (keeping every r-th token for r in {1,2,3,4}) and randomly erased in short spans, teaching the model to predict under timing jitter and missing context. Third, delayed coarse-then-fine auxiliary losses: audio tokens are clustered into coarse buckets, and the model is trained with both a coarse-label loss and a next-code loss, biasing it to plan coarse content structure before committing to fine acoustic detail.
Three variants are trained on LibriLight (57k hours) plus the clean subset of People’s Speech (~20k hours): a 0.7B speech-only model (56k text tokens + 4096 speech tokens, no text vocabulary needed), a 1.0B speech-only model (full 262k text vocabulary retained but unused for text), and a 1.0B interleaved model that alternates speech and text spans from paired audio-transcript data, with thinning/erasure applied only to the audio subsequence. Evaluation is likelihood-based: length-normalized negative log-likelihood is used to score whether the model prefers natural recordings over versions with a mid-utterance acoustic or content change, following the SALMON protocol.
Key Results
On SALMON acoustic consistency, the 0.7B speech-only CAST model scores highest across most factors (90.8 speaker, 90.0 gender, 81.8 sentiment), outperforming not only the 1.0B speech-only variant but also 7B baselines: SpiritLM 7B (81.0 speaker, 85.0 gender) and Twist 7B (71.0 speaker, 70.0 gender). Interleaving speech and text in the 1.0B model reduces consistency by 5-9 points across all SALMON factors relative to the 1.0B speech-only model (e.g., speaker drops from 90.0 to 83.5) while improving semantic-acoustic alignment (sentiment alignment rises from 48.5 to 54.5, background from 51.5 to 59.0) and lexical probes (sWUGGY rises from 67.0 to 73.7). Twist 7B still reaches the highest sWUGGY score overall (82.8) but does not lead on sBLiMP, and no system approaches human-level scores on any SALMON dimension (human consistency and alignment scores exceed 88 and 93 respectively). An ablation removing the coarse/next-code auxiliary losses drops 0.7B speaker consistency from 90.8 to 83.5 and 1.0B speaker consistency from 90.0 to 83.0, with background and room degrading more moderately. Linear probes on frozen LM features show the SSL-initialized variant gains roughly +16% relative accuracy on content-leaning tasks (ESC-50, US8K, RAVDESS) but loses about 7% relative on prosody-leaning tasks (VIVAE, EMOVO) versus a randomly initialized baseline.
Novelty Assessment
The architecture itself is not new: a standard decoder-only transformer over an off-the-shelf codec is the same recipe used by Twist, SpiritLM, and related speech LMs. The contribution is a training-recipe bundle: SSL-centroid embedding initialization with a stop-gradient alignment loss, multi-rate thinning/span-erasure augmentation, and delayed coarse-then-fine auxiliary objectives, combined and ablated specifically to target acoustic consistency rather than downstream task accuracy. Individually, SSL-informed initialization and coarse-to-fine (semantic-then-acoustic) modeling are established ideas in the neural codec LM literature (e.g., AudioLM’s separate semantic/acoustic streams); this paper’s contribution is showing that these ideas, applied purely to the LM’s embeddings and objectives without any change to codec, tokenizer, or decoder structure, can substitute for scale. The headline result, a 0.7B model matching or beating 7B baselines on acoustic consistency, is a genuine and useful empirical finding, though the baselines being compared against differ in tokenizer and training data as well as scale, so the comparison isolates the training recipe less cleanly than the paper’s framing suggests.
Field Significance
Moderate — the paper provides a controlled demonstration that acoustic stability in speech language models is governed more by embedding initialization and auxiliary training objectives than by parameter count, and that mixing text into the training stream trades consistency for semantic grounding in a way that is tunable rather than fixed by architecture. This is a useful data point for anyone designing speech LM training recipes, but the evaluation is confined to likelihood-based consistency probes rather than generation quality or downstream task performance, and the paper is a preprint without independent replication.
Claims
- supports: Acoustic consistency in autoregressive speech language models can be improved through LM-side embedding initialization and auxiliary training objectives, independent of parameter count.
Evidence: The 0.7B speech-only CAST model scores 90.8 speaker and 90.0 gender consistency on SALMON, exceeding the 7B SpiritLM (81.0/85.0) and 7B Twist (71.0/70.0), despite using an order of magnitude fewer parameters and the same evaluation protocol. (§4, Table 1)
- complicates: Interleaving text and speech tokens in a shared-vocabulary language model improves semantic and lexical grounding at the cost of acoustic consistency.
Evidence: The 1.0B interleaved model drops SALMON consistency by 5-9 points across every measured factor relative to the 1.0B speech-only model (e.g., speaker 83.5 vs. 90.0) while sWUGGY rises from 67.0 to 73.7 and semantic-acoustic alignment scores improve by 6-7.5 points. (§4, Tables 1-2)
- supports: Auxiliary objectives that require the model to plan coarse content structure before predicting fine acoustic detail materially contribute to speaker- and gender-identity stability beyond what embedding initialization alone provides.
Evidence: Removing the delayed coarse-label and next-code auxiliary losses drops speaker consistency from 90.8 to 83.5 in the 0.7B model and from 90.0 to 83.0 in the 1.0B model, with background and room consistency declining more moderately. (§4, Table 4)
- refines: Initializing speech-token embeddings from self-supervised acoustic features biases the resulting representation toward content structure at the expense of fine-grained prosodic detail.
Evidence: Linear probes on frozen LM features show the SSL-initialized variant gains on average +16% relative accuracy on content-leaning tasks (ESC-50, US8K, RAVDESS) while losing about 7% relative accuracy on prosody-leaning tasks (VIVAE, EMOVO), compared to a randomly initialized baseline. (§4, Table 3)
Limitations and Open Questions
The paper evaluates consistency exclusively through likelihood-based pairwise preference scoring (SALMON) rather than through generated-sample quality or human listening tests; no MOS or subjective naturalness evaluation is reported, so it is untested whether the consistency gains translate into perceptibly better generations. The comparison against Twist, SpiritLM, LAST, and Flow-SLM baselines is confounded by differences in tokenizer, training data, and architecture beyond scale, which the paper acknowledges only implicitly by noting these systems “differ in tokenization, architecture, and training objectives.” The 1.0B ablation shows a single anomalous result for room consistency that the authors attribute to run instability rather than investigate further. All experiments use English-only, largely read-speech and audiobook data (LibriLight, People’s Speech), leaving open whether the training recipe generalizes to conversational, multilingual, or noisier real-world speech.
Wiki Connections
- Spoken Language Model — proposes an LM-side training recipe (SSL-distilled initialization, alignment loss, thinning augmentation, coarse-then-fine auxiliary losses) for improving acoustic consistency in decoder-only speech LMs without changing tokenizer or architecture.
- Neural Audio Codec — treats WavTokenizer as a frozen black-box vocoder and shows that consistency problems attributed to codec token discretization can instead be mitigated entirely on the language-model side.
- Self-Supervised Speech — initializes and aligns LM speech-token embeddings against frozen HuBERT features via a stop-gradient distillation loss, and finds this biases representations toward content over prosody.
- Evaluation Metrics — relies on the SALMON acoustic-consistency benchmark, sWUGGY/sBLiMP lexical-syntactic probes, and ARCH linear probes to compare speech LMs without any human listening evaluation.