arXiv · 2026 · Preprint
Luca Della Libera et al. (Concordia University / Mila-Quebec AI Institute / Université Laval) · → Paper · Demo: ✓ · Code: ?
Distills a pretrained WavLM checkpoint into an autoregressive speech language model that jointly represents semantic and acoustic information in a single token stream, without any text supervision or text-pretrained initialization.
Problem
Text language models achieve scalable, coherent generation from a single next-token prediction objective over a single token stream. Extending this simplicity to speech has proven difficult because speech entangles semantic, prosodic, and acoustic information across multiple time scales. Prior speech language models (SLMs) that model this jointly typically depart from the single-stream paradigm: they bootstrap from text-pretrained LLMs, interleave or hierarchically stack multiple token streams with distinct semantic/acoustic roles (as in Moshi’s dual temporal/depth-wise transformers or LLaMA-Mimi’s interleaved tokens), or add explicit text-alignment objectives. These design choices add architectural complexity, and where simplicity is preserved, it typically comes at the cost of requiring orders of magnitude more data and compute to reach comparable quality. The paper asks whether comparable performance can instead come from better representations rather than increased scale or architectural complexity.
Method
WavSLM builds on WavLM-large representations, specifically the 6th transformer layer, which the authors treat as a balance point between low-level acoustic detail and higher-level semantic content. Rather than training a new tokenizer, WavSLM reuses FocalCodec-Stream, an existing streamable neural codec that operates directly on WavLM-6 features via a causally distilled encoder, compressor, quantizer, decompressor, and decoder, producing a single stream of discrete tokens at 50 Hz. Decoded tokens can be projected back into a continuous feature space compatible with WavLM’s upper layers, which lets the tokenizer act as an interface between raw audio and WavLM’s own representation hierarchy.
The remaining WavLM layers (7-24) are repurposed as the language modeling backbone: a causal attention mask is applied and the layers are fine-tuned with a next-token prediction objective, with a lightweight linear head mapping features to a distribution over the token vocabulary. The entire distillation is speech-only, with the SLM initialized directly from the WavLM checkpoint rather than from a text-pretrained LLM. To match the tokenizer’s temporal resolution and speed up inference, the model predicts chunks of C = 4 consecutive tokens at each autoregressive step (next-chunk rather than next-token prediction), implemented via chunked causal attention and a left-shifted target sequence. A sliding-window attention mechanism bounds the context used at each step, enabling constant-latency, unbounded streaming generation. No explicit BOS/EOS tokens are used; utterances are zero-padded at the waveform level, which maps to silence tokens and lets generation continue producing silence without a hard termination signal.
Three model variants (WavSLM-2k, WavSLM-4k, WavSLM-65k; 305M-370M params) are trained, one for each of FocalCodec-Stream’s three codebook vocabulary sizes, on Libri-Light (~60k hours of unlabeled speech, with LibriSpeech dev-clean used for validation). Training uses AdamW with a learning rate of 1e-4, weight decay 0.01, gradient clipping at L2 norm 5.0, batch size 16, and a validation-loss-triggered learning-rate schedule, on a single NVIDIA H100 GPU.
Key Results
On likelihood-based evaluation (SALMon acoustic/sentiment/speaker/gender consistency, ZeroSpeech sWUGGY/sBLiMP, and Topic Story-Cloze), WavSLM-4k reaches the best average score (69.5) among the three WavSLM variants and matches or exceeds several substantially larger, text-pretrained baselines, including TWIST 7B (66.3), SpiRit LM (64.2) and SpiRit LM Expressive (69.4), and LLaMA-Mimi 1.3B (69.0), despite using 307M parameters and roughly 60-94k hours of speech (including WavLM’s own pretraining data) versus hundreds of thousands to millions of hours for the baselines (§4.1, Table 1).
On generation-based evaluation (continuation from a LibriSpeech test-clean prompt), WavSLM-2k obtains the best naturalness and speaker-similarity scores among all compared systems (UTMOS 3.72, speaker similarity 91.8), ahead of LLaMA-Mimi 8B (UTMOS 3.56, similarity 91.5). LLaMA-Mimi 8B still achieves lower perplexity (122 vs. 161-210 for WavSLM variants), indicating a residual linguistic-modeling gap despite WavSLM’s advantage in perceived quality. WavSLM’s real-time factor (5.8-5.9) is markedly higher than LLaMA-Mimi’s (1.1-2.0), reflecting faster generation from the smaller model and next-chunk prediction (§4.2, Table 2). The largest-vocabulary variant, WavSLM-65k, underperforms the smaller-vocabulary variants on generation metrics, which the authors attribute to increased modeling complexity relative to the fixed training data budget.
An ablation over context window and chunk size (§4.3, Table 3) shows that increasing the attention window from 512 to 2048 tokens yields modest, consistent gains in spoken-content metrics while preserving acoustic consistency, whereas increasing chunk size from 4 to 8 or 16 speeds up generation but substantially degrades both likelihood-based and generation-based metrics.
Novelty Assessment
The core contribution is a training recipe rather than a new architectural module: WavSLM reuses an existing self-supervised model (WavLM) and an existing streaming codec (FocalCodec-Stream) largely unchanged, and its novelty lies in distilling WavLM’s own upper layers directly into a causal, single-codebook language model without text supervision or text-pretrained initialization. The paper’s own framing, that this is the first SLM to jointly capture semantic and acoustic information in a single codebook without hierarchical or multi-stream tokenization, is a genuine methodological simplification relative to Moshi’s dual-transformer design and LLaMA-Mimi’s role-differentiated interleaved tokens. The results are competitive rather than dominant: WavSLM matches or exceeds several billion-parameter baselines on consistency benchmarks and perceived generation quality, but still trails the strongest baseline on generation perplexity, and the paper is transparent that further gains from decoding-hyperparameter tuning were not explored.
Field Significance
moderate — WavSLM provides a data point that a single-stream, speech-only distillation of an existing self-supervised model can match larger, text-pretrained speech language models on several consistency and generation benchmarks at a fraction of the parameter count and training data. It demonstrates a design simplification (single codebook, no text supervision, no hierarchical tokenization) as a viable alternative to the more common hybrid or interleaved architectures in this space, though the study is conducted at modest scale on a single GPU and English-only data.
Claims
- supports: Distilling the upper layers of a pretrained self-supervised speech model directly into an autoregressive next-token predictor can yield a competitive speech language model without requiring text supervision or text-pretrained initialization.
Evidence: WavSLM-4k (307M params, initialized from WavLM-large layers 7-24, trained speech-only on ~60k hours) matches or exceeds TWIST 7B, SpiRit LM, and LLaMA-Mimi 1.3B average scores on SALMon and ZeroSpeech benchmarks, all of which are text-pretrained and an order of magnitude larger. (§4.1, Table 1)
- supports: A single discrete token stream can jointly encode enough semantic and acoustic information to support coherent speech continuation, without needing separate hierarchical or multi-codebook streams.
Evidence: WavSLM’s single-codebook variants (1×2048 to 1×65536) achieve generation-based UTMOS and speaker-similarity scores that match or exceed LLaMA-Mimi’s 4-codebook design (4×2048) on the same continuation task. (§4.2, Table 2)
- complicates: Enlarging a single-stream speech token vocabulary does not monotonically improve downstream language modeling quality; excess vocabulary size can degrade generation without additional training data.
Evidence: WavSLM-65k (1×65536 codebook) scores noticeably lower on UTMOS, speaker similarity, and perplexity than the smaller-vocabulary WavSLM-2k and WavSLM-4k variants, despite identical architecture and training data. (§4.2, Table 2)
- complicates: Chunked next-chunk prediction trades inference speed against fidelity: larger prediction chunks reduce the number of autoregressive steps but degrade both acoustic and semantic quality.
Evidence: Increasing WavSLM-4k’s chunk size from 4 to 8 or 16 tokens raises real-time factor from 5.8 to 10.9 and 16.4 respectively, while UTMOS drops from 3.69 to 2.92 and 1.97, and spoken-content scores (sWUGGY, sBLiMP, tSC) also decline. (§4.3, Table 3)
- refines: Speech-only, text-free training narrows but does not close the linguistic coherence gap with text-pretrained speech language models, even when perceived audio quality and speaker consistency are competitive or superior.
Evidence: WavSLM outperforms LLaMA-Mimi 8B on UTMOS and speaker similarity but is outperformed on generation perplexity (161-210 vs. 122), which the authors attribute to LLaMA-Mimi’s stronger linguistic modeling from text pretraining. (§4.2, Discussion)
Limitations and Open Questions
The study is conducted at a comparatively small scale: all models are trained on a single H100 GPU on ~60k hours of speech (~94k hours when counting WavLM’s own pretraining data), which leaves open whether the reported gains stem primarily from representation quality, as the authors hypothesize, or would diminish or compound differently at the training scale of the larger baselines it is compared against. Evaluation is limited to English (Libri-Light, LibriSpeech); no multilingual results are reported. Generation-based evaluation reuses decoding hyperparameters (top-k sampling, temperature) from LLaMA-Mimi’s setup without WavSLM-specific tuning, which the authors note may understate WavSLM’s achievable quality. The largest-vocabulary variant (WavSLM-65k) underperforms the smaller variants, and the paper does not further investigate why beyond attributing it to insufficient training data for the larger vocabulary. Code and checkpoints are stated to be released publicly but were not confirmed available at the time of this paper’s publication.
Wiki Connections
- Spoken Language Model — WavSLM is itself a speech language model, distilling a self-supervised model into a single-stream autoregressive next-chunk predictor as an alternative to hierarchical or text-pretrained SLM designs.
- Self-Supervised Speech — The entire system is built on WavLM self-supervised representations, distilling and fine-tuning its upper transformer layers to form the language modeling backbone.
- Neural Audio Codec — WavSLM relies on FocalCodec-Stream’s single-stream tokenization and studies how codebook vocabulary size trades off against downstream language modeling quality.
- Streaming TTS — Chunked next-token prediction combined with sliding-window attention gives WavSLM constant-latency, unbounded streaming generation.
- FocalCodec-Stream — WavSLM adopts this streaming neural codec unchanged as its tokenizer, distilling WavLM’s upper layers into the same single-stream token space it defines.
- Llama-Mimi — Serves as WavSLM’s primary generation-quality baseline and evaluation protocol source; its multi-codebook interleaved design is the main point of contrast against WavSLM’s single codebook.
- MiMo-Audio — Cited as an example of the large-scale-compensates-for-complexity approach that WavSLM contrasts against its own efficiency-focused, text-free design.
- DyCAST — Cited as supporting evidence, from the same research group, for selecting WavLM’s mid-level layer representations as the basis for the tokenization pipeline.
- Late Fusion and Multi-Level Fission — Represents the joint speech-text training objective approach that WavSLM’s speech-only, text-free design is positioned against.