arXiv · 2025 · Preprint
Labrak et al. · → Paper · Demo: ? · Code: ✓
A systematic empirical study training 51 speech-extended LLMs across encoder architectures, cluster counts, model scales, and k-means data sources to identify optimal discretization strategies for continual speech pre-training of text LLMs. (Published in the International Conference on Text, Speech, and Dialogue 2025.)
Problem
Adapting pre-trained text LLMs to process speech through vocabulary expansion with discrete units has emerged as a practical approach for building spoken language models (SpeechGPT-style), but the design space for these discrete representations is poorly characterized. Which self-supervised encoder provides the most useful features? How many k-means clusters are appropriate at different model scales? Does the data source for k-means training affect downstream robustness? These questions have remained largely unexplored, leaving practitioners without principled guidance for the critical continual pre-training stage.
Method
The paper adapts SmolLM, a family of small text LLMs at three parameter scales (135M, 360M, 1.7B), to the speech modality using a vocabulary expansion approach following the SpeechGPT paradigm. Speech is first encoded by one of four self-supervised models: WavLM, HuBERT, XLS-R, or Wav2Vec 2.0, all extracting features at 50 Hz from the final hidden layer. These frame-level representations are then discretized into k discrete tokens via k-means clustering, with k ranging across {125, 250, 500, 1000, 2500, 5000}.
The LLMs are trained with LoRA adapters (rank 64, alpha 16) on an autoregressive next-token prediction objective over the LibriSpeech 960h corpus tokenized with the discrete units. Training runs for 300 steps (approximately 157M tokens) on 16 H100 GPUs. To study the effect of the clustering data source, four k-means configurations are compared: clusters built from 2,000 hours drawn separately from LibriHeavy, GigaSpeech, People’s Speech, or CommonVoice 19.
Model quality is evaluated using Negative Log-Likelihood (NLL) on LibriSpeech test-clean. Robustness is assessed by applying Gaussian noise (two SNR levels) and random pitch shifts to the evaluation set. Phoneme alignment quality is analyzed by computing, for each discrete unit, the probability distribution over MFA-aligned phonemes.
Key Results
Among encoders, WavLM achieves the best NLL (2.05 at k=500 for 135M, 1.83 at k=500 for 1.7B) followed by HuBERT, with XLS-R and Wav2Vec 2.0 trailing substantially. The vocabulary size effect is strong and consistent: performance is competitive for k at or below 1,000, but degrades sharply at k=2,500 and k=5,000. For WavLM with SmolLM-135M, NLL rises from 2.05 (k=500) to 4.01 (k=2500) at step 300.
Larger models handle higher cluster counts more gracefully. SmolLM-1.7B maintains NLL of 1.83 to 1.95 across k=125 to k=1,000, while the 135M model already suffers at k=1,000. Both very large cluster counts trigger out-of-memory conditions for the 1.7B model in the experimental setup.
Cluster utilization analysis (Table 4) shows WavLM and HuBERT use 77-92% of available clusters, whereas XLS-R and Wav2Vec 2.0 achieve only 52-68% utilization on test-other, indicating inefficient vocabulary use under challenging acoustic conditions.
Domain matching emerges as a decisive factor for robustness. k-means built from LibriHeavy (read English, closely matched to LibriSpeech) achieves NLL=2.62 on clean speech and degrades only to 2.70 under heavy noise. GigaSpeech and CommonVoice-trained k-means start with higher baseline NLL (3.07 and 2.85) and degrade more under perturbations, despite containing noisier training audio. Training the k-means on noisy data does not confer noise robustness; it simply reduces overall quality.
Phonemic alignment analysis shows a clear diagonal structure in the unit-to-phoneme confusion matrix, with distinctive phonemes (vowels /AH/, /IY/, /UW/; consonants /S/, /F/, /M/; silence) each specializing in small sets of discrete units. This phonetic structure persists across encoder choices and cluster counts, and does not require explicit phonetic supervision.
Novelty Assessment
The contribution is purely empirical: no new architecture, training objective, or evaluation methodology is introduced. The paper’s value is the systematic sweep, training 51 models across all combinations of encoder, cluster count, model size, and data source, which is more comprehensive than prior ablations in the SLM literature. The findings consolidate and quantify intuitions that were individually present in the literature (WavLM tends to work well; smaller vocabularies are more stable; domain match matters) into a coherent practical guide. The phonemic alignment analysis is a useful diagnostic but is not novel in methodology. Reproducibility is supported by releasing tokenized datasets and clustering models.
Field Significance
Moderate: this paper provides practical design guidance for researchers adapting text LLMs to speech via discrete unit vocabulary expansion. The empirical sweep is thorough and the domain-matching finding (that noisy k-means data hurts rather than helps robustness) is a useful correction to a plausible prior assumption. The work is incremental relative to the SpeechGPT and GSLM lineage, and its scope is limited to the pre-training stage with no evaluation on downstream tasks, but the findings are directly actionable for practitioners building spoken language models.
Claims
-
supports: Discrete vocabulary size is the dominant design variable in speech LM continual pre-training, with compact cluster sets substantially outperforming fine-grained discretizations regardless of encoder choice.
Evidence: WavLM NLL degrades from 2.05 (k=500) to 4.01 (k=2500) and 4.21 (k=5000) for SmolLM-135M at step 300; the same degradation pattern holds for HuBERT, XLS-R, and Wav2Vec 2.0 across all model sizes. (§3.1, Table 1)
-
complicates: Optimal discrete vocabulary granularity for speech LM pre-training is not fixed across model capacities: larger models tolerate higher cluster counts where smaller models fail.
Evidence: SmolLM-135M degrades sharply at k=1,000 (NLL=2.19) versus k=500 (NLL=2.05), while SmolLM-1.7B maintains stable NLL of 1.83 to 1.94 across k=125 to k=1,000, suggesting capacity-dependent discretization regimes. (§3.2, Table 2)
-
supports: Domain matching between the clustering training corpus and the target speech domain is a critical factor for both discrete unit quality and robustness to acoustic perturbations.
Evidence: LibriHeavy-trained k-means (read English, in-domain with LibriSpeech) achieves NLL=2.62 clean and 2.70 under heavy noise; GigaSpeech-trained k-means degrades from 3.07 to 3.09 and CommonVoice from 2.85 to 3.11, showing that exposure to noisier audio during k-means construction does not improve robustness. (§3.3, Table 3)
-
supports: Self-supervised discrete speech units encode phoneme-level distinctions without explicit phonetic supervision, as evidenced by their systematic alignment with forced-alignment phoneme sequences.
Evidence: Phoneme confusion matrices on LibriSpeech test-clean (WavLM, k=125) show a strong diagonal structure where vowels and consonants each specialize into distinct unit subsets; the pattern persists across all four k-means data sources and holds at k=250. (§3.5, Figure 2)
Limitations and Open Questions
Warning
Evaluation uses only NLL on LibriSpeech, a proxy metric. No downstream task evaluation (spoken question answering, spoken language understanding, ASR) is included, so the practical implications of the observed NLL differences for real applications remain untested.
The study uses the SmolLM family exclusively, trained only on read English speech from LibriSpeech. Whether the encoder and cluster-count findings generalize to larger models, multilingual data, conversational speech, or production-scale training runs is unknown. The 1.7B model encounters out-of-memory limitations at k=2,500 and above, so the full picture for large models at high cluster counts is missing. The relationship between cluster utilization efficiency and downstream performance is posited but not directly measured.
Wiki Connections
- Spoken Language Model — provides empirical design guidance for the discrete unit vocabulary expansion approach to speech LM continual pre-training, examining how encoder choice and cluster granularity interact with model scale.
- Self-Supervised Speech — directly depends on WavLM, HuBERT, XLS-R, and Wav2Vec 2.0 as the feature encoders whose representations are discretized; encoder quality is a primary determinant of SLM pre-training quality.
- Evaluation Metrics — uses NLL on LibriSpeech test-clean as its primary evaluation criterion, arguing this metric correlates with downstream semantic speech understanding performance.