ACL · 2025 · Conference (Findings)
Maimon et al. (The Hebrew University of Jerusalem) · → Paper · Demo: ✓ · Code: ✓
A systematic recipe (Slam) for training competitive speech language models on a single academic GPU in 24 hours, combining text-LM initialization, synthetic TTS data, and preference optimization to far exceed compute-optimal performance predicted by SLM scaling laws.
Problem
Training high-quality speech language models has demanded substantial compute: TWIST required up to 160 V100 GPU-days, SpiritLM used 560,000 hours of speech data across 64 A100s, and Moshi trained on 7 million hours. Published SLM scaling laws (Cuervo and Marxer, 2024) predict that reaching text-LM-level performance requires three times more tokens than comparable text models, painting a pessimistic picture for academic labs. Efficient training recipes for NLP (Cramming, ModernBERT) had no equivalent in the speech domain, leaving the question of compute-limited SLM training unexplored.
Method
Slam trains a decoder-only autoregressive speech LM over discrete speech tokens extracted by a multilingual HuBERT model running at 25 Hz, quantized to 500 units via k-means. The base architecture is Qwen2.5-0.5B, resized to 358M parameters after vocabulary reduction to speech units. The system uses TWIST-style initialization from the pretrained text LM as the starting point, then trains entirely within a 24-hour budget on a single A5000 GPU (24 GB VRAM).
The final recipe integrates four components derived from systematic ablation across model families, optimizers, schedulers, data mixes, and training objectives. For the model, Qwen2.5-0.5B with TWIST initialization consistently outperforms all other families (OPT, SmolLM2, MobileLLM, Pythia) both with and without initialization. For optimization, removing dropout, switching to a cosine decay LR schedule with AdamW (peak LR 1e-3), and using a context length of 1024 with RoPE theta of 10,000 together improve final performance substantially over the original TWIST recipe. For data, the core training mix is LibriLight plus LibriSpeech plus sTinyStories, a synthetic dataset generated by running a TTS model over the TinyStories text corpus; diverse multi-domain data (VoxPopuli, TED-LIUM, PeopleSpeech, SWC) consistently degrades performance under low-compute conditions, attributed to the difficulty of modeling acoustically diverse speech within the budget. For alignment, 30 minutes of DPO training on SpokenSwag (47k synthetically generated preference pairs derived from the SWAG commonsense text corpus) substantially improves generation quality; allocating more budget to DPO does not yield further gains and can degrade results.
Text-speech interleaving is also explored but underperforms speech-only training at the single-GPU budget: the larger vocabulary increases model parameters to 500M and reduces the effective number of speech tokens seen per step, limiting total steps to roughly 11k versus 18k for speech-only.
Key Results
On a single A5000 in 24 hours, Slam (358M params) achieves TSC 82.04, SSC 58.04, sBLIMP 58.86, and GenPPL 62.8. For context, TWIST-1.3B (1B params, 32 V100s) scores TSC 70.6 and GenPPL 131.8, while TWIST-7B (7B params) scores TSC 74.1 and GenPPL 93.7. Slam without DPO already matches or exceeds TWIST-7B on TSC and surpasses it on GenPPL. The predicted optimal performance from scaling laws for the same compute budget is TSC 70.49 and sBLIMP 56.85, which Slam surpasses by a substantial margin (Table 2).
When scaled to 2 A100s for 48 hours (Slam scaled), TSC reaches 84.18 and GPTScore 2.69, comparable to SpiritLM (7B params, 64 A100s) at TSC 82.9 and within range of AlignSLM-7B (GPTScore 3.50, 64 A100s). On out-of-domain generalization (People Speech GenPPL), Slam (scaled) scores 128.78 against TWIST-7B at 128.74, despite TWIST being trained explicitly on People Speech data (Table 11).
Novelty Assessment
No component of Slam is architecturally novel: TWIST initialization, DPO from preference data, HuBERT tokenization, and TinyStories synthetic augmentation all come from prior work. The contribution is the systematic ablation within a strict compute budget and the resulting reproducible recipe, analogous to Cramming (Geiping and Goldstein, 2023) for text LMs. The empirical finding that scaling-law predictions are overly pessimistic under this combination of techniques is meaningful: it reframes the feasibility of SLM research for resource-constrained settings. The open-sourced code, model weights, and synthetic datasets add practical value beyond the recipe itself.
Field Significance
Moderate — Slam demonstrates that the compute barrier to SLM research is lower than scaling-law extrapolations implied, providing concrete evidence that well-chosen initialization, synthetic data, and preference optimization can substitute for large data and multi-GPU training. The contribution is primarily engineering and empirical rather than architectural, but the accessibility argument is well-supported and the ablation study is thorough enough to inform other SLM training efforts.
Claims
-
supports: Text LM initialization substantially accelerates SLM convergence and improves final model quality within fixed compute budgets.
Evidence: TWIST-style initialization from Qwen2.5-0.5B outperforms uninitialised variants across all evaluated model families; by end of training, Qwen2.5 with initialization surpasses alternatives even with fewer training tokens. (§4.1, Figure 2, Figure 3)
-
supports: Synthetic TTS data improves semantic modeling in SLMs when combined with real speech, but replacing real data with synthetic data alone degrades performance.
Evidence: Adding sTinyStories to the training mix boosts TSC from 71.14 to 78.01 and GenPPL from 145.4 to 88.3 for Qwen-0.5B; training on synthetic data exclusively yields sBLIMP 52.35 versus 56.45 for the mixed baseline. (§4.2, Table 1)
-
supports: Preference optimization with synthetically generated preference data substantially improves SLM generation quality even under tight compute constraints.
Evidence: 30 minutes of DPO training on SpokenSwag improves TSC from 78.01 to 82.04 and GenPPL from 88.3 to 62.8; the gain saturates quickly and additional DPO budget can degrade results. (§4.4, Figure 5, Table 2)
-
contradicts: SLM scaling laws accurately predict compute-optimal performance boundaries for academic-scale training.
Evidence: Slam on one A5000 GPU-day achieves TSC 82.04 against the scaling-law predicted optimal of TSC 70.49 for the same compute, and sBLIMP 58.86 against the predicted 56.85; the authors attribute the gap to text LM initialization and synthetic data not being factored into scaling law derivations. (§5, Table 2)
-
complicates: Diverse multi-domain speech training data improves SLM generalization.
Evidence: Augmenting LibriLight and LibriSpeech with VoxPopuli, TED-LIUM, PeopleSpeech, and SWC consistently degrades TSC and SSC for both OPT-125M and Qwen-0.5B under the single-GPU budget; the authors hypothesize that modelling acoustic diversity requires more compute than the Slamming budget allows. (§4.2, Table 1)
Limitations and Open Questions
Warning
The evaluation uses only semantic modeling benchmarks (TSC, SSC, sBLIMP) and generation perplexity. Acoustic and prosodic quality are not evaluated; the authors note that benchmarks such as SALMon may reveal further limitations of low-resource SLM training in these dimensions. (§7)
The recipe relies exclusively on HuBERT semantic tokens, leaving open whether the same efficiency gains apply with newer tokenizers such as Mimi or SylBoost. Text-speech interleaving underperforms speech-only training at the single-GPU budget due to vocabulary expansion and fewer speech tokens per step; whether this changes at higher compute is unresolved. Out-of-domain generalization (People Speech) shows that Slam scaled matches but does not clearly outperform TWIST-7B, despite Slam using only audiobook-domain training data that TWIST also trained on.
Wiki Connections
- Spoken Language Model — Slam directly advances SLM accessibility by establishing that competitive performance does not require industrial compute, with systematic ablation of every major training component.
- Self-Supervised Speech — The paper’s tokenization pipeline is built on multilingual HuBERT, making HuBERT discrete units a load-bearing dependency of the entire Slam recipe.
- RLHF Speech — DPO on synthetically generated preference pairs is a core component of the final Slam recipe, contributing the largest single performance boost over pre-training alone.