Speech Generation Research Wiki

Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback

7 min read

ACL · 2025 · Conference

Lin et al. · → Paper · Demo: ? · Code: ?

Align-SLM applies preference optimization (DPO with LLM-generated AI feedback) to textless spoken language models, demonstrating for the first time that RLHF-style training substantially improves semantic coherence in end-to-end speech-to-speech generation without any text token injection.

Problem

Textless spoken language models (SLMs) trained on next-speech-token prediction generate continuations that are often repetitive, grammatically inconsistent, and semantically irrelevant to the input prompt. The root cause is structural: speech tokens derived from self-supervised models are finer-grained and less informationally compact than text subwords, so the next-token objective fails to capture long-range semantic dependencies. Prior work has addressed this through text injection at inference (speech-text interleaving, intermediate ASR-LM-TTS decoding chains), but these approaches break the end-to-end property and introduce latency. Whether purely textless SLMs can generate semantically coherent speech without text guidance is the central question the paper pursues.

Method

Align-SLM builds on TWIST, a decoder-only transformer (available in 1.3B and 7B parameter versions) initialized from a text LLM and trained on next-speech-token prediction. Speech is represented as discrete units extracted by applying K-means clustering (500 clusters) to HuBERT features at 25 Hz from the model’s 11th layer, with consecutive duplicates removed. A HiFi-GAN unit vocoder synthesizes waveforms from these discrete units.

The training framework proceeds in three stages. Given a speech prompt, N=5 continuations are sampled via nucleus sampling from the pre-trained SLM. An automatic preference data selection pipeline then creates (prompt, chosen, rejected) triples: each candidate is transcribed by Whisper-large-v2, then scored by Mistral 7B (instruct-v02) on a 1–5 scale for semantic coherence and continuation likelihood. Candidates with auto-BLEU above a threshold are rejected as repetitive regardless of score. DPO is then applied using LoRA adapters (rank 32, alpha 8) with the backbone frozen; the reference model is the original pre-trained SLM, so no separate reward model is needed.

An optional curriculum learning stage iteratively raises the preference data quality thresholds (s_c from 3 to 4, s_r from 1 to 2 for the Mistral score) and re-runs DPO from the previous Align-SLM checkpoint. The paper explores two rounds of curriculum learning on the 7B model, finding consistent improvement across iterations.

The illustration of the Align-SLM framework. Firstly, tokenize the speech prompt into speech tokens using a speech tokenizer. Then, generate and sample multiple speech continuations from the pre-trained SLM. To create the preference data pairs, the framework uses a unit-based vocoder to synthesize the speech continuations back to waveforms, an ASR model to transcribe the waveforms to text, and an LLM evaluator to rate the quality of the semantics. These preference data pairs are then used for direct preference optimization to train the LoRA adapter in the SLM. This alignment process can be coupled with curriculum learning to further improve performance.

Key Results

On the 7B model with curriculum learning and MLS data augmentation (Align-SLM-mls+CL), the framework achieves 77.9% on sWUGGY, 61.1% on S-StoryCloze, and 86.8% on T-StoryCloze (Table 2) — state-of-the-art among textless SLMs on all three metrics. T-StoryCloze performance (86.8%) approaches human-level accuracy (90.2%) and surpasses speech-text interleaving approaches (SPIRITLM: 82.9%, Moshi: 83.0%) despite operating purely in the speech domain.

Human evaluations using MMOS (Meaningfulness MOS) show Align-SLM 7B+CL scoring 3.73 ± 0.06 versus 3.48 ± 0.07 for the pre-trained TWIST 7B baseline and 3.50 ± 0.07 for re-synthesized ground truth continuations (Table 3). GPT-4o-evaluated semantic quality improves from 2.70 to 3.56 on the 7B model, and MOSnet audio quality scores remain comparable throughout (4.08–4.09 vs 3.80 for the baseline).

LLM-based preference selection (Mistral score) substantially outperforms perplexity-based selection: on the 1.3B model, the Mistral variant improves T-StoryCloze by +4.5 points over the pre-trained baseline while the PPL variant degrades it by 2.0 points (Table 1). The cascaded ASR+LLM topline (79.2% sWUGGY, 94.8% T-StoryCloze) remains substantially ahead, indicating remaining headroom.

Novelty Assessment

The primary contribution is the first application of preference optimization to textless spoken language models — a meaningful transfer of DPO and RLAIF from text LLMs to the speech-only domain. The automatic preference selection pipeline (Whisper transcription + Mistral LLM scoring + auto-BLEU repetition filter) is a practical adaptation of RLAIF methodology to a domain where human annotation of preference pairs would be prohibitively expensive. The curriculum learning coupling is straightforward but demonstrably effective.

The underlying architecture (TWIST with LoRA) is not novel; the contribution is the training procedure and preference data construction strategy. The evaluation is limited to English audiobook speech, and comparisons with text-injecting models (SPIRITLM, Moshi) are not controlled for compute or data budget, leaving the cost-benefit tradeoff of textless versus text-guided approaches unresolved.

Field Significance

Moderate — Align-SLM fills a clear gap by demonstrating that alignment techniques from text LLMs transfer to textless SLMs and yield meaningful semantic improvements without text token injection. The framework is general enough to apply to other SLM backbones and lays groundwork for extending preference optimization in speech to dimensions beyond semantics — prosody, speaking style, paralinguistics.

Claims

  • supports: Preference optimization with LLM-based semantic feedback improves long-range semantic coherence in textless spoken language models beyond next-token prediction training.

    Evidence: DPO with Mistral-score preference data raises T-StoryCloze from 69.7% to 74.2% (1.3B model) and from 75.4% to 83.8% (7B model) over the pre-trained TWIST baseline; curriculum learning further improves both to 76.1% and 85.6% respectively. (§5.2, Table 2)

  • supports: LLM-based semantic scoring provides more effective alignment signal for speech continuation than perplexity-based preference selection.

    Evidence: On the 1.3B model, Align-SLM with Mistral score improves T-StoryCloze by +4.5 and GPT4-o score by +0.24 over the pre-trained baseline, whereas the PPL-based variant degrades T-StoryCloze by 2.0 and improves GPT4-o by only +0.03. (§5.1, Table 1)

  • supports: Curriculum learning with iteratively tightened preference thresholds yields progressive semantic improvements in preference-optimized spoken language models.

    Evidence: Two curriculum learning iterations on the 7B model with LibriSpeech data improve T-StoryCloze from 83.8% (Align-SLM) to 85.6% (Align-SLM+CL) and GPT4-o from 3.50 to 3.56; further iterations continue to improve most metrics. (§5.3, Appendix E, Table 2)

  • complicates: Semantic gains from preference optimization do not transfer uniformly to lexical and grammatical aspects of spoken language modeling.

    Evidence: Align-SLM shows only marginal sBLIMP improvement (+1.3 for the 1.3B Mistral variant) while AudioLM and SyllableLM, which use no preference training but have better speech token designs, outperform on grammatical correctness. (§5.5, Table 2)

  • complicates: Textless end-to-end spoken language models remain substantially behind cascaded ASR+LLM pipelines in semantic understanding even with preference optimization.

    Evidence: The best Align-SLM variant achieves 77.9% sWUGGY and 86.8% T-StoryCloze, compared to 79.2% and 94.8% for the ASR+LLM cascade topline, a gap that preference optimization alone does not close. (§5.5, Table 2)

Limitations and Open Questions

Training and evaluation are restricted to English audiobook speech (LibriSpeech, MLS), a clean and structured domain. Generalisation to spontaneous, noisy, or multi-speaker speech is untested, and the automatic preference pipeline’s reliance on ASR transcription quality may degrade under real-world acoustic conditions.

The paper does not compare against text-injecting SLMs (SPIRITLM, Moshi, VoxtLM) under matched data and compute budgets, so the relative cost of maintaining the textless constraint is not quantified. Only the semantic dimension of SLM quality is addressed; paralinguistic aspects such as prosody, speaking style, and emotion are explicitly out of scope and represent open directions for extending the framework.

Wiki Connections

  • Spoken Language Model — the paper directly improves textless SLM semantics by introducing preference optimization as an alternative training objective.
  • RLHF Speech — the paper provides the first demonstration of DPO and RLAIF applied to end-to-end speech-only language models, extending the alignment paradigm from TTS to SLMs.
  • Self-Supervised Speech — HuBERT serves as the core speech tokenizer throughout the framework, converting waveforms to discrete units that form the SLM’s generation vocabulary.
  • Speech-to-Speech — the framework preserves an end-to-end speech-to-speech pipeline, distinguishing it from approaches that inject text tokens during inference.
  • Subjective Evaluation — MMOS (Meaningfulness MOS) human evaluation confirms that preference-optimized speech continuations are rated as more meaningful than both the pre-trained baseline and re-synthesized ground truth.

Backlinks

Graph View