arXiv · 2024 · Preprint
Tu Anh Nguyen et al. (Meta AI) · → Paper · Demo: ✓ · Code: ✓
Spirit LM extends a 7B text LLM (LLaMA 2) to freely interleave speech and text tokens in a single sequence, enabling cross-modal generation, few-shot learning across modalities, and expressive speech output that preserves sentiment across modality boundaries.
Problem
Existing speech language models were trained either on speech alone or on specific tasks like ASR and TTS, causing them to trail text LLMs in semantic reasoning while losing expressive capabilities. Cascade pipelines (ASR → text LLM → TTS) solve the semantics problem but sever expressivity from the language model entirely: prosody, emotion, and speaking style are generated by a downstream synthesizer that operates independently of the model’s semantic choices. The field lacked a single unified model that could reason over both speech and text while retaining the expressive qualities of raw speech.
Method
Spirit LM is built by continuously pretraining LLaMA 2 7B on three interleaved data streams: text-only sequences (300B tokens from the LLaMA training corpus), speech-only sequences (460K hours, 30B speech tokens), and aligned speech+text sequences (110K hours, 7B speech tokens + 1.5B text tokens). The key design decision is word-level interleaving: at each training step, modality switches are triggered randomly at word boundaries in the aligned corpus, so the model sees inputs like [TEXT]the cat [SPEECH][Hu3][Hu7]...[TEXT]the mat. This forces the model to develop shared internal representations that span both modalities.
For the Base version, speech is encoded into 501 HuBERT tokens at 25 Hz (deduplicated), and text uses the standard LLaMA BPE tokenizer. At inference, a HiFi-GAN vocoder conditioned on HuBERT tokens and 1-hot speaker embeddings (from the 4 Expresso voices) reconstructs the waveform.
The Expressive version supplements HuBERT tokens with two additional token streams: pitch tokens from a VQ-VAE trained on F0 (64-unit codebook at 12.5 Hz), and style tokens from speaker-normalized SpeechProp features clustered into 100 units at 1 Hz. All three token types are merged by timestamp into a single sequence. The Expressive model uses the same LLaMA 2 7B backbone and identical training procedure, extended with the larger token vocabulary. Both models were trained for 200K steps on 64 A100 GPUs (approximately 2 weeks), though ablations in the paper used a 100K-step budget.
The paper also introduces the Speech-Text Sentiment Preservation (STSP) benchmark, which measures whether a model preserves the sentiment of a prompt when generating continuations within and across modalities, evaluated using fine-tuned speech and text sentiment classifiers.
Key Results
On standard speech comprehension benchmarks, Spirit LM Base outperforms prior speech-only LMs (GSLM, TWIST, AudioLM) on StoryCloze, which requires high-level semantic understanding, while remaining competitive on WUGGY (lexical) and BLIMP (grammatical). Crucially, Spirit LM BASE achieves 36.9% accuracy on MMLU (5-shot, text-only), demonstrating that the text LLM’s capabilities are largely preserved despite pretraining on 30B+ speech tokens. The cascade topline (Whisper + LLaMA 2) remains well above on both speech comprehension and MMLU, highlighting the semantic cost of tokenizing speech through discrete units.
For few-shot generation tasks, Spirit LM Base reaches 21.9% WER on LibriSpeech clean (ASR, 10-shot) and 45.5% CER on LibriSpeech clean (TTS, 10-shot), far below the 3.7% / 4.0% cascade topline but clearly above chance. Adding explicit ASR+TTS parallel training data to the recipe dramatically improves these numbers (6.0% WER, 6.7% CER), showing the base interleaved model is not optimized for task-specific generation.
On the STSP expressivity benchmark, Spirit LM Expressive achieves 0.48 average accuracy across all directions (zero-shot), outperforming Spirit LM Base (0.41) and the cascade topline (0.42). The Expressive model particularly excels at S→S (0.54 vs. 0.33 for Spirit LM Base), the only direction where it clearly beats cascades. Cross-modal sentiment preservation (S→T and T→S) remains below same-modality performance for all systems.
Ablations confirm that interleaving is the critical training ingredient: removing interleaved data while keeping speech and text separately drops cross-modal StoryCloze performance by more than 10 points. Parallel ASR+TTS training without interleaving also underperforms the full recipe on semantic tasks.
Novelty Assessment
The core novelty is the word-level interleaving training scheme applied to a large pretrained text LLM. Mixing discrete speech tokens into a text LLM’s token stream had been explored in smaller systems (SpeechGPT, VioLA, VoxtLM), but Spirit LM is the first to demonstrate this at 7B scale on a competitive text-pretrained backbone with thorough ablations isolating the role of interleaving versus parallel training alternatives. The STSP benchmark fills a genuine evaluation gap: prior work had no standardized way to measure cross-modal sentiment or emotion preservation.
The Expressive variant’s multi-stream tokenization (HuBERT + pitch VQ-VAE + style cluster tokens merged by timestamp) is a pragmatic engineering design rather than a structural breakthrough. The HiFi-GAN vocoder synthesis pipeline is off-the-shelf. The main limitation of the approach is that speech quality and speaker coverage are constrained by the 4-voice Expresso dataset used to train the vocoder, and the model falls substantially short of cascade baselines on all task-specific metrics.
The contribution is genuine but sits at the intersection of architectural novelty (the interleaving recipe as a scalable approach to unified speech-text modeling) and conceptual reframing (demonstrating that a single LM can be both semantically capable and expressively aware without modality separation).
Field Significance
Tip
High — Spirit LM demonstrates that a single large language model can operate fluidly over interleaved speech and text tokens, preserving semantic capabilities from text pretraining while acquiring expressive speech generation. This provides an influential existence proof for natively multimodal speech LMs as an alternative to cascade architectures, and introduces the STSP benchmark as a reusable tool for evaluating cross-modal expressivity preservation.
Claims
- Continuous pretraining of a text LLM on interleaved speech and text tokens transfers the text model’s few-shot learning and semantic reasoning abilities to the speech modality. (§4.2, §4.3, Table 4)
- Word-level interleaving of speech and text during training is more effective than parallel ASR/TTS training or speech-only fine-tuning for cross-modal semantic understanding. (§4.2, Table 6)
- Expressive speech properties (sentiment, pitch, style) can be modeled in a language model through discrete token streams that supplement phonetic tokens, enabling cross-modal sentiment preservation. (§5, Table 3)
- Adding expressive speech tokens to a speech LM improves expressivity at the cost of moderate degradation in lexical and grammatical speech understanding. (§4.2, Table 4)
- Cascade speech pipelines remain substantially stronger than end-to-end unified models on task-specific metrics such as ASR WER and TTS intelligibility at equivalent model scale. (§4.3, Table 5)
Limitations and Open Questions
Warning
Spirit LM’s vocoder is conditioned on only 4 speaker voices from the Expresso dataset, which severely constrains the diversity and quality of synthesized speech; the model cannot generalize to arbitrary target speakers at inference without retraining the vocoder.
The STSP benchmark is evaluated using fine-tuned automatic classifiers rather than human listeners, which may not capture perceptual sentiment fidelity accurately. The model was evaluated only in English, leaving multilingual capabilities untested. The 7B scale represents a reasonable starting point, but the authors note that scaling beyond 7B could substantially improve both semantic and expressive performance. Toxicity analysis shows the model can add harmful speech content at levels comparable to cascades, with higher MUTOX scores in S→S, warranting safety work before deployment.
Wiki Connections
Core methods and architectures: spoken-language-model · autoregressive-codec-tts · self-supervised-speech · speech-to-speech · emotion-synthesis · disentanglement
In-corpus papers this work builds on: 2210.13438 (EnCodec — used as a baseline in resynthesis evaluation) · 2308.05725 (Expresso benchmark — used for vocoder training and expressivity evaluation) · 2302.13971 (LLaMA — the backbone model) · 2301.02111 (VALL-E — cited as a related speech LM approach)
2507.16632 — Step-Audio 2 Technical Report