arXiv · 2025 · Preprint
Sugiura et al. · → Paper · Demo: ✓ · Code: ✓
Llama-Mimi demonstrates that flattening multi-level RVQ tokens into a single autoregressive sequence can match or surpass hierarchical speech LM designs on acoustic tasks, while revealing a fundamental trade-off with linguistic performance.
Problem
Most speech language models (SpeechLMs) that operate on RVQ codec tokens adopt hierarchical architectures, typically derived from the RQTransformer, which use separate Transformer decoders along the temporal (frame) and depth (quantizer) dimensions. While effective, these designs increase architectural complexity, complicate hyperparameter tuning, and may limit the cross-level interactions that a single unified model could learn. The parallel trend in NLP toward simpler, more scalable single-decoder Transformers raises the question of whether the same simplification can work for multi-level speech token sequences.
Method
Llama-Mimi takes the Mimi neural audio codec as its tokenizer and flattens its RVQ output into a single one-dimensional sequence for autoregressive modeling with a Transformer decoder. Given a waveform, Mimi encodes it at 12.5 frames per second into Q-level RVQ tokens per frame; within each frame, semantic tokens (first quantizer, distilled from WavLM) are ordered before acoustic tokens, preserving the coarse-to-fine structure of RVQ. The concatenated sequence is then processed by a single Llama 3 Transformer decoder trained with the standard next-token prediction objective. The Llama 3 vocabulary is extended with all RVQ tokens and two special <audio> and </audio> markers; Mimi weights are kept frozen throughout training.
The primary controlled comparison trains a 1.3B flattened model (Llama-3.2-1B backbone) against a CSM-1.3B hierarchical model of equivalent parameter count, both on the same ~240k hours of English speech drawn from Libri-Light, The People’s Speech, VoxPopuli, and Emilia. Both models use Q=4 quantizers, yielding 50 tokens per second. A larger Llama-Mimi-8B variant (Llama-3.1-8B backbone) is also trained to examine scaling behaviour.
Key Results
Llama-Mimi-1.3B outperforms CSM-1.3B on all acoustic consistency tasks from the SALMon evaluation suite (e.g., room consistency 92 vs. 81.5, speaker consistency 85 vs. 77.5) and on speaker similarity (0.346 vs. 0.320). Its spoken content quality score (3.01) also exceeds CSM-1.3B’s (2.80). This holds despite both models sharing identical training data and comparable parameter counts, indicating that direct cross-level attention in the flat architecture is beneficial for acoustic coherence.
However, Llama-Mimi-1.3B underperforms SSL-phonetic-token approaches on linguistic benchmarks: sWUGGY 68.7 vs. 71.7 for TWIST-1.3B, T-Story Cloze 64.0 vs. 69.9. The gap is attributed to the Q-fold increase in sequence length that accompanies RVQ flattening, which dilutes modeling capacity for higher-level linguistic structure.
Ablation results reveal three additional trade-offs. Scaling quantizers from Q=2 to Q=8 consistently improves Audiobox-Aesthetics audio quality and speaker similarity (0.201 to 0.474) but degrades content quality (3.53 to 2.54). Increasing the semantic token loss weight from λ=1 to λ=100 improves sBLIMP (54.3 to 55.4) and T-Story Cloze (64.0 to 68.4) while reducing room consistency (92 to 74) and speaker similarity (0.346 to 0.196). Scaling from 1.3B to 8B parameters improves all tasks, with the largest gain in content quality (3.01 to 4.03).
Novelty Assessment
The core contribution is an architectural comparison: flattened single-Transformer versus hierarchical dual-Transformer modeling of RVQ codec tokens, on matched data with controlled conditions. The flattening strategy itself is not novel (SpiritLM uses a related approach with SSL tokens), and Llama-Mimi’s components (Llama 3 backbone, Mimi codec, standard AR training) are all established. What is new is the first systematic evaluation of a purely RVQ-based flattened speech LM, alongside detailed ablations of quantizer count, loss weighting, and model scale. The trade-off findings (acoustic-linguistic, sequence-length-linguistic, quantizer-count-quality) are the paper’s primary scientific contribution.
Field Significance
Moderate — This paper provides a well-controlled characterisation of the flat-versus-hierarchical design choice for RVQ-based speech LMs. The finding that a single-Transformer design can exceed hierarchical models on acoustic consistency challenges the assumption that hierarchical structure is necessary for multi-level token modeling. At the same time, the persistent linguistic gap relative to SSL-based approaches highlights that sequence length remains a core bottleneck in flat RVQ designs.
Claims
-
supports: A single-Transformer flattened architecture can match or surpass hierarchical speech LM designs on acoustic consistency tasks when both use identical data and comparable parameter budgets.
Evidence: Llama-Mimi-1.3B outperforms CSM-1.3B on all SALMon acoustic consistency dimensions and speaker similarity (0.346 vs. 0.320) under controlled training conditions. (§4.3, Table 1)
-
complicates: Flattening RVQ tokens into a single autoregressive sequence creates an inherent acoustic-linguistic trade-off: strong acoustic performance comes at the cost of weaker linguistic benchmarks relative to SSL-based phonetic-token approaches.
Evidence: Llama-Mimi-1.3B achieves best acoustic consistency but underperforms TWIST-1.3B on sWUGGY (68.7 vs. 71.7) and T-Story Cloze (64.0 vs. 69.9), attributed to the Q-fold sequence length increase from RVQ token flattening. (§4.3, Table 1)
-
complicates: Increasing the number of RVQ quantizers in a flattened speech LM improves audio quality but degrades spoken content coherence, because longer token sequences shift modeling capacity toward acoustic reconstruction.
Evidence: Ablation with Q∈{2,4,8} shows Q=8 achieves best Audiobox-Aesthetics scores and speaker similarity (0.474) but worst content quality (2.54), while Q=2 yields content quality (3.53) comparable to TWIST-1.3B. (§4.4, Table 5)
-
refines: Applying a higher loss weight to semantic tokens in speech LM training shifts the acoustic-linguistic balance toward linguistic accuracy, but causes measurable degradation in acoustic consistency and speaker similarity.
Evidence: Llama-Mimi-1.3B with semantic weight λ=100 gains on sBLIMP (55.4 vs. 54.3) and T-Story Cloze (68.4 vs. 64.0), but loses substantially on room consistency (74 vs. 92) and speaker similarity (0.196 vs. 0.346). (§4.4, Table 3)
-
supports: Scaling model size in flattened speech LMs consistently improves performance across both acoustic and linguistic tasks, with the largest gains in spoken content quality.
Evidence: Llama-Mimi-8B improves content quality (4.03 vs. 3.01) and T-Story Cloze (67.6 vs. 64.0) over the 1.3B model, with qualitative analysis showing more semantically coherent long-form continuations. (§4.4, Table 4)
Limitations and Open Questions
Warning
Evaluations are conducted exclusively on English speech using LibriSpeech prompts; generalisation to other languages, speakers, or acoustic conditions is untested. No human listening tests are reported; all acoustic and linguistic evaluations rely on automated metrics.
The paper evaluates only speech continuation, not text-conditioned generation or dialogue. Whether the acoustic-linguistic trade-off in flattened designs persists when text tokens are added to the sequence (as in Moshi’s inner monologue approach) is left as an open question. The 8B model is evaluated only on a subset of tasks, and the cost of the Q-fold sequence length increase at large scale is not fully characterised. Training Mimi weights end-to-end jointly with the LM is not explored; the frozen codec is an architectural constraint, not a systematic choice.
Wiki Connections
- Spoken Language Model — Llama-Mimi directly investigates the flat-versus-hierarchical architecture decision for RVQ-based speech language modeling, providing empirical evidence for the flat design.
- Autoregressive Codec TTS — The paper applies autoregressive next-token prediction over RVQ codec tokens to speech generation, with a focus on how token sequence structure affects both acoustic and linguistic performance.
- Neural Audio Codec — Mimi serves as the fixed tokenizer backbone; the paper’s design choices (quantizer count, frame rate, semantic distillation) are directly conditioned on Mimi’s RVQ structure.
- Evaluation Metrics — The paper reports results on SALMon (acoustic consistency and alignment), sWUGGY, sBLIMP, T-Story Cloze, and Audiobox-Aesthetics, providing a multi-dimensional evaluation framework for speech LMs.