arXiv · 2025 · Preprint
Zhen Ye et al. · → Paper · Demo: ? · Code: ✓
Llasa introduces a single-Transformer TTS system fully aligned with standard LLM architecture (Llama backbone + single-VQ codec), and provides the first systematic investigation of both train-time and inference-time compute scaling for speech synthesis.
Problem
Dominant high-quality TTS systems as of early 2025 are multi-stage pipelines that combine an autoregressive language model with a separate non-autoregressive or diffusion-based decoder. This complexity makes it difficult to study scaling laws: any scaling decision must be applied to multiple components, and the components interact in ways that obscure clean causal attribution. Meanwhile, the text LLM community had converged on a minimalist, single-Transformer paradigm that enables systematic investigation of scaling behavior. The gap motivates a correspondingly simple TTS framework where scaling effects on both training compute and inference compute can be studied cleanly.
Method
Llasa reduces TTS to two components: a speech tokenizer and a single Transformer. The Transformer is initialized from Llama 3.2 (1B and 3B) or Llama 3.1 (8B), inheriting both its architecture and text tokenizer. Speech tokens are appended to the text token sequence and the model is trained with next-token prediction loss over the speech portion only, yielding a straightforward conditional language model.
The speech tokenizer, called X-codec2, is the key technical contribution. It fuses semantic and acoustic information in a single codebook by running two parallel encoders: a pre-trained Wav2Vec2-BERT for semantic features and a residual convolutional encoder for acoustic features. The concatenated representation is quantized with Finite Scalar Quantization (FSQ), which replaces residual vector quantization and enforces a single codebook layer with 65,536 entries at a 50 Hz frame rate. The single-layer design is essential for compatibility with the left-to-right autoregressive factorization of the LLM: RVQ would introduce multi-stream dependencies that break the 1D causal structure. The decoder follows the Vocos design, predicting STFT magnitude and phase for reconstruction via inverse STFT. Training combines adversarial losses (multi-period and multi-scale STFT discriminators) with semantic reconstruction and a perceptual loss in the final training phase.
For zero-shot TTS, a reference speech clip is prepended as context (in-context learning), and the model generates speech tokens conditioned on both the text and the audio prompt.
Train-time scaling is studied at fixed data (250k hours) across three model sizes (1B, 3B, 8B) and at fixed model size (1B) across three data scales (80k, 160k, 250k hours). Inference-time scaling uses off-the-shelf speech understanding models as verifiers. Two search strategies are compared: Best-of-N (output reward model, ORM) and beam search with a process reward model (PRM) that evaluates speaker similarity at each 0.5-second step. A hybrid strategy (partial PRM for early timesteps, then ORM for final selection) is found to balance speaker similarity improvement with content accuracy.
Key Results
On the Seed-TTS-Eval benchmark (Table 2), Llasa-8B-250k with direct inference achieves WER of 2.97% (English) and CER of 1.59% (Chinese), competitive with multi-stage baselines like CosyVoice 2 (WER 2.57%) and MaskGCT (WER 2.62%). However, speaker similarity (SIM-o) for direct inference is notably weaker than these baselines, reflecting the acoustic reconstruction ceiling of a single-VQ codec versus multi-layer RVQ or mel-based vocoders.
Scaling both model size and training data consistently improves all metrics: WER on test-en drops from 3.71% (1B-80k) to 2.97% (8B-250k), and emotion similarity on ESD rises from 0.753 to 0.778 (English) across model sizes (Table 4).
Inference-time scaling with the partial-PRM+ORM(WER) strategy for Llasa-8B-250k pushes WER to 1.84% (test-en) and SIM-o to 0.783, which surpasses all listed baselines on speaker similarity while maintaining competitive WER (Table 2). On the hard subset, chunked inference with the best strategy achieves WER 3.12%, demonstrating that inference compute allocation is particularly beneficial for difficult synthesis cases.
On LibriSpeech test-clean continuation (Table 3), Llasa-8B-250k direct achieves SIM-r of 0.626, close to the codec resynthesis ceiling (0.638), suggesting that the single-Transformer architecture is not intrinsically disadvantaged on content fidelity relative to AR+NAR hybrids.
The X-codec2 codec at 50 tokens/second (Table 1) achieves UTMOS 4.13 and WER 2.47%, outperforming all compared codecs at that token rate, including BigCodec (UTMOS 4.11, WER 2.76%) and X-codec (UTMOS 4.05, WER 3.42%).
Novelty Assessment
Llasa’s core architectural contribution is the X-codec2 tokenizer: the combination of semantic (Wav2Vec2-BERT) and acoustic encoders with FSQ single-VQ quantization is not standard, and the motivation (enabling 1D causal LLM compatibility) is clear and well-executed. The TTS Transformer itself is straightforward Llama fine-tuning with an expanded vocabulary, which is established territory (VALL-E family), though the reduction to a truly single-stage single-model pipeline removes the NAR decoder used in prior work.
The most original contribution is the inference-time compute study. Best-of-N search for TTS is not new, but the PRM formulation (step-wise speaker similarity scoring during generation), the partial-PRM hybrid strategy, and the systematic comparison of ORM vs PRM search under matched compute budgets is genuinely new for speech. The finding that PRM beam search can cause WER degradation due to mode collapse (overfitting to speaker similarity at the cost of content accuracy), and that partial PRM mitigates this, is a practically useful result.
The train-time scaling study is primarily empirical: it confirms that larger models and more data help, which is expected. The Chinese text understanding benchmark (seven categories including polyphonic characters and tongue twisters) is a useful addition to the evaluation toolkit for expressive Chinese TTS.
The acknowledged limitation on SIM-o relative to RVQ-based baselines is honest and consequential: single-VQ codecs sacrifice fine-grained acoustic reconstruction for LLM alignment simplicity, and this trade-off is not fully closed by inference-time scaling.
Field Significance
Tip
High — Llasa provides evidence that a maximally simplified TTS framework (single Transformer, single-VQ codec, next-token prediction only) is competitive with carefully engineered multi-stage pipelines on most metrics, and opens TTS to the scaling law and inference-time compute research directions that have proven productive in the text LLM community. The open model releases (1B, 3B, 8B) and training code lower the barrier for subsequent scaling studies. The paper’s framework also demonstrates that the SIM-o gap attributable to single-VQ codecs is recoverable at inference time, which has practical implications for system design trade-offs.
Claims
- Single-stage autoregressive TTS trained with next-token prediction over discrete speech tokens is competitive with multi-stage AR+NAR pipelines on intelligibility and speaker similarity in continuation mode, though SIM-o gaps remain due to codec acoustic reconstruction limits. (§3.2.4, Table 3)
- Both model scale and training data volume independently improve TTS quality across naturalness, prosody, and text comprehension, consistent with scaling laws observed in text LLMs. (§2.3, §3.2.2, Tables 2, 4)
- Inference-time compute scaling via speech understanding verifiers can substantially improve speaker similarity and emotional expressiveness beyond what train-time scaling alone achieves, at the cost of additional inference compute. (§2.4, §3.2.3, Figure 2, Table 2)
- Pure process reward model beam search for TTS is prone to mode collapse that degrades content accuracy (WER), and a hybrid partial-PRM strategy is needed to preserve both speaker similarity and intelligibility. (§3.2.3, Figure 2)
- Single-VQ codecs can achieve intelligibility and naturalness competitive with multi-layer RVQ codecs at the same token rate, but acoustic fidelity (speaker similarity) remains the limiting factor for single-VQ reconstruction. (§3.1.2, Table 1)
Limitations and Open Questions
Warning
The SIM-o gap between Llasa and RVQ-based baselines is intrinsic to the single-VQ design: acoustic reconstruction from a 65,536-entry single codebook at 50 Hz is weaker than 8-layer RVQ codecs, and this gap is only partially recovered by inference-time search. Systems requiring high timbre fidelity in a single inference pass would need a different codec design.
Inference-time compute scaling requires running multiple candidates (beam search or Best-of-N) with auxiliary verifier models, which increases latency and compute cost substantially and makes the approach unsuitable for real-time or low-resource applications. The paper does not characterize latency or wall-clock overhead of the search strategies.
The text understanding evaluation uses expert-rated 3-point discrete scores, which are not directly comparable across papers and rely on a small number of sentences per category. The evaluation methodology is adapted from BASE TTS but the inter-rater reliability is not reported.
Models are trained on mixed Mandarin/English data, but the language coverage and balance are not fully documented. The internal data component of the 250k-hour corpus is not described, limiting reproducibility.
Wiki Connections
This paper directly builds on 2301.02111 (VALL-E, which established the AR codec LM paradigm for TTS) and extends it toward a maximally LLM-aligned, single-stage design. It compares against 2406.02430 (Seed-TTS), 2409.00750 (MaskGCT), 2407.05407 (CosyVoice), and 2412.10117 (CosyVoice 2). The X-codec2 tokenizer extends 2210.13438 (EnCodec) and relates to neural codec work. Moshi (2410.00037) is cited as a related single-Transformer audio-LM. Emotion evaluation uses 2312.15185 (emotion2vec), and UTMOS (2204.02152) is used for codec quality assessment. 2006.04558 (FastSpeech 2) is referenced in related work context.
Concept connections: autoregressive-codec-tts, neural-codec, zero-shot-tts, self-supervised-speech, prosody-control, evaluation-metrics