arXiv · 2025 · Preprint
Wenjie Tian et al. (Northwestern Polytechnical University / HKUST) · → Paper · Demo: ✗ · Code: ✓
Llasa+ adds two plug-and-play Multi-Token Prediction (MTP) modules to a frozen Llasa backbone and a causal streaming codec decoder (XCodec2-S), achieving 1.48x inference speedup without quality degradation through a novel token verification algorithm.
Problem
Large LLM-based TTS models like Llasa suffer from slow autoregressive inference: generation steps scale linearly with target speech length, and the single-token-per-step bottleneck is the primary latency bottleneck regardless of model parallelism improvements. Existing acceleration approaches either require expensive retraining on large datasets (grouped code modeling in VALL-E 2) or rely on data-distribution-sensitive transition matrices (Viterbi-like approaches). A practical, training-efficient acceleration method that preserves full model quality on arbitrary speakers has been lacking. Separately, streaming waveform reconstruction from single-codebook tokens requires a causal codec decoder, which Llasa’s original XCodec2 does not support.
Method
Multi-Token Prediction (MTP) Modules. Llasa+ freezes the Llasa-1B backbone (LLaMA-based, 16 decoder layers, hidden size 2048) and adds two MTP modules in cascade. Each MTP module is a single LLaMA decoder layer that takes the last hidden states from the previous module as input. All MTP modules share the frozen LM head from Llasa. At training time, only the MTP modules are optimized via cross-entropy loss against speech tokens offset by k+1 steps (for the k-th MTP). The backbone is never updated, ensuring no degradation of the original generation quality.
Verification Algorithm. Because MTP predictions are imperfect, a verification loop validates generated tokens. At each AR step, the backbone produces a “trusted” token S_t, while MTP1 and MTP2 produce “untrusted” tokens S’{t+1} and S’{t+2}. At the next step, the backbone’s logits are used to check whether S’{t+1} falls in the top-k candidates. If yes, it is accepted and S’{t+2} is further validated against MTP1’s logits. If not, both untrusted tokens are discarded and a fresh trusted token is sampled. End-of-sequence (EOS) tokens from MTP modules are always verified before acceptance. This verification is critical: without it, WER degrades from 3.07 to 14.372 and SIM from 0.570 to 0.463.
Streaming Codec (XCodec2-S). The original XCodec2 transformer decoder is converted to a causal architecture to support streaming waveform reconstruction. The encoder and VQ module are frozen from pretrained XCodec2; only the decoder is unfrozen for fine-tuning as a causal adapter. Some conv1d layers retain local lookahead within a fixed window to improve streaming quality without breaking causality. Trained on LibriTTS (all training splits), XCodec2-S preserves ~95% of XCodec2 performance: WER 3.239 vs. 2.47, STOI 0.913 vs. 0.919, UTMOS 4.029 vs. 4.127.
MTP modules trained on 4x A800 GPUs for 20 epochs; XCodec2-S trained on 8x 4090 GPUs for 280k steps.
Key Results
At topk=100 (quality-maximizing setting), Llasa+ achieves WER 2.499% and SIM 0.575 on Seed-TTS-eval-en — both better than the Llasa baseline (WER 3.22, SIM 0.572) — with 1.42x speedup. At topk=500 (throughput-maximizing setting), Llasa+ achieves 1.48x speedup with WER 3.07 and SIM 0.570, essentially matching baseline performance. The improvement in WER at topk=100 is attributed to the MTP modules enabling the model to better leverage longer historical context, acting as a form of improved look-ahead planning.
Ablation studies confirm that: (1) attention-based MTP layers substantially outperform MLP replacements (WER 2.745 vs. 4.375 for MTP1 at topk=500); (2) the DeepSeek-V3-style input (feeding ground-truth tokens to MTP) underperforms the proposed hidden-state-only input due to train/inference mismatch; (3) a third MTP module provides less than 10% additional speedup at disproportionate cost.
Novelty Assessment
The core contribution is the verification algorithm, which transforms MTP from a quality-degrading trick into a lossless acceleration method. This is a practically important insight: accepting multiple tokens per step without verification is well-known to cause cascading errors in TTS due to high codec bitrate and temporal sensitivity. The plug-and-play property (backbone frozen) is valuable for deployment — no large dataset or retraining is required. The XCodec2-S streaming adapter completes the pipeline for real-time use. The paper is primarily an inference engineering contribution; the underlying model architecture follows Llasa without innovation.
Field Significance
Moderate — Llasa+ provides a practical template for adding speculative-decoding-style acceleration to frozen autoregressive TTS models without retraining the backbone. The verification-gated MTP approach generalises beyond Llasa: the authors demonstrate it applies to any LLM-based speech model, making this a reusable engineering pattern for latency reduction in production TTS. The XCodec2-S causal adapter separately provides a lightweight pathway to streaming waveform reconstruction from single-codebook tokens.
Claims
- Token verification is necessary for multi-token prediction to be effective in autoregressive TTS: without it, WER increases from 3.07% to 14.37% and speaker similarity drops from 0.570 to 0.463. (§V, Table III)
- Plug-and-play MTP modules trained on a modest dataset can accelerate a frozen autoregressive TTS backbone by up to 1.48x without sacrificing generation quality on standard benchmarks. (§IV.B, Table I)
- Under quality-maximizing inference settings, MTP with verification can improve intelligibility beyond the backbone baseline, likely due to extended look-ahead context from the cascaded hidden states. (§IV.A, Table I)
- Converting a non-causal codec decoder to a causal streaming architecture via lightweight fine-tuning preserves approximately 95% of reconstruction quality, making streaming reconstruction viable without full retraining. (§IV.C, Table II)
- Attention-based MTP modules substantially outperform MLP-based equivalents of similar parameter count in both intelligibility and speaker similarity for TTS acceleration. (§IV.B, Table I)
Limitations and Open Questions
Evaluation is English-only (LibriTTS training, Seed-TTS-eval-en test), and speaker generalization to out-of-distribution languages or accents is untested. The verification overhead (one additional LM forward pass for verification at each step) partially offsets the MTP speedup, especially at strict topk values. The 1.48x figure assumes topk=500, which allows some quality degradation; the fully lossless speedup (topk=100) is closer to 1.42x. Scaling MTP to larger models (Llasa-3B, Llasa-8B) is not investigated.
Wiki Connections
This paper makes a direct contribution to streaming-tts by enabling streaming waveform reconstruction from Llasa’s single-codebook tokens, and to autoregressive-codec-tts by demonstrating a verification-based multi-token prediction framework applicable to any LM-based TTS. The XCodec2-S design is relevant to neural-codec. The MTP-and-verification framework is a general acceleration technique that overlaps conceptually with speculative decoding approaches used in concurrent work (VocalNet). The paper builds directly on Llasa.