ACL · 2025 · Conference
Sambal Shikhar et al. (MBZUAI) · → Paper · Demo: ✓ · Code: ✓
LLMVoX is a 30M-parameter, LLM-agnostic, autoregressive streaming TTS module that decouples speech synthesis from text generation via multi-queue streaming, achieving WER 3.70% and 475ms end-to-end latency without modifying the base LLM.
Problem
Existing speech-enabled LLMs fall into two camps: (1) end-to-end models (Moshi, LLaMA-Omni, Mini-Omni) that fine-tune or modify the LLM with speech-specific components, degrading its linguistic reasoning capabilities; (2) cascaded pipelines (Whisper+LLM+XTTS) that preserve LLM quality but incur prohibitive latency (4200ms for XTTS) because conventional TTS models require full text before synthesis can begin. The core tension is that autoregressive LLMs produce text tokens incrementally, but standard TTS decoders are non-incremental. Additionally, most speech-enabled LLMs are LLM-dependent — their speech components condition on LLM hidden states, requiring retraining for each new base LLM.
Method
LLMVoX is a plug-and-play autoregressive TTS frontend that converts streaming LLM text output into speech tokens in real time, decoupled from the LLM.
Speech tokenization. WavTokenizer (single-layer RVQ, 4096-entry codebook) encodes speech at 40–75 tokens/second at 24 kHz. Using a single RVQ level (vs. multi-level RVQ in EnCodec) keeps the autoregressive sequence manageable.
Phoneme-aware input. Text from the LLM is passed through the embedding layer of a pretrained ByT5-based G2P model (100+ language coverage) to obtain byte-level phoneme embeddings (256-dim). These are L2-normalized and concatenated with the latent feature of the previous WavTokenizer speech token (512-dim), producing a 768-dim input to the transformer.
Decoder-only transformer. A lightweight 4-layer decoder-only transformer (n_embd=768, n_head=8, context=8192 tokens) autoregressively predicts the next speech token. Trained with cross-entropy loss on ground-truth WavTokenizer tokens. Runs on 4×A100 for ~3 days with bfloat16 and flash-attention.
Multi-queue streaming inference. Incoming LLM text tokens are enqueued into two FIFO queues Q1 and Q2 partitioned by sentence boundaries. Two replica LLMVoX instances process Q1 and Q2 concurrently. Every n speech tokens generated are decoded to audio by the WavTokenizer decoder and placed in producer queues P1/P2 for immediate streaming. The chunk size starts at n and doubles after each segment, leveraging playback time for extra processing. This achieves uninterrupted infinite-length speech without context-length constraints.
Training data. English model: VoiceAssistant-400K (GPT-4o Q&A pairs, synthetic speech, ~2200 hours, single-speaker). Arabic model: 450K text entries synthesized via XTTS (~1500 hours). No architectural changes needed for new languages — only data.
Key Results
On the joint General+Knowledge QA streaming benchmark (Table 1), Whisper+LLaMA3.1-8B+LLMVoX achieves:
- GPT-4o General QA score: 6.14 (highest among streaming models; Freeze-Omni: 3.48; Moshi: 2.71)
- GPT-4o Knowledge QA score: 7.62 (highest; vs. 4.98 for Freeze-Omni)
- WER: 3.70% (lowest; Freeze-Omni: 14.05%; LLaMA-Omni: 9.18%; CosyVoice2: 17.2%)
- UTMOS: 4.05 (competitive; Freeze-Omni: 4.38; MiniCPM-o 2.6: 3.87)
- End-to-end latency: 475ms (vs. Whisper+XTTS: 4200ms; LLaMA-Omni: 220ms)
For Arabic (Table 3), LLMVoX streaming CER of 8.2% competes with non-streaming XTTS (CER 1.7%) while operating ~10× faster.
For VLM integration (Table 4), LLMVoX+Qwen2.5-VL-7B on LLaVA-Bench: WER 4.2%, GPT score 6.41, latency 1.05s — outperforming MiniCPM-o 2.6 (WER 5.3%, GPT 6.32, latency 1.45s) by 28% on latency.
Robustness study (Table 5) shows WER stable at 3.72–4.1% across 1–5 sentence responses, while CosyVoice2 degrades from 12.2% to 17.1%.
Novelty Assessment
The primary contribution is architectural: the multi-queue streaming design that runs TTS strictly in parallel with LLM text generation, without conditioning on LLM hidden states. This is a clean engineering solution to a well-identified problem. The use of a ByT5 G2P embedding as a fixed lookup (rather than training G2P end-to-end) is a practical efficiency trick. The model is tiny (30M vs. 7B+ for end-to-end systems). The main limitation in novelty: the underlying components (WavTokenizer, decoder transformer, ByT5 G2P) are all off-the-shelf; the innovation is their integration and the streaming scheduler. The paper lacks voice cloning, which is a major gap vs. zero-shot TTS systems.
Field Significance
Moderate — LLMVoX provides a practical template for attaching streaming TTS to any LLM without modifying the base model’s weights, directly addressing a deployment bottleneck in spoken conversational agents. Its contribution is primarily a systems integration result: the components are known, but their combination with the dual-queue streaming scheduler is a clean, reproducible engineering solution that others can adopt or extend. The evaluation against a broad set of end-to-end and cascaded baselines provides useful comparative data for practitioners choosing between latency-quality trade-offs.
Claims
- Decoupling speech synthesis from LLM text generation via a lightweight autoregressive module can preserve the base LLM’s reasoning quality while achieving competitive streaming latency. (§6.4, Table 1)
- End-to-end speech-enabled LLMs that fine-tune or condition the base LLM on speech data consistently show degraded language understanding compared to systems that keep the LLM frozen. (§6.4, Table 1)
- A single-layer RVQ codec is sufficient for low-latency autoregressive TTS generation when paired with a compact decoder transformer, avoiding the complexity of multi-codebook prediction. (§3.1)
- Streaming TTS quality improves with larger decode chunk sizes, with WER and UTMOS gains achievable without substantially increasing end-to-end latency. (§6.4, Figure 6)
- Language adaptation of a codec-based TTS module can be achieved by replacing training data alone, without architectural changes or explicit G2P conversion for the new language. (§6.5, Table 3)
Limitations and Open Questions
LLMVoX is single-speaker — no voice cloning or speaker reference support. The Arabic model was trained on XTTS-synthesized data, so XTTS acts as an upper bound (CER 1.7% vs. LLMVoX 8.2%). The streaming pipeline does not yet extend to the ASR front-end. Latency with 70B LLMs exceeds 1.9s, making real-time use marginal. The quality improvement from larger chunk sizes (UTMOS 3.75→4.41) suggests that the 475ms latency figure is somewhat optimistic for maximum quality operation.
Wiki Connections
- streaming-tts — LLMVoX directly addresses low-latency streaming speech generation, with the dual-queue scheduler as its core mechanism.
- spoken-language-model — the plug-and-play design targets spoken conversational agent pipelines without modifying the LLM backbone.
- autoregressive-codec-tts — the token-prediction approach follows the VALL-E family but operates on streaming text input rather than full-context generation.
- neural-codec — WavTokenizer (single-layer RVQ) is the discrete speech representation used throughout.
- speech-to-speech — integration with ASR and VLM pipelines demonstrates multi-modal spoken dialogue capability.