arXiv · 2025 · Preprint
Nikita Torgashov et al. (KTH Royal Institute of Technology) · → Paper · Demo: ✓ · Code: ✓
VoXtream is a fully autoregressive zero-shot streaming TTS system that begins speaking from the first received word, achieving a first-packet latency of 102 ms on GPU — the lowest reported among publicly available streaming TTS models at the time of publication.
Problem
Most high-quality TTS models are designed for offline operation, requiring the complete input text before speech generation can begin. Systems that do support streaming either (1) generate in output-streaming mode where text is already fully known before decoding starts, merely chunking the output, or (2) interleave text and speech at fixed chunk granularities using non-autoregressive decoders that accumulate tokens before audio playback, reintroducing latency at the decoding stage. No prior publicly available system demonstrated simultaneously low first-packet latency (under 200 ms), full-stream operation (processing word-by-word input as it arrives), and zero-shot speaker generalization.
Method
VoXtream is built around three transformer components that operate in a cascade, all using a Llama-style decoder-only architecture.
The Phoneme Transformer (PT, 6 layers, 8 heads) encodes a growing phoneme sequence incrementally as words arrive from the text stream. Rather than waiting for N phonemes of look-ahead before emitting any output, PT starts immediately after the first word and allows a maximal look-ahead of up to 10 phonemes. Crucially, the look-ahead is a soft bound — the model does not block on it. Text is converted to phonemes at the word level using g2pE.
The Temporal Transformer (TT, 12 layers, 16 heads, 1024-dim, 4096 FFN) is autoregressive over time and is conditioned on the phoneme embeddings from PT. Audio is tokenized by the Mimi codec (12.5 Hz, 24 kHz). TT jointly predicts the first Mimi codebook (semantic token) and a duration token encoding a monotonic alignment flag. The duration token encodes two decisions: a stay/go flag (continue current phoneme or advance) and a count (1 or 2 phonemes per frame) to handle speaking-rate variation. An acoustic delay of one step is applied following Mimi’s original design.
The Depth Transformer (DT, 4 layers, 8 heads, 8192 FFN) generates acoustic tokens for codebooks 2–12 of Mimi, conditioned on TT’s output embedding and the semantic token. DT also receives a speaker embedding from a ReDimNet model trained on 100k+ identities, enabling zero-shot speaker conditioning. Critically, the DT weights are borrowed frozen from the CSM model trained on a large-scale dataset — this cross-model distillation substantially improves output quality without requiring large-scale training.
The entire system is trained on 9k hours (4.5k from Emilia, 4.5k from HiFiTTS-2) with AdamW optimization over 9 epochs on two A100-80GB GPUs. Loss is the negative log-likelihood of TT and DT outputs combined. Montreal Forced Aligner provides initial phoneme-to-audio alignment for supervision.
At inference, the Mimi decoder converts tokens frame by frame in a streaming fashion, producing 80 ms of audio per frame. With torch.compile, the full system achieves a first-packet latency of 102 ms and an RTF of 0.17, meaning it runs more than 5 times faster than real time.
Key Results
On SEED-TTS test-en and LibriSpeech test-clean (Table 1), VoXtream (441M params, 9k hours) is compared against mid-scale and large-scale baselines in non-streaming mode. Among mid-scale systems (comparable training data), VoXtream-NS (non-streaming variant) achieves the best SPK-SIM (0.537 on SEED) and UTMOS (3.89 on SEED) and attains the second-best WER on LibriSpeech test-clean (2.99%) across all systems evaluated. The streaming variant VoXtream (full-stream) introduces only minor degradations: WER rises from 2.99% to 3.15% on LibriSpeech test-clean; SPK-SIM and UTMOS are essentially unchanged.
In output-streaming comparison (Table 1), VoXtream outperforms XTTS-v2 on all three metrics and is second only to CosyVoice2 — a system trained on 167k hours of multilingual data (18x more data).
On LibriSpeech long (Table 2, full-stream evaluation against CosyVoice2 under matched conditions), VoXtream achieves substantially lower WER (3.24% vs. 6.11%) and wins the naturalness preference study (57% vs. 31%, p < 5e-10), though CosyVoice2 retains higher SPK-SIM (0.685 vs. 0.564) due to its non-autoregressive flow-matching decoder.
Latency results (Table 3) show VoXtream at 171 ms FPL without optimization and 102 ms with torch.compile, compared to CosyVoice2 at 1643 ms and XTTS-v2 at 295 ms (output-stream only).
Ablation (Table 4) confirms that both the frozen CSM-DT weights and the ReDimNet speaker encoder contribute independently: CSM-DT improves quality (UTMOS from 3.39 to 3.9), and SPK-ENC improves SPK-SIM by 19% when DT is unfrozen and by 6% even with frozen DT.
Novelty Assessment
The primary contribution is engineering and system design rather than a fundamentally new architecture. The three-transformer cascade (phoneme encoding, temporal prediction, depth prediction) recombines established ideas — monotonic alignment from VALL-E R, depth transformers from Moshi/CSM, and speaker embeddings — in a configuration specifically optimized for minimal first-packet latency. The incremental phoneme encoding with bounded look-ahead (not a hard wait) is a meaningful design choice that differentiates VoXtream from prior work. The borrowing of frozen CSM depth transformer weights as a form of cross-model knowledge transfer is practical and effective. The overall contribution is a well-engineered point on the quality-latency frontier, validated with rigorous user studies on Prolific with attention checks. The work is primarily a recipe contribution rather than an architectural breakthrough.
Field Significance
Moderate — VoXtream demonstrates that a mid-scale autoregressive system trained on 9k hours can match or exceed larger non-streaming baselines on intelligibility and naturalness while achieving first-packet latency an order of magnitude lower than competing streaming systems. It provides a concrete and reproducible design for the quality-latency trade-off in streaming TTS, where prior publicly available systems required either significantly more training data or accepted substantially higher latency. The fully released code and model make it a practical reference point for future streaming TTS work.
Claims
- Fully autoregressive streaming TTS can achieve first-packet latencies under 150 ms without sacrificing intelligibility relative to non-streaming operation. (§4, Table 1, Table 3)
- Training data scale is the primary driver of speaker similarity in zero-shot TTS, and systems trained on an order-of-magnitude less data show measurable SPK-SIM gaps even when naturalness scores are competitive. (§4, Table 1, Table 2)
- Borrowing frozen depth transformer weights from a large-scale pretrained model provides substantial quality improvement for a mid-scale system without requiring additional large-scale training. (§3, Table 4)
- Full-stream input processing introduces only marginal quality degradation relative to output-streaming when a bounded phoneme look-ahead is used, suggesting that input latency and output quality are largely decoupled in autoregressive codec TTS. (§4, Table 1)
- Non-autoregressive flow-matching decoders used in chunk-based streaming systems incur first-packet latencies exceeding 1.5 seconds on standard hardware, which is prohibitive for real-time spoken conversational agents. (§4, Table 3)
Limitations and Open Questions
Speaker similarity remains lower than large-scale systems trained on hundreds of thousands of hours, particularly in full-stream mode (SPK-SIM 0.458 on LibriSpeech test-clean vs. 0.587 for CosyVoice2 trained on 167k hours). The system is English-only; multilingual extension is not addressed. Prosody and speaking rate are not explicitly controllable at inference. The use of a frozen CSM depth transformer introduces a dependency on an external large-scale model. Long-form streaming beyond 10-15 second utterances is identified as future work. Performance in adverse or spontaneous speech conditions (outside the training domain) is untested.
Wiki Connections
This paper advances the streaming-tts concept by establishing a new latency frontier for publicly available systems. The three-stage architecture places it squarely in autoregressive-codec-tts, using the Mimi neural-codec as the token representation. Zero-shot speaker generalization is achieved via speaker-adaptation through the ReDimNet embedding. The evaluation methodology — WER, SPK-SIM, UTMOS, and MUSHRA user studies — connects to evaluation-metrics and subjective-evaluation. The zero-shot generalization capability links to zero-shot-tts.