arXiv · 2026 · Preprint
Hangrui Hu et al. (Alibaba / Qwen Team) · → Paper · Demo: ✓ · Code: ✓
Qwen3-TTS introduces a dual-track autoregressive LM family — 0.6B and 1.7B variants — with two purpose-designed tokenizers (25 Hz single-codebook and 12.5 Hz 16-layer RVQ) enabling sub-100 ms first-packet streaming TTS across 10 languages, achieving state-of-the-art WER on Seed-TTS and top speaker similarity across all evaluated languages.
Problem
Prior large-scale TTS systems using discrete speech tokenization face a trade-off between semantic richness and acoustic detail of the tokenizer representation; purely semantic tokens lack expressivity while purely acoustic tokens inject low-level noise that complicates autoregressive LM modeling and causes long-horizon error accumulation. Additionally, achieving ultra-low first-packet latency (<100 ms) while maintaining synthesis quality and supporting multilingual voice cloning and instruction-following control simultaneously remained unsolved at scale.
Method
Qwen3-TTS builds on the Qwen3 LM family (0.6B and 1.7B) and introduces two speech tokenizers. Qwen-TTS-Tokenizer-25Hz is a 25 Hz single-codebook tokenizer trained in two stages: (1) ASR-supervised pre-training of a Qwen2-Audio encoder augmented with a resampling and vector quantization layer; (2) fine-tuning with a convolution-based mel-spectrogram decoder to inject acoustic information into token representations. Streaming is enabled by a sliding-window block DiT (diffusion transformer with flow matching), followed by BigVGAN. The codebook has 32,768 entries. Qwen-TTS-Tokenizer-12Hz is a 12.5 Hz multi-codebook (16 RVQ layers) tokenizer with a GAN-based training framework; it uses WavLM as a semantic teacher for the first codebook layer and 15-layer RVQ for acoustic details; fully causal design enables streaming with no lookahead.
The main model (Qwen3-TTS) uses the Qwen3 tokenizer for text and the appropriate speech tokenizer for audio. A jointly trained learnable speaker encoder provides identity control. For real-time streaming, the model concatenates textual and acoustic tokens along the channel axis (dual-track) and predicts acoustic tokens immediately upon receiving a textual token. Qwen3-TTS-12Hz uses a Multi-Token Prediction (MTP) module to hierarchically predict the zeroth codebook via the backbone and residual codebooks in a single-frame instant generation step.
Training comprises three pre-training stages (general, high-quality, long-context — context window extended from 8,192 to 32,768 tokens) plus three post-training stages: Direct Preference Optimization (DPO) on multilingual preference pairs, GRPO with rule-based rewards for stability and capability, and lightweight speaker fine-tuning (SFT) for specific voice adoption.
Inference is measured end-to-end on a vLLM V0 backend with torch.compile and CUDA Graph acceleration. At concurrency 1, Qwen3-TTS-12Hz-0.6B achieves a first-packet latency of 97 ms and RTF of 0.288.
Key Results
On the Seed-TTS benchmark (zero-shot voice cloning), Qwen3-TTS-12Hz-1.7B-Base achieves WER 1.24% on test-en and 0.77% on test-zh — the best reported among open-source models, surpassing CosyVoice 3 (1.45% / 0.71%) and comparable to or better than closed-source systems. On multilingual evaluation (10 languages), Qwen3-TTS achieves the highest speaker similarity scores in all 10 languages versus MiniMax-Speech and ElevenLabs Multilingual v2. On cross-lingual voice cloning (CV3-Eval), the 12Hz-1.7B variant reduces zh-to-ko error rate to 4.82 vs. 14.4 for CosyVoice 3 (66% reduction). On InstructTTSEval instruction following, the 12Hz-1.7B-VoiceDesign variant achieves APS 85.2 / DSD 81.1 on Chinese, outperforming GPT-4o-mini-tts by 28+ APS points. Long-form synthesis (>10 min audio) shows WER 1.533 (zh) and 1.571 (en) for the 25Hz-1.7B variant, outperforming all open-source baselines including Higgs-Audio-v2 and VibeVoice.
The 12Hz tokenizer demonstrates new state-of-the-art reconstruction quality on LibriSpeech test-clean: PESQ-WB 3.21, PESQ-NB 3.68, STOI 0.96, UTMOS 4.16, SPK-SIM 0.95 — all outperforming prior semantic-aware codecs including Mimi and FireredTTS 2 Tokenizer.
Novelty Assessment
The dual-tokenizer architecture (25 Hz and 12 Hz variants serving different quality/latency trade-offs) is a practical engineering contribution rather than a fundamental algorithmic advance. The key novelty is the balanced integration of semantic and acoustic information in Qwen-TTS-Tokenizer-25Hz via the two-stage training process, and the fully causal 12Hz RVQ design enabling streaming at 80 ms per token without lookahead. The MTP module for multi-codebook prediction, DPO + GRPO post-training pipeline, and the probabilistically activated thinking pattern for instruction following are practical contributions. The work is primarily a systems and data-at-scale paper: training on 5M hours is the dominant driver of multilingual and cross-lingual breadth. Architectural novelty is incremental over CosyVoice/Qwen2 foundations.
Field Significance
Tip
High — Qwen3-TTS demonstrates that a unified autoregressive LM with purpose-designed dual tokenization can simultaneously achieve sub-100 ms first-packet latency, state-of-the-art zero-shot voice cloning, and strong multilingual performance across 10 languages. It provides a comprehensive open-source reference point for the full stack of production streaming TTS, covering tokenizer design, multi-stage training, RLHF alignment, and instruction-conditioned voice control. The scale of evaluation across zero-shot, cross-lingual, instruction-following, and long-form tasks sets a new empirical standard for comparing open-source large-scale TTS systems.
Claims
- Tokenizers that combine semantic supervision with acoustic reconstruction objectives achieve better downstream TTS quality than purely semantic or purely acoustic tokenizers. (§2.1, §4.1.1)
- Reducing codec frame rate from 25 Hz to 12.5 Hz improves autoregressive LM stability in long-horizon TTS generation by reducing sequence length and error accumulation. (§4.2.1)
- Multi-stage post-training combining preference optimisation and rule-based reinforcement learning improves both the naturalness and instruction-following capability of large-scale TTS systems. (§3.2)
- Causal codec decoders with no lookahead enable first-packet latencies under 100 ms in streaming TTS without sacrificing reconstruction quality. (§3.4, Table 2)
- Natural language instruction following in TTS generalises to fine-grained voice design, enabling open-source systems to match or exceed commercial baselines on standardised instruction-following benchmarks. (§4.2.4, Table 8)
Limitations and Open Questions
The model relies on proprietary 5M-hour multilingual training data; results may not transfer to lower-resource settings. Despite 10 languages covered, Japanese and some European languages show higher WER than Chinese/English. The 25Hz variant performs better for long-form synthesis, suggesting the 12Hz variant has difficulty with long-horizon dependencies — an open research question for ultra-low-bitrate streaming TTS. Voice design via natural language instruction (thinking pattern) has not been rigorously benchmarked for robustness against adversarial or ambiguous prompts.
Wiki Connections
This paper significantly extends the autoregressive-codec-tts and streaming-tts concept pages — it demonstrates that a dual-tokenizer approach with 97 ms first-packet latency is achievable at scale. The 12Hz RVQ tokenizer advances the neural-codec landscape, pushing SOTA on reconstruction while halving frame rate versus prior work. The DPO + GRPO training pipeline connects to rlhf-speech, and the voice design and target-speaker editing capabilities are central to instruction-conditioned-tts and zero-shot-tts. Multilingual breadth across 10 languages is relevant for multilingual-tts.
Citing papers in this corpus (integrated 2026-05-29): Fish Audio S2 benchmarks against Qwen3-TTS on Seed-TTS-Eval and Long-Audio benchmarks; LongCat-AudioDiT compares SIM on Seed-ZH and Seed-EN. Back-linked by MOSS-TTS, which positions its open-source large-scale AR TTS directly against Qwen3-TTS and evaluates on the same Seed-TTS-Eval benchmark.