arXiv · 2026 · Preprint
Gabriel Clark et al. (Zyphra) · → Paper · Demo: ? · Code: ✓
ZONOS2 is an open-weight (Apache 2.0) 8B-parameter mixture-of-experts autoregressive codec-token TTS model (900M active parameters) trained on 6.2M hours of multilingual speech, released together with ZTTS1-Eval, a zero-shot TTS benchmark that pairs a 9-language read-speech set with a 17-language in-the-wild spontaneous set and replaces the Seed-TTS-Eval scoring models.
Problem
The authors argue that TTS systems tend to optimize one of stability, naturalness, controllability, multilinguality, or inference cost at the expense of the others, and that their earlier phoneme-based Zonos-v0.1 (1.6B parameters, 200K hours) was limited by G2P frontends that fail silently on code-switched text, rare vocabulary, and lower-resource languages (§II-B, Table I). On the evaluation side, they argue that Seed-TTS-Eval covers only English and Chinese read speech and scores with older models (Whisper-Large, Paraformer, WavLM), and that its extensions (CV3-Eval, the MiniMax multilingual set) keep the same scoring stack, include no spontaneous in-the-wild speech, and do not measure prosodic distribution or generation diversity (§I, §V, Table II).
Method
ZONOS2 is a decoder-only transformer that takes UTF-8 bytes of the input text and autoregressively predicts DAC audio tokens (9 RVQ codebooks), arranged with a MusicGen-style delay pattern so that codebook j is shifted by j frames; streaming decoding therefore needs a lookahead of N-1 = 8 generated frames before a full frame can be passed to the DAC decoder (§II-A). DAC is used as an existing tokenizer and is not retrained. Text is tokenized as raw bytes rather than phonemes: the authors report that a byte-level variant first matched and then surpassed a phoneme-based counterpart as data and model size grew, and adopt bytes to avoid G2P failures (§II-B); no numbers are given for that comparison.
The backbone (Table V) has 28 layers with hidden size 2048: the first three and the final layer are dense SwiGLU blocks, the rest are MoE blocks with 16 experts and top-1 routing, except the last routed layer which uses top-2. The router follows ZAYA1-8B (down projection, Exponential Depth Averaging across layers, RMSNorm, three-layer GELU MLP). Attention is GQA (16 query heads, 4 KV heads) with RoPE, QK-normalization, and headwise Qwen gating (§II-F, Appendix C). Training minimizes masked next-frame NLL over audio tokens (text and conditioning positions are context only), with tanh logit soft-capping (τ = 15) and an auxiliary-loss-free style expert-balancing bias trained with a separate AdamW optimizer (§II-G).
Conditioning is prepended to the sequence. Voice cloning uses a 2048-dimensional ECAPA-TDNN speaker embedding extracted with the pretrained speaker module from Qwen3-TTS, projected by an LDA transform to 1024 dimensions to suppress within-speaker nuisance factors (duration, noise, lexical content, pauses), then mapped into the model by a learned linear projection; the reference clip needs no transcript and its length does not grow the context (§II-C). Speaking rate (bytes per second of normalized text) is quantized into bucket tokens prepended to the text, and the same bucketing scheme encodes bandwidth, volume, leading/trailing silence, and estimated SNR; a separate “Augmented Embedding” token marks clone audio that was corrupted with noise, music, codec compression, or reverberation, and a “Quality Mode” token is attached to a high-quality subset during the final anneal (§II-D, §II-E).
Training runs in four stages (§IV): pre-training without conditioning (77,500 steps, 2.9T tokens, Muon optimizer, 6,144-frame sequences, 121.8 hours of audio per global batch), mid-training with stricter inter-ASR transcript agreement (15,000 steps, about 560B tokens), and two 10,000-step annealing stages. The first anneal introduces speaker, rate, and quality conditioning, computing the speaker embedding on a random crop of the target and masking the loss on that crop to limit shortcut leakage; the second computes the embedding on the full target, removes the loss mask, and adds Quality Mode. Rate and quality tokens are dropped with probabilities 0.4 and 0.25. Training data (6.2M hours, majority English, Figure 3) are segmented by VAD and transcribed by multiple ASR systems, with utterances discarded when pairwise inter-ASR WER exceeds a stage-dependent threshold; the transcript used as input is also varied across training (§III).
ZTTS1-Eval (§V) has a Clean set of 500 FLEURS-R utterances in each of 9 languages (en, zh, de, es, fr, it, ja, ko, ru, about 13 hours, with hard English and Chinese portions) and an in-the-wild (ITW) set of 1,618 VoxBlink2 utterances (2.86 hours) across 17 languages (Table VI). It scores WER with Qwen3-ASR, speaker similarity with ReDimNet, quality with MSR-UTMOS, generation diversity with DS-WED between pairs of generations from the same prompt, and prosody with TTSDS2-Prosody. The authors state that ZONOS2 is not trained on ZTTS1-Eval audio but cannot speak to baselines with undisclosed training data.
Key Results
ZTTS1-Eval Clean (Table III). On English, base ZONOS2 reaches WER 2.76 and speaker similarity 78.6, the highest speaker similarity among open-source systems (Fish Audio S2 Pro 76.9, Qwen3-TTS 1.7B 68.3, VoxCPM2 65.2) and second overall behind Cartesia Sonic 3.5 (79.9). Its English WER is above Qwen3-TTS (1.94), ElevenLabs V3 (2.35), Gemini 3.1 Flash (2.50), and Cartesia (2.56), and below Fish S2 Pro (3.60), Inworld TTS 2 (3.14), and VoxCPM2 (4.23). Outside English, base ZONOS2 has much higher WER on several languages (zh 15.62, hard_zh 26.23, ko 18.96, fr 13.18, hard_en 15.04). Quality Mode lowers WER on every Clean column except English (en rises from 2.76 to 3.99), e.g. zh 15.62 to 6.73 and ko 18.96 to 4.14, and raises UTMOS on every column except Spanish (2.96 to 2.94); its effect on speaker similarity is mixed (up for zh, fr, it, ja, ko; down for en, hard_en, hard_zh, de, es, ru). Even with Quality Mode, ZONOS2 trails Qwen3-TTS on WER and UTMOS in every Clean column, and Fish S2 Pro and Cartesia have higher speaker similarity than ZONOS2 in most non-English columns. ElevenLabs V3 and Gemini 3.1 Flash do not support zero-shot cloning, so their speaker-similarity scores are not comparable.
ZTTS1-Eval In-the-wild (Table IV). Quality Mode lowers WER and raises UTMOS in all 17 languages relative to base ZONOS2 (English WER 4.70 to 2.21) while lowering speaker similarity in all 17 (English 67.0 to 56.9). On English ITW, ZONOS2 Quality Mode WER (2.21) is higher than every baseline listed (Qwen3-TTS 1.05, ElevenLabs 1.35, Cartesia 1.40, VoxCPM2 1.69, Inworld 1.89, Fish 2.09, Gemini 2.12). Base ZONOS2 English speaker similarity (67.0) is below VoxCPM2 (68.1) and Cartesia (70.2) and above Fish (65.0), Qwen3-TTS (61.5), and Inworld (53.0). Some baselines fail on specific languages (Qwen3-TTS WER 82.04 on pl and 83.20 on tr; Fish 74.12 on th), and several closed systems do not support all 17 languages.
Prosody and diversity (Figures 5 to 7). The authors report that ZONOS2 has the best mean TTSDS2 prosody score on the English ITW set and a weaker but competitive score on the English Clean set, much higher DS-WED generation diversity than all other models on both English sets, and an Allosaurus SR distribution closer to the source audio. These results are shown only as plots; no numeric values are tabulated.
CV3-Eval and Seed-TTS-Eval (Table VII). ZONOS2 reports Seed-TTS-Eval WER 2.05 (Test-EN), 2.55 (Test-ZH), and 11.15 (Test-ZH-Hard) with speaker similarity 47.6, 58.2, and 56.2, and CV3-Eval zero-shot WER between 4.48 (en) and 25.55 (hard_zh), cross-lingual WER up to 23.27 (to_hard_zh), and emotion accuracy 37.3 (en) and 36.7 (zh). Table VII lists ZONOS2 only, without baseline rows, and the speaker-similarity scorer used for these two benchmarks is not stated.
No human listening test is reported; quality is measured only with automatic predictors (MSR-UTMOS, DNSMOS). Streaming latency is described as good in the abstract but no latency, real-time factor, or throughput number is reported, and no ablation isolates the effect of the MoE backbone, LDA projection, or byte tokenization.
Novelty Assessment
The model is a combination of existing parts: delayed-codebook autoregression over a pretrained DAC codec, an LLM-style MoE backbone and router taken from ZAYA1-8B, byte-level text input, and a pretrained ECAPA-TDNN speaker encoder. The authors present MoE in open-source TTS as their architectural novelty; it is an application of an LLM architecture to codec-token TTS rather than a new structure. The more specific contributions are training-recipe ones: the LDA projection of a high-bandwidth speaker embedding plus a two-stage anneal (crop with loss masking, then full-target embedding) to delay shortcut overfitting when the embedding is computed from the target utterance, quality and acoustic-attribute tokens to decouple low-bandwidth controls from the speaker embedding, and an inter-ASR agreement filter tuned per training stage. These are described qualitatively, without ablation numbers. ZTTS1-Eval is a new test set and scoring stack (newer ASR, speaker, and MOS predictors, a spontaneous multilingual set, and distributional prosody and diversity measures); it assembles existing metrics rather than defining new ones, and the scoring choices are not validated against human judgments in this report. The discussion of MoE routing instability on delayed DAC tokens (entropy collapse, the fix of dense first and last layers plus top-2 final routing) and of MHA being more stable than GQA is a useful engineering record.
Field Significance
Moderate. The report provides open weights for a large sparse-MoE codec-token TTS model and a multilingual zero-shot benchmark that covers spontaneous in-the-wild prompts in 17 languages with updated automatic scorers, so it can serve as an open baseline and as an evaluation resource. Its own results show a system that is strongest on English Clean speaker similarity and on figure-reported prosody diversity, while trailing several open and closed baselines on intelligibility and predicted quality, with no human evaluation and no ablations to attribute gains to the MoE backbone.
Claims
- complicates: Mixture-of-experts routing is harder to keep balanced on delayed multi-codebook audio tokens than on text, requiring architectural workarounds for stable training.
Evidence: During ZONOS2 pre-training, normalized router entropy of several layers collapsed to as low as 0.6; the authors stabilized training by making the first three and the final layers dense, using top-2 routing on the final MoE layer, and manually adjusting balancing-bias and router learning rates, and report balancing on delayed DAC tokens was much less stable than on text at similar size. (§IV-A, §VII-B, Figure 4)
- supports: At sufficient data and model scale, raw byte-level text input can replace phoneme input for multilingual TTS, avoiding silent G2P frontend failures on code-switched and rare text.
Evidence: The authors report a byte-level variant first matched and then surpassed its phoneme-based counterpart as scale increased, and Table I shows silent G2P failures (a Chinese span under an English tag, an overmatched morphology rule, a Spanish inventory dropping “sh”); no numeric comparison is given. (§II-B, Table I)
- complicates: Conditioning on a speaker embedding computed from the target utterance leaks utterance-specific content and timing, giving the model a shortcut that limits how long cloning can be trained before instability.
Evidence: The authors observed causal leakage of length, lexical content, and pause distribution that caused silent or babble outputs; an LDA projection of the embedding and a two-stage anneal (cropped embedding with loss masking, then full-target embedding) were the most effective mitigations. (§II-C, §IV-C, §VII-C)
- complicates: A single quality or intelligibility control token can trade speaker similarity for intelligibility and predicted quality, and its effect on intelligibility is not uniform across languages.
Evidence: ZONOS2 Quality Mode lowers WER and raises UTMOS in all 17 in-the-wild languages but lowers speaker similarity in all 17 (English 67.0 to 56.9); on the Clean set it lowers WER on every column except English, where WER rises from 2.76 to 3.99. (§VI, Tables III and IV)
- refines: Zero-shot TTS benchmarks restricted to read English and Chinese speech with older ASR and speaker scorers under-describe multilingual and spontaneous-prompt performance, and rankings change between read and in-the-wild prompts.
Evidence: ZTTS1-Eval adds a 17-language VoxBlink2 in-the-wild set alongside a 9-language FLEURS-R read set and rescoring with Qwen3-ASR, ReDimNet, and MSR-UTMOS; ZONOS2 is second overall in English speaker similarity on the Clean set (78.6) but below VoxCPM2 and Cartesia on English in-the-wild (67.0 vs. 68.1 and 70.2). (§V, Table II, Tables III and IV)
Limitations and Open Questions
No human listening test, no latency numbers, and no ablations
Quality is measured only with automatic predictors (MSR-UTMOS, DNSMOS), the streaming-latency claim has no reported measurement, and none of the design choices (MoE backbone, LDA projection, byte tokenization, two-stage anneal) is isolated by a quantitative ablation, so the report’s attribution of quality and latency gains to these choices is not testable from its own results.
- Intelligibility is uneven: base ZONOS2 has high WER on several Clean languages (zh 15.62, ko 18.96, fr 13.18, hard_zh 26.23) and Quality Mode, which fixes much of this, costs speaker similarity.
- ZTTS1-Eval is designed and run by the same team that built ZONOS2; its scorer choices are not validated against human ratings, and baseline training-data overlap with FLEURS-R or VoxBlink2 cannot be ruled out.
- TTSDS2, DS-WED, and Allosaurus results appear only in plots, so the prosody and diversity advantages cannot be checked numerically from the report.
- Table VII has no baseline rows, and the speaker-similarity scorer for CV3-Eval and Seed-TTS-Eval is not stated, so those numbers are not directly comparable to published results.
- The training corpus (6.2M hours) is not itemized beyond source types and a language chart, and the speaking-rate and acoustic-attribute control tokens are described but not evaluated.
- The authors report MHA was more stable and higher quality than GQA but chose GQA for speed; the size of that quality cost is not reported.
Wiki Connections
- Autoregressive Codec TTS — ZONOS2 autoregressively predicts delayed DAC RVQ tokens with a sparse MoE decoder-only backbone, and documents routing instability specific to this token stream.
- Zero-Shot TTS — cloning from an untranscribed reference via an LDA-projected ECAPA-TDNN embedding is evaluated on unseen FLEURS-R and VoxBlink2 prompts across 17 languages.
- Multilingual TTS — a single byte-tokenized model trained on 6.2M multilingual hours is evaluated in 9 read languages (en, zh, de, es, fr, it, ja, ko, ru) and 17 in-the-wild languages (adding ar, hi, id, pl, pt, th, tl, tr).
- Qwen3-TTS — open-source baseline that leads ZONOS2 on WER and UTMOS across the Clean set; its pretrained speaker module is reused for ZONOS2’s speaker embedding.
- Fish Audio S2 — open-source baseline (S2 Pro) with higher speaker similarity than ZONOS2 on most non-English Clean languages but lower on English.
- VoxCPM2 — open-source baseline that exceeds ZONOS2 in in-the-wild speaker similarity on English and most other languages.
- Seed-TTS — source of Seed-TTS-Eval, the benchmark ZTTS1-Eval is designed to extend, and on which ZONOS2 reports its own scores.
- CosyVoice 3 — source of CV3-Eval, compared against in Table II and used for ZONOS2’s zero-shot, cross-lingual, and emotion results.
- MiniMax-Speech — source of the 24-language MiniMax multilingual test set that ZTTS1-Eval is contrasted with.
- TTSDS2 — distributional prosody score adopted as a ZTTS1-Eval measure.
- DS-WED — prosody-diversity metric used in ZTTS1-Eval to compare generation variation across systems.