arXiv · 2026 · Preprint
Bajian Xiang et al. (Alibaba Token Foundry) · → Paper · Demo: ✓ · Code: ?
A CosyVoice3-lineage LM plus flow-matching TTS system that halves the speech-token rate to 12.5 Hz and trains in five progressive stages (separate pretraining, joint LM-FM training on LM hidden states, LM reinforcement learning, FM prompt-robustness training, FM reinforcement learning) to combine zero-shot cloning, 16-language coverage, free-style instructions, 3-minute one-pass synthesis and robustness to degraded prompts in one production model.
Problem
Current zero-shot TTS paradigms each trade something away. Discrete-token autoregressive systems lose acoustic detail through quantization and pay a decoding cost that grows with token rate. Non-autoregressive flow or diffusion systems complicate streaming. Hybrid LM-plus-flow-matching systems such as CosyVoice 2 and CosyVoice 3 pass only a single-codebook token sequence from the LM to the acoustic model, which the authors treat as an information and optimization bottleneck. Continuous-autoregressive systems avoid quantization but are more sensitive in stability and long-form consistency. The authors also argue that short, clean benchmarks hide failures on dialects, long-form text, text normalization and noisy or reverberant reference prompts, which a production system must handle together.
Method
The system keeps the three-part CosyVoice design: an autoregressive text-to-token LM (bi-streaming), a chunk-based DiT flow-matching model (FM) that maps LM output to mel-spectrograms, and a causal BigVGAN vocoder. The speech tokenizer follows the CosyVoice3 supervised design: a causal SenseVoice encoder (32 Transformer layers, 1280 hidden dimensions) with a 10-dimensional, three-level FSQ bottleneck (3^10 = 59,049 codes), trained through a MinMo-style multi-task pipeline (ASR, language ID, emotion, audio events, speaker, general audio analysis) on top of Qwen2.5-7B-Instruct. Training runs continuous first, then activates FSQ. The token rate drops from 25 Hz to 12.5 Hz, with the larger codebook and broader supervision meant to offset the stronger temporal compression (§2.1, §3.1).
Training runs in five stages, each initialized from the previous checkpoint (§2.2):
- Independent pretraining. LM and FM are pretrained separately on large multilingual, dialect and instruction data, as in CosyVoice2/3 (§2.2.1).
- Joint LM-FM training with data annealing. Following JoyVoice, the FM is conditioned on continuous LM hidden states rather than discrete token embeddings, and the token-prediction and flow-matching losses are optimized jointly, so the FM loss also shapes LM representations. Training then anneals onto a curated high-quality subset (§2.2.2).
- LM reinforcement learning. With FM and vocoder frozen, online GRPO with a KL penalty optimizes token rollouts under a composite reward of token-domain ASR content, duration outliers, diversity and a prosody term (alignment progression, pause timing). All rewards are computed before FM inference. A DiffRO branch (Gumbel-Softmax) adds token-level gradients only for rollouts with non-negative group advantage, and degenerate rollouts (repetition, missing stop) are excluded. A second curriculum phase adds a dialect-classification reward (§2.2.3).
- Acoustic robustness training. With the LM frozen, the FM learns to produce clean speech from prompts degraded by noise, reverberation, device and codec responses, packet loss, echo and compound conditions, so prompt enhancement sits inside the cloning path rather than in a separate denoiser (§2.2.4).
- FM reinforcement learning. FlowTTS-GRPO converts the ODE sampler into a marginal-preserving SDE over an early-step window and optimizes a reward combining speaker-verification similarity, ASR intelligibility and DNSMOS, each normalized by its batch standard deviation; classifier-free guidance is dropped during rollouts (§2.2.5).
Control comes from free-style natural-language instructions (role, emotion, style, rate, timbre, accent) and 86 new inline tags for phrase-level emotion, style, speed and non-verbal events such as laughter and coughing. Coverage is 16 languages (seven added over CosyVoice3) and 20 Chinese dialect regions, with one-pass synthesis up to 3 minutes. A separate two-stage speaker-adaptation SFT (joint LM-FM rounds with duration-matched replay, then FM-only refinement) and a 48 kHz super-resolution vocoder with a multi-scale STFT discriminator support target-voice deployment (§2.3). Model size and training-data hours are not reported.
Key Results
Tokenizer ablation (§4.1, Tables 1-2). At a fixed 6,561-code vocabulary, moving from 25 Hz to 12.5 Hz degrades downstream TTS (test-zh CER 1.45 to 2.59, test-zh SIM 80.6 to 72.44). Enlarging the codebook recovers it: the 59,049-code 12.5 Hz tokenizer gives the lowest CER/WER of the 12.5 Hz variants on all three SEED-TTS-Eval sets (1.23 / 2.37 / 6.68), while the 19,683-code variant keeps marginally higher SIM on all three (83.25 / 77.58 / 80.85 vs. 83.09 / 77.49 / 80.61). In intrinsic ASR the 59,049-code tokenizer is best among 12.5 Hz variants on five of six columns, but trails the 25 Hz CosyVoice3 tokenizer on CV-ko (11.70 vs. 11.35) and FLEURS-zh (3.85 vs. 3.77).
SEED-TTS-Eval (§4.2, Table 3). Content error is competitive but not lowest: test-zh CER 0.84 is second to Qwen3-TTS-12Hz-1.7B-Base (0.77); test-en WER 1.54 is fifth (behind Qwen3-TTS 1.24, Qwen3.5-Omni-Plus 1.26, Dots.TTS 1.30, LongCat-AudioDiT 1.50); test-hard CER 7.00 is fourth (behind CosyVoice3 5.83, LongCat-AudioDiT 6.04, Dots.TTS 6.60). ERes2Net speaker similarity is the highest among systems with an ERes2Net score on all three sets (0.847 / 0.815 / 0.824 vs. next-best CosyVoice3-1.5B 0.837 on zh, Dots.TTS 0.792 on en, CosyVoice3 0.816 on hard). Under WavLM similarity it ranks lower (test-zh 0.792, sixth; test-en 0.762, tied with Seed-TTS behind LongCat-AudioDiT 0.786 and Dots.TTS 0.771). Several baselines (Seed-TTS, MiniMax-Speech, Qwen3-TTS) have no ERes2Net score, and the authors note the two similarity models rank systems differently. They also state that pushing CER/WER lower costs naturalness and that the model deliberately targets a trade-off.
CV3-Eval multilingual (§4.3.1, Tables 4-5). Across 16 languages the system has the lowest error on 6 (ja 4.78, ko 4.30, ru 4.68, ar 3.36, ms 2.62, th 1.45; ar by 0.02 over MiniMax-Speech-2.8-HD’s 3.38). The MiniMax-Speech-2.8-HD API is lowest on 9 (en, de, es, fr, it, id, pt, vi, tl) and Qwen3-TTS on zh. On hard-zh and hard-en it has the highest SIM (78.7 / 76.6, narrowly above CosyVoice3-1.5B’s 78.5 / 76.1) and highest DNSMOS (3.93 / 4.04); hard-zh CER 7.44 is second to MiniMax (7.42) and hard-en WER 6.71 third (ElevenLabs-v3 5.84, Qwen3-TTS 6.53).
CV3-Eval cross-lingual (§4.3.2, Table 6). It is lowest-error in 8 of 12 transfer directions and second in the other 4, where Qwen3-TTS-12Hz-1.7B-Base is lowest (to-zh from en and ko, to-en from ja and ko). The mean error over the 12 directions is 4.05%, against 10.09% for CosyVoice3-1.5B (computed from Table 6, consistent with the text) and 4.42% for Qwen3-TTS (computed here from Table 6).
Qwen-Audio-TTS-Eval (§4.4, Tables 7-10). This in-house diagnostic benchmark is introduced in the paper.
- Text normalization (1,375 cases, Gemini-2.5-Pro judge): highest overall accuracy in zh (68.7 vs. CosyVoice3 59.3) and en (65.7 vs. Qwen3-TTS 60.5); highest in 8 of 10 category columns, second on Num.-en (78.2 vs. 78.7) and Acr.-en (59.4 vs. 63.3).
- Long-form (100 zh and 100 en paragraphs, about 1.5-3 minutes): all-bucket zh CER 2.22 is second to VoxCPM2 (0.54); en WER 5.00 is third (VoxCPM2 3.20, Qwen3-TTS 4.81). The zh long bucket rises to 5.62 CER. Prompt and segment similarity are second to CosyVoice3-1.5B in both languages, but CosyVoice3 has far higher content error (zh 25.41, en 23.24).
- Acoustic robustness (894 real degraded prompts): Reverb SIM 74.12 is the highest in the table; the system is second on Noisy and Unclear SIM (behind Dots.TTS, 76.69 on both), second on all three DNSMOS columns (behind ElevenLabs-v3 in denoise mode), second on Reverb WER and fourth on Noisy and Unclear WER. Without a denoise mode it reaches DNSMOS near ElevenLabs-v3·Denoise (3.962 vs. 3.981 Noisy) at much higher SIM (76.14 vs. 46.07). Denoise modes in the commercial APIs raise DNSMOS but cut SIM (MiniMax Reverb 61.56 to 56.53).
- Instruction following under zero-shot cloning (440 cases, Gemini-2.5-Pro judge): highest overall in zh (78.94 vs. CosyVoice3 75.91, IndexTTS2 54.39) and en (80.45 vs. 64.09, 59.39), with the clearest margin on natural-language multi-attribute instructions. It trails CosyVoice3 on zh structured-attribute instructions (65.33 vs. 68.67) and on speaker similarity (73.27 vs. 75.60 zh, 66.66 vs. 68.45 en). The judge agrees with humans on 70.0% of single-attribute cases and 56.7% of complex-instruction criteria.
Speaker adaptation (§4.5, Figure 5). Adapted models of four anonymized speakers have lower CER/WER than adapted CosyVoice3 models for every speaker (for example test-zh 1.25 to 1.07, test-en 2.04 to 1.51).
Human evaluation (§4.6, Tables 11-12). In a 20-dialect test (50 prompts per dialect, three native-speaker annotators), mean severity scores are 3.639 for dialect authenticity, 3.935 for pronunciation and 3.68 for prosodic naturalness on a 1-4 scale. Arena win rates against an unnamed previous-generation baseline are 44.8% vs. 30.9% (instruction following) and 55.6% vs. 42.9% (prosodic naturalness). The authors also report that the system, listed as Qwen-Audio-3.0-TTS-Plus, ranks first by point estimate on the Artificial Analysis Text-to-Speech Arena snapshot of July 16, 2026 (Elo 1,237, rank range 1-2, confidence interval overlapping Simba 3.2) (§1, Figure 1).
Novelty Assessment
The contribution is primarily engineering integration and training recipe. Each component comes from prior work, much of it by the same group: the supervised FSQ tokenizer (CosyVoice3), hidden-state LM-to-FM conditioning with joint loss (JoyVoice), DiffRO (CosyVoice3), GRPO (DeepSeekMath), and FlowTTS-GRPO (the authors’ own prior paper). What is new is the 12.5 Hz tokenizer with a larger codebook, the ordering of the five stages, the LM reward design (pre-FM token-domain rewards, DiffRO restricted to non-negative advantages, dialect reward curriculum), and folding prompt enhancement into FM training. None of these changes is ablated stage by stage: the paper reports no results for the stage 1 Cascade, stage 2 joint model, or either RL stage in isolation. The individual contribution of each stage is therefore not measured. The tokenizer study (Tables 1-2) is the only controlled ablation.
The evaluation is broad, and the new Qwen-Audio-TTS-Eval sets (text normalization, real degraded prompts, long-form, instruction following) cover deployment failures that SEED-TTS-Eval misses. The paper does not say whether they will be released. Comparisons are mixed. On standard benchmarks the system is usually competitive rather than best in content error, and its clearest wins are on ERes2Net similarity, DNSMOS, cross-lingual cloning and the authors’ own benchmarks.
Field Significance
Moderate — The report shows that a hybrid LM plus flow-matching pipeline can run at a 12.5 Hz token rate without losing downstream accuracy if the codebook grows, and that stacked post-training (LM GRPO, FM robustness training, FM GRPO) can deliver multilingual, instruction-controlled, long-form and degraded-prompt cloning in a single model. It also adds evidence that the speaker-similarity model choice (WavLM vs. ERes2Net) changes system rankings on SEED-TTS-Eval. Its value lies mainly in this integrated recipe and evaluation breadth rather than in a new modeling idea.
Claims
- supports: In supervised speech tokenizers for LM-based TTS, a larger codebook can offset the content and speaker losses caused by halving the token frame rate.
Evidence: Moving from 25 Hz to 12.5 Hz at 6,561 codes raises SEED-TTS-Eval test-zh CER from 1.45 to 2.59 and lowers SIM from 80.6 to 72.44; the 12.5 Hz tokenizer with 59,049 codes reaches 1.23 CER and 83.09 SIM, better than the 25 Hz baseline on all three test sets. (§4.1, Tables 1-2)
- refines: Accuracy and speaker similarity respond differently to tokenizer capacity, so token-vocabulary choices involve a content-versus-timbre trade-off rather than a single optimum.
Evidence: Among 12.5 Hz tokenizers, the 59,049-code variant gives the lowest CER/WER on all three SEED-TTS-Eval sets, while the 19,683-code variant gives marginally higher SIM on all three (for example 83.25 vs. 83.09 on test-zh). (§4.1, Table 2)
- complicates: System rankings in zero-shot TTS speaker similarity depend on which speaker-verification model computes the score.
Evidence: On SEED-TTS-Eval the proposed system has the highest ERes2Net similarity on all three sets but ranks sixth on test-zh under WavLM (0.792 vs. LongCat-AudioDiT 0.818); the authors note that WavLM and ERes2Net often order systems differently. (§4.2, Table 3)
- complicates: Inference-time denoising of degraded voice-cloning prompts raises perceptual quality scores at the cost of speaker similarity, whereas training the acoustic model on degraded prompts can retain both.
Evidence: MiniMax-Speech-2.8-HD’s denoise mode raises Reverb DNSMOS from 3.065 to 3.343 but lowers SIM from 61.56 to 56.53; the proposed system, trained with prompt-degradation augmentation in the FM stage, reaches 3.925 DNSMOS with 74.12 SIM on the same set without a denoise mode. (§2.2.4, §4.4.3, Table 9)
- complicates: LLM judges used to score TTS instruction following agree only moderately with human listeners, which limits how much small differences in judged scores can be trusted.
Evidence: On a calibration subset, Gemini-2.5-Pro agrees with human judgments on 70.0% of single-attribute cases (64.9% recall, conservative by McNemar’s test, p = 0.007) and on 56.7% of complex-instruction criteria. (§4.4.4)
Limitations and Open Questions
Warning
Model size, training-data volume and sources, and code or weight availability are not reported, and the five training stages are never ablated individually. The per-stage effects of joint LM-FM training, LM RL, robustness training and FM RL therefore cannot be checked from the paper.
- Text-normalization and instruction-following scores come from a Gemini-2.5-Pro judge with moderate human agreement; only Tables 11 and 12 use human raters.
- The arena comparison (Table 12) is against an unnamed “previous-generation baseline”, and the dialect evaluation (Table 11) has no baseline at all.
- Low-latency streaming is claimed from the design (12.5 Hz tokens, chunk-based FM, causal vocoder), but no latency or real-time-factor numbers are reported.
- The external leaderboard rank is a single snapshot with an overlapping confidence interval, reported by the authors.
- Content error on SEED-TTS-Eval and long-form English is not the lowest among baselines; the authors attribute this to a deliberate naturalness trade-off but do not report naturalness metrics that would show it.
- Qwen-Audio-TTS-Eval is in-house, and the paper does not state whether it will be released.
Wiki Connections
- Flow Matching — The acoustic renderer is a chunk-based DiT flow-matching model conditioned on LM hidden states and post-trained with SDE-based GRPO.
- Zero-Shot TTS — Evaluated on prompt-based cloning on SEED-TTS-Eval and CV3-Eval, including cross-lingual and degraded-prompt cloning.
- Multilingual TTS — One model trained across 16 languages and 20 Chinese dialect regions and evaluated on all 16 languages plus 12 cross-lingual directions.
- Instruction-Conditioned TTS — Accepts free-style natural-language instructions and inline tags, evaluated with an LLM judge and a human arena test.
- RLHF Speech — Applies GRPO with DiffRO to the LM and FlowTTS-GRPO to the flow-matching model, with multi-term rewards for content, duration, prosody, similarity and quality.
- Subjective Evaluation — Reports a three-annotator native-speaker test on 20 dialects and an arena win-rate test for instruction following and naturalness.
- CosyVoice 3 — Direct predecessor: the tokenizer design, DiffRO and stage-1 recipe come from it, and it is the main baseline throughout.
- CosyVoice 2 — Source of the bi-streaming LM and chunk-based FM pretraining recipe.
- JoyVoice — Source of the method of conditioning the FM on continuous LM hidden states with joint optimization, here placed after independent pretraining.
- Qwen3-TTS — Compared as Qwen3-TTS-12Hz-1.7B-Base; it has lower content error on several SEED-TTS-Eval and CV3-Eval columns.
- VoxCPM2 — Continuous-autoregressive baseline with the lowest long-form content error in Table 8.
- LongCat-AudioDiT — Non-autoregressive baseline with the highest WavLM similarity on SEED-TTS-Eval test-zh and test-en.
- IndexTTS2 — Instruction-following baseline in Table 10.
- Seed-TTS — Source of the SEED-TTS-Eval benchmark and an autoregressive baseline in Table 3.