arXiv · 2025 · Preprint
Du et al. (Alibaba) · → Paper · Demo: ✓ · Code: ?
CosyVoice 3 scales zero-shot multilingual TTS to 1M hours of training data and 1.5B parameters, introducing a supervised multi-task speech tokenizer and a differentiable reward optimization method that together yield strong content consistency and speaker similarity across 9 languages and 18 Chinese dialects.
Problem
CosyVoice 2, despite achieving competitive quality and low-latency streaming for Chinese and English, faced significant gaps in language coverage, domain diversity, training data volume, text format handling, and post-training techniques. Existing reinforcement learning approaches for TTS relied on full audio synthesis for reward computation, making them computationally prohibitive and unable to discriminate positive from negative outputs due to high output similarity. Standard tokenizers trained on ASR objectives alone also failed to adequately capture paralinguistic information such as emotion and pronunciation style, limiting prosody naturalness.
Method
CosyVoice 3 uses the same two-stage hybrid architecture as its predecessors: an autoregressive language model generates discrete speech tokens from text, and a conditional flow matching (CFM) model with a Diffusion Transformer (DiT) backbone renders those tokens into Mel spectrograms, followed by a vocoder. The LM grows from 0.5B to 1.5B parameters, and the CFM DiT grows from 100M to 300M.
The central architectural contribution is a new speech tokenizer built on MinMo, a large-scale multimodal speech understanding model trained on over 1.4M hours. A Finite Scalar Quantization (FSQ) module is inserted into MinMo’s voice encoder, and the entire tokenizer is then fine-tuned via supervised multi-task learning on 530K hours covering ASR, language identification (LID), speech emotion recognition (SER), audio event detection (AED), and speaker analysis. Operating at 25 Hz, this tokenizer embeds richer paralinguistic content than the SenseVoice-based tokenizer in CosyVoice 2.
The post-training method, Differentiable Reward Optimization (DiffRO), bypasses full audio synthesis by operating directly on discrete speech tokens. A Token2Text model is trained to predict text from token sequences; Gumbel-Softmax sampling from the LM’s output distribution allows gradients to flow back through the token prediction step without a discrete RL training loop. A KL divergence penalty computed over per-timestep token logits prevents the model from drifting too far from the reference policy. Multi-task reward extensions incorporate SER, MOS prediction, and audio event detection as additional reward signals, enabling fine-grained control of attributes such as emotion.
Three additional capabilities round out the system: pronunciation inpainting (mixing word and phoneme tokens in training data to correct polyphonic characters), self-training for text normalization (using Qwen-Max to generate paired raw-text and audio examples), and expanded instruction-following covering over 100 speaking styles. Continual pretraining transfers these capabilities to speaker fine-tuned (SFT) models while mitigating catastrophic forgetting.
Key Results
On SEED-TTS-Eval, CosyVoice 3-1.5B with DiffRO reaches 0.71% CER (test-zh) and 1.45% WER (test-en), representing 44% and 51% relative improvements over CosyVoice 2 on the respective sets. Speaker similarity (WavLM) reaches 0.775–0.781 on test-zh and 0.695–0.720 on test-en. The only model that exceeds CosyVoice 3’s speaker similarity in this evaluation is Seed-TTS, which the authors attribute to larger speaker diversity in Seed-TTS pretraining data. DiffRO alone contributes 12–35% relative WER improvements across conditions, with gains exceeding 50% in half of cross-lingual and low-resource scenarios.
On the newly introduced CV3-Eval multilingual benchmark, CosyVoice 3 is the only system able to cover all 9 evaluated languages. Competing open-source models (F5-TTS, SparkTTS, GPT-SoVits) support only Chinese and English. DiffRO adds a further 20–50% relative WER reduction across most language pairs, particularly pronounced for Korean (68.7% relative improvement for the 0.5B model). A 200-sample subjective evaluation shows CosyVoice 3-1.5B MOS scores meeting or exceeding human speech in English, and above 4.45 for Chinese (Figure 4).
Emotion cloning results are more mixed: text-related emotion accuracy is strong (0.84–0.98 for happy/angry/sad with DiffRO-EMO), but text-unrelated emotion accuracy drops substantially for sad and angry categories, revealing that current models infer emotional tone primarily from text semantics rather than from the reference prompt.
MOS comparisons between CosyVoice 3 and baselines (Table 8) use a 200-sample internal test set rather than the SEED-TTS-Eval set, so direct numerical comparison to prior work MOS values is not straightforward.
Novelty Assessment
The key genuine contributions are the DiffRO post-training method and the MinMo-based supervised multi-task tokenizer. DiffRO is novel in that it enables end-to-end differentiable reward optimization over discrete token outputs without a full audio synthesis pass, making RL-style post-training tractable at scale for two-stage codec TTS systems. The tokenizer design, inserting FSQ into a large pre-trained speech understanding model and jointly training on five tasks, is a substantive extension over the simpler ASR-encoder approach in CosyVoice 2.
The broader system integration (scaling to 1M hours, adding 9 languages, increasing model size, pronunciation inpainting, text normalization self-training) is engineering-integration work rather than architectural novelty. The CV3-Eval benchmark addresses a real gap in multilingual and in-the-wild evaluation, which existing benchmarks based on audiobook-quality reference speech did not cover. The scaling investigation itself is valuable as empirical evidence but does not depart from established practice.
Field Significance
Tip
High: CosyVoice 3 demonstrates that RL post-training can be made tractable and broadly effective for two-stage discrete-token TTS systems by operating on token logits rather than synthesized audio. The DiffRO method is general enough to apply to other LLM-based TTS pipelines, which distinguishes it from purely system-specific engineering. The paper also provides one of the more thorough multilingual in-the-wild evaluations through CV3-Eval, covering 9 languages and noisy reference conditions absent from standard audiobook benchmarks.
Claims
- Differentiable reward optimization over discrete speech tokens, rather than synthesized audio, enables scalable post-training for LLM-based TTS without the computational overhead of full audio generation. (§2.2)
- Supervised multi-task training of a speech tokenizer on tasks including emotion recognition, language identification, and speaker analysis captures richer paralinguistic content than ASR-only objectives, with measurable downstream gains in content consistency and speaker similarity. (§2.1, §5.3)
- Current zero-shot TTS systems infer emotional tone primarily from text semantics; expression of text-unrelated emotions from a reference prompt remains a significant unsolved challenge. (§5.2.3, Table 9)
- Scaling training data and model size in two-stage codec TTS produces consistent gains up to a point of diminishing returns, with data scaling showing stronger marginal returns than model size scaling beyond 0.5B parameters on current benchmarks. (§4.2, §5.1)
- Existing TTS benchmarks based on studio-quality audiobook data underestimate the difficulty of in-the-wild voice cloning, where noisy reference audio and domain diversity expose robustness gaps not visible in standard evaluations. (§4.4)
Limitations and Open Questions
Warning
Speaker similarity lags Seed-TTS across all SEED-TTS-Eval conditions, and the authors attribute this to data volume differences rather than architectural factors. Since pretraining data is not released, the degree to which CosyVoice 3’s improvements are reproducible without Alibaba’s internal 1M-hour corpus is unclear.
The DiffRO reward mechanism exhibits a trade-off between intelligibility and speaker similarity: RL post-training slightly reduces speaker similarity while improving WER. Multi-reward DiffRO with emotion targets further degrades pronunciation, suggesting that reward balancing in multi-objective post-training for TTS is an open engineering problem. Timbre control via textual instructions is absent and noted as future work. Singing voice generation remains unsatisfactory due to insufficient training data coverage in the tokenizer and LM stages. Performance on Korean and Japanese lags other languages due to data volume limitations and language-specific encoding challenges (kana conversion, multiple pronunciations).
Wiki Connections
Concepts: autoregressive-codec-tts · flow-matching · zero-shot-tts · multilingual-tts · rlhf-speech · evaluation-metrics
Prior CosyVoice work: 2407.05407 (CosyVoice) · 2412.10117 (CosyVoice 2)
Baselines and context: 2406.02430 (Seed-TTS) · 2409.00750 (MaskGCT) · 2410.06885 (F5-TTS) · 2406.18009 (E2 TTS) · 2503.01710 (SparkTTS) · 2504.02407 (F5R-TTS)
VALL-E family (related architectures): 2301.02111 · 2406.05370 · 2406.07855