arXiv · 2024 · Preprint
Zhihao Du et al. (Alibaba Group) · → Paper · Demo: ✓ · Code: ✓
CosyVoice introduces supervised semantic speech (S3) tokens — derived by inserting a vector quantizer into a supervised ASR encoder (SenseVoice-Large) — as the discrete speech representation for an LLM + OT-CFM pipeline, demonstrating that supervision-derived tokens significantly improve content consistency and speaker similarity over unsupervised tokens such as HuBERT or EnCodec.
Problem
LLM-based TTS systems discretize speech into token sequences, but existing tokens (HuBERT, EnCodec, etc.) are learned unsupervised and lack explicit semantic alignment with text. This misalignment leads to higher WER and reduced content consistency in zero-shot synthesis. Additionally, most flow-matching TTS systems require explicit phoneme duration predictors and forced aligners, adding pipeline complexity.
Method
CosyVoice consists of four components: (1) a supervised S3 tokenizer, (2) a text encoder, (3) an autoregressive LLM, and (4) a conditional OT-CFM model with HiFi-GAN vocoder.
S3 Tokenizer: A fine-tuned SenseVoice-Large ASR encoder is split into two parts with a VQ layer inserted between them. The first part encodes the mel spectrogram into context-aware representations; VQ discretizes these into semantic tokens from a single codebook of 4096 codes; the second encoder part continues processing for the ASR decoder. The VQ codebook is updated via exponential moving average. This supervised training ensures the tokens preserve strong semantic/textual alignment — the VQ-inserted model achieves WER of 3.18%/7.56% on LibriSpeech test-clean/other, vs. 2.89%/6.57% for the vanilla ASR model (only a minor degradation).
LLM: An autoregressive transformer models the sequence [SOS, speaker-embedding, text-encodings, speech-turn-token, speech-tokens, EOS]. Text and speech tokens occupy different semantic levels, so a text encoder aligns them into a shared space before the LLM. Speaker embedding (x-vector) is prepended to separate timbre modeling from content.
OT-CFM model: Generates mel spectrograms from Gaussian noise conditioned on speaker embedding, semantic tokens, and masked mel spectrogram (prompt frames set to zero from a random start point to end). A cosine timestep scheduler concentrates more generation steps at the beginning. CFG is applied by randomly dropping all conditions (p=0.2) during training; guidance strength β=0.7 at inference.
Zero-shot ICL: For same-language synthesis, the prompt speech tokens are treated as pre-generated and prepended to the LLM context. For cross-lingual cloning, the prompt text and tokens are omitted to prevent source-language prosodic leakage; only speaker embedding and prompt mel are passed to the flow model.
CosyVoice-instruct: Built on CosyVoice-base with instruction fine-tuning on 556h of curated data covering speaker identity descriptions, speaking style (emotion, gender, rate, pitch), and fine-grained paralinguistics (laughter, breath, emphasis). The speaker embedding is not used in this variant; instructions replace it.
Key Results
Token comparison (Table 7, LibriTTS test-clean):
- VALL-E (Encodec tokens): WER 18.7%, SS 53.19
- HuBERT-based (Exp-1): WER 7.41%, SS 67.85
- S3 tokens, BPE text (Exp-3): WER 3.93%, SS 67.85 — best content consistency with same speaker similarity
- S3 tokens, large-scale multilingual data (Exp-4-Large): WER 3.17%, SS 69.49 — human parity (original: 3.01%, SS 69.67)
CosyVoice vs. ChatTTS (Tables 8, 9):
- EN: CosyVoice WER 2.89% vs. ChatTTS 8.32%; SS 74.30 (no ChatTTS SS reported)
- ZH: CosyVoice CER 3.82% vs. ChatTTS 3.87%, but ChatTTS has far more ins/del errors (111 vs. 24.4), indicating speaker leakage
- With 5x ASR re-ranking: EN WER 1.51%, ZH CER 1.84%
Emotion control (Table 10, CosyVoice-instruct):
- Happy: 1.00→1.00 (base→instruct), Sad: 0.45→0.98, Angry: 0.59→0.83, Surprised: 0.26→0.64
ASR data generation: ASR model trained on CosyVoice-synthesized LibriSpeech achieves WER 3.0%/6.59% (test-clean/other) vs. 2.79%/5.97% on real data; combining real + synthetic on MLS text yields 2.04%/4.53%, a strong improvement.
Novelty Assessment
The primary novelty is the supervised S3 tokenizer concept — inserting VQ into an ASR encoder to obtain tokens with strong text-semantic alignment. This is the first paper to demonstrate this idea in a TTS context and show it directly translates to lower WER and higher SS. The OT-CFM component and LLM component draw on Voicebox and VALL-E respectively; the integration is straightforward but well-executed. The instruction fine-tuning with paralinguistic controls (laughter, breath, emphasis) extends controllability beyond what contemporaneous codec LM systems offered.
The scaling finding (Exp-4-LibriTTS vs. Exp-4-Large-scale) is clean evidence that the architecture scales well with data volume.
Field Significance
Tip
High — CosyVoice demonstrates that replacing unsupervised discrete speech tokens with supervision-derived tokens directly closes the content consistency gap in LLM-based TTS, an insight that has influenced the design of numerous subsequent codec-LM systems. It provides a practical blueprint for combining an AR language model with a flow-matching acoustic model in a single end-to-end pipeline, and is widely used as a competitive baseline across multilingual and zero-shot TTS benchmarks.
Claims
- Inserting a vector quantizer into a supervised ASR encoder yields discrete speech tokens that preserve significantly stronger text-semantic alignment than unsupervised alternatives such as HuBERT or EnCodec tokens. (§2.1, §5.1, Table 7)
- In autoregressive codec-LM TTS, both the text tokenizer and the speech tokenizer independently contribute to content consistency, while speaker similarity is primarily controlled by the speaker embedding and acoustic model conditioning. (§5.2, Table 7)
- ASR re-ranking is an effective post-hoc method for improving content consistency in autoregressive TTS without any model retraining, at the cost of increased inference-time compute. (§5.3, Tables 8, 9)
- Instruction fine-tuning on a modest amount of labelled data (556h) enables a TTS system to control fine-grained paralinguistic features — including laughter, breath, and word emphasis — with substantially improved accuracy over the base model. (§2.4, §5.4, Table 10)
- High-quality TTS-synthesized speech can serve as effective training data augmentation for ASR, with text diversity of the synthesis prompts contributing more to downstream ASR gains than the raw duration of the synthetic corpus. (§5.5, Table 11)
Limitations and Open Questions
- Only a single VQ codebook (4096 codes) is used; multi-level RVQ and its effect on quality vs. compression is left for future work.
- The choice of VQ insertion layer (after layer 6 of 12) is not ablated — optimal placement is unresolved.
- Cross-lingual cloning omits prompt prosody to prevent leakage, which may reduce naturalness in target language.
- Instruction fine-tuning data amounts (556h) are modest; broader paralinguistic coverage remains open.
- No subjective (MOS) evaluation in the main paper; relies entirely on objective WER/CER/SS metrics.
Wiki Connections
CosyVoice is the predecessor to 2412.10117 (CosyVoice 2), which extends it with finite-scalar quantization, LLM backbone, and chunk-aware causal flow matching for streaming. It builds on the codec LM paradigm established by 2301.02111 (VALL-E), replacing unsupervised Encodec tokens with supervised S3 tokens. It is cited as a baseline by 2410.06885 (F5-TTS) and 2406.02430 (Seed-TTS) as a representative of the LLM+flow-matching hybrid architecture. The S3 tokenizer connects to self-supervised-speech (by contrasting with it) and neural-codec. Instruction fine-tuning aspects connect to instruction-conditioned-tts.
Citing papers in this corpus: 2506.21619 (IndexTTS2) extends the AR+FM cascade with emotion disentanglement and duration control; 2510.12210 (DiSTAR) uses CosyVoice as a comparison baseline on SeedTTS; MELLE compares against CosyVoice’s tokenizer approach; Controllable TTS Survey surveys CosyVoice as a representative supervised-semantic-token system; 2603.08823 (Fish Audio S2) benchmarks against CosyVoice in multilingual evaluation; 2512.04720 (M3-TTS) and 2512.13251 (DisCo-Speech) compare CosyVoice as a baseline; 2603.29339 (LongCat-AudioDiT) compares SPK-SIM on Seed benchmark. Further citing papers: 2508.04141 (Parallel GPT) benchmarks its MOS against CosyVoice; 2511.12347 (VoiceCraft-X) compares against CosyVoice on Seed-TTS test-en; 2601.03888 (IndexTTS 2.5) evaluates its multilingual system relative to CosyVoice baseline; 2603.18090 (MOSS-TTS) positions its AR codec TTS relative to CosyVoice family; 2603.26364 (LLaDA-TTS) replaces the CosyVoice AR stage with masked discrete diffusion. New citing papers (integrated 2026-05-30): interspeech-2025-0575 (VoiceMark) uses CosyVoice as one of three zero-shot VC attack models to test watermark survival; interspeech-2025-0704 (DiffRO) builds directly on CosyVoice 2.0 as the base codec LM for differentiable reward optimization. Further citing papers (integrated 2026-06-02): 2508.02849 (SecoustiCodec) proposes frame-level contrastive learning as an alternative to CosyVoice’s supervised semantic tokenizer approach; 2508.07426 (Scalable Controllable Accented TTS) fine-tunes XTTS-v2 as a CosyVoice-comparable zero-shot TTS backbone for accent control; 2508.07711 (FreeGAN) benchmarks against CosyVoice in vocoder quality comparison. New citing papers (integrated 2026-06-03): interspeech-2025-2447 (SSD) cites CosyVoice in context of AR TTS systems for speculative decoding. Integration pass 7 (2026-06-05): 2508.17623 (EMO-Reasoning) uses CosyVoice to synthesize evaluation stimulus material for the emotional coherence benchmark.
2407.08551 — Autoregressive Speech Synthesis without Vector Quantization 2501.06282 — MinMo: A Multimodal Large Language Model for Seamless Voice Interaction 2502.11946 — Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction 2502.17239 — Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction 2507.16632 — Step-Audio 2 Technical Report