arXiv · 2026 · Preprint
Bowen Li et al. (Amap, Alibaba Group / The Chinese University of Hong Kong, Shenzhen) · → Paper · Demo: ? · Code: ✓
Assembles a compact zero-shot TTS system entirely from existing open components (Qwen3-0.6B over CosyVoice 3 tokens, a CFM decoder, HiFi-GAN) and attributes its Seed-TTS Eval results to a 200K-hour corpus curated with an open-tool data pipeline plus a two-path speaker/style conditioner (CAMPPlus embedding and a Q-Former over frozen w2v-BERT 2.0 features).
Problem
Recent zero-shot TTS systems reach their quality with hundreds of thousands to millions of hours of often proprietary data, multi-codebook tokenizers, multi-stage training, and separate specialised models for emotion, paralinguistic, or dialect control. The authors argue this raises the barrier for resource-constrained teams and that most commercial deployments need a simpler system that still meets practical quality requirements. They also identify a conditioning trade-off in prior work: audio-token continuation (prompt continuation) copies fine acoustic detail but degrades with noisy or short prompts and grows costly with long ones, while a fixed speaker embedding is robust but discards timbre detail and dynamic style such as speaking rate and prosodic contour.
Method
Data pipeline (§2). Raw web audio passes through three stages built from public tools: (1) quality assessment and enhancement (format standardisation, speech activity and speaker-change detection via pyannote-style models, DNSMOS, a speech/non-speech classifier, SNR estimation, and denoising of segments below thresholds such as predicted MOS ≤ 3.5); (2) label annotation (multi-system ASR with Paraformer, FireRedASR, Whisper, and internal models for cross-system consistency checks, overlapping-speech detection, forced alignment, prosody annotation built on Qwen3-Force-Alignment, speaker tagging with 3D-Speaker-Toolkit, and spectral-rolloff bandwidth checks); (3) quality filtering (truncation detection, a synthetic-speech detector, and a joint filter over all signals). About 200K hours of Chinese and English speech survive for pre-training. Rejected data is kept with its tags rather than discarded.
Model (§3). The system has four parts:
- Speech tokenizer: the single-codebook FSQ tokenizer of CosyVoice 3 (25 Hz, 6,561-entry codebook, multi-task supervised on ASR, LID, SER, AED, and speaker analysis), reused without modification.
- Text-to-token LM: Qwen3-0.6B autoregressively predicts these discrete speech tokens from an input sequence of a speaker embedding, 32 style tokens, language and emotion control tags, and text tokens. The speaker embedding comes from a frozen CAMPPlus (CAM++) speaker verification encoder; the 32 style tokens come from a Q-Former “Semantic Content Adapter” whose learnable queries cross-attend (with a Conformer block and linear projection) to features from a frozen w2v-BERT 2.0 encoder applied to the reference audio. The reference is therefore compressed to a fixed-length condition rather than prepended as a token prompt.
- Cross-sample paired training: the reference for each training target is a different utterance of the same speaker, intended to force the conditioner to carry only content-independent speaker and style attributes.
- Decoder and vocoder: a ~300M-parameter conditional flow matching decoder with a DiT backbone maps tokens to mel spectrograms in 10 steps, conditioned on the reference mel and CAMPPlus embedding; a HiFi-GAN vocoder produces the waveform. The paper does not say whether the decoder and vocoder were trained by the authors or reused.
Post-training (§3.3.3–3.3.5). Separate supervised fine-tuning stages add explicit emotion control (~2,200 hours; 7 primary plus 4 extended categories via an emotion tag), paralinguistic events (~200 hours; laughter, breathing, crying, coughing, and a “LAUGH_SPAN” mode where laughter overlays an entire utterance, triggered implicitly from context or explicitly by onomatopoeia in the text), and 14 Chinese dialects (~16,000 hours from dialectal ASR corpora). For dialects, the pretrained model synthesises three Mandarin utterances per dialect speaker to build dialect/Mandarin pairs, and fine-tuning draws the prompt with equal probability from a Mandarin or dialect utterance of the same speaker so that a Mandarin prompt can drive dialect output.
Key Results
Zero-shot (Seed-TTS Eval, Table 1). PilotTTS reports CER 0.87% / SIM 0.862 on test-zh and WER 1.50% / SIM 0.815 on test-en.
- On test-en WER, 1.50% is the lowest in the table, ahead of Qwen3-TTS-25Hz-0.6B (1.64%), MiniMax-Speech (1.65%), F5-TTS (1.83%), and VoxCPM-0.5B (1.85%).
- On test-zh CER it is second: MiniMax-Speech is lower (0.83%), and VoxCPM-0.5B (0.93%), Seed-TTS (1.12%), and FireRedTTS-2 (1.14%) follow.
- Its SIM is the highest among the six baselines that report SIM, by +0.066 over Seed-TTS on test-zh (0.796) and +0.053 on test-en (0.762). Qwen3-TTS and MiniMax-Speech report no SIM, so the comparison is incomplete.
Baseline numbers appear to be taken from the respective papers. The paper describes SIM only as “cosine similarity between speaker embeddings” and does not name the verification model. Because SIM values are sensitive to that model, it is unclear whether the margin reflects a common protocol. The paper reports no naturalness MOS or CMOS for zero-shot synthesis.
Emotion control (Tables 2–3, human judgments). Success requires both preserved timbre and a recognisable target emotion.
- Averages: PilotTTS leads the averages with 88.1% on the primary categories and 85.7% on all categories. CosyVoice 3 scores 83.8% and 82.5%, and Fish-Speech S2 scores 60.0% and 60.2%.
- Where it trails: CosyVoice 3 is ahead on Sad (96.4 vs 90.5), Contempt (88.2 vs 81.2), Blue (86.4 vs 79.1), and Concern (83.6 vs 82.9), and Fish-Speech S2 is ahead on Surprise (96.4 vs 93.2). Psychology is tied with CosyVoice 3 (98.2).
- Text/table mismatch: the §4.3 prose says CosyVoice 3 leads the all-category average “81.4% vs. 80.2%”, but the table image gives 82.5 vs 85.7. Both averages in the table recompute correctly from the per-category cells, so the prose figures appear to be stale.
- Speaker similarity under emotion control: PilotTTS has the highest SIM both without (0.8101) and with (0.7329) emotion control. The text says it has the smallest drop “among all systems”, but Fish-Speech S2’s SIM is essentially unchanged (0.5727 to 0.5731) at a much lower level. PilotTTS’s drop (0.077) is smaller than that of CosyVoice 3 (0.102), VoxCPM (0.162), and IndexTTS (0.345).
Paralinguistics (Table 4).
- Overall: PilotTTS reports 85.1% across laughter, coughing, and breathing, against 80.4% for CosyVoice 3 and 64.3% for Fish-Speech S2. In the table image, Overall does not equal the unweighted mean of the three category columns for any system (for PilotTTS the mean is about 81.0), and the aggregation is not described.
- Per category: it leads on LAUGH (97.6 vs 83.3 and 54.8), ties Fish-Speech S2 on COUGH (64.3), and is lowest of the three on BREATH (81.0, against 95.2 for CosyVoice 3 and 83.3 for Fish-Speech S2).
- PilotTTS-only modes: LAUGH_SPAN (94.6%) and CRY (61.9%) have no baseline.
Dialects (Table 5). Dialect accuracy is 91.80% for same-dialect, 86.46% for Mandarin-to-dialect, and 85.38% for cross-dialect prompts. The authors deliberately include no baseline because they consider the judgments too subjective.
Conditioning ablation (Table 6, 60K-hour subset, 200K steps).
- Removing both conditioning paths: errors rise on test-zh (1.130 to 1.412 CER), test-en (1.940 to 2.710 WER), and test-hc (7.830 to 10.623 CER).
- Removing only the CAMPPlus embedding: CER/WER drops slightly on test-zh (1.022) and test-en (1.860) but rises on test-hc (8.866). SIM falls on all three sets (largest on test-hc, 0.8470 to 0.8355).
- No conditioner-only ablation: there is no setting that removes the Q-Former tokens while keeping the speaker embedding, so the effect the paper attributes to the condition tokens is measured jointly with the embedding. The text-only “w/o both” variant still reaches SIM 0.8617 on test-zh. This suggests that much of the measured timbre fidelity comes from the decoder’s reference-mel and speaker-embedding conditioning rather than from the LM conditioner.
Novelty Assessment
By the authors’ own description, every component is an established open-source module and the contribution is integration plus data engineering, not architectural novelty. The most distinctive design choice is replacing prompt-token continuation in a CosyVoice-style LM with a fixed-length Q-Former condition over SSL features plus a global speaker embedding, trained with cross-sample references. Q-Former conditioning and same-speaker cross-utterance references are both known techniques, and the ablation does not isolate the Q-Former’s own contribution. The data pipeline is described as a recipe of public tools but key thresholds beyond the DNSMOS cut-off, the synthetic-speech and truncation detectors, and the source corpora are not specified in the paper itself. The release of code, weights, and pipeline recipe is the main practical contribution.
Field Significance
Moderate. The paper is an engineering data point showing that a ~0.6B LM over an off-the-shelf single-codebook supervised tokenizer, trained on ~200K hours of carefully filtered public data, can report Seed-TTS Eval intelligibility and similarity numbers in the same range as systems trained on larger corpora. That supports the view that data curation, not scale or architectural complexity alone, drives zero-shot TTS benchmark scores. Its evidence is limited to objective metrics with an unspecified SIM model for the main benchmark, plus in-house human tests for controllability.
Claims
- supports: Curated training data of moderate scale can bring a compact autoregressive zero-shot TTS model to intelligibility and speaker-similarity levels reported by systems trained on substantially larger corpora.
Evidence: Trained on ~200K hours filtered by an open-tool pipeline, the 0.6B-backbone system reports test-en WER 1.50% (lowest in the table) and test-zh CER 0.87% (second to MiniMax-Speech at 0.83%), with SIM above every baseline that reports it. Baseline values come from their papers and the SIM extractor is unspecified. (§2.3, §4.2, Table 1)
- refines: Compressing a reference prompt into a fixed set of learned condition tokens, rather than prepending it as token continuation, can preserve content accuracy and speaker similarity in autoregressive TTS. The gains in content accuracy are largest on acoustically hard inputs.
Evidence: Removing the CAMPPlus embedding and the Q-Former tokens together raises test-hc CER from 7.83% to 10.62% and test-en WER from 1.94% to 2.71%. Removing only the speaker embedding lowers SIM on all three sets but slightly lowers test-zh/test-en errors. No setting ablates the Q-Former alone. (§4.6, Table 6)
- complicates: In a cascaded LM-plus-decoder TTS system, speaker similarity can be largely determined by the acoustic decoder’s own reference conditioning, so ablations of LM-side speaker conditioning may show only small similarity changes.
Evidence: The variant with both LM conditioning paths removed (text only) still reaches SIM 0.8617 on test-zh against 0.8626 for the full model, while its content errors rise substantially. The CFM decoder independently receives the reference mel and speaker embedding. (§3.4, §4.6, Table 6)
- complicates: Explicit emotion control trades off against timbre preservation in zero-shot TTS, and the size of the trade-off varies widely across systems.
Evidence: Speaker similarity falls from 0.8101 to 0.7329 for PilotTTS, 0.7963 to 0.6940 for CosyVoice 3, and 0.7680 to 0.4233 for IndexTTS when emotion control is applied; Fish-Speech S2 is unchanged but at a much lower level (0.573). (§4.3, Table 3)
- supports: Pairing pseudo-parallel synthetic prompt/target data with mixed-language prompt sampling enables dialect speech generation from prompts in a different variety when dialect data is scarce.
Evidence: Fine-tuning on 16,000 hours of dialectal ASR data with Mandarin or dialect prompts drawn at equal probability yields 86.46% dialect accuracy from Mandarin prompts and 85.38% cross-dialect, against 91.80% same-dialect. These come from human judgment with no baseline. (§3.3.5, §4.5, Table 5)
Limitations and Open Questions
Warning
The headline zero-shot comparison rests only on objective CER/WER/SIM: baseline numbers appear to be copied from other papers, the speaker-verification model behind SIM is not named, and two strong baselines report no SIM. No naturalness MOS or CMOS is reported for zero-shot synthesis.
- Text/table mismatches: the §4.3 prose gives an all-category emotion average favouring CosyVoice 3 (81.4 vs 80.2) that contradicts Table 2 (82.5 vs 85.7). The Table 4 “Overall” paralinguistic score is not the unweighted mean of its category columns, and the aggregation is not explained.
- Controllability tests: all three tests (emotion, paralinguistic, dialect) use in-house test sets with human judges. The number of raters, rater agreement, and confidence intervals are not reported, and dialect synthesis has no baseline.
- Ablation design: there is no Q-Former-only ablation, and the ablation is on a 60K-hour subset at a fixed step budget. The authors note that the content-error advantage of removing the speaker embedding narrows with longer training, but do not show it.
- Missing reproducibility details: the pipeline is described as built from public tools, but the raw source corpora and several components (the synthetic-speech and truncation detectors, the enhancement model, the internal ASR systems) are not identified in the paper.
- Limits named by the authors: there is no explicit style encoder, so fine-grained expressiveness relies on implicit Q-Former capture. The single-codebook FSQ tokenizer limits extension to singing and background music, and the mel-plus-vocoder cascade adds reconstruction loss.
- Not evaluated: inference latency, streaming, and robustness to long inputs are not reported.
Wiki Connections
- Autoregressive Codec TTS — Uses a Qwen3-0.6B LM to predict single-codebook FSQ speech tokens, conditioned through a fixed-length Q-Former condition instead of prompt-token continuation.
- Zero-Shot TTS — Evaluates prompt-based cloning on Seed-TTS Eval test-zh/test-en/test-hc and reports high speaker similarity with ~200K hours of training data.
- Self-Supervised Speech — The style conditioner queries features from a frozen w2v-BERT 2.0 encoder, and ablating the conditioning paths changes content accuracy, most strongly on hard cases.
- Emotion Synthesis — Adds 11-category explicit emotion control via post-training and measures joint emotion-plus-timbre success and the speaker-similarity cost of emotion control against four systems.
- Multilingual TTS — Trains on Chinese and English and evaluates both, then extends to 14 Chinese dialects with Mandarin-to-dialect and cross-dialect prompting.
- Subjective Evaluation — Human listeners judge emotion, paralinguistic, and dialect success rates under joint timbre-and-attribute criteria, without reported rater counts or agreement.
- CosyVoice 3 — Supplies the unmodified FSQ speech tokenizer and serves as the strongest emotion and paralinguistic baseline.
- Seed-TTS — Provides the Seed-TTS Eval benchmark and the second-best SIM baseline in Table 1.
- MiniMax-Speech — Baseline with the only lower test-zh CER (0.83%), with no SIM reported.
- Qwen3-TTS — Similar-scale 0.6B baseline that trails PilotTTS on test-en WER (1.64%) and reports no SIM.
- VoxCPM — Compact zero-shot baseline (0.93% test-zh CER) and an emotion-control comparison system.
- Fish Audio S2 — Emotion and paralinguistic baseline that leads on Surprise and ties PilotTTS on coughing.
- IndexTTS — Emotion-control baseline whose speaker similarity drops most under emotion conditioning.
- F5-TTS — Flow-matching zero-shot baseline in Table 1.