NCMMSC · 2025 · Conference
Zhou, Zhang, Chen & Qian (Shanghai Jiao Tong University) · → Paper · Demo: ? · Code: ?
A systematic study distilling a small zero-shot TTS model (Matcha-TTS) purely on synthetic speech generated by large teacher TTS models (XTTS-v2, CosyVoice2, ChatTTS), finding that synthetic-data training can outperform both the teacher and equivalent real-data training, and isolating which data properties (speaker diversity, text richness, noise, speaking style) drive that gap.
Problem
High-quality TTS depends on large-scale, high-fidelity training data, but real-world speech datasets are costly and slow to collect, raise privacy concerns, and are inherently noisy in ways that can degrade model performance. Synthetic data generated by existing large TTS models offers an alternative that sidesteps these issues, and has already been used to augment training for ASR, speaker verification, and speech enhancement, but prior work on synthetic-data TTS training has been narrow (e.g., targeting specific cross-style or accent-transfer problems) rather than systematically explaining why synthetic data helps and which of its properties matter most.
Method
The study uses a generation-training-evaluation framework: for each data property under study, a matched synthetic dataset is generated, a TTS model is trained on it, and the trained model is evaluated with a fixed metric suite. All synthetic training data is generated by XTTS-v2 (a 750M-parameter autoregressive TTS model), prompted to independently and precisely control text content and speaker identity via shuffled text-speaker pairing unseen by the generator itself, keeping dataset size matched to a 100-hour real baseline (LibriSpeech train-100, extended to up to 1,000 speakers using train-360). The student model, Matcha-TTS (a ~21M-parameter non-autoregressive flow-matching TTS architecture, augmented with WeSpeaker speaker embeddings for zero-shot capability), is trained separately on each generated variant. Four data properties are manipulated independently: speaker diversity (varying the number of unique speakers, each contributing one prompt utterance reused across the dataset); text richness (varying the text-reuse rate while holding total word count fixed); noise level and noise ratio (mixing WHAM! noise and RIRS reverberation at controlled SNR and proportion); and speaking style (using three different teacher models, XTTS-v2 for reading-style, CosyVoice2 for natural-style, and ChatTTS for oral/conversational-style, to generate style-matched training sets while holding other factors constant). Evaluation uses WER (via Whisper-large-v3 transcription), speaker similarity (cosine distance of WavLM-large speaker embeddings), and DNSMOS and UTMOS for perceptual quality, on both the in-domain LibriSpeech test-clean set and the out-of-domain Seed-TTS test set.
Key Results
A Matcha-TTS student trained purely on XTTS-v2-generated synthetic data achieves lower WER (3.493 vs. 4.639) and higher UTMOS than the much larger XTTS-v2 teacher itself on LibriSpeech test-clean, and also outperforms an identical student trained on the same scale of real LibriSpeech data (WER 3.493 vs. 4.428), though speaker similarity trails both due to the gap in model and training-data scale relative to the teacher. Performance improves consistently with more speakers and richer text, but each factor saturates at a different point: WER and UTMOS plateau around 50 speakers while speaker similarity keeps improving to roughly 500; text-richness gains plateau once the training text, even with 99% repetition, still covers all 69 English phonemes. The model is highly sensitive to noise, degrading sharply once noisy utterances exceed roughly 0.1% of the training set, and reverberation degrades performance similarly as its proportion increases. Speaking style matters independently of data cleanliness: synthetic data from the conversational-style teacher (ChatTTS) trains a worse model on WER and speaker similarity than data from the more standardized reading-style (XTTS-v2) or natural-but-non-oral (CosyVoice2) teachers, despite scoring reasonably on DNSMOS, which the authors attribute to conversational speech’s contractions and irregular pronunciations being harder training targets. On the out-of-domain Seed-TTS test set, models trained purely on synthetic data show a slight drop in speaker similarity but comparable WER, DNSMOS, and UTMOS relative to in-domain performance.
Novelty Assessment
The paper does not introduce a new TTS architecture (it uses existing Matcha-TTS as the student and existing XTTS-v2/CosyVoice2/ChatTTS as teachers/data generators); its contribution is the systematic, controlled sensitivity analysis itself, isolating speaker diversity, text richness, noise, and speaking style as independently manipulable data properties and showing each has a distinct saturation point and effect on different evaluation metrics. The central, somewhat counterintuitive finding, that a small student trained purely on synthetic data can beat both its own larger teacher and an equivalently-sized real-data-trained model, is a genuinely useful empirical result that the paper grounds in a specific, measured mechanism (synthetic data’s lower background noise, directly compared via DNSMOS between real and synthetic training sets) rather than leaving as an unexplained anomaly.
Field Significance
high — This is a rigorous, well-controlled empirical study addressing a practically important and previously under-examined question: not just whether synthetic data works for TTS training, but specifically which data properties drive that effect and where each one saturates.
The finding that a distilled student model can outperform its own teacher, driven by synthetic data’s inherent cleanliness relative to real recordings, has direct implications for how future TTS training pipelines should weigh data curation (diversity, noise control, style standardization) against simply collecting more real-world data, and the systematic factor-by-factor methodology is a template other groups could apply to their own architectures and languages.
Claims
- supports: A small TTS model distilled purely on synthetic data generated by a much larger teacher TTS model can outperform both a model trained on matched-scale real data and the teacher itself on transcription accuracy and perceptual quality.
Evidence: A ~21M-parameter Matcha-TTS student trained on purely synthetic data generated by XTTS-v2 achieves lower WER (3.493 vs. 4.639) and higher UTMOS than the 750M-parameter XTTS-v2 teacher itself, and also outperforms an identical student trained on the same amount of real LibriSpeech data (WER 3.493 vs. 4.428). (§4.1, Table 2)
- supports: Increasing speaker diversity and text diversity in TTS training data each improve model performance up to a point, after which further diversity yields diminishing returns, with the saturation point differing across data dimensions and across metrics within the same dimension.
Evidence: Speaker-similarity improvements continue to roughly 500 speakers while WER and UTMOS improvements plateau earlier around 50 speakers; text-richness gains plateau once the training text retains all 69 English phonemes, occurring at a 99% text-reuse rate. (§4.2-4.3, Figures 1-2, Table 3)
- supports: TTS models trained on synthetic data are highly sensitive to even small proportions of noisy or reverberant training samples, with degradation occurring well before noise dominates the dataset.
Evidence: Model performance degrades significantly once the proportion of noisy training utterances exceeds roughly 0.1%, and reverberation degrades performance similarly as its proportion in training data increases. (§4.4, Figures 3-5)
- complicates: Synthetic training data generated to sound more natural or conversational does not necessarily train a better TTS model than synthetic data generated in a more standardized reading style, even when the conversational data scores well on automated quality metrics.
Evidence: The student trained on synthetic data from a conversational-style teacher (ChatTTS) underperforms models trained on reading-style (XTTS-v2) or natural-but-non-oral (CosyVoice2) synthetic data on WER and speaker similarity, despite ChatTTS-derived data scoring reasonably on DNSMOS. (§4.5, Table 2)
- complicates: A TTS model trained exclusively on synthetic data generalizes reasonably but not fully to out-of-domain conditions, with degradation concentrated in speaker similarity rather than intelligibility or perceptual quality.
Evidence: Evaluated on the out-of-domain Seed-TTS test set, models trained purely on synthetic data show a slight drop in speaker similarity relative to in-domain performance, while WER, DNSMOS, and UTMOS remain comparable to in-domain results. (§4.6)
Limitations and Open Questions
All experiments use a single student architecture (Matcha-TTS), which the authors explicitly note as a resource constraint; the generality of the findings across other TTS architectures (autoregressive, neural-codec-based, or larger flow-matching models) is untested. The study is similarly limited to a single scale (100-hour-equivalent datasets, ~21M-parameter student), and the authors identify scaling both dataset size and model capacity as a direction for validating whether the same sensitivity patterns and synthetic-data advantage hold at larger scale. Because all synthetic training data derives from XTTS-v2 for the main sensitivity experiments, some findings may be specific to that generator’s particular error and noise profile rather than universal across synthetic-data sources.
Wiki Connections
- Zero-Shot TTS — trains and evaluates a zero-shot TTS student model (speaker-embedding-conditioned Matcha-TTS) purely on synthetic data, measuring speaker similarity generalization to both in-domain and out-of-domain speakers.
- Seed-TTS — its test set is used as the out-of-domain evaluation benchmark for assessing generalization beyond the in-domain LibriSpeech distribution.
- CosyVoice 2 — used directly as one of three large-scale teacher TTS models to generate style-matched synthetic training data.