arXiv · 2026 · Preprint
Shengfan Shen et al. (Nanjing University, MiLM Plus (Xiaomi), HKUST) · → Paper · Demo: ? · Code: ?
Introduces Iterate to Differentiate (I2D), an evaluation protocol that recursively resynthesizes speech using a zero-shot TTS model’s own prior output as the reference, amplifying otherwise-saturated quality gaps between state-of-the-art systems.
Problem
TTS synthesis quality has advanced faster than the methods used to measure it. Objective metrics such as WER/CER and speaker similarity (SIM), along with neural MOS predictors like UTMOS and DNSMOS, were designed when quality gaps between systems were large. Among current state-of-the-art zero-shot systems, these metrics saturate: scores cluster within a narrow numerical range, so the intrinsic measurement noise of the evaluation models themselves becomes comparable to, or larger than, the true performance differences between systems. Subjective listening tests remain the most trustworthy signal but are costly, slow, and hard to reproduce across labs. Recent LLM-as-judge and large speech language model (LSLM) judging approaches are promising but still exploratory and computationally expensive. The paper asks whether existing, cheap objective metrics can be made discriminative again without inventing new metrics or collecting more human ratings.
Method
I2D reframes evaluation as an iterative process rather than a single synthesis pass. Given a TTS model and a triplet of reference audio, reference transcript, and target text, the model synthesizes speech once, and that output is then fed back in as the reference audio for the next round while the target text is held fixed; this repeats for a fixed number of iterations (10, with 5 recommended as a lower-cost alternative that reaches comparable discriminability). Because any content or speaker error introduced at one iteration corrupts the reference for the next, the protocol acts as an implicit error-amplification mechanism: models with weaker robustness accumulate mismatch between synthesized audio and target text faster, producing a divergence in degradation trajectories that a single-pass evaluation cannot reveal.
Objective metrics (WER/CER, SIM, DNSMOS, UTMOSv2, and an emotion-classification F1 score on an emotion subset) are computed at every iteration and then aggregated across the trajectory using one of four strategies: a simple mean, a linearly weighted average that up-weights later iterations, an exponentially weighted average with decay factor 0.9, and the trapezoidal area under the iteration-wise curve. The evaluation corpus combines three existing sources: a Chinese subset built from Seed-TTS-Eval’s test-zh split (2,020 utterances, 1,010 speakers, derived from DiDiSpeech), an English subset from LibriTTS test-clean filtered to 3-15 second utterances (2,915 utterances, 38 speakers), and an emotion subset from CV3-Eval’s Emotion Cloning split (300 samples spanning happy, sad, and angry in Chinese and English). A 100-sample subset of the Chinese data is additionally rated by human annotators (5-6 raters per sample, 11,752 annotation records after outlier filtering) across three dimensions: Content Accuracy, Speaker Consistency, and Overall Naturalness, at both the first and tenth iteration.
Key Results
At the first iteration, the standard deviation across 11 evaluated open-source TTS models is compressed to 3.83 (SIM), 0.64 (CER), 0.12 (UTMOSv2), and 0.07 (DNSMOS), and several models reach or exceed the ground-truth reference score, confirming severe saturation. Correspondingly, system-level Spearman correlation (SRCC) between UTMOSv2 and human naturalness ratings is only 0.118 at iteration 1. After 10 iterations, the same standard deviations rise to 12.15, 17.62, 0.48, and 0.52 respectively, and system-level SRCC for all four objective metrics with their corresponding human dimension exceeds 0.8. Aggregating scores across iterations (Mean strategy) and correlating against first-iteration human ratings raises the UTMOSv2 SRCC from 0.118 to 0.464 and the DNSMOS SRCC from 0.091 to 0.255, while SIM and 1-CER change little from their already-adequate baseline correlation.
Across the 11-model comparison (CosyVoice, CosyVoice2, CosyVoice3, CosyVoice3-RL, F5-TTS, FireRedTTS2, GLM-TTS, IndexTTS2, MaskGCT, Qwen3-TTS, VoxCPM1.5), CosyVoice3-RL attains the best CER (0.90) and SIM (67.32/62.17 en/zh), IndexTTS2 the lowest WER (2.22) and highest first-iteration Content Accuracy (4.88) and Naturalness (4.30), and Qwen3-TTS the highest UTMOSv2 (4.04/3.68) with the most stable tenth-iteration Naturalness (3.79). FireRedTTS2 degrades most severely, at points producing WER above 40 due to hallucinated or empty audio segments that compound across iterations; F5-TTS shows rising speaking rate and flattening intonation; CosyVoice accumulates electrical-noise artifacts. A targeted cross-model experiment swaps reference audio between the most robust system (CosyVoice3-RL) and the least robust (F5-TTS) at iteration 6: F5-TTS’s quality jumps toward CosyVoice3-RL’s level immediately after receiving its higher-quality reference but re-diverges within a few further iterations, indicating the degradation is driven mainly by progressive reference-quality decay rather than an irreversible distributional shift, while also confirming a genuine gap in intrinsic model robustness.
Novelty Assessment
I2D does not introduce a new metric or a new evaluation model; every underlying signal (WER/CER, SIM, DNSMOS, UTMOSv2, emotion F1, MOS) is an existing tool. The contribution is procedural: chaining a zero-shot TTS model’s own output back into itself as the reference across multiple rounds, then aggregating the resulting trajectory rather than reading a single score. This is a comparatively simple experimental design change, but it is applied carefully, with a systematic ablation of four aggregation strategies, a dedicated cross-model swap experiment to separate reference-quality decay from distributional shift, and a sizeable human evaluation (11,752 ratings) used to validate the resulting correlation gains rather than assumed. The honest characterization is a methodological/evaluation contribution rather than an architectural or modeling one, and its value depends on the field adopting iterative resynthesis as a standard evaluation add-on rather than a metric replacement.
Field Significance
Moderate — the paper documents a real and underexamined failure mode (objective-metric saturation among strong zero-shot TTS systems) and provides a low-cost, reproducible fix that restores ranking reliability without new training or annotation infrastructure. Its contribution is confirmatory and procedural rather than a new theoretical framework, but it is validated across three datasets, 11 models, and a substantial human evaluation, which gives the discriminability claim more evidentiary weight than a typical single-dataset evaluation study.
Claims
- supports: Aggregating objective metrics across iterative self-resynthesis restores ranking discriminability among TTS systems whose single-pass scores have saturated.
Evidence: System-level SRCC between UTMOSv2 and human naturalness ratings rises from 0.118 (iteration-1 baseline) to 0.464 under Mean aggregation over 10 iterations of resynthesis. (§5.2, Table 3)
- complicates: Standard single-pass objective TTS metrics lose their ability to rank systems once those systems reach comparable quality levels.
Evidence: At iteration 1, the standard deviations of SIM, CER, UTMOSv2, and DNSMOS across 11 SOTA models are compressed to within roughly 2% of each other, with several models at or above the ground-truth reference score. (§5.1)
- supports: Recursively conditioning a zero-shot TTS model on its own prior output surfaces latent robustness differences between systems that are invisible in a single generation pass.
Evidence: In a cross-model reference-swap experiment, a weaker model’s quality jumps toward a stronger model’s level immediately after receiving its reference audio but re-diverges within a few subsequent iterations, showing that degradation reflects intrinsic model robustness and not only reference-audio quality. (§5.4, Figure 5)
- complicates: Evaluation protocols that reward stability under repeated self-conditioning can conflate robustness with a preference for less expressive, more homogeneous output.
Evidence: The authors note that I2D’s iterative design favors model stability over expressive diversity and recommend pairing it with diversity-oriented measures for a holistic assessment. (§6)
- refines: Aggregate classification metrics for emotion-conditioned TTS can mask systematic category bias rather than reflecting genuine emotion controllability.
Evidence: One evaluated model attains the highest F1 on the Sad emotion category purely because its outputs converge toward a sad-sounding tone under iteration, inflating recall for that class while its performance on other emotions degrades. (§5.3, Table 5)
Limitations and Open Questions
Warning
The framework is validated only on open-source TTS systems (commercial models were inaccessible for cost and access reasons), requires repeated synthesis and metric computation for every model under test (multiplying compute cost roughly by the iteration count), and its iterative design structurally rewards stability over expressive diversity, so results should not be read as a complete quality assessment on their own.
Naturalness scores under I2D are also confounded by reference-audio quality: because later-iteration references are themselves synthetic and progressively degraded, the protocol creates a tension between naturalness and speaker similarity in zero-shot settings, particularly once the reference audio is already suboptimal. The recommended reduction to 5 iterations trades off some of the discriminability gains for lower compute cost, and the paper does not establish how the choice of maximum iteration count should be set for architectures very different from the 11 evaluated here.
Wiki Connections
- Evaluation Metrics — proposes an iterative aggregation protocol that restores the discriminative power of existing objective metrics (WER/CER, SIM, DNSMOS, UTMOSv2) among saturated state-of-the-art zero-shot TTS systems.
- Subjective Evaluation — validates the iterative protocol against a large-scale human MOS study (11,752 ratings across Content Accuracy, Speaker Consistency, and Naturalness) to confirm improved human alignment.
- Zero-Shot TTS — benchmarks 11 zero-shot voice-cloning systems spanning autoregressive, non-autoregressive, and hybrid paradigms under a common iterative evaluation protocol.
- Emotion Synthesis — evaluates emotion-cloning capability across 11 systems and surfaces a category-bias failure mode in aggregate emotion F1 scoring.
- Towards Responsible Evaluation for TTS — cited as directly related prior work diagnosing evaluation methodology gaps that motivate this paper’s iterative protocol.
- UrgentMOS — cited as independent confirmation that neural MOS predictors struggle to rank strong TTS systems reliably, the same saturation phenomenon I2D targets.
- SP-MCQA — related contemporaneous work on improving intelligibility evaluation beyond word-level metrics for TTS.
- F5-TTS — one of 11 evaluated systems; shows rising speaking rate and flattening intonation under iterative resynthesis and is used as the weaker system in the cross-model reference-swap experiment.
- Qwen3-TTS — one of 11 evaluated systems; achieves the highest UTMOSv2 scores and the most stable tenth-iteration naturalness among all models tested.
- FireRedTTS2 — one of 11 evaluated systems; shows the most severe degradation under iteration, with hallucinated or empty audio segments driving WER above 40.
- CosyVoice3 — one of 11 evaluated systems (also tested with an RL-tuned variant); attains the best CER and SIM scores among all models evaluated.
- Seed-TTS — Seed-TTS-Eval’s test-zh split, derived from DiDiSpeech, is used as this paper’s Chinese evaluation subset.