arXiv · 2025 · Preprint
Yifan Yang et al. · → Paper · Demo: ✓ · Code: ?
Introduces a human-annotated benchmark and a discrete-token edit-distance metric for measuring prosody diversity in zero-shot TTS, then uses it to benchmark seven open-source systems and isolate the factors that drive prosodic variation.
Problem
Prosody diversity, the extent to which a TTS system produces varied pitch, rhythm, and stress patterns across repeated generations of the same text and speaker, is central to naturalness and expressiveness in zero-shot TTS, but it has no reliable measurement. The standard proxy, log F0 RMSE computed after dynamic time warping (DTW) alignment, captures only pitch, ignores rhythm and intensity, and is expensive to compute. No prior work had validated any objective prosody-diversity metric against human perception at scale, leaving system comparisons in this dimension essentially unverified.
Method
The paper builds two artifacts. First, ProsodyEval, a dataset of 1000 speech samples generated by seven zero-shot TTS systems spanning three generative paradigms (autoregressive next-token prediction, flow matching, and masked generative modeling), using prompts and text from LibriSpeech test-clean and Seed-TTS test-en. Each system produces groups of five samples per input with different random seeds; 20 graduate raters perform pairwise comparisons across all ten pairs within each group on a five-point Likert scale for perceived prosodic difference, yielding 2000 Prosody Mean Opinion Score (PMOS) ratings, alongside the conventional log F0 RMSE and MCD metrics computed via FastDTW alignment.
Second, the paper proposes Discretized Speech Weighted Edit Distance (DS-WED), an objective metric for prosody diversity between a pair of samples. Each waveform is trimmed with a voice-activity detector to remove silence, then discretized into semantic tokens via k-means clustering (50 clusters) over hidden representations from a frozen self-supervised speech encoder (HuBERT-base or WavLM-base, 8th transformer layer in the reported configuration). Prosodic variation is then computed as a weighted Levenshtein (edit) distance between the two token sequences, with substitution, insertion, and deletion operations each carrying a tunable cost (set to 1 in all reported experiments). The authors motivate the choice of semantic tokens over supervised tokens (which distort duration due to CTC-style losses) or acoustic codec tokens (which encode low-level signal detail irrelevant to prosody), and argue the metric is representation-agnostic and could in principle use other discretization schemes. No new model is trained: DS-WED is a training-free metric built from an off-the-shelf SSL encoder plus a k-means model fit on LibriSpeech 960h.
Key Results
On ProsodyEval, DS-WED reaches a Pearson correlation of r̄ = 0.77 with PMOS, versus r̄ = 0.66 for MCD and r̄ = 0.30 for log F0 RMSE, all aggregated via Fisher’s Z transformation across groups (Table 1). DS-WED also has a lower measured real-time factor (0.110) than log F0 RMSE (0.549) or MCD (0.203), since it requires only a GPU forward pass and discrete edit distance rather than CPU-bound DTW alignment. Ablations across SSL backbone, encoder layer, and cluster count (Figure 1) show correlations remaining near 0.7 across most settings, with middle layers (6-9) and smaller cluster counts performing best.
Applying DS-WED to benchmark seven zero-shot TTS systems on LibriSpeech test-clean and Seed-TTS test-en (Table 2), autoregressive systems (XTTS-v2, CosyVoice, CosyVoice 2) consistently outperform flow-matching NAR systems (E2 TTS, F5-TTS, ZipVoice) in prosody diversity, but a masked generative modeling system (MaskGCT) matches or exceeds the AR systems despite training on the same 100k-hour Emilia corpus as the flow-matching baselines, indicating that modeling paradigm rather than training data scale drives the difference. Duration perturbation (varying playback rate by 0.8-1.2x) increases DS-WED diversity for both tested NAR systems, with F5-TTS gaining nearly 30% (Table 3), though it still trails AR and MGM systems even with perturbation applied. Applying DPO-based preference alignment for intelligibility reduces DS-WED diversity by 2.9-18.8% across the two systems tested (Table 4). Finally, prompting Gemini 2.5 Pro to judge relative prosodic difference produces only a weak, though statistically significant, correlation with PMOS (r̄ = 0.27) and weaker, less stable correlations with the objective metrics (Table 5).
Novelty Assessment
The contribution is an evaluation-methodology and benchmark paper rather than a generative architecture paper. The core technical idea, weighted edit distance over discretized SSL tokens, closely follows an existing preliminary approach (SpeechTokenDistance) that the authors cite and explicitly extend by validating it against a purpose-built human-rated dataset and testing its robustness across SSL backbones, layers, and cluster sizes. The benchmark itself (comparing AR, flow-matching, and MGM systems, and probing duration control and DPO effects) is a genuine empirical contribution: the corpus-wide comparison and the finding that data scale is not the dominant factor in prosody diversity are new observations not established in prior single-system papers. The human-annotated ProsodyEval dataset, with 2000 ratings across seven diverse TTS systems, is the paper’s most durable artifact, since it enables independent validation of future prosody metrics without requiring new listening tests.
Field Significance
Moderate — the paper fills a concrete measurement gap (no validated objective metric for prosody diversity existed) and produces both a metric and a reusable human-rated dataset, but its scope is narrow (English only, seven systems, one aspect of TTS quality) and its architectural insights are observational rather than novel modeling contributions. Its benchmarking results, that masked generative modeling matches autoregressive systems in prosody diversity while flow-matching NAR systems lag due to implicit alignment, and that preference-optimization for intelligibility trades off against prosody diversity, provide useful data points for practitioners choosing among TTS paradigms.
Claims
- supports: Acoustic proxy metrics for prosodic variation correlate weakly with human perception, while distance measures computed over discretized self-supervised speech tokens correlate substantially better.
Evidence: Averaged Pearson correlation with human PMOS ratings is r̄ = 0.30 for log F0 RMSE and r̄ = 0.66 for MCD, versus r̄ = 0.77 for the proposed token-edit-distance metric, aggregated via Fisher’s Z transformation across 1000 samples and 2000 ratings. (§4.1, Table 1)
- supports: Autoregressive generation provides greater output diversity in prosody than non-autoregressive flow-matching models with implicit text-speech alignment, but this advantage does not extend to non-autoregressive masked generative modeling.
Evidence: On the DS-WED benchmark across LibriSpeech test-clean and Seed-TTS test-en, three AR systems (XTTS-v2, CosyVoice, CosyVoice 2) outperform three flow-matching NAR systems (E2 TTS, F5-TTS, ZipVoice), while the masked generative modeling system (MaskGCT) surpasses all AR systems on LibriSpeech and remains competitive on Seed-TTS despite training on the same Emilia corpus as the flow-matching systems. (§4.4-4.5, Table 2)
- complicates: Explicit duration control during inference is a significant, but not sufficient, factor in restoring prosodic diversity to non-autoregressive TTS systems with implicit alignment.
Evidence: Applying duration perturbation (0.8-1.2x) to two flow-matching/MGM NAR systems increases DS-WED diversity by 13.8-28.5%, but the perturbed flow-matching system (F5-TTS) still lags behind AR and MGM systems evaluated without perturbation, suggesting the shortfall is architectural (implicit alignment) rather than purely a missing duration-control step. (§4.5, Table 3)
- complicates: Preference optimization that targets one quality dimension (intelligibility) can measurably suppress output diversity along an unrelated axis (prosody) as a side effect.
Evidence: Applying DPO for intelligibility to CosyVoice 2 and MaskGCT reduces DS-WED prosody-diversity scores by 18.8% and 2.9-3.2% respectively across two test sets, with no diversity-specific reward term in the alignment objective. (§4.5, Table 4)
- contradicts: A general-purpose large audio language model with strong multimodal reasoning capability can serve as a reliable automatic judge of fine-grained prosodic variation.
Evidence: Gemini 2.5 Pro, prompted to rate relative prosodic difference within groups of five samples, achieves only a weak correlation with human PMOS ratings (r̄ = 0.27) and unstable, wide-confidence-interval correlations with objective acoustic and token-based metrics. (§4.5, Table 5)
Limitations and Open Questions
The authors state that DS-WED has only been validated on English speech; cross-lingual applicability is untested. The benchmark itself covers only seven open-source systems and two evaluation corpora (LibriSpeech test-clean, Seed-TTS test-en), so its paradigm-level conclusions (AR vs. flow-matching vs. MGM) rest on a small number of representative systems per category rather than an exhaustive sweep. The DPO and duration-perturbation exploration each cover only two systems, limiting how confidently the reported trends generalize across the broader space of NAR TTS architectures. The paper also does not report whether code or the ProsodyEval dataset itself will be released beyond the audio-sample demo page, which affects reproducibility of the correlation analysis.
Wiki Connections
- Evaluation Metrics — introduces DS-WED, a token-edit-distance objective metric for prosody diversity, and validates it against human ratings as an alternative to log F0 RMSE and MCD.
- Subjective Evaluation — constructs ProsodyEval, a 2000-rating human listening test dataset specifically targeting perceived prosodic difference rather than overall naturalness or similarity.
- Self-Supervised Speech — the proposed metric depends entirely on discretizing speech into semantic tokens via frozen HuBERT/WavLM representations and k-means clustering.
- Zero-Shot TTS — benchmarks prosody diversity across seven zero-shot TTS systems spanning autoregressive, flow-matching, and masked generative modeling paradigms.
- Flow Matching — finds that flow-matching NAR systems with implicit text-speech alignment under-perform on prosody diversity relative to autoregressive and masked generative modeling systems.
- RLHF Speech — shows that DPO-based preference alignment for intelligibility reduces prosody diversity as a side effect in both autoregressive and masked generative modeling systems.
- F5-TTS — benchmarked as a fully non-autoregressive flow-matching system; ranks lowest in prosody diversity among tested systems, with duration perturbation only partially closing the gap.
- CosyVoice 2 — benchmarked as an AR+flow-matching hybrid system; used as one of the two systems in the DPO ablation, showing an 18.8% diversity reduction after preference alignment.
- MaskGCT — benchmarked as the masked generative modeling system; found to match or exceed AR systems in prosody diversity despite sharing training data with weaker flow-matching systems.
- E2 TTS — benchmarked as a fully non-autoregressive flow-matching system; used as an example of implicit-alignment architecture underperforming in prosody diversity.
- XTTS — benchmarked as an AR+VQ-VAE system trained on comparatively little data; its strong prosody diversity despite smaller training scale supports the paper’s claim that modeling paradigm dominates data scale.
- Emilia — the shared 100k-hour training corpus for the NAR systems compared in the benchmark, used to argue that prosody-diversity differences stem from architecture rather than data.
- Seed-TTS — its test-en benchmark set is used as one of the two evaluation corpora for the prosody-diversity benchmark.
- ZipVoice — benchmarked as a fully non-autoregressive flow-matching system with explicit average upsampling alignment, showing partial mitigation of the flow-matching prosody-diversity shortfall.