Position: Towards Responsible Evaluation for Text-to-Speech

arXiv · 2025 · Preprint

Yifan Yang et al. · → Paper · Demo: ? · Code: ?

This position paper argues that TTS evaluation must expand beyond technical performance metrics to a three-level “Responsible Evaluation” framework covering measurement fidelity, cross-study comparability, and ethical/risk oversight.

Problem

TTS systems now routinely generate speech that is close to indistinguishable from human recordings, yet the paper argues that evaluation methodology has not kept pace with this capability jump. Current practice still centers on a narrow set of naturalness, intelligibility, robustness, and speaker-similarity metrics inherited from earlier eras of the field, while three distinct problems compound: (1) both objective metrics (WER, SIM, predicted MOS, F0-RMSE) and subjective protocols (MOS) have known validity limitations that are routinely ignored when interpreting score differences; (2) evaluation datasets, task definitions, and metric computation protocols vary silently across papers, making numbers that look comparable actually incomparable; and (3) the ethical and societal risks of highly realistic synthetic speech (deepfakes, biometric spoofing, bias, unclear data provenance) are almost never assessed as part of standard evaluation, being relegated to brief disclaimers if mentioned at all.

Method

The paper does not propose a new model, metric, or benchmark. Its contribution is a critical diagnosis of existing TTS evaluation practice, organized into a three-level conceptual framework called Responsible Evaluation:

• Level One (Fidelity and Accuracy): evaluation metrics should faithfully reflect a system’s true capabilities and limitations. The paper works through specific objective metrics (WER, SIM, predicted MOS, F0-RMSE) and subjective MOS-family protocols, identifying known failure modes for each (§3.1–3.2), and flags underexplored evaluation dimensions such as mathematical/symbolic verbalization, long-form coherence, emotional expressiveness, and punctuation sensitivity (§3.3).
• Level Two (Comparability, Standardization, Transferability): evaluation should follow scientific rigor sufficient for fair cross-system comparison. The paper documents concrete sources of incomparability in current zero-shot TTS evaluation: divergent LibriSpeech test-clean subset sizes across papers, inconsistent definitions of the “Continuation” inference task, divergent SIM-o computation protocols, and underspecified MOS/RTF reporting (§4.1–4.3).
• Level Three (Ethical and Risk Oversight): evaluation should incorporate societal and ethical implications. The paper discusses training-data provenance and consent, traceability/watermarking, bias auditing, and adversarial-misuse testing as evaluation dimensions that are essentially absent from current practice (§5.1–5.4).

To support the Level One and Level Two arguments with concrete numbers rather than assertion alone, the paper runs three small original case studies in its appendices: (A) re-evaluating a released MaskGCT checkpoint on two commonly used LibriSpeech test-clean subset variants (40 vs. 1234 utterances) to show how much WER shifts purely from dataset-variant choice; (B) recomputing SIM-o for a representative continuation-task output under the VALL-E protocol (prompt excluded) versus the VALL-E 2 protocol (prompt included) to show how much the score shifts purely from a protocol choice; and (C) collecting a small MOS-style listening test (10 graduate raters) focused specifically on perceived intelligibility (“WER-MOS”) for MELLE output versus ground truth, to test whether a lower objective WER corresponds to a perceptible intelligibility gain. Each level closes with a short set of actionable recommendations (§3.4, §4.4, §5.5), and a dedicated “Alternative Views” section (§6) engages counterarguments about evaluation complexity and the tension between ethical rigor and low-resource-language progress.

Key Results

The paper’s evidence is illustrative rather than a systematic benchmark, but the three appendix case studies are concrete: (A) the same MaskGCT model’s WER on LibriSpeech test-clean cross-sentence synthesis ranges from 2.63% to 4.22% depending solely on which commonly cited test-clean subset variant is used (Table 1); (B) SIM-o for the same synthesized output differs by an absolute 0.151 (0.754 vs. 0.905) depending solely on whether the prompt segment is included in the similarity computation (Table 2); and (C) MELLE achieves a lower objective WER than ground-truth recordings (1.47% vs. 1.61%) while the corresponding perceptual WER-MOS is nearly unchanged (1.06% vs. 1.09%), suggesting the two measures diverge once error rates are already low (Table 3). Beyond these appendices, the “results” of the paper are its synthesis of the state of TTS evaluation practice and its recommendations, which are argued rather than empirically benchmarked at scale.

Novelty Assessment

The individual observations in this paper are mostly not new in isolation: MOS saturation, metric-perception mismatch, WER/SIM protocol inconsistency, and DNSMOS domain mismatch have each been raised in prior work that the paper itself cites (e.g., Chiang et al., 2023; Kirkland et al., 2023; Cooper et al., 2022). The paper’s contribution is primarily conceptual: organizing these scattered critiques into a structured, three-level taxonomy (fidelity, comparability, ethics) that treats them as a coherent evaluation-maturity ladder rather than a list of unrelated complaints, and pairing that taxonomy with concrete, if small-scale, original demonstrations (the LibriSpeech-variant, SIM-o-protocol, and WER-vs-WER-MOS case studies) rather than citing the inconsistency only in the abstract. It does not introduce a new metric, dataset, or architecture, and the empirical appendices are single-case illustrations (one model, one protocol pair, ten raters) rather than systematic audits across the field.

Field Significance

Moderate — the paper functions as a critical synthesis and call-to-action rather than a technical advance; its contribution is reframing known evaluation weaknesses into an actionable three-level agenda and providing small concrete demonstrations of specific comparability failures (dataset-variant and protocol dependence) that individual TTS papers rarely surface on their own.

Claims

• supports: Once objective intelligibility metrics reach an already-low error rate, further numerical improvement does not necessarily correspond to a perceptible gain in synthesized speech quality.

  Evidence: An original 10-rater listening test finds MELLE’s synthesized speech has a lower WER than ground-truth recordings (1.47% vs. 1.61%) but a nearly identical perceived-intelligibility WER-MOS (1.06% vs. 1.09%). (§Appendix C, Table 3)

• complicates: Speaker-similarity (SIM) scores are not directly comparable across zero-shot TTS papers unless the similarity-computation protocol is held fixed, since the protocol itself materially changes the score.

  Evidence: Recomputing SIM-o for the same continuation-task output under the VALL-E protocol (prompt excluded from the similarity computation) versus the VALL-E 2 protocol (prompt included) yields 0.754 and 0.905 respectively, an absolute difference of 0.151. (§Appendix B, Table 2)

• complicates: Reporting a benchmark name and a metric value alone (e.g., “LibriSpeech test-clean WER”) is not sufficient for reproducible cross-paper comparison, because different papers silently evaluate on different subset variants of the same nominally shared benchmark.

  Evidence: The same released MaskGCT checkpoint evaluated on two commonly cited LibriSpeech test-clean variants (40 vs. 1234 utterances) for the identical cross-sentence task produces WER of 2.63% and 4.22% respectively. (§Appendix A, Table 1)

• complicates: Neural MOS-prediction models trained for one purpose transfer unreliably when repurposed for a different evaluation target, and typically report no uncertainty around their point estimates.

  Evidence: The paper’s analysis identifies DNSMOS, trained on speech-enhancement data, as a domain-mismatched but widely used predictor for TTS naturalness, and notes that predicted-MOS models generally provide only point estimates without confidence intervals. (§3.1)

• supports: Provenance and traceability safeguards for synthetic speech are largely absent from mainstream TTS evaluation despite documented real-world misuse.

  Evidence: The paper’s risk analysis cites documented synthetic voice fraud in the financial and communications sectors as evidence that current evaluation practice rarely reports verifiable markers of machine-generated speech or tests for traceability. (§5.2)

Limitations and Open Questions

Warning

The paper is an argued position piece, not a systematic survey or meta-analysis: it does not report a reproducible literature-search methodology, and its supporting appendix experiments are single-case illustrations (one model per case study, one protocol pair, ten raters) rather than field-wide audits. Readers should treat the appendix numbers as existence proofs of specific failure modes, not as effect-size estimates generalizable across systems.

The recommendations themselves (uncertainty reporting for predicted MOS, mandatory data-provenance disclosure, watermarking adoption, standardized adversarial-risk testing) are not accompanied by concrete protocol specifications or reference implementations, leaving their translation into practice to future work. The paper’s own “Alternative Views” section (§6) acknowledges that expanding evaluation dimensions risks fragmenting comparability in industrial settings, and that strict data-provenance requirements could disproportionately burden low-resource-language TTS development, without fully resolving either tension.

Wiki Connections

• Evaluation Metrics — the paper’s central contribution is a critical audit of WER, SIM, predicted MOS, and F0-RMSE as objective TTS metrics, and a taxonomy for what these metrics do and do not capture.
• Subjective Evaluation — argues that MOS-family protocols suffer from score saturation, rater variability, and underspecified reporting, and calls for more discriminative, better-documented listening-test methodology.
• Zero-Shot TTS — uses zero-shot TTS evaluation (test-clean variants, SIM-o protocols, continuation/cross-sentence tasks) as its primary case study domain for cross-paper incomparability.
• Spoken Language Model — situates the rise of LLM-based and foundation-model TTS systems as the trigger for evaluation practice falling behind model capability.
• VALL-E — used as the source of the original zero-shot continuation/cross-sentence task definitions and the “prompt-excluded” SIM-o protocol that the paper shows is incompatible with later variants.
• VALL-E 2 — cited as introducing the “prompt-included” SIM-o protocol that the paper directly contrasts against VALL-E’s protocol in its Appendix B case study.
• MaskGCT — used as the evaluated system in the paper’s Appendix A case study demonstrating WER sensitivity to LibriSpeech test-clean subset choice.
• F5-TTS — cited as another system using yet another LibriSpeech test-clean subset variant, reinforcing the paper’s argument about fragmented evaluation datasets.
• MELLE — used as the evaluated system in the paper’s Appendix C case study comparing objective WER against a dedicated perceptual WER-MOS listening test.
• E2 TTS — cited as redefining VALL-E’s “Continuation” task (using the end of an utterance as the prompt rather than the start), illustrating inconsistent inference-task definitions across papers.
• SpeechLLM-as-Judges — cited as an example of the LLM-as-judges paradigm the paper points to as a promising direction for more transferable, interpretable evaluation.
• Measuring Prosody Diversity in Zero-Shot TTS — cited as an exemplar of the transferable, perceptually grounded objective metrics the paper’s recommendations call for.