arXiv · 2025 · Preprint
Cedric Chan et al. (University of Pennsylvania) · → Paper · Demo: ? · Code: ?
Introduces a linguistically motivated, two-tier objective evaluation framework for TTS prosody that separates discrete event placement from continuous acoustic realization and validates it against human perceptual ratings.
Problem
Existing TTS prosody evaluation relies mainly on subjective methods (MOS, MUSHRA, pairwise comparison) that are costly, inconsistent across studies, and collapse many perceptual dimensions into a single opaque number, offering no insight into why a system sounds unnatural. Prior objective alternatives either predict MOS from acoustic features (inheriting the same opacity and inconsistency) or compute rigid frame-level distances between synthetic and reference speech, which unfairly penalize legitimate prosodic variation: two human speakers, or even the same speaker across repetitions, can realize the same communicative target with different pitch ranges, timing, or voice quality, all of which are perceptually valid. No existing objective method is grounded in the two-layered structure of prosodic theory (discrete structural events versus their continuous phonetic realization) or explicitly models the natural variability across speakers and acoustic cues.
Method
The framework extends and validates an evaluation method the authors first introduced in an earlier workshop paper, applying it to a new corpus and pairing it with fresh human-subject validation. It operates in two tiers that mirror the discrete/continuous distinction from Autosegmental-Metrical intonational phonology: a binary event tier and a continuous signal tier.
For the binary tier, prosodic events (pitch accents, phrase accents, boundary tones, pauses) are detected automatically as local extrema in word-level acoustic signals using a moving-median filter threshold (window size 7 words, threshold shift of half a standard deviation). Because multiple human realizations of the same sentence can differ in where they place events, the framework defines an “agreement score” per word: the proportion of reference human speakers who place an event at that word. A model is scored “correct” at a word if this agreement exceeds a fixed threshold (c = 0.5, i.e., majority agreement), enabling standard precision/recall/F1 computation. A smoothed variant replaces the hard threshold with a continuous correctness score computed from a generalized Gaussian function of the agreement score, awarding partial credit that decays sharply as human agreement falls, so minority-but-plausible placements are not treated as outright failures.
For the continuous tier, the framework computes a per-word normalized (z-scored) error between the TTS system’s acoustic measurement and the distribution of measurements across all reference human speakers at that word, dividing by the human standard deviation so that words with high natural human variability are not penalized as harshly as words where humans are highly consistent.
Audio is force-aligned at the word level with the Charsiu transformer-based aligner. Five word-level acoustic features are extracted with Praat: word/pause duration, mean F0 (pitch), mean intensity, spectral tilt (alpha ratio and L1-L0), and cepstral peak prominence-smoothed (CPPS, a voice-quality/periodicity measure). All measurements are z-normalized by speaker and sentence before scoring.
The reference corpus is a LibriVox recording of a chapter from Jane Austen’s Emma (136 sentences, 5 North American English female speakers). The same 136 sentences were synthesized with five commercial and open-source TTS systems (Google TTS, OpenAI TTS, Amazon Polly, Microsoft Azure TTS, and VITS via the facebook/mms-tts-eng checkpoint) as of February 2025. To validate the framework, the authors separately collected MOS ratings (140 participants) and pairwise preference judgments (97 participants) for the same systems, and ran a leave-one-out self-validation in which each of the five human speakers was scored against the other four using the same metrics.
Key Results
Perceptual results (Tables 1-2) rank the systems OpenAI > Google > Azure > VITS > Polly on MOS, with even the best system (OpenAI, MOS 3.55) rated statistically below the worst-rated human speaker (p = 0.00011). The pairwise-comparison ranking (Table 3, Bradley-Terry scores) largely agrees but swaps VITS and Polly, illustrating that the same listeners can produce inconsistent rankings depending on whether systems are rated in isolation or compared directly.
The objective framework reproduces essentially the same overall ranking (OpenAI > Google > Azure > Polly > VITS) while adding diagnostic detail the perceptual tests cannot: OpenAI has the most balanced profile but its weakest dimension is pitch (F1 = 0.311, normalized error = 0.582), consistent with perceptual reports of less natural intonation. Azure achieves higher binary event-placement F1 than Google on several cues (e.g., duration F1 0.723 vs. 0.684) but higher continuous error (e.g., pitch error 0.660 vs. 0.468), which the authors argue explains why Google was judged more natural overall despite weaker event placement.
In the human self-validation experiment (Table 4), human speakers scored statistically better than every TTS model on all 18 tested feature/metric combinations (smoothed zero-one loss, F1, normalized error, across five acoustic cues), and every human speaker achieved a smoothed zero-one loss of exactly 0.000 while no TTS model ever did. Comparing the single best model against the single worst human speaker per setting (Table B.17), humans still won 15 of 18 comparisons.
Novelty Assessment
The contribution is evaluation methodology, not a new TTS architecture. The genuinely new elements are: (1) grounding objective TTS evaluation explicitly in the discrete-event/continuous-realization distinction from intonational phonology, rather than treating prosody as a single undifferentiated signal; and (2) encoding natural human variability directly into the scoring function (via the agreement-score threshold and per-word z-normalization against the human reference distribution) rather than penalizing any deviation from a single canonical realization. The underlying acoustic features (F0, duration, intensity, spectral tilt, CPPS) and force-alignment pipeline are standard tools; the novelty is in how they are aggregated and compared. The paper is presented as an extension and validation (new corpus, new human-subject experiments, human self-validation analysis) of a two-tier method the same authors introduced in an earlier workshop paper, rather than an entirely new method.
Field Significance
moderate — This paper provides a reproducible, diagnostic alternative to MOS for prosody evaluation that correlates with perceptual rankings while exposing per-cue weaknesses (e.g., pitch versus voice-quality control) that a single MOS number cannot reveal. Its validation is demonstrated on one narrative audiobook corpus (a single chapter, one narrative genre, all-female North American English speakers) against five commercial or off-the-shelf systems, so the paper itself establishes the method’s diagnostic value in this specific setting rather than across languages, speaking styles, or a broader model pool.
Claims
- supports: A two-tier evaluation scheme that separately scores discrete prosodic event placement and continuous acoustic realization can reproduce perceptual naturalness rankings of TTS systems while providing per-cue diagnostic detail that a single perceptual or MOS-style score cannot.
Evidence: The framework’s aggregate rankings across five commercial/open TTS systems (OpenAI > Google > Azure > Polly > VITS) matched the MOS-based perceptual ranking, while per-feature F1/error breakdowns identified specific weaknesses (e.g., OpenAI’s weak pitch control vs. strong duration/intensity control) invisible in the single MOS number. (§3.2, Figures 3-4)
- supports: Encoding the natural variability of human prosodic realization into an objective scoring function, rather than scoring against a single reference target, allows the metric to correctly treat genuine human variation as acceptable while still separating human from synthetic speech.
Evidence: In leave-one-out self-validation, all five human reference speakers achieved a smoothed zero-one loss of exactly 0.000 (indicating the variability-aware metric does not penalize natural inter-speaker differences) while still scoring statistically higher than every TTS model on all 18 tested feature/metric combinations. (§3.3, Table 4)
- complicates: Perceptual evaluation methods that are treated as a single “gold standard” can disagree with each other even when administered to the same listener pool, undermining their use as a stable ground truth for validating objective metrics.
Evidence: The same participant pool produced a different naturalness ranking under MOS versus pairwise comparison testing, with VITS and Amazon Polly swapping relative rank between the two protocols. (§3.1.2)
- complicates: Event-placement accuracy and continuous acoustic-realization accuracy are separable dimensions of prosodic quality, so a system can excel at one while lagging on the other, meaning naturalness is not fully explained by either dimension alone.
Evidence: Azure achieved higher binary F1 than Google for several cues (e.g., duration F1 0.723 vs. 0.684) yet had higher continuous error (e.g., pitch error 0.660 vs. 0.468) and was judged less natural overall than Google, indicating precise continuous realization mattered more for perceived naturalness than correct event placement alone. (§3.2, §4)
Limitations and Open Questions
The validation corpus is a single audiobook chapter (136 sentences) read by five North American English female narrators in one narrative genre; the paper does not test cross-linguistic, conversational, or emotionally expressive speech, and the authors explicitly flag this generalizability question as future work.
The method also depends on a fixed threshold (c = 0.5) for binary event correctness and on a specific peak-detection procedure (moving-median filter with fixed window size and shift), both of which were set by the authors rather than derived from the data, and their sensitivity is not explored. A phone-level version of the analysis was attempted but discarded due to high variation and reduced relevance, which the authors note but do not quantify further. The five evaluated TTS systems used a single fixed voice/persona each, so the framework has not yet been tested for its ability to track prosodic quality across multiple voices, speaking styles, or expressive settings within the same system.
Wiki Connections
- Evaluation Metrics — introduces a novel objective, per-cue diagnostic metric intended as an alternative to acoustic-distance and MOS-prediction approaches for scoring TTS prosody.
- Subjective Evaluation — directly compares the proposed objective framework against MOS and pairwise human preference tests collected from the same listener pool, and documents inconsistency between the two perceptual protocols.