arXiv · 2026 · Preprint

Antonis Asonitis et al. (AGIGO / ETH Zurich / Sapienza University of Rome / University of Zurich) · → Paper · Demo: ✗ · Code: ✗

Reference-free quality predictors agree with listeners when one TTS clip carries audible defects but fall to chance when both clips are clean, and optimizing a single such score as a policy-optimization reward games the metric while an independent judge and human listeners judge the output worse.

Problem

Reference-free MOS predictors (UTMOS, UTMOSv2, DNSMOS, NISQA, SCOREQ and others) serve two roles in TTS work: they rank systems in place of listening tests, and they act as rewards for preference optimization. Both roles assume the predicted score tracks human preference. Most of these predictors were developed on material whose main variation is signal degradation, often by contrasting a clean utterance with degraded copies of itself. The paper asks whether they still track preference on the clean audio that modern TTS produces. It also asks what happens when a TTS policy is optimized against them. Prior MOS benchmarks did not read predictor accuracy against a measured human-reliability ceiling across a controlled quality gradient, and did not stress-test the same scores as rewards (§II-D).

Method

Benchmark. The authors reuse six public human-rated corpora, ordered by audible synthesis quality (§III-A, Table I): artifact-rich BVCC (187 systems, 27.8k pairs) and SOMOS (200 systems, 89k pairs), singing-synthesis SingMOS (37.7k pairs), and three defect-free corpora: SpeechJudge (multilingual zero-shot systems including cross-lingual cloning, 7.6k pairs), TTS-Arena (5.2k pairs), and TTS-HP (four commercial systems, 18 voices, 15 forced-choice votes per pair, 2.7k pairs). Pairs are same-text cross-system comparisons. Targets are delta-MOS on the MOS corpora and majority vote on the vote corpora. TTSDS2 is excluded because its systems sit at a low-artifact midpoint of the gradient (§III-A, footnote 1).

Predictors and protocol. Thirteen reference-free predictor families (33 score variants), thirteen Praat prosody descriptors, and eighteen classical signal-processing descriptors are scored after loudness normalization to -23 LUFS (§III-B). The task is pairwise: does the clip a predictor scores higher match the clip listeners prefer? Chance is 0.5 (§III-C). The SpeechJudge generative judge is also run as one evaluator; it sees both clips and the transcript.

Human ceiling. The ceiling is the fraction of individual listeners agreeing with the majority verdict on clear-majority pairs (§III-D). A stricter split-half floor is also reported for TTS-HP and BVCC.

Controls and composite. Controlled degradations of 500 clean clips test whether predictors remain sensitive within a clip (Table III). A calibrated composite (ridge regression on graded preference over score differences, a Bradley-Terry construction) is fit on at most min(20%, 1000) in-domain pairs per corpus and evaluated on group-disjoint held-out folds that share no system, voice, or clip with the training pool (§V-E, Table IV). A leave-one-corpus-out variant tests transfer (§V-D).

Reward stress test. Qwen3-TTS-12Hz (0.6B), a codec language model emitting 12.5 Hz acoustic tokens, is post-trained with Group Sequence Policy Optimization (GSPO) for 300 steps (group size 16, AdamW, learning rate 1e-6, no KL penalty by default), changing only the reward (§VI-A). Generation is pinned to one reference voice by in-context cloning. Each policy is read at its peak-reward checkpoint and judged on held-out prompts by a panel the reward never sees, with held-out UTMOS-v1 as the primary automatic judge. A human pairwise listening test (30 held-out utterances across 11 speakers, 30 raters, 300 comparisons chosen by information-gain maximization) yields Elo ratings. The equal-weight composite reward combines WER, Distill-MOS and UTMOSv2 with fixed equal weights (§VI-C).

Key Results

Collapse across the quality gradient (Table II). Rankings differ by corpus, so they are stated per corpus:

  • BVCC: SCOREQ 0.909, UTMOSv2 0.899, UTMOS 0.892, against a 0.887 human ceiling.
  • SOMOS: SCOREQ 0.672, UTMOS 0.667, UTMOSv2 0.654 (ceiling 0.836).
  • SingMOS: SCOREQ 0.683, UTMOS 0.674, UTMOSv2 0.653.
  • SpeechJudge: SCOREQ 0.705, SCOREQ-nat 0.694, UTMOS and WV-MOS 0.690 each.
  • TTS-Arena: SCOREQ 0.585, UTMOS 0.577, WV-MOS 0.573.
  • TTS-HP: UTMOSv2 0.528, then SIGMOS and the content-blind argmax-duration baseline at 0.524 each, against a 0.764 ceiling. The authors report bootstrap 95% intervals of about ±0.02 (n=2666) and treat predictor differences there as not significant.

Several scores fall below 0.5 on TTS-HP, including SQUIM-PESQ 0.470 (p=0.002 per §IV), spectral flux 0.468, and the SpeechJudge model 0.45, against its in-domain 0.75 (§IV-A). Clip duration is itself anti-correlated with preference on SOMOS (0.368) and SpeechJudge (0.370). Restricting each corpus to both-clean pairs pushes agreement toward chance in all six; on TTS-HP even the full 64-feature composite reaches about 0.54 (§IV). Split-half agreement among TTS-HP annotators is 0.53, against 0.81 on BVCC (§III-D).

Predictors remain sensitive within a clip (Table III). Predictors prefer the original over a degraded copy at 0.998 (noise at 15 dB SNR) and 0.983 (25 dB), 0.863 and 0.771 for band-limiting to 3 and 5 kHz, 0.950 and 0.977 for +3 and +6 semitone pitch shifts, 0.927 and 0.947 for pause insertion, and 0.566, 0.672 and 0.856 for 6%, 12% and 25% time-stretching. The authors read the cross-clip collapse as a property of the task on clean audio rather than lost sensitivity (§IV-A). Whisper transcribes 72% of TTS-HP clips with zero word errors, and the lower-WER clip wins at chance. Untuned Qwen2-Audio shows complete position bias in pairwise mode and 0.485 and 0.496 in single-clip rating (§IV-A).

Calibrated composite (Table IV). With in-domain pairs, the held-out composite exceeds the best single metric selected on the same pairs on every corpus: BVCC 0.921 vs. 0.907, SOMOS 0.714 vs. 0.670, SingMOS 0.693 vs. 0.678, SpeechJudge 0.721 vs. 0.694, TTS-Arena 0.611 vs. 0.575, TTS-HP 0.523 vs. 0.513 (relative gains +1.5% to +6.6%). On TTS-HP the 0.010 gain covers a small part of the 0.264 span from chance to the ceiling, and the authors place it within fold-to-fold spread (§V-A). Without in-domain data the composite does not transfer: leave-one-corpus-out training reaches 0.52 on TTS-HP (0.54 weighting corpora equally), and transfer between the two defect-free corpora is 0.53, or 0.48 from TTS-HP to SpeechJudge (§V-D).

Reward stress test (Table V). Against the baseline’s held-out UTMOS-v1 of 4.51 and Elo 2098 ± 101:

  • SCOREQ reward: SCOREQ rises from -4.71 to -0.19 while UTMOS-v1 falls to 1.26 and Elo to 1675 ± 115; the audio degenerates into padding. Adding a KL penalty (β=0.1) does not prevent it, with UTMOS-v1 still falling to 1.23 (§VI-B).
  • Crest-factor and HF-energy-ratio rewards: UTMOS-v1 1.23 and 1.30, Elo 1582 ± 123 and 1635 ± 101.
  • SQUIM-PESQ: UTMOS-v1 3.54, Elo 2008 ± 98. UTMOSv2: UTMOS-v1 4.30, Elo 1915 ± 109.
  • DNSMOS and Distill-MOS: UTMOS-v1 4.41 and 4.48, Elo 2010 ± 98 and 2115 ± 94, which the authors describe as within the baseline interval.
  • Equal-weight composite (WER, Distill-MOS, UTMOSv2): UTMOS-v1 4.56, Elo 2125 ± 95, a tie with the baseline within the reported intervals.

The human listening test is small (300 comparisons), and only the SCOREQ and signal-processing rewards have Elo intervals that do not overlap the baseline’s.

Novelty Assessment

The contribution is a measurement study, not a new metric or model. Pairwise accuracy, MOS-predictor benchmarking, and reward over-optimization are established ideas. What is new is putting them together: reading reference-free predictors against a measured human-reliability ceiling across a quality gradient from artifact-rich to commercial TTS, adding a content-blind duration baseline and within-clip sensitivity controls, and closing the loop with a GSPO reward stress test judged by a held-out predictor and human listeners. The authors position SpeechJudge as the closest concurrent work, which also reports poor fixed-predictor agreement near the top of the scale but trains a judge instead (§II-D). The composite evaluator is a standard ridge head, and the equal-weight reward is a hand-chosen mix. The reward experiment is framed by the authors as an existence proof under one recipe on one 0.6B policy, not a general law (§VI).

Field Significance

Moderate — The paper gives a corpus-stratified, ceiling-referenced account of where reference-free MOS predictors stop tracking human preference on TTS, and pairs it with a direct demonstration that single-score rewards can be gamed during TTS post-training while an equal-weight multi-signal reward holds quality. Its main value is cautionary and methodological. It supplies a reusable protocol (human ceiling, quality stratification, duration baseline) for reading automatic TTS evaluations of clean systems, rather than a better metric.

Claims

  • contradicts: Reference-free neural MOS predictors reliably identify the listener-preferred clip among high-quality TTS outputs.

    Evidence: The three predictors that lead on BVCC (SCOREQ, UTMOS, UTMOSv2) score 0.892 to 0.909 on artifact-rich BVCC but 0.502 to 0.528 on commercial TTS-HP, where a content-blind longer-clip baseline reaches 0.524 against the top predictor’s 0.528 and a human ceiling of 0.764. (§IV, Table II)

  • refines: Disagreement between automatic quality predictors and human preference on clean synthetic speech reflects low agreement among listeners on such pairs, not loss of predictor sensitivity to manipulations.

    Evidence: Predictors still prefer an original clean clip over its degraded copy at 0.77 to 0.998 for injected artifacts and 0.93 to 0.98 for pitch shifts and pause insertion, while split-half agreement among TTS-HP annotators is only 0.53 against 0.81 on BVCC. (§III-D, §IV-A, Table III)

  • supports: Combining complementary quality signals evaluates TTS preference better than any single selected metric, but the combination must be calibrated in domain.

    Evidence: A ridge composite fit on at most 1000 in-domain pairs beats the best single metric chosen from the same pairs on all six corpora (relative gains 1.5% to 6.6% on group-disjoint folds), whereas a leave-one-corpus-out composite falls to 0.52 on TTS-HP, below the strongest single predictor. (§V-D, §V-E, Table IV)

  • supports: Optimizing a TTS model against a single automatic quality score as an RL reward induces reward hacking that degrades quality under independent automatic and human judgment.

    Evidence: GSPO with a SCOREQ reward drives SCOREQ near its optimum while held-out UTMOS-v1 falls from 4.51 to 1.26, human Elo falls from 2098 ± 101 to 1675 ± 115, and outputs degenerate into padding; a KL penalty of 0.1 does not prevent it. (§VI-B, Table V)

  • supports: A multi-signal reward that pairs an intelligibility anchor with naturalness predictors from different lineages resists reward hacking in TTS policy optimization.

    Evidence: An unfitted equal-weight reward of WER, Distill-MOS and UTMOSv2 keeps held-out UTMOS-v1 at 4.56 and Elo at 2125 ± 95, within the baseline intervals, while UTMOSv2 optimized alone lowers held-out UTMOS-v1 to 4.30 and Elo to 1915 ± 109. (§VI-C, Table V)

Limitations and Open Questions

  • The evidence is drawn from English-heavy corpora; the authors leave broader confirmation to future work (§VII).
  • The reward study uses one policy (Qwen3-TTS-12Hz, 0.6B), one algorithm (GSPO), 300 steps, and one fixed reference voice. The authors call it an existence proof of exploitability. DNSMOS and Distill-MOS rewards also held quality on their own, so which single scores are safe is not predictable from the score itself (§VI-B).
  • The human listening test is small (30 utterances, 30 raters, 300 comparisons), and several Elo intervals overlap the baseline.
  • The human ceiling can be estimated only for BVCC, SOMOS and TTS-HP; it is not reported for SingMOS, SpeechJudge or TTS-Arena (Table II).
  • On TTS-HP, the composite’s gain over the best single metric lies within fold-to-fold spread (§V-A), and the commercial systems are near-indistinguishable in aggregate (vote share 0.49 to 0.51, §III-D). How much preference signal clean commercial TTS pairs contain remains open.
  • No code or data release is stated in the paper.

Wiki Connections

  • Evaluation Metrics — Benchmarks 13 reference-free predictor families plus prosody and signal-processing features against human pairwise preference, stratified by audio quality and read against a human ceiling.
  • Subjective Evaluation — Builds its analysis on existing crowd ratings of TTS output, measures their reliability ceiling, and runs its own pairwise listening test with Elo ratings on RL-trained outputs.
  • RLHF Speech — Shows single-score GSPO rewards on a TTS policy can be gamed, while an equal-weight multi-signal reward holds quality.
  • Autoregressive Codec TTS — Post-trains a 12.5 Hz codec language model TTS with metric-based rewards and documents padding as a reward-hacking failure mode.
  • UTMOS — Evaluated as a predictor and used, as UTMOS-v1, as the held-out judge that no reward run optimizes.
  • SpeechJudge — Closest concurrent work; its human-rated corpus is one of the six benchmarks, and its trained judge is evaluated and found near chance on TTS-HP.
  • Qwen3-TTS — The 12Hz 0.6B model serves as the policy in every reward stress-test run.
  • Qwen2-Audio — Used untuned as an audio language model judge and found to carry no preference signal on clean pairs.
  • From Scores to Preferences — Related reward-model benchmark that recasts MOS datasets as preference data, cited as a parallel strand.
  • Objective Metrics for Neural Audio Codecs — Cited for the finding that SCOREQ and PESQ agree closely with MUSHRA on codecs, the regime where this paper finds predictors still work.
  • TTSDS2 — Considered and excluded from the benchmark because its systems sit at a low-artifact midpoint of the quality gradient.
  • No Verifiable Reward for Prosody — Cited as speech-specific preference optimization that uses learned scores as training signals.