Abstract

Evaluation determines which improvements we can see and which failures remain hidden. Across 346 reviewed papers, the main finding is consistent: no single automatic score measures all aspects of speech quality. A metric is useful only when it matches the target property, domain, level of detail, and intended use.

Current State

  • Automatic metrics are useful tools, but they do not replace targeted judgment. WER and CER can disagree with human intelligibility. Speaker embeddings can rank a synthetic voice above its reference. Automatic MOS predictors can reverse human rankings. PESQ, STOI, and MCD can also disagree with perceived quality (VoiceCraft, E2 TTS, 2508.00317, 2507.01611).
  • Speech quality has several separate dimensions. Naturalness, intelligibility, identity, prosody, diversity, emotion, latency, and robustness can change independently. An evaluation should use a small set of metrics chosen for the intended use. A long generic leaderboard does not solve the problem.
  • Clean, in-domain tests hide real-world failures. Performance gaps often grow under noise, reverberation, spontaneous speech, unfamiliar speakers, or domain shift (VoiceCraft, WildSpeech-Bench).
  • Text-only tests miss important parts of spoken interaction. Transcripts and text judges do not capture timing, emotion, speaker traits, background sound, or non-verbal behavior. This is a major limit for spoken agents and full-duplex systems (MTalk-Bench, Full-Duplex-Bench v1.5).
  • Learned judges work well only for some capabilities. SSL-based MOS predictors are useful and mature. LLM and audio-LM judges work better for instruction following and rubric checks than for fine prosody or raw-waveform cues (UTMOS, InstructTTSEval, 2509.19928).
  • New tests cover behavior and social impact. Recent work measures multi-turn reasoning, full-duplex timing, fairness, clinical suitability, and watermark robustness. Aggregate MOS or accuracy cannot expose these failures.
  • Q4 makes evaluation more specific to each intervention. Full-duplex tests separate fast response from false interruption. Fairness audits expose group differences hidden by averages. Watermark tests use realistic transformations, not only basic noise (Full-Duplex-Bench v1.5, A Watermark for Auto-Regressive Speech Generation Models).

Method Landscape

Evaluation layerBest useMain limit
Human listening and interaction testsNaturalness, preference, identity, emotion, and complete behaviorExpensive and sensitive to test design; scores can saturate near the quality ceiling
Task proxiesWER/CER for content, speaker embeddings for identity, and classifiers for emotionMeasures the proxy model as well as the speech
Signal and perceptual metricsFast checks of distortion, reconstruction, or intelligibilityRankings can change across architectures, sampling rates, and domains
Learned quality and LLM judgesLarge-scale screening and rubric scoringSensitive to domain shift, level of detail, and fine acoustic differences
Capability-specific benchmarksDialogue, timing, fairness, clinical use, and codec behaviorProtocols remain fragmented across papers

Key Trade-offs

  • Scale versus validity. Automatic scores support large tests. Human tests remain necessary for subtle, perceptual, or poorly modeled properties.
  • Common metrics versus task fit. MOS or WER helps comparison across papers. A metric designed for prosody, accent, identity, timing, or clinical speech can be more informative.
  • System ranking versus sample diagnosis. A predictor can rank complete systems well but fail on individual utterances (2507.06116).
  • Reconstruction versus downstream use. A codec can reconstruct speech well and still produce poor tokens for a generative model (2025.naacl-srw.6, 2506.10274).
  • Target-score gains versus hidden losses. Preference optimization, distillation, or alignment can improve one metric while reducing prosodic diversity (2507.14988, 2509.19928).
  • Standard protocols versus population validity. Shared tests improve comparison. Clinical, accented, low-resource, and assistive speech still need validation for the target population.

Open Questions

  • Can LLM and audio-LM judges measure fine prosody and non-verbal audio reliably?
  • What should complement or replace MOS and MUSHRA near the quality ceiling?
  • Can full-duplex and spoken-dialogue evaluation use shared protocols instead of separate lab pipelines?
  • How should codec tests combine reconstruction, semantic retention, noise robustness, and downstream generation?
  • Will fairness, accessibility, and multi-turn tests become standard parts of evaluation?

Go Deeper

Read Evaluation Metrics: In Depth for the detailed evidence on metric disagreement, learned judges, subjective-test limits, codec evaluation, spoken-agent benchmarks, and fit-for-purpose test design.

Scope

This assessment covers 346 integrated papers through Q4 2025. Many entries are TTS, voice conversion, codec, or spoken-agent papers that expose one evaluation problem rather than introduce a metric. The graph is broad, but it is not a formal meta-analysis. Citations are representative; complete evidence remains in wiki/_claims/evaluation-metrics.yaml.


This page is generated from wiki/_claims/evaluation-metrics.yaml (digest date: 2026-09-20). For complete structured claims and paper-level provenance, use the source YAML.