arXiv · 2025 · Preprint
Wolfgang Mack et al. (Cisco Systems) · → Paper · Demo: ? · Code: ?
Benchmarks 45 objective speech-quality metrics against MUSHRA-1S listening-test scores across 17 neural-codec conditions, showing that neural-network-based metrics track human quality judgments best but saturate at high subjective-quality levels.
Problem
Objective speech-quality metrics are the fast, cheap substitute for human listening tests during codec development and operational monitoring, but classical intrusive metrics (PESQ, POLQA, WARP-Q) were designed and validated against DSP-era coding artifacts, not the generative distortions (temporal inconsistencies, perceptual hallucinations, timbre shifts) produced by neural audio codecs. Prior comparisons of objective-metric reliability for neural codecs were limited in the number of metrics and codec conditions considered, leaving practitioners without clear, evidence-based guidance on which metrics to trust when selecting or monitoring neural codecs, particularly ultra-low-bitrate systems.
Method
The paper does not propose a model; it proposes an evaluation protocol. The authors process 100 clean speech files from the low-resource audio coding (LRAC) challenge Track 1 blind test set through 17 codec conditions spanning 12 neural codecs (AudioDec, BigCodec, DAC, DAC Tiny, DualCodec, EnCodec, ESC, an internal codec, Mimi, SemantiCodec, SNAC, StableCodec) chosen to differ in network design, training, compute, latency, bitrate, and spectral bandwidth. Each processed file is scored by 45 objective metrics (both intrusive/reference-based and non-intrusive/reference-free) computed via the VERSA toolkit, and by human listeners using the MUSHRA-1S protocol, a single-condition variant of MUSHRA designed to scale to many conditions while retaining finer-grained discrimination than ACR/MOS tests. Listening tests are run at 24 kHz to match the sampling rate of the codecs under test, with an Opus-6kbps anchor and mean votes per file of approximately 7. For each objective metric, the authors compute Pearson, Spearman, and Kendall correlations between the metric’s 1,700 scores (17 conditions x 100 files) and the corresponding MUSHRA-1S scores, reporting all three to separately capture linear association, monotonic rank agreement, and tie-robust concordance. A secondary analysis varies the number of audio files used to compute per-condition mean scores and tracks the resulting confidence-interval width for two representative metrics (scoreq_ref, scoreq_nr) across a high-, medium-, and low-quality codec condition.
Key Results
Neural-network-based metrics dominate the correlation rankings: scoreq_ref (intrusive) achieves the highest overall Pearson correlation at 0.87, followed by utmos (0.82), sheet_ssqa (0.81), nomad (0.83), scoreq_nr (0.81), and audiobox aesthetics’ ce (0.80). The strongest classical baselines, warpq and pesq, reach only 0.73. Weakest correlations are reverberation-focused predictors (srmr, sigmos_reverb), F0-error metrics, and waveform-matching measures (se_sdr, se_sar, se_si_snr), consistent with neural codecs producing non-waveform-preserving, generative distortions rather than classical coding artifacts. For the two top non-intrusive metrics (utmos, scoreq_nr), Spearman and Kendall correlations drop substantially relative to Pearson, indicating ranking deficiencies not visible from Pearson alone. Confidence-interval analysis shows the intrusive scoreq_ref produces tighter confidence intervals than the non-intrusive scoreq_nr across sample sizes, most pronounced for the high-quality codec condition, and a discontinuity between sample sizes 10 and 20-30 for low- and medium-quality codecs suggests those conditions need more samples for reliable ranking. Condition-wise plots reveal that non-intrusive metrics (scoreq_nr, ce, sheet_ssqa, utmos) become nearly constant in the high-MUSHRA range (a saturation effect the authors attribute to the coarse 5-point ACR scale underlying their MOS training targets), while intrusive metrics like scoreq_ref remain near-linear. Conversely, in low-to-medium MUSHRA ranges, several metrics (most prominently pesq, utmosv2, and warpq) also produce near-identical scores across codecs that differ substantially in subjective quality, with pesq showing the widest such blind spot (~35 MUSHRA points).
Novelty Assessment
The contribution is empirical and methodological rather than architectural: no new metric, codec, or model is introduced. The paper’s value is in scale and protocol design, combining a broader set of objective metrics (45) and codec conditions (17, spanning 12 codecs with diverse designs and bitrates) than prior comparisons, and pairing them with a scalable subjective-testing methodology (MUSHRA-1S) specifically chosen to resolve fine-grained quality differences at the high-quality end of the scale, where coarser ACR-based tests struggle. The correlation findings themselves (neural metrics correlating better than classical ones, non-intrusive metrics saturating near ceiling) are consistent with prior small-scale observations cited by the authors rather than a new discovery, so the paper’s honest contribution is confirmatory evidence at a larger scale plus a reproducible protocol, not a conceptual breakthrough.
Field Significance
Moderate — the paper provides systematic, large-scale evidence for which objective metrics practitioners can trust when evaluating neural audio codecs on clean speech, and documents a specific measurement failure mode (non-intrusive metric saturation near ceiling) with a plausible mechanistic explanation tied to MOS-based training targets. Its contribution is confirmatory and infrastructural (a reusable evaluation protocol) rather than introducing new architectures or metrics.
Claims
- supports: Neural-network-based objective quality metrics correlate more strongly with human subjective judgments than classical DSP-based metrics when evaluating neural audio codec output.
Evidence: Across 1,700 data points from 17 codec conditions, scoreq_ref reaches a Pearson correlation of 0.87 with MUSHRA-1S scores and utmos, nomad, scoreq_nr, sheet_ssqa, and audiobox ce all exceed 0.80, while the strongest classical baselines (warpq, pesq) reach only 0.73; all metrics with correlation above 0.8 are neural-network-based. (§4)
- complicates: Non-intrusive objective metrics trained on MOS-based subjective ratings saturate near the top of the quality scale, weakening their ability to discriminate between near-transparent codec conditions.
Evidence: Spearman and Kendall correlations for utmos and scoreq_nr drop sharply relative to their Pearson correlation, and condition-wise plots show scoreq_nr, ce, sheet_ssqa, and utmos producing near-constant scores in the high-MUSHRA range (red-ellipse regions of Fig. 3), attributed to the coarse 5-point ACR scale underlying the MOS targets these metrics are trained on. (§4)
- refines: Reference-based (intrusive) objective metrics remain more statistically stable and discriminative than reference-free (non-intrusive) metrics for high-quality neural codec conditions, even though non-intrusive metrics are the only option when no reference signal is available.
Evidence: Confidence-interval analysis across a high-, medium-, and low-quality codec condition (DAC-8kbps, SemantiCodec-1.4kbps, EnCodec-1.5kbps) shows intrusive scoreq_ref producing consistently smaller confidence intervals than non-intrusive scoreq_nr across sample sizes, an effect most pronounced for the high-quality condition. (§4)
- complicates: Reference-based classical metrics can also fail to discriminate between codec conditions that differ substantially in perceived subjective quality, despite having access to the clean reference signal.
Evidence: pesq shows the widest observed blind spot among all evaluated metrics, mapping codecs spanning roughly 35 MUSHRA points to nearly identical objective scores in the low-to-medium quality range (green-ellipse regions of Fig. 3), a limitation the authors note is a previously known phenomenon for pesq. (§4)
Limitations and Open Questions
Warning
All results are obtained under clean speech conditions only; the authors explicitly leave analysis of noise and reverberation for future work, so the metric rankings and saturation findings here should not be assumed to transfer to noisy or reverberant deployment scenarios.
The study is restricted to 12 codecs and 17 conditions selected by the authors, and to a single test set of 100 files drawn from the LRAC challenge Track 1 blind set; generalization to other speech domains, languages, or codec families is untested. Sample-size guidance is derived from only three representative codec conditions and two metrics (scoreq_ref, scoreq_nr), leaving open how confidence-interval behavior generalizes across the full metric suite. The authors also note unexplained run-to-run variability for utmosv2, possibly from internal random frame selection not currently exposed by the toolkit used, which they flag rather than resolve.
Wiki Connections
- Neural Audio Codec — evaluates 12 neural audio codecs spanning diverse architectures, training regimes, and bitrates, and characterizes which objective metrics reliably capture their generative distortions.
- Evaluation Metrics — systematically benchmarks 45 objective speech-quality metrics against human listening scores, providing practical guidance on metric selection for neural codec evaluation.
- Subjective Evaluation — uses the MUSHRA-1S crowdsourced listening-test methodology as the ground truth against which all 45 objective metrics are correlated.