arXiv · 2025 · Preprint
Lanzendörfer et al. (ETH Zurich) · → Paper · Demo: ? · Code: ?
Runs a MUSHRA listening test on neural-audio-codec outputs and correlates the results against a wide battery of objective quality metrics to determine which ones reliably predict human perceptual judgments of codec-reconstructed speech.
Problem
Neural audio codecs (NACs) are increasingly used as the compression backbone for generative speech and audio models, but assessing their perceptual quality still relies on either costly, slow human listening tests or objective metrics whose reliability for this specific use case is largely unverified. Prior work on objective perceptual audio quality measures found that most such metrics are domain-dependent, performing well only within the application domain they were designed and validated for (traditional, non-neural coding and source separation). Whether that domain dependence also holds for reconstructions from modern NACs, which introduce different artifact types than classical codecs, had not been directly tested.
Method
The authors evaluate six codec/vocoder systems spanning the sub-10 kbps range: EnCodec (24 kHz, 6 kbps), Multi-Band Diffusion and Vocos (two vocoders that reconstruct audio from EnCodec’s latent space), DAC (44.1 kHz, 7.74 kbps), SNAC (44.1 kHz, 2.6 kbps), and Mimi (24 kHz speech compressed to 4.4 kbps, producing joint acoustic and semantic tokens). Test material is drawn from ODAQ: 11 clean speech samples and 11 combined speech-plus-background-audio samples, resampled to 48 kHz and processed per-channel since none of the evaluated systems natively support stereo.
Each sample is encoded and decoded by all six systems, then rated in a double-blind MUSHRA listening test with a hidden reference and two low-pass anchors (3.5 kHz and 7 kHz). Participants were recruited via a crowd-sourced MUSHRA tool; after applying the standard MUSHRA reference-rating filter (removing raters who scored the hidden reference below 90 at least twice), 11 participants remained for the speech-only condition and 17 for the combined-audio condition. In parallel, the same reconstructions are scored with a battery of eighteen objective metrics spanning distortion-based measures (SNR, SI-SNR, SDR, SI-SDR, SA-SDR and their variants), perceptual measures (PESQ, PEAQ, the 2f-Model), intelligibility (STOI), spectrogram similarity (ViSQOL, computed separately for speech and general audio), and non-intrusive/reference-free MOS predictors (DNSMOS, NISQA, NORESQA, NORESQA-MOS, WARP-Q, SCOREQ). Pearson’s ρ and Kendall’s τ are computed between each metric’s per-sample scores and the mean MUSHRA ratings, separately for the speech-only and combined-audio conditions.
Key Results
In the MUSHRA test, DAC consistently rated highest among the six systems, occasionally judged indistinguishable from the hidden reference by experienced listeners in the speech-only condition; ratings were uniformly lower in the combined-audio condition than in the speech-only condition. Vocos substantially outperformed EnCodec on speech-only audio but only matched it on combined audio, consistent with Vocos being trained predominantly on clean speech.
For correlation with the subjective scores, SCOREQ achieved the highest Pearson correlation on speech-only audio (ρ = 0.937), while PESQ, a metric from 2001, achieved the highest correlation on combined audio (ρ = 0.903) and was close to the top on speech-only audio (ρ = 0.886), outperforming most metrics developed decades later (Table 1, Table 2). PEAQ, DNSMOS, NISQA, and ViSQOL-Audio showed weak correlations in both conditions (ρ below roughly 0.6, with DNSMOS and NORESQA-MOS approaching zero on combined audio), indicating they do not reliably track NAC-introduced distortions. ViSQOL-Speech dropped sharply from ρ = 0.85 on speech-only audio to ρ = 0.593 on combined audio, and WARP-Q and NORESQA-MOS showed a similar pattern of strong speech-only performance that degraded on mixed content. Simple signal-level distortion metrics (SNR, SDR, SI-SDR) tracked MUSHRA scores about as well as several perceptually-motivated metrics, and in some comparisons outperformed them.
Novelty Assessment
The contribution is empirical rather than methodological: no new metric, model, or listening-test protocol is proposed. The novelty lies in applying an existing evaluation methodology (Torcoli et al.’s domain-dependence framework, extended with a fresh MUSHRA study) to a set of NACs and vocoders that had not previously been jointly benchmarked against this metric battery, and in surfacing a counterintuitive finding (an older, non-neural metric outperforming several purpose-built or non-intrusive MOS predictors on NAC-reconstructed audio).
Field Significance
moderate — This paper does not introduce new methods but provides a direct, apples-to-apples correlation study between eighteen objective metrics and fresh MUSHRA human ratings specifically for modern neural audio codecs, a combination not previously reported at this scope. It gives practitioners evaluating NACs for speech generation pipelines a concrete basis for choosing which automatic metric to trust, and which ones (PEAQ, DNSMOS, NISQA, ViSQOL-Audio) to treat with caution for this use case.
Claims
- supports: Signal-processing-era perceptual metrics can correlate with human quality judgments on modern neural-codec-reconstructed speech as well as, or better than, metrics specifically designed for neural or non-intrusive quality prediction.
Evidence: PESQ, proposed in 2001, achieved the highest Pearson correlation with MUSHRA scores on combined audio (ρ = 0.903) and the second-highest on speech-only audio (ρ = 0.886), outperforming DNSMOS, NISQA, NORESQA-MOS, and ViSQOL in both conditions. (§3.2, Table 1, Table 2)
- complicates: Objective quality metrics validated on clean or narrowly-scoped speech content do not necessarily generalize to mixed audio scenes containing background sound.
Evidence: ViSQOL-Speech’s Pearson correlation with MUSHRA dropped from ρ = 0.85 on speech-only samples to ρ = 0.593 on combined speech-plus-background samples, and several other speech-only-strong metrics (WARP-Q, NORESQA-MOS) showed the same degradation pattern. (§3.2, Table 1, Table 2)
- complicates: Non-intrusive, reference-free MOS predictors can fail to track quality differences introduced specifically by neural audio codec compression, even though they are designed for general speech-quality prediction.
Evidence: DNSMOS correlated at ρ = 0.683 with MUSHRA on speech-only audio and effectively at chance (ρ = -0.073) on combined audio; NISQA and PEAQ likewise showed weak correlations (ρ ≤ 0.6) in both conditions. (§3.2, Table 1, Table 2)
- refines: Simple signal-fidelity metrics (SNR, SDR and their scale-invariant variants) remain informative proxies for perceptual quality even when compared against metrics explicitly modeling human auditory perception.
Evidence: SNR, SDR, SI-SDR, and SA-SDR achieved Pearson correlations in the 0.7-0.77 range on both test conditions, comparable to or exceeding several perceptually-motivated metrics such as ViSQOL-Audio, DNSMOS, and PEAQ. (§3.2, Table 1, Table 2)
Limitations and Open Questions
The listening test used a small stimulus set (11 clean speech and 11 combined-audio samples from ODAQ) and a modest, crowd-sourced participant pool (11 and 17 raters after MUSHRA-protocol filtering, with an 18% rater failure rate exceeding the protocol's 15% recommendation), which limits the statistical power of the per-metric correlation estimates.
The six evaluated systems are described by the authors as far from exhaustive relative to the fast-growing space of neural speech codecs, and all operate below roughly 8 kbps with none of the systems natively supporting stereo, so channel-separated encoding may itself introduce artifacts not present in genuinely joint-stereo codecs. The study is also limited to a single listening-test protocol (MUSHRA) and does not examine whether the same metric rankings hold under other subjective paradigms (e.g., ABX, MOS-based ACR tests) or at bitrates outside the sub-10 kbps range studied here.
Wiki Connections
- Neural Audio Codec — evaluates six representative neural audio codecs and vocoders (EnCodec, Multi-Band Diffusion, Vocos, DAC, SNAC, Mimi) as the subjects of the correlation study, providing perceptual-quality validation data for this class of systems.
- Evaluation Metrics — systematically benchmarks eighteen objective speech and audio quality metrics against human ratings, directly informing which metrics are reliable choices for evaluating codec-based generative speech systems.
- Subjective Evaluation — conducts an original double-blind MUSHRA listening test with human raters to establish the ground truth against which objective metrics are validated.