arXiv · 2025 · Preprint
Arijit Biswas et al. (Dolby) · → Paper · Demo: ? · Code: ?
Systematically compares Fréchet Audio Distance (FAD) and Maximum Mean Discrepancy (MMD) computed in neural audio codec embedding spaces against MUSHRA subjective scores, finding FAD consistently more reliable and codec fidelity a strong predictor of embedding quality for perceptual evaluation.
Problem
Reference-free, embedding-based distance metrics such as FAD and MMD are widely used to score generative audio systems, but the field lacks consensus on which distance measure and which embedding space best track human perception. Prior studies disagree on whether FAD or MMD is superior, and high-quality full-bandwidth subjective datasets specifically for generative audio are scarce, since existing sets are either bandlimited or contain lossy-codec artifacts. This makes it hard to validate a metric’s core assumption: that its embedding space actually reflects perceptual quality before it is trusted in a true zero-shot, reference-free setting.
Method
The paper isolates the embedding-quality question from the reference-free evaluation question by first testing FAD and MMD in a controlled, signal-to-signal setting: computing distances between a coded test signal and its uncoded reference, then correlating those distances with MUSHRA subjective scores from the mono MPEG USAC verification listening test (24 excerpts of speech, music, and mixed content, 66 listeners, evaluated across USAC, HE-AAC, and AMR-WB+ at 8-24 kb/s). Embeddings are drawn from the final encoder layer (prior to residual vector quantization) of three neural audio codecs of increasing fidelity: EnCodec (music-only, 48 kHz), the publicly released Descript Audio Codec (DAC) at 8 and 16 kb/s, and a new enhanced variant, DACe. DACe extends DAC’s training with up to 32 10-bit codebooks (~30 kb/s per channel), a diverse 720-hour, 48 kHz mono training set spanning 18 music genres, speech, and isolated instruments, plus 32 hours of real tonal audio and on-the-fly generated synthetic tonal excerpts (33% of each mini-batch) to target DAC’s known weakness on tonal material such as glockenspiel. DACe is trained with the same reconstruction-plus-adversarial objective as DAC, only with a smaller batch size (48 vs. 72). The same signal-to-signal correlation protocol is then repeated using popular general-purpose audio embeddings (CLAP, CLAP LAION audio- and music-trained variants, and OpenL3 at two Mel resolutions) for comparison. Prediction accuracy is measured with Pearson (linearity) and Spearman (monotonicity) correlation against the MUSHRA scores.
Key Results
Across all three neural codec embedding spaces, FAD correlates more strongly with MUSHRA scores than MMD, and correlation improves monotonically with codec fidelity: EnCodec embeddings are weakest (FAD Rp = 0.38, Rs = 0.66), DAC 16 kb/s embeddings are markedly better (Rp = 0.68, Rs = 0.81), and DACe embeddings improve further (Rp = 0.70, Rs = 0.82) (Table 1). A separate internal MUSHRA test confirms DACe subjectively outperforms DAC, particularly on tonal content and a panned speech excerpt, and that DAC in turn outperforms EnCodec. However, general-purpose embeddings trained with contrastive or self-supervised objectives on much larger datasets beat every codec embedding tested: FAD with CLAP LAION Music (CLAP-M) reaches Rp = 0.85, Rs = 0.88, and OpenL3-128M reaches Rp = 0.84, Rs = 0.84, both ahead of DACe. Fixed-bandwidth ablations for MMD show correlation peaking near σ = 100 but never matching the median-heuristic bandwidth used elsewhere in the paper, and correlation dropping more sharply for a too-small bandwidth than for a too-large one.
Novelty Assessment
The core evaluation methodology, FAD and MMD as embedding-distance metrics, is not new; both are established tools from the generative audio evaluation literature. The paper’s contribution is a systematic empirical comparison that the authors position as the first to jointly vary distance measure and codec fidelity in a controlled signal-to-signal setting, plus DACe, an incremental fidelity improvement to DAC via additional real and synthetic tonal training data rather than a structural architecture change. The comparison against CLAP and OpenL3 is a useful negative-adjacent result: it shows that the paper’s own proposed codec embeddings, despite the fidelity improvement, still lag the best general-purpose embeddings, which the authors attribute honestly to training objective and data-scale gaps rather than overstating DACe’s competitiveness.
Field Significance
Moderate — the paper provides controlled evidence, in a signal-to-signal setting rather than a true reference-free generative setting, that FAD is a more stable perceptual-quality predictor than MMD and that codec embedding fidelity matters for this use case. It documents where neural-codec embeddings currently fall short of contrastive/self-supervised general-purpose embeddings for this task, which is useful guidance for choosing evaluation metrics but does not itself validate the metrics on true generative (as opposed to lossy-coded) audio.
Claims
- supports: Fréchet-distance metrics computed on audio embeddings correlate more reliably with human perceptual quality judgments than kernel-based maximum mean discrepancy metrics.
Evidence: Across every embedding space tested (EnCodec, DAC, DACe, CLAP, CLAP LAION, OpenL3), FAD achieved equal or higher Pearson and Spearman correlation with MUSHRA scores than MMD on the same embeddings, e.g. DACe FAD Rp = 0.70 vs. DACe MMD Rp = 0.65. (§4.1, Table 1)
- supports: Higher-fidelity neural audio codec embeddings track human-perceived audio quality more accurately than lower-fidelity codec embeddings.
Evidence: Correlation with MUSHRA scores rose monotonically with codec quality: EnCodec (Rp = 0.38) < DAC 16 kb/s (Rp = 0.68) < DACe (Rp = 0.70) under FAD, mirrored by a separate subjective MUSHRA test confirming DACe > DAC > EnCodec. (§4.1, Table 1; §3, Figs. 1-2)
- complicates: Codec-derived embeddings, even from high-fidelity codecs, are not automatically competitive with embeddings trained for perceptual or semantic similarity when used for generative audio quality prediction.
Evidence: General-purpose CLAP LAION Music and OpenL3-128M embeddings outperformed the best neural codec embedding (DACe) in both Pearson and Spearman correlation with MUSHRA scores, attributed to codecs’ reconstruction-based training objective and roughly 10x less training data than the contrastively/self-supervised-trained alternatives. (§4.2)
- complicates: Kernel-based statistical distance metrics for audio evaluation are sensitive to bandwidth hyperparameter choice in a way that moment-based distances are not.
Evidence: Fixed MMD kernel bandwidths (σ = 1, 10, 100, 1000, 10000) tested on DACe embeddings peaked in correlation near σ = 100 but never reached the correlation achieved with the median-distance heuristic bandwidth, with performance degrading more sharply for undersized than oversized bandwidths. (§4.1)
Limitations and Open Questions
Warning
Validation is performed in a signal-to-signal, reference-based setting (coded audio vs. its own uncoded reference under classical lossy coding artifacts), not on true generative-model outputs. The paper explicitly motivates this as a proxy because full-bandwidth subjective datasets for generative audio are scarce, but this means the reported correlations characterize embedding sensitivity to coding artifacts rather than to the artifact types produced by generative models, and the paper does not close that gap.
The mono-only USAC test excludes stereo perceptual effects, which the authors note most public codecs and embedding models do not model anyway. DACe’s training data and exact architecture changes beyond codebook count and batch size are only partially disclosed (internal 720-hour dataset, internal MUSHRA test sets), limiting external reproducibility, and no code or demo release is mentioned for DACe itself.
Wiki Connections
- Neural Audio Codec — introduces DACe, an enhanced DAC-family codec trained with additional real and synthetic tonal data, and evaluates codec embeddings (EnCodec, DAC, DACe) as perceptual-quality feature extractors.
- Evaluation Metrics — systematically compares FAD and MMD as embedding-distance quality metrics and benchmarks them against popular general-purpose audio embeddings (CLAP, OpenL3).
- Subjective Evaluation — validates all embedding-distance metrics against MUSHRA listening test scores from the MPEG USAC verification test and an internal DAC-vs-DACe MUSHRA test.