arXiv · 2025 · Preprint
Deng et al. (Fudan University) · → Paper · Demo: ? · Code: ✓
CodecBench introduces a multi-domain evaluation framework that assesses audio codecs on both acoustic reconstruction fidelity and semantic information preservation, exposing a consistent trade-off between codecs optimised for signal quality and those designed to carry linguistic or paralinguistic content.
Problem
Existing audio codec benchmarks — most notably Codec-SUPERB — rely predominantly on clean, controlled datasets and a limited set of acoustic metrics. As audio codecs are increasingly embedded inside speech language models (SLMs), their requirements have expanded: a codec must now encode not just clean speech but multi-speaker dialogues, emotionally expressive audio, background noise, music, and general environmental sounds. Existing benchmarks do not test these conditions, and their semantic evaluation — when present — measures downstream task performance directly from reconstructed audio rather than probing what information the codec’s internal representations actually carry to a language model.
Method
CodecBench assembles 18 open-source datasets and one self-collected dataset across four audio domains: Speech (KeSpeech, LibriSpeech, Libri2Mix, MELD, CREMA-D, RAVDESS), Music (NSynth, GTZAN, Musical Instrument Chord Classification), Sound (Laughterscape, VocalSound, ESC-50, CatDog, Gunshot Triangulation), and General Audio (AudioSet, Air-Bench Chat, WavCaps Soundbible, Clotho-AQA, and a self-collected dataset of expressive multi-speaker audio from Bilibili). The self-collected set specifically targets codec stress conditions — quarrels, speech over background music — not covered by existing corpora.
Acoustic evaluation uses eight signal-level metrics: Mel Loss, PESQ (narrowband and wideband), Spectral Convergence, SDR, SI-SDR, Speaker Similarity (SPK-SIM), STOI, and ViSQOL. All audio is resampled to 16 kHz for fair cross-model comparison.
Semantic evaluation proceeds through two complementary embedding-based protocols. The first is an ASR probing task adapted from the SUPERB framework: codec quantised embeddings (upsampled to ≥50 Hz when needed for CTC alignment) are fed to a frozen two-layer bidirectional LSTM trained with CTC loss on LibriSpeech train-clean-100, and WER on dev-clean measures semantic alignment. The second is a classification task inspired by ARCH: codec embeddings are used to train linear classifiers on labelled subsets across eight datasets (emotion, music genre, instrument type, environmental sound), measuring accuracy as a proxy for paralinguistic content retention.
Fourteen codec configurations are evaluated: DAC (three configurations at 24 kHz and 44 kHz), MaskGCT Codec (two configurations), BigCodec, Mimi (two configurations), Stable-Codec (two configurations), X-Codec-2.0, FlowDec (two configurations), and the Baichuan-Audio tokenizer.
Key Results
At high bitrates, DAC-24k with 32 quantisers achieves the best overall acoustic performance (PESQ-NB 4.383, STOI 0.988 on Speech), though DAC-44k-9nq leads on SDR and SI-SDR, which the paper attributes to higher temporal resolution reducing alignment errors. FlowDec slightly outperforms DAC-24k-8nq on Music datasets despite identical codebook and bitrate configurations, suggesting flow-matching codecs handle non-speech audio more consistently. At low bitrates, BigCodec performs best on Speech while Mimi-8 leads on other domains; Stable-Codec consistently underperforms across all conditions.
The acoustic-semantic trade-off is the paper’s most informative finding. DAC and BigCodec, which rank highest acoustically, show significantly worse WER in the ASR probing task than Mimi, X-Codec-2.0, and the Baichuan-Audio tokenizer — a direct consequence of their training objective, which does not incorporate semantic information. X-Codec-2.0 presents an interesting inversion: its single-codebook, low-bitrate design yields strong WER but poor classification accuracy across music, emotion, and sound datasets, suggesting it preserves text-aligned content at the expense of paralinguistic features.
Note
WER comparisons in the ASR probing task use a model trained specifically for this benchmark and are not directly comparable to codec-vendor-reported ASR metrics; the relative rankings are informative but absolute values depend on the downstream LSTM architecture and training data.
Novelty Assessment
The contribution is primarily methodological and data-scale: CodecBench extends the codec evaluation surface beyond clean speech by aggregating diverse real-world domains and introducing an embedding-level semantic evaluation protocol. Neither the acoustic metrics nor the ASR probing methodology is original — both adapt prior work (SUPERB, ARCH) — but their combination in a single benchmark targeting SLM integration scenarios fills a genuine gap that Codec-SUPERB leaves open. The self-collected dataset of expressive multi-speaker audio is a modest but useful addition to the evaluation corpus. The key insight — that acoustic fidelity and semantic content preservation are distinct axes that no single codec optimises simultaneously — is reinforced by systematic evidence rather than being new.
Field Significance
Moderate — CodecBench makes a useful case that the field has been evaluating audio codecs on too narrow a data distribution and too few metrics for SLM-era use cases. The finding that acoustic and semantic quality constitute orthogonal design axes, surfaced with controlled evidence across 14 codec configurations, is a useful reference point for codec design choices. The work is primarily additive infrastructure rather than a paradigm shift.
Claims
- Acoustic reconstruction quality and semantic information retention are largely orthogonal objectives in neural audio codec design, and optimising for one does not reliably improve the other. (§4.3, Figure 2)
- Codecs trained without explicit semantic objectives achieve state-of-the-art signal reconstruction but exhibit substantially higher word error rates than semantically-trained codecs of comparable bitrate. (§4.3.1)
- Single-codebook, low-bitrate codecs that preserve textual semantic content tend to sacrifice paralinguistic information such as speaker emotion and music characteristics. (§4.3.2, Table 5)
- Codec performance on clean speech benchmarks does not generalise to noisier, more expressive, or musically rich audio domains, with performance gaps widening at low bitrates. (§4.2, Table 4)
- Flow-matching-based audio codecs achieve competitive perceptual quality on non-speech audio relative to traditional RVQ-GAN-based codecs at equivalent bitrate and codebook size. (§4.2)
Limitations and Open Questions
The benchmark’s semantic evaluation relies on two embedding-based proxy tasks — ASR probing and classification — that do not directly measure how codec representations affect generation quality or dialogue coherence in a deployed SLM. The authors acknowledge this, noting that semantic information is “inherently broad and multifaceted” and their current methods are “relatively simplistic.” The self-collected dataset of 400 entries is small and may not generalise beyond the specific Bilibili content sources. All audio is downsampled to 16 kHz for fair comparison, which disadvantages high-sample-rate codecs (DAC-44k, FlowDec-48k) and blurs the bitrate advantage those models provide at their native resolution. Token-based semantic evaluation was explored but proved impractical for large-codebook single-VQ models (Stable-Codec, X-Codec-2.0) given the embedding mapping difficulty on LibriSpeech alone.
Wiki Connections
- neural-codec
- evaluation-metrics
- spoken-language-model
- VALL-E — evaluated as a predecessor to the codec generation paradigm this benchmark targets
- SpeechTokenizer — codec with explicit semantic-acoustic disentanglement; evaluated in the benchmark
- MaskGCT — provides the MaskGCT Codec evaluated in acoustic and semantic tasks
- BigCodec — low-bitrate codec evaluated; leads on Speech at low bitrate
- Moshi — provides the Mimi codec evaluated across all benchmark domains
- Stable-Codec — low-bitrate codec configuration evaluated in benchmark
- LLaSA — provides X-Codec-2.0 evaluated for semantic content retention
- Baichuan-Audio — provides the Baichuan-Audio tokenizer evaluated in semantic tasks
- GLM-4-Voice — cited as speech-to-speech dialogue model motivating SLM-era codec requirements
- SpeechGPT — cited as early speech LM motivating need for semantic codec evaluation