arXiv · 2025 · Preprint
Mousavi et al. (Concordia University) · → Paper · Demo: ✓ · Code: ?
A systematic taxonomy and multi-domain benchmark of discrete audio tokenizers that reveals no single tokenizer dominates across reconstruction, downstream tasks, and acoustic language modeling, providing unified comparative guidance across speech, music, and general audio.
Problem
The field of discrete audio tokenization has produced a proliferating variety of methods, but existing benchmarks evaluate tokenizers in isolation: some focus exclusively on reconstruction quality (Codec-SUPERB), others on downstream discriminative performance (DASB), and others on acoustic language modeling (SALMon). Crucially, all prior benchmarks constrain their comparison to a single audio domain (usually speech) and evaluate tokenizers trained under inconsistent conditions, making cross-system conclusions unreliable. The common binary classification of tokenizers as either “acoustic” or “semantic” also blurs important architectural distinctions, since acoustic tokenizers can encode semantic information and semantic tokenizers have been used effectively in generative tasks.
Method
The paper is structured around three studies: taxonomy, benchmark evaluation, and controlled ablation.
The taxonomy (Section 2) classifies tokenizers along five dimensions: quantization method (K-means, RVQ, SVQ, GVQ, FSQ, MSRVQ, CSRVQ, PQ), encoder-decoder architecture (CNN, CNN+RNN, CNN+T, T), training paradigm (separate post-training vs. joint end-to-end), streamability, and target domain. This supersedes the acoustic-vs.-semantic dichotomy by capturing which design choices actually differentiate systems.
The benchmark (Section 3) evaluates seven publicly available pre-trained tokenizers (EnCodec, DAC, SpeechTokenizer, Mimi, Discrete WavLM, SQ-Codec, WavTokenizer) under a consistent protocol across three domains. Reconstruction quality is assessed using Codec-SUPERB and VERSA on LibriSpeech, MUSDB, and AudioSet. Downstream performance uses the DASB benchmark with frozen tokenizer representations and lightweight task-specific heads across 12 tasks (ASR, speaker ID/verification, emotion recognition, keyword spotting, intent classification, speech enhancement, speech separation, music genre classification, music source separation, sound event classification, audio source separation). Acoustic language modeling is evaluated via SALMon and the ZeroSpeech benchmark, using a 357M Qwen-2.5-based SLM trained on LibriHeavy (56k hours). TTS is evaluated via VALL-E trained on LibriTTS.
The ablation study (Section 4) uses ESPnet-Codec with a controlled DAC backbone to isolate the effects of quantization method (RVQ, SVQ, FSQ), training domain (speech, audio, music, multi-domain), sampling rate (16 kHz vs. 44.1 kHz), and semantic distillation from SSL features.
Key Results
Reconstruction. Higher bitrates consistently improve signal reconstruction, but SDR and SI-SNR are unreliable for models not optimized with time-domain losses (DAC, WavTokenizer show poor SDR despite perceptually acceptable output). SQ-Codec at 3 kbps matches or exceeds DAC and Mimi at much higher bitrates on several speech metrics.
Downstream tasks. Discrete WavLM consistently leads on discriminative speech tasks due to strong phonetic preservation; DAC leads on speaker recognition tasks. Increasing codebook count (bitrate) often hurts downstream performance by adding redundancy. Continuous WavLM-large representations remain the ceiling, outperforming all discrete tokenizers across most tasks, with the gap widening in low-resource settings.
Speech language modeling. HuBERT is the strongest semantic tokenizer (sBLIMP, sWUGGY). Mimi with 100x semantic stream overweighting narrows the gap to HuBERT on sBLIMP (60.17% vs. 60.89%). Purely acoustic tokenizers (EnCodec, DAC) show near-chance performance on semantic tasks. No tokenizer achieves strong results on semantic-acoustic alignment.
TTS. Speech-specific EnCodec (Enc-S-24, trained on LibriTTS) achieves UTMOS 3.77 in a VALL-E setup. Discrete WavLM achieves the best text adherence (dWER 4.32). Acoustic tokenizers trained on multi-domain data underperform speech-specific or semantic tokenizers for TTS.
Ablation. RVQ outperforms SVQ and FSQ for reconstruction across most settings. Training domain alignment is the single largest determinant of performance: even balanced multi-domain training yields worse domain-specific performance than single-domain training. Higher sampling rate (44.1 kHz) improves RVQ but degrades FSQ performance. Semantic distillation helps signal reconstruction on some metrics but reduces cross-domain generalization.
Novelty Assessment
The paper’s contribution is primarily a rigorous, unified evaluation framework rather than a new architecture. The taxonomy is a genuine conceptual contribution: the five-axis classification captures design dimensions that the acoustic-vs.-semantic binary misses, particularly streamability and training paradigm. The benchmark’s novelty is in scope: it is the first to simultaneously evaluate reconstruction, downstream representation quality, and acoustic language modeling across speech, music, and general audio under consistent conditions with the same tokenizer set. The controlled ablation study (Section 4) adds quantitative evidence for long-suspected but previously uncontrolled factors: domain alignment, sampling rate, and quantization method choice. The TTS evaluation is a useful addition but uses VALL-E in a constrained training regime, so absolute numbers should not be read as representing state-of-the-art TTS capability.
Field Significance
Tip
High — This paper fills a genuine gap: prior benchmarks for discrete audio tokenizers were domain-specific and methodologically inconsistent, making comparative guidance difficult. By evaluating seven tokenizers across reconstruction, downstream tasks, and LM-based generation in a single controlled framework, it provides a reference comparison that can anchor future tokenizer development and selection decisions. The taxonomy also offers the field a more expressive classification vocabulary than the acoustic-vs.-semantic binary. The finding that no single tokenizer dominates across all criteria, and that continuous representations still outperform discrete ones on most discriminative tasks, is a useful calibration against overly optimistic views of tokenizer-based audio representation.
Claims
- No discrete audio tokenizer consistently outperforms others across reconstruction, downstream discriminative tasks, and acoustic language modeling; the optimal tokenizer is task- and domain-dependent. (§3.4, Figure 4)
- Increasing the number of codebooks improves signal reconstruction quality but often degrades downstream task performance by adding redundancy that burdens representation-level learning. (§3.2, §3.2 “Impact of Codebook Size”)
- Semantic distillation (aligning early RVQ layers with SSL features) improves phonetic content preservation in acoustic tokenizers but may reduce cross-domain generalization when the distillation source is speech-specific. (§4.2 “Distillation Effect”, Table 16)
- Domain alignment between tokenizer training data and evaluation domain is the dominant factor in discrete audio codec performance, outweighing quantization method or bitrate choices. (§4.2 “Data Domains”)
- Continuous speech representations (e.g., WavLM-large) consistently outperform all discrete tokenizers on discriminative tasks, with the gap widening in low-resource conditions. (§3.2, Table 7)
Limitations and Open Questions
Warning
The TTS and audio LM evaluations are conducted with constrained training budgets (VALL-E on LibriTTS only; 300M audio LM at half the original training compute), which limits the practical transferability of findings on TTS tokenizer ranking to large-scale production settings.
The acoustic LM evaluation uses a single architecture (Qwen-2.5 based, 357M) and a single dataset (LibriHeavy), so findings on which tokenizers support better SLMs may not generalise to other LM architectures or training scales. The ablation study (Section 4) is restricted to a DAC-backbone framework; FSQ and SVQ conclusions may not transfer to other encoder-decoder designs. Evaluation metrics for audio generation (FAD, KLD, CLAP) conflate vocoder quality with language model quality, making it difficult to attribute performance differences to the tokenizer’s representational properties versus its decoder quality. The paper does not evaluate streaming tokenizers under actual latency constraints, limiting guidance for real-time deployment. Trustworthiness considerations (voice deepfakes, bias) are raised as open concerns but not empirically evaluated.
Wiki Connections
- neural-codec — provides the most comprehensive cross-tokenizer benchmark to date, comparing reconstruction quality, downstream representation utility, and LM-based generation for seven major codecs.
- autoregressive-codec-tts — includes a VALL-E-based TTS evaluation showing that speech-specific and semantic tokenizers outperform multi-domain acoustic tokenizers in constrained TTS regimes.
- self-supervised-speech — benchmarks SSL-derived tokenizers (Discrete WavLM, Mimi, SpeechTokenizer) against acoustic codecs, finding SSL-derived representations lead on semantic and discriminative tasks.
- spoken-language-model — evaluates how tokenizer choice affects SLM quality on SALMon and ZeroSpeech benchmarks, finding no tokenizer dominates across both semantic and acoustic LM criteria.
- evaluation-metrics — introduces a unified evaluation pipeline spanning reconstruction (Codec-SUPERB, VERSA), downstream (DASB), and LM evaluation (SALMon, ZeroSpeech) across three audio domains.
- disentanglement — surveys and taxonomises disentanglement-based auxiliary objectives in tokenizer training (speaker-content separation, semantic distillation), placing FACodec, LSCodec, and TiCodec in a unified framework.
- 2410.00037 (Moshi) — Mimi (used in Moshi) is one of the seven benchmarked tokenizers; the study provides controlled comparisons of Mimi against other codecs not available in the original Moshi paper.
- 2407.05407 (CosyVoice) — cited as an example of supervised semantic tokenization applied to multilingual zero-shot TTS; provides architectural context for the supervised semantic tokenization category in the taxonomy.
- 2412.10117 (CosyVoice 2) — cited alongside CosyVoice as a scalable streaming TTS system using supervised semantic tokens; represents the downstream application context for speech tokenizer design.
- 2502.06490 (Recent Advances in Discrete Speech Tokens) — a parallel survey reviewed in the introduction; the paper explicitly positions itself as more comprehensive by covering multi-domain evaluation that this survey lacks.
- 2306.12925 — cited as an acoustic tokenizer included in the benchmark taxonomy (SpeechTokenizer, RVQ with semantic distillation).
- 2307.09288 — cited in the codec taxonomy (DAC, one of the primary benchmarked tokenizers).
- 2402.13236 — cited as a benchmark in the evaluation pipeline (DASB, used for downstream task evaluation).
- 2412.02612 — cited as part of the acoustic LM evaluation context (SALMon benchmark suite used in §3.3).