arXiv · 2024 · Preprint
Mousavi et al. · → Paper · Demo: ? · Code: ✓
DASB introduces a comprehensive benchmark for evaluating discrete audio tokenizers directly on downstream tasks, without relying on waveform reconstruction, across 11+ discriminative and generative tasks spanning speech, music, and general audio.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
DASB addresses a systematic gap in the evaluation of discrete audio tokens: prior benchmarks measured tokenizer quality primarily through waveform reconstruction metrics, which can mask deficiencies because a strong decoder may compensate for poor token representations. DASB instead evaluates semantic, acoustic, and hybrid tokenizers (EnCodec, DAC, SpeechTokenizer, Mimi, Discrete HuBERT/WavLM/Wav2Vec2, WavTokenizer, SQ-Codec) by training lightweight downstream models directly on token embeddings across tasks including ASR, speaker verification, emotion recognition, TTS, speech enhancement, and source separation. The benchmark reveals a consistent performance gap between discrete and continuous representations, with semantic tokens outperforming acoustic tokens on most discriminative tasks while acoustic tokens better preserve speaker identity, providing TTS researchers with structured evidence for tokenizer selection decisions. DASB also provides ablation results on the number of codebooks and bitrate, showing that medium bitrate settings generally outperform higher ones on downstream tasks.
Wiki Connections
- neural-codec — DASB benchmarks acoustic tokenizers (EnCodec, DAC, Mimi, WavTokenizer) as one of the two major token families evaluated across downstream tasks
- self-supervised-speech — DASB benchmarks semantic tokenizers derived from self-supervised models (HuBERT, WavLM, Wav2Vec2) as the competing token family
- evaluation-metrics — DASB introduces a task-based evaluation protocol for discrete audio tokens that complements reconstruction-quality metrics such as PESQ and STOI
- autoregressive-codec-tts — TTS is one of the 11+ downstream tasks in the DASB evaluation suite, providing direct evidence for token choice in codec-based TTS systems
- 2504.08528 (SLM Landscape Survey) — cited as a related survey of spoken language models
- 2311.07919 (Qwen-Audio) — cited as an audio-language model baseline evaluated under the benchmark
- 2410.00037 (Moshi) — cited as a speech LM whose Mimi codec is evaluated in DASB
- 2312.11805 (Gemini) — cited as a large multimodal baseline in the evaluation context
- 2502.06490 (Discrete Speech Tokens Review) — cited as a concurrent survey on discrete speech token evaluation
- 2106.04624 (SpeechBrain) — toolkit used for implementing downstream task training recipes in DASB
- 2301.02111 (VALL-E) — cited as a key autoregressive codec TTS system motivating the benchmark’s TTS task
- 2402.13236 (Codec-SUPERB) — cited as a related codec evaluation benchmark that DASB extends and contrasts with
- 2409.05377 (BigCodec) — one of the acoustic tokenizers evaluated in the benchmark
- 2408.16532, 2403.03100, 2309.15505, 2212.04356, 2411.18803, 2305.02765 — additional tokenizers or baselines evaluated in the DASB suite