arXiv · 2025 · Preprint
Bai, Lu & Guo (NYU Shanghai) · → Paper · Demo: ? · Code: ?
A DAC-based neural speech codec (SemDAC) that designates the first RVQ quantizer as a HuBERT-distilled semantic codebook and explicitly conditions the decoder on those semantic tokens via FiLM, achieving better perceptual quality and lower downstream ASR word error rate than DAC at less than half the bitrate.
Problem
Conventional neural speech codecs are optimized purely for waveform fidelity, allocating bits to preserve acoustic detail even when much of that detail (timbre and prosody variation around a phonetic backbone) could in principle be inferred from linguistic structure. This produces inefficient compression and hurts performance on downstream tasks like ASR. Self-supervised speech models such as HuBERT capture phonetic and semantic content at very low bitrates, but their tokens alone discard the fine-grained acoustic cues needed for waveform reconstruction. Prior codecs that do disentangle semantic and acoustic tokens (e.g. SpeechTokenizer, XCodec) treat the semantic tokens as auxiliary features extracted alongside the acoustic stream, without using them to actively guide how the decoder reconstructs the waveform.
Method
SemDAC builds directly on the DAC framework, retaining its encoder and discriminator designs, and modifies two things. First, rather than treating all RVQ quantizer layers uniformly, it designates the first quantizer as a dedicated semantic codebook: a lightweight projection head maps the first quantizer’s latent into the feature space of a frozen, pretrained HuBERT model (9th-layer features), and a Euclidean distillation loss aligns this latent with the corresponding HuBERT embedding at each timestep. Because semantic content requires fewer distinct codewords than fine acoustic detail, this semantic codebook uses only 256-512 entries versus 1024 for each subsequent acoustic quantizer, which models the residual acoustic detail not captured semantically. Second, and identified by the authors as the more important design choice, the decoder is explicitly conditioned on the semantic tokens via Feature-wise Linear Modulation (FiLM): semantic and acoustic codes are concatenated as decoder input, while a separate FiLM generator (a stack of convolutional layers) projects and upsamples the semantic latents into modulation parameters that scale and shift the decoder’s acoustic feature maps at a chosen injection point, rather than letting the decoder process semantic information only implicitly through concatenation. Through ablation, the authors find inserting this FiLM modulation between the pre-convolutional layer and the first decoder block (rather than deeper in the decoder, or at multiple points) is most effective, since early injection provides a phonetic scaffold that shapes every subsequent decoding stage. Training follows DAC’s standard multi-scale mel-spectrogram, adversarial, feature-matching, and codebook/commitment loss combination, with an added semantic distillation loss term.
Key Results
Across the tested bitrate range, SemDAC consistently outperforms both the traditional Opus codec and a retrained DAC baseline on PESQ, STOI, ViSQOL, Si-SNR, and Whisper-measured WER. Most notably, SemDAC at 0.95 kbps surpasses DAC at 2.5 kbps on every objective metric and achieves performance comparable to DAC at 3 kbps, roughly a 2.5-3x bitrate reduction for equivalent quality. WER for SemDAC approaches that of raw (uncompressed) audio (4.25%) once bitrate exceeds roughly 2 kbps, indicating near-transparent intelligibility at that point. An ablation on semantic codebook size finds 256-512 entries strike the best balance, with 128 entries substantially degrading quality and entries beyond 512 providing no further gain. The FiLM-placement ablation is the paper’s central empirical finding: a variant with the same HuBERT-distilled semantic codebook but without FiLM conditioning (semantic and acoustic codes simply concatenated into the decoder) performs only comparably to the plain DAC baseline, while adding FiLM conditioning at the earliest decoder injection point improves PESQ from 2.72 to 3.24 and WER from 4.83% to 4.20% at matched bitrate, isolating FiLM conditioning (not semantic distillation alone) as the source of SemDAC’s gains. Injecting FiLM modulation at multiple decoder positions simultaneously provides no additional benefit over the single earliest-position injection.
Novelty Assessment
Both underlying components, HuBERT-distilled semantic quantization and FiLM conditioning, are individually established techniques; SemDAC’s contribution is combining them specifically to have semantic tokens actively steer decoder reconstruction rather than serving as a passive auxiliary stream. The paper directly positions this against SpeechTokenizer and XCodec, which disentangle semantic and acoustic tokens but do not use the semantic stream to condition decoding, and its own ablation (the no-FiLM variant regressing to DAC-baseline performance despite still having semantic distillation) is designed specifically to isolate this distinction as the actual source of the reported gains, rather than merely claiming it.
Field Significance
moderate — SemDAC offers a focused, well-ablated architectural insight (semantic tokens are more useful as an active decoder-conditioning signal than as a passive auxiliary codebook) with a clean empirical demonstration on a standard benchmark (LibriSpeech, DAC/Opus baselines, PESQ/STOI/WER). Its validation is narrower than more comprehensive low-bitrate codec papers: a single training/evaluation corpus, no subjective listening test, and no multi-domain (music/general audio) robustness check, and the paper’s own claims are limited to reconstruction and ASR-proxy intelligibility rather than direct downstream speech-language-model integration.
Claims
- supports: Conditioning a codec’s decoder directly on semantic tokens, rather than merely distilling those tokens from a self-supervised teacher and passing them into the decoder as an undifferentiated input, is necessary to realize the quality benefits of semantic-aware speech compression.
Evidence: A SemDAC variant with the same HuBERT-distilled semantic quantizer but without FiLM decoder conditioning performs only comparably to the plain DAC baseline (PESQ 2.72 vs. DAC’s 2.74), while adding FiLM conditioning at the same bitrate raises PESQ to 3.24 and lowers WER from 4.83% to 4.20%. (§Experimental Results, Table 3)
- supports: A neural speech codec can achieve substantially better perceptual quality and lower downstream ASR word error rate than a strong RVQ baseline at less than half the bitrate, by allocating a small dedicated semantic codebook and using it to actively shape acoustic reconstruction rather than increasing overall bit allocation to acoustic detail.
Evidence: SemDAC at 0.95 kbps outperforms a retrained DAC baseline at 2.5 kbps across PESQ, STOI, ViSQOL, Si-SNR, and Whisper-measured WER, and achieves quality comparable to DAC at 3 kbps. (§Experimental Results, Table 1)
- complicates: The effectiveness of semantic-token decoder conditioning depends on where in the decoder the conditioning signal is injected, with earlier injection providing consistently larger gains than later or multi-point injection.
Evidence: FiLM conditioning inserted between the pre-convolutional layer and the first decoder block (F0) outperforms insertion later in the decoder (F1-F3), and combining multiple injection points (F0+F1, F0+F2, F0+F3) performs at best on par with the single earliest-position injection, never exceeding it. (§Experimental Results, Table 3)
- complicates: A semantic codebook’s usefulness saturates well below the codebook sizes typically used for pure acoustic quantization, so allocating additional codebook capacity to the semantic stream beyond a modest size yields no further benefit.
Evidence: Varying the semantic codebook size across 128, 256, 512, and 1024 entries shows 256-512 entries strike the best balance, with 128 substantially degrading quality and sizes beyond 512 (up to the 1024 used for acoustic quantizers) offering no further improvement. (§Experimental Results, Table 2)
Limitations and Open Questions
All experiments are conducted on a single training and evaluation corpus (a 360-hour subset of LibriSpeech), leaving cross-domain robustness (music, noisy or reverberant speech, other languages) and generalization to out-of-domain speech unexamined. Evaluation relies entirely on objective metrics and a Whisper-based ASR-WER proxy for intelligibility, with no subjective listening test to validate perceptual quality claims. The paper does not evaluate SemDAC’s tokens in an actual downstream speech-language-model or TTS pipeline, so its claimed benefit for such systems remains inferred from reconstruction and ASR-proxy metrics rather than directly demonstrated.
Wiki Connections
- Neural Codec — introduces an asymmetric RVQ design with a HuBERT-distilled semantic quantizer whose tokens explicitly condition the decoder via FiLM, rather than serving as a passive auxiliary codebook.
- Self-Supervised Speech Representations — distills its semantic quantizer’s codebook from frozen HuBERT features, using self-supervised phonetic representations as a training target for codec tokenization.
- SpeechTokenizer — directly contrasted as a prior semantic-acoustic disentanglement approach that, unlike SemDAC, does not use its semantic tokens to actively condition decoder reconstruction.