DM-Codec: Distilling Multimodal Representations for Speech Tokenization

EMNLP · 2025 · Conference

Md Mubtasim Ahasan et al. · → Paper · Demo: ? · Code: ✓

Introduces a speech tokenizer that distills contextual representations from a language model, in addition to the acoustic and semantic representations existing codecs already capture, into the discrete tokens of a residual-vector-quantized codec.

Problem

Discrete speech tokenizers generally fall into two families: acoustic tokens from neural audio codecs (e.g., EnCodec, DAC) that reconstruct waveforms faithfully but carry little linguistic structure, and semantic tokens distilled from self-supervised speech models (e.g., HuBERT) that capture phonetic content but sacrifice acoustic fidelity. Prior unification efforts (SpeechTokenizer, FACodec) combine acoustic and semantic tokens but leave out contextual information, i.e., the meaning a token carries given its surrounding linguistic context, which language models learn well but codecs do not. The authors show empirically that omitting contextual representations leads to elevated Word Error Rate (WER) and Word Information Lost (WIL) when reconstructed speech is transcribed, indicating that existing tokenizers lose information needed to recover accurate linguistic content.

Method

DM-Codec builds on a standard RVQ-GAN backbone (encoder, decoder, Residual Vector Quantizer with 8 quantization layers, codebook size 1024, 50 Hz frame rate), following the design of EnCodec and SpeechTokenizer, with a multi-discriminator setup (MSD, MPD, MS-STFT) adapted from HiFi-Codec and HiFi-GAN. The paper’s contribution is not the codec backbone itself but a set of distillation strategies applied only during training (removed at inference, so they add no inference-time cost or parameters):

• LM-guided distillation: raw speech is transcribed to text via an ASR model, the text is passed through a pretrained language model (BERT), and the layer-averaged hidden representations are distilled into the first RVQ layer’s quantized output via a dimension-wise cosine-similarity loss that does not require temporal alignment between text tokens and RVQ frames.
• Combined LM+SM-guided distillation: the LM-guided loss is jointly optimized alongside a symmetric distillation loss from a pretrained self-supervised speech model (HuBERT), averaged across all 8 RVQ layers, so that the quantized tokens jointly encode acoustic, semantic, and contextual information.
• [CLS]-token-guided distillation: an alternative that distills only the sequence-level [CLS] token representation from the LM (repeated across the RVQ sequence length), avoiding any need for fine-grained temporal alignment.

Training combines these distillation losses with the standard RVQ-GAN objective (time- and frequency-domain reconstruction, adversarial, feature-matching, and commitment losses). The codec is trained on LibriSpeech-100h clean speech. To demonstrate downstream generalizability, the authors also build DM-Codec-TTS, a VALL-E-style two-stage (autoregressive + non-autoregressive transformer decoder) zero-shot TTS system that predicts the DM-Codec-quantized tokens conditioned on a phoneme sequence and a 3-second acoustic prompt. DM-Codec-TTS is trained on LibriHeavy (50k hours) and a smaller variant on LibriTTS (585 hours).

Key Results

On LibriSpeech test-clean reconstruction, the combined LM+SM-guided variant, DM-Codec (LM+SM), achieves the best results among all compared tokenizers: WER 4.05 and WIL 6.61 (vs. SpeechTokenizer 4.49/7.10, FACodec 4.68/7.33, EnCodec 4.53/7.17), and the best speech-quality scores (ViSQOL 3.26, MOS 3.72, UTMOS 3.52) (Table 1). The LM-only and [CLS]-token variants also outperform most baselines but trail the combined variant. An ablated “DM-Codec (Baseline)” without any distillation performs worse than all baselines on WER/WIL, and an “SM Baseline” (semantic-only distillation) underperforms DM-Codec (LM+SM), which the authors use to argue for the specific contribution of contextual distillation.

For downstream zero-shot TTS, DM-Codec-TTS outperforms VALL-E and USLM (SpeechTokenizer’s TTS model) on both LibriSpeech (WER 5.08, MOS 3.70, SMOS 3.89) and VCTK (WER 3.58, MOS 3.78, SMOS 3.85) (Table 2). A smaller DM-Codec-TTS trained on LibriTTS (585h) still outperforms USLM (libri), a comparably-sized baseline, across all reported metrics despite the modest training budget.

Novelty Assessment

The codec backbone (RVQ-GAN encoder-decoder with multi-discriminator training) is not new; it follows EnCodec, SpeechTokenizer, and HiFi-Codec closely. The genuine contribution is the distillation recipe: this is, per the authors, the first attempt to distill acoustic, semantic, and contextual representations into a single codec’s tokens simultaneously, extending SpeechTokenizer’s semantic-only distillation with an LM-guided contextual signal and a temporal-alignment-free, dimension-wise loss formulation. The [CLS]-token-guided variant is a secondary but distinct contribution addressing the same alignment problem differently. DM-Codec-TTS is primarily an engineering integration (VALL-E-style AR/NAR TTS trained on DM-Codec tokens) used to validate that the improved tokenizer transfers to a downstream generation task, rather than a novel TTS architecture in its own right.

Field Significance

moderate — This paper adds a concrete, training-only distillation mechanism to the neural codec toolbox, demonstrating that contextual (language-model) representations, not just acoustic and semantic ones, materially reduce transcription errors and improve reconstructed speech quality relative to established tokenizers on a shared LibriSpeech benchmark. It provides a reusable recipe (LM/SM-guided, alignment-free distillation) that other codec designs can adopt, and its downstream TTS results give initial evidence that better tokenizers translate into better zero-shot synthesis quality.

Claims

• supports: Distilling contextual representations from a pretrained language model into a neural codec’s quantized tokens, in addition to acoustic and semantic representations, reduces transcription error and improves perceived speech quality relative to codecs that omit contextual distillation.

  Evidence: DM-Codec (LM+SM) achieves WER 4.05 / WIL 6.61 and MOS 3.72, outperforming SpeechTokenizer (4.49/7.10, MOS 3.67), FACodec (4.68/7.33, MOS 3.70), and EnCodec (4.53/7.17, MOS 3.09) on the same LibriSpeech test-clean evaluation protocol. (§5.1, Table 1)

• supports: Contextual and semantic knowledge distillation can be applied to a residual vector quantizer’s output without requiring temporal alignment between text/speech-model tokens and quantized frames, by matching representations at the feature-dimension level via cosine similarity instead of the time-step level.

  Evidence: The distillation loss maximizes cosine similarity across the feature dimension D, aggregated over all time steps, rather than performing per-timestep matching; a separate [CLS]-token variant that discards all temporal structure still improves WER/WIL over baselines without distillation. (§3.1, Appendix B)

• complicates: Semantic-only distillation from a self-supervised speech model, without a complementary contextual (language-model) signal, underperforms a combined contextual-plus-semantic approach on speech reconstruction quality.

  Evidence: The “SM Baseline” variant (HuBERT-only distillation) shows higher transcription error and lower quality scores than DM-Codec (LM+SM) in Table 1, which the authors attribute to the missing contextual signal. (§5.1, Discussion)

• supports: Improvements in a codec’s token representation quality can transfer to downstream zero-shot text-to-speech synthesis, improving both intelligibility and naturalness of generated speech relative to codecs lacking that representation.

  Evidence: DM-Codec-TTS, using DM-Codec (LM+SM) tokens, outperforms VALL-E and USLM (built on SpeechTokenizer tokens) in WER, MOS, and SMOS on both LibriSpeech and VCTK, and a version trained on the smaller LibriTTS corpus still beats a comparably-trained USLM (libri) baseline across all reported metrics. (§5.2, Table 2)

Limitations and Open Questions

All experiments are conducted on English read speech from LibriSpeech/LibriHeavy/LibriTTS and evaluated on LibriSpeech/VCTK; the paper does not test multilingual, code-switched, noisy, or conversational speech, so generalization beyond clean, read, monolingual English audio is not established.

The authors also note that their distillation relies solely on masked/encoder-style language models (BERT) for the contextual signal, leaving decoder-based LLMs unexplored as a contextual teacher. The reported gains over baselines, while consistent, are incremental (roughly 4-13% relative reduction in WER/WIL and single-digit percentage gains in quality scores) rather than a step change, and the DM-Codec-TTS comparisons partly rely on VALL-E and USLM results taken from prior papers rather than fully reproduced under identical conditions for VALL-E.

Wiki Connections

• Neural Audio Codec — proposes a training-time distillation recipe that layers contextual (LM), semantic (SM), and acoustic supervision onto a standard RVQ-GAN codec without adding inference-time cost.
• Autoregressive Codec TTS — DM-Codec-TTS is a VALL-E-style AR+NAR codec language model built on top of the improved DM-Codec tokens to validate downstream transfer.
• Self-Supervised Speech — uses a pretrained HuBERT model as the semantic-representation teacher (M_SM) in the combined LM+SM distillation loss.
• Zero-Shot TTS — DM-Codec-TTS performs zero-shot speaker cloning via a short acoustic prompt, following the VALL-E problem formulation.
• SpeechTokenizer — DM-Codec’s continuous-representation distillation loss formulation and RVQ-GAN training setup directly follow SpeechTokenizer, which it also uses as its closest baseline and reproduces via official training code.
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (VALL-E) — DM-Codec-TTS’s problem formulation and AR/NAR training objective follow VALL-E, which is also used as a zero-shot TTS baseline.
• NaturalSpeech 3 — FACodec’s factorized-subspace disentanglement approach is compared against and used as a reconstruction baseline in Table 1.
• High Fidelity Neural Audio Compression (EnCodec) — DM-Codec’s encoder-decoder-RVQ backbone design is inspired by EnCodec, which is also used as a reconstruction baseline.
• BigCodec — used as a low-bitrate codec baseline for speech reconstruction quality in Table 1.
• Moshi — the Mimi codec from Moshi is used as a reconstruction baseline in Table 1.
• CosyVoice — cited as related work on retaining semantic information in discrete speech representations for both understanding and generation.
• FuseCodec — a follow-up work by the same authors that further develops semantic-contextual fusion and supervision for neural codecs.