arXiv · 2025 · Preprint
Xue et al. · → Paper · Demo: ? · Code: ✓
Introduces HH-Codec, a single-quantizer neural codec reaching 24 tokens per second at 24 kHz audio through a specialized VQ space for spoken language modeling, an asymmetric audio-to-mel-to-audio decoder, and a two-phase progressive dual-supervision training strategy.
Problem
Multi-quantizer neural codecs generate 300–900 tokens per second, sequences 10–40x longer than equivalent text tokenized with byte-pair encoding. This mismatch in granularity complicates integration with large language model backbones in speech generation systems. Existing single-quantizer alternatives (WavTokenizer at 40–75 tokens/s, Single-Codec at 75 tokens/s) close some of the gap, but push quality well below usable thresholds at token rates below 30 per second. Four specific failure modes emerge at extreme compression: adversarial training collapse requiring special scheduling strategies, minimal benefit from larger training datasets, severe codebook underutilization (43% at 8192 entries), and a 63% UTMOS reduction when token rate drops below 30 per second. No prior single-quantizer codec had demonstrated competitive quality at token rates close to text LM granularity.
Method
HH-Codec follows an Audio-VQ-Mel-Audio pipeline organized around an asymmetric encoder-decoder. The encoder uses a convolutional stack with strides (8, 8, 4, 4) followed by BiLSTM layers and a 1D projection to a 512-dimensional embedding space, compressing 24 kHz waveforms to 24 tokens per second. Longer training windows (rather than standard one-second clips) are used to preserve word-level context.
The central novelty is SLM-VQ, a vector quantization space designed for spoken language models. It builds on SimVQ’s frozen codebook with a learnable MLP and replaces the standard straight-through gradient estimator with the rotation trick for improved gradient flow. During training, a two-layer residual VQ structure is used: both quantizers receive the VQ loss, but only the first-layer output is decoded. The second quantizer layer acts as a “virtual class” regularizer that improves the compactness and utilization of the first layer without increasing inference complexity. The first quantizer output is additionally supervised via HuBERT semantic distillation, aligning learned representations with pre-trained semantic features through a cosine similarity objective.
The decoder is intentionally heavier than the encoder. ConvNeXt blocks and attention layers reconstruct an intermediate mel-spectrogram from the quantized representation; BigVGAN then converts this mel-spectrogram to audio. Dual supervision operates at both stages: mel reconstruction loss is applied to the intermediate output and the final BigVGAN-synthesized waveform. Training follows a two-phase progressive strategy. In phase one, the BigVGAN decoder is frozen while the encoder and VQ converge using only feature matching, mel, and generator losses. Once reconstruction loss drops below a threshold (roughly 20 epochs), phase two unfreezes the full system for joint fine-tuning. Training uses 4 A100 GPUs, 30 epochs, and a learning rate of 1e-4.
Key Results
On LibriTTS test-clean, HH-Codec achieves UTMOS 3.61, STOI 0.89, and SPK-SIM 0.73 using a single quantizer at 0.3 kbps and 24 tokens/s. This exceeds WavTokenizer-Big (UTMOS 3.58 at 0.5 kbps, 40 tokens/s) and vastly outperforms SpeechTokenizer’s single-quantizer configuration at 0.75 kbps (UTMOS 1.26). DAC at 9 kbps with 9 quantizers scores UTMOS 4.03, indicating remaining headroom, but HH-Codec matches DAC’s 4-quantizer variant (UTMOS 3.41 at 4 kbps) while using one quantizer and 17x less bandwidth.
On LibriTTS test-other (noisier in-domain) and Seed-TTS-eval (out-of-domain), results are consistent: HH-Codec UTMOS 3.21 and 3.33 respectively, competitive with or better than WavTokenizer-Big (3.08 and 3.23) at a lower token rate. Human voice UTMOS baselines for these sets are 3.48, 4.05, and 3.57.
Ablations (Table 2) show that dual supervision and progressive training are the most critical components: removing either reduces UTMOS by more than 1.1 points and SPK-SIM by 0.31. SLM-VQ’s multi-layer training scheme provides 0.31 UTMOS improvement over Classic VQ and nearly doubles codebook utilization at large codebook sizes (98% vs. 56% at 8192 entries, Table 3). A downstream experiment combining HH-Codec with Qwen2.5-7B shows faster training loss reduction than WavTokenizer or CosyVoice 2 under identical training conditions.
Novelty Assessment
The compression level (24 tokens/s, single quantizer) is genuinely below prior work, and the diagnostic analysis of training failures at extreme compression rates is a useful empirical contribution in itself. However, every individual component draws from existing work: SimVQ provides the VQ framework, the rotation trick replaces the standard estimator, HuBERT distillation has been used in prior codecs, and BigVGAN is an established vocoder. The “virtual class” multi-layer training trick adapts an idea from computer vision rather than introducing a new principle. The contribution is in identifying which combination of these techniques resolves ultra-high-compression instability, not in the techniques themselves. Evaluation is thorough with fair comparisons using official open-source weights; the three test sets (in-domain noisy, in-domain clean, out-of-domain) provide reasonable coverage.
Field Significance
Moderate — HH-Codec advances the practical lower bound for single-quantizer codec token rates, demonstrating that 24 tokens/s is achievable with competitive quality through careful management of training instability. Its SLM-VQ design provides a concrete technique for reaching token rates closer to text LM granularity. The progressive training methodology and dual-supervision scheme address failure modes that are not yet well characterized in the codec literature.
Claims
-
supports: Single-quantizer neural codecs can match multi-quantizer systems in perceived speech quality at far lower bitrates when combined with stabilized VQ spaces and asymmetric decoder architectures.
Evidence: HH-Codec achieves UTMOS 3.61 on LibriTTS test-clean at 0.3 kbps with a single quantizer, competitive with DAC’s 4-quantizer configuration at 4 kbps (UTMOS 3.41) and markedly above SpeechTokenizer’s single-quantizer variant at 0.75 kbps (UTMOS 1.26). (§4.3, Table 1)
-
supports: Multi-layer VQ training with single-layer inference, where auxiliary quantizer layers act as regularizers, substantially improves codebook utilization in high-compression single-quantizer settings.
Evidence: SLM-VQ achieves 98% codebook utilization at 8192 entries versus 56% for Classic VQ and 92% for single-layer SLM-VQ, while improving UTMOS from 2.76 (Classic VQ) to 3.07 on LibriTTS test-other. (§4.4, Tables 2-3)
-
supports: Dual-domain supervision combining intermediate mel-spectrogram and final audio reconstruction objectives is critical for stable high-compression neural codec training.
Evidence: Reducing to single audio-domain supervision drops UTMOS from 3.07 to 1.85 and SPK-SIM from 0.64 to 0.33 on LibriTTS test-other, the largest degradation across all ablation variants. (§4.4, Table 2)
-
complicates: Standard adversarial codec training recipes break down at extreme compression ratios, requiring architectural and procedural modifications to avoid collapse.
Evidence: Below 0.3 kbps with existing methods, the paper documents adversarial training collapse, a 63% UTMOS drop below 30 tokens/s, 43% codebook utilization at 8192 entries, and minimal benefit from expanding training data, all addressed in HH-Codec through SLM-VQ and progressive training. (§1)
Limitations and Open Questions
HuBERT-based semantic distillation is trained on English, so multilingual performance of SLM-VQ is unknown. The downstream spoken language modeling experiment measures only training loss reduction rather than end-to-end TTS quality, leaving it unclear how the 24-token-per-second compression affects downstream synthesis intelligibility and naturalness. Training data conditions differ across compared baselines (WavTokenizer and DAC use larger or different datasets), which limits direct attribution of gains to architecture versus data. The ablation study is conducted on a subset of training data only (LibriTTS train-100/360, not the full training set including Emilia), which may underestimate some component contributions.
Wiki Connections
- Neural Audio Codec — introduces SLM-VQ, a VQ space specifically optimized for spoken language model integration at extreme single-quantizer compression ratios.
- Spoken Language Model — HH-Codec is designed to bring neural audio token rates closer to text LM granularity, enabling more efficient joint speech-text model training.
- Autoregressive Codec TTS — provides a codec with 24 tokens/s suitable as a drop-in discrete representation for autoregressive speech language models.
- Self-Supervised Speech — uses HuBERT features as a semantic distillation target during SLM-VQ training to preserve linguistic content at ultra-high compression.
- Seed-TTS (Seed-TTS) — used as an out-of-domain evaluation benchmark to test generalization of codec reconstruction quality.
- CosyVoice2 (CosyVoice 2) — cited as a supervised semantic tokenization approach and used as a downstream spoken language model training baseline comparison.
- VALL-E (VALL-E) — cited as an early approach to managing multi-quantizer codec complexity in autoregressive TTS via non-autoregressive prediction of residual codebooks.