arXiv · 2025 · Preprint
Jiaqi Li et al. (The Chinese University of Hong Kong, Shenzhen / Baidu) · → Paper · Demo: ✓ · Code: ✓
DualCodec proposes a dual-stream encoding framework that separately encodes speech through a frozen SSL model (for semantic content) and a waveform encoder (for acoustic fidelity), unifying both in a single end-to-end codec that achieves 12.5Hz frame rate with high audio quality and strong semantic preservation.
Problem
Existing neural audio codecs used as tokenizers in speech language models face a three-way tension: semantic richness, audio quality, and frame rate cannot all be optimised simultaneously. Distillation-based approaches (SpeechTokenizer, Mimi) improve semantic content in the first RVQ layer but still leave RVQ-1 tokens lacking sufficient pitch information, which causes failures in tonal languages like Mandarin. Furthermore, prior low-frame-rate codecs at 12.5Hz suffer from poor audio reconstruction quality, especially at low bitrates below 1 kbps. These gaps translate directly into higher word error rates and lower speaker similarity in downstream TTS systems.
Method
DualCodec addresses the three-way tension through a dual-stream architecture that directly assigns SSL features to the first RVQ layer rather than distilling into it. The two streams operate in parallel on the same speech input.
The SSL encoding stream takes 16kHz waveform, extracts the 16th layer of a frozen w2v-BERT-2.0 model (600M parameters), then passes it through average-pooling downsampling (4x for 12.5Hz, 2x for 25Hz), a 13M-parameter ResNet encoder, a VQ module with optional 1024 or 16384 codebook entries using L2-normalised projection (following DAC), and a ResNet decoder. The output is the RVQ-1 feature, which directly encodes rich pronunciation and pitch information.
The waveform encoding stream is a reproduced DAC architecture (CNN encoder with snake activation, strided convolution blocks, N-1 residual VQ layers) that operates on 24kHz audio. Its input to the RVQ module is the residual between the waveform encoder output and the RVQ-1 feature produced by the SSL stream. This residual formulation concentrates acoustic detail (timbre, noise, fine structure) in the waveform stream’s quantizers, while the SSL stream is responsible solely for content and prosody. To decode, RVQ-1 features are re-summed with the waveform RVQ codebook vectors and passed through the DAC decoder.
Training uses an end-to-end GAN objective combining multi-scale mel spectrogram loss, adversarial loss (MPD + MS-STFTD), feature matching loss, quantization loss, and an additional MSE SSL reconstruction loss on the downsampled SSL features. RVQ dropout is applied during training so that the model degrades gracefully from 0 to N-1 waveform quantizers.
During TTS training, both encoding streams are active to produce tokens. During TTS inference, only the codec decoder is needed, keeping deployment lightweight. The paper open-sources DualCodec in both 12.5Hz and 25Hz variants, and integrates it with VALL-E, AR+SoundStorm, FlattenAR, and MaskGCT TTS architectures, all trained on the 100K-hour multilingual Emilia dataset.
Key Results
Semantic content (Table 2, §4.2). On RVQ-1-only decoding evaluated with Whisper-large-v3 (EN) and Paraformer-zh (ZH) on Seed-TTS-Eval, dual encoding at 25Hz with 16384 codebooks (C3) achieves 2.98% EN WER and 2.91% ZH WER, near ground-truth levels (2.13/1.25%). The same setup with distillation (C2) gives 17.8/14.4%, confirming that direct encoding from SSL features substantially outperforms distillation. At 12.5Hz (D1), dual encoding achieves 6.94/6.36% WER, a large improvement over non-semantic baselines.
Audio reconstruction (Table 3, §4.3). At a standardised 75 tokens/second and approximately 0.75 kbps on LibriSpeech test-clean, DualCodec 12.5Hz with 16384+4096 codebooks (G5) achieves MUSHRA 88.2, UTMOS 4.11, PESQ-nb 3.11, and STOI 0.92, outperforming WavTokenizer-large (MUSHRA 81.0) and Mimi (MUSHRA 72.8). Dual encoding adds perceptual quality gains over plain DAC at the same frame rate (MUSHRA 79.5 vs 68.8 at 25Hz).
TTS performance (Table 4, §4.4). VALL-E paired with DualCodec 25Hz achieves 3.4% EN WER and 0.57 EN SIM on Seed-TTS-Eval, compared to 15.4/0.47 with SpeechTokenizer and 8.16/0.48 with Mimi. RTF at 0.30 on an A100 is slower than Mimi (0.16) due to the 25Hz model’s longer AR sequences, whereas DualCodec 12.5Hz matches Mimi’s RTF. In the broader Table 6 comparison against CosyVoice, MaskGCT, and AR+SoundStorm baselines using their original tokenizers, DualCodec-based systems are competitive, with DualCodec-MaskGCT 25Hz achieving 1.85% ZH WER.
Novelty Assessment
The core contribution is architectural: replacing SSL distillation in the first RVQ layer with direct VQ-VAE-style encoding of SSL features constitutes a structural departure from prior work (SpeechTokenizer, Mimi). The key insight, that distillation leaves semantic tokens lacking pitch information (critical for tonal languages), is supported by a targeted analysis of Mandarin WER from RVQ-1-only decoding. The residual formulation that subtracts RVQ-1 features before the waveform stream’s quantizers is a clean design choice that enforces content-acoustic separation without requiring an explicit disentanglement loss beyond the SSL reconstruction objective.
The paper’s low-frame-rate focus is also well-motivated and empirically grounded: the reproduced 12.5Hz DAC baseline itself outperforms official Encodec and SpeechTokenizer at the same token budget, establishing a strong baseline before introducing the dual-stream design. The codebook size ablation (1024 vs 16384 vs 4096 for RVQ-rest) is thorough and practically useful.
The work does not introduce a fundamentally new codec architecture type; it builds on DAC and VQ-VAE components. The novelty is specifically in the integration strategy (dual streams, residual design) and the accompanying analysis demonstrating semantic failure modes in distillation-based approaches for tonal languages. Open-sourcing both the codec and four derived TTS systems adds practical value.
Field Significance
Tip
High — DualCodec introduces a dual-stream encoding strategy that directly aligns with the AR+NAR decomposition of token-based TTS (RVQ-1 for content, RVQ-rest for acoustics), and demonstrates that direct SSL encoding outperforms distillation for both semantic quality and tonal language intelligibility. As one of the first open-source 12.5Hz codecs with strong audio reconstruction quality, it provides a practical foundation for efficiency-focused autoregressive speech generation systems.
Claims
- Directly encoding SSL features into the first RVQ layer preserves significantly more semantic content than distilling SSL representations into codec tokens, particularly for tonal languages where pitch information is phonemically critical. (§4.2, Table 2)
- Neural audio codecs operating at lower frame rates with more quantization layers achieve superior audio quality at equivalent bitrates compared to higher frame rate codecs with fewer quantization layers. (§4.3, Table 3)
- Semantic enhancement of the first RVQ layer improves downstream TTS speaker similarity as well as intelligibility, because higher semantic fidelity in RVQ-1 enables the waveform stream’s quantizers to focus on acoustic detail rather than recovering content information. (§4.4, Table 4)
- The quality gap between distillation-based and direct-encoding semantic codecs is substantially larger in Mandarin than in English, revealing a systematic limitation of distillation approaches for tonal languages. (§4.2, Table 2)
Limitations and Open Questions
Warning
The 12.5Hz DualCodec-based TTS systems consistently underperform their 25Hz counterparts on both WER and speaker similarity (Table 4, Table 6), indicating that the quality upper bound of the 12.5Hz variant is not yet competitive with the best open-source systems at 50Hz despite the frame rate reduction improving inference speed.
The paper evaluates TTS only on Seed-TTS-Eval; no subjective TTS listening tests are reported, so the MUSHRA gains in codec reconstruction may not fully translate to perceived TTS naturalness. Speaker similarity scores with DualCodec-VALLE remain below those of MaskGCT baselines that use separate semantic and acoustic tokenizers, suggesting the unified approach has not yet matched the best-performing two-stage pipeline design. The DualCodec encoder is substantially heavier than baselines (628M vs 38M for Mimi) due to the frozen w2v-BERT-2.0 model, increasing training-time compute, though the decoder remains lightweight for inference.
Wiki Connections
- neural-codec — DualCodec proposes a dual-stream encoding design as an alternative to distillation-based semantic enhancement, advancing the state of semantic codec design.
- autoregressive-codec-tts — the codec is validated end-to-end through VALL-E and AR+SoundStorm TTS systems, demonstrating that improved RVQ-1 semantics directly lower downstream TTS WER and raise speaker similarity.
- self-supervised-speech — the system’s SSL stream directly uses frozen w2v-BERT-2.0 features as the source of RVQ-1 content tokens, making SSL pre-training a core architectural component rather than a training auxiliary.
- zero-shot-tts — all evaluated TTS systems perform zero-shot voice cloning by conditioning on a prompt speech clip, with DualCodec improving both intelligibility and speaker similarity over prior codecs.
- multilingual-tts — evaluation on Seed-TTS-Eval covers both English and Mandarin, and the paper identifies tonal language intelligibility as a key axis where distillation-based codecs fail.
- 2411.18803 (TS3-Codec) — cited as a related low-bitrate single-codebook approach; DualCodec takes the complementary direction of multiple RVQ layers at low frame rate.
- 2301.02111 (VALL-E) — used as one of the primary TTS evaluation architectures to demonstrate DualCodec’s downstream benefit.
- 2406.05370 (VALL-E 2) — referenced as a prior codec-based TTS system the work contextualises itself against.
- 2305.09636 (SoundStorm) — the AR+SoundStorm TTS system is used as the second evaluation framework and shows the largest gains from DualCodec over SpeechTokenizer and Mimi.
- 2406.02430 (Seed-TTS) — provides the Seed-TTS-Eval benchmark used for all TTS evaluation in this paper.
- 2407.05407 (CosyVoice) — compared against in Table 6 as a strong multilingual TTS baseline using supervised semantic tokens and flow matching.
- 2502.06490 (Recent Advances in Discrete Speech Tokens survey) — cited for broader context on the discrete speech tokenization landscape that DualCodec contributes to.