arXiv · 2025 · Preprint
Yuancheng Wang et al. (The Chinese University of Hong Kong, Shenzhen) · → Paper · Demo: ✓ · Code: ✓
TaDiCodec is a speech tokenizer that achieves an extreme compression rate of 6.25 Hz and 0.0875 kbps using a single-layer codebook, trained end-to-end via a flow-matching diffusion autoencoder that jointly learns quantization and reconstruction — without adversarial training, semantic distillation from SSL models, or multi-stage training pipelines.
Problem
Current speech tokenizers impose a difficult trade-off: compression-oriented codecs (EnCodec, DAC) operate at high frame rates (75–150 Hz) with multi-layer RVQ, making autoregressive language modeling over their tokens computationally expensive; semantic-enhanced tokenizers achieve lower rates but depend on pre-trained SSL models for distillation and require two-stage training that separates the VQ model from the diffusion decoder. The reconstruction-generation gap — where tokens that decode faithfully to speech fail to remain useful predictive targets under a language model — also remains poorly characterized and largely unaddressed by existing designs. No single-stage, distillation-free system had achieved token rates below 20 Hz while matching the reconstruction quality of heavier multi-codebook alternatives.
Method
TaDiCodec reframes the codec problem as an end-to-end diffusion autoencoder trained with a single flow-matching loss. The encoder is an 8-layer Llama-style bidirectional Transformer operating on mel-spectrogram frames; its output is downsampled and projected to a 14-dimensional latent, then quantized using Binary Spherical Quantization (BSQ) — an implicit-codebook scheme that projects encoder embeddings onto a unit hypersphere and binarises each dimension, yielding a codebook of size 2¹⁴ = 16,384 tokens without a learnable codebook or commitment loss. The 16-layer decoder, also a Transformer, uses a flow-matching objective to reconstruct the mel-spectrogram from the discrete token sequence.
The defining design choice is text-aware de-tokenization: the decoder receives the corresponding transcript concatenated with the token sequence along the time axis. In speech LM contexts — TTS, spoken dialogue — the target text is always available, so conditioning on it effectively offloads content information from the compressed token to the conditioning signal, enabling much more aggressive compression. A prompt mechanism further supplies a prefix segment of the mel-spectrogram as a noise-free conditioning signal during decoding, reducing the token’s burden to encode global speaker identity. Together, text guidance and prompt conditioning enable the system to operate at 6.25 Hz where text-free alternatives fail entirely (ablating text conditioning at 12.5 Hz pushes WER above 10%). The entire system is trained jointly with the diffusion loss alone; after convergence, the encoder and VQ are frozen and the decoder is fine-tuned for an additional 400K steps focused on reconstruction quality.
For downstream TTS, an autoregressive LM (TaDiCodec-AR) or masked generative model (TaDiCodec-MGM) predicts TaDiCodec tokens from text, then the same text-conditioned diffusion decoder reconstructs the waveform. AR models are initialised from pretrained LLMs (Qwen2.5 and Phi-3.5) and extended with a speech token vocabulary; variants range from 0.2B to 4B parameters. The low token rate (6.25 Hz vs. 25–50 Hz for typical AR TTS) means each second of speech is predicted as only 6–7 tokens, drastically shortening the generation sequences.
Key Results
On SeedTTS test-en reconstruction (Table 1), TaDiCodec achieves WER 3.02 (2.73 with decoder fine-tuning), SIM 0.67 (0.69), and UTMOS 3.68 (3.73) at 0.0875 kbps — the lowest bitrate of any reported system. The nearest competitor in compression, Ints Tokenizer, operates at 0.175 kbps (2× higher), with worse WER (7.14) and UTMOS (3.37). Among single-stage, distillation-free systems with single-layer codebooks (BigCodec, WavTokenizer, BiCodec), all run at bitrates 7–10× higher and perform worse on WER and SIM.
Multilingual reconstruction on Common Voice shows TaDiCodec achieving the best SIM across all six languages and best WER on English, Chinese, German, and Korean (Table 2). The CMOS subjective study (Table 3) ranks TaDiCodec highest among codec baselines: 0.00 (reference), versus −0.92 for DualCodec and −1.07 for X-codec 2.
For zero-shot TTS (Table 5), TaDiCodec-AR (4B) achieves WER 2.28 on SeedTTS test-en and 1.19 on test-zh, outperforming all baselines including CosyVoice 2 (2.89/1.29) and MaskGCT (2.40/2.28). On challenging test sets — articulatory, code-switching, cross-lingual — the improvements are larger still: Code-switching en WER 9.16 vs. 15.03 for the next-best system. TaDiCodec-MGM (0.6B) matches or beats state-of-the-art NAR systems with RTF 0.12, and the 4B AR model achieves RTF 0.13 with vLLM (Table 6). Reconstruction-generation gap analysis (Figure 3) shows TaDiCodec retaining nearly all reconstruction quality at generation time, while Mimi degrades by 104.5% on English WER from reconstruction to generation.
Note
Speaker similarity (SIM 0.65) for TaDiCodec-AR is slightly below MaskGCT (0.71) and CosyVoice 2 (0.66), which run at 50 Hz and 25 Hz respectively. The intelligibility gains appear to come partly at the cost of the top speaker similarity tier.
Novelty Assessment
The core novelty is the unification of quantization and reconstruction into a single-objective training loop — prior work with diffusion decoders (CosyVoice, FireRedTTS, Vevo) always treated these as separate stages, with the VQ trained first and the diffusion decoder trained independently. The text-aware conditioning of the decoder is not unprecedented (similar ideas appear in CosyVoice and E2 TTS), but applying it inside the codec itself as the mechanism that enables extreme compression is genuinely new. BSQ is borrowed from visual tokenisation (it was proposed for image/video) and its application here is an adaptation, not an invention.
The compression ratio achieved (6.25 Hz, 0.0875 kbps) is a clear quantitative outlier — roughly 2× lower bitrate than any prior published single-layer single-stage system. The downstream TTS results are strong, and the reconstruction-generation gap analysis is a useful empirical contribution that contextualises what makes a tokenizer “language-model-friendly.” The evaluation is thorough and comparisons appear fair: baselines use official checkpoints and are evaluated on the same test sets.
The one significant constraint: the system is explicitly designed for contexts where text is available during decoding. Pure audio reconstruction without text (e.g., for audio compression or codec transmission) is out of scope, and the performance at 6.25 Hz without text conditioning degrades sharply.
Field Significance
Tip
High — TaDiCodec demonstrates that text-conditioned diffusion decoding can unlock an order-of-magnitude compression improvement over the prior generation of speech tokenizers, without sacrificing language-model friendliness. The reduction from 25–75 Hz to 6.25 Hz directly reduces AR sequence lengths for TTS by 4–12×, which matters for training cost, inference latency, and multi-language scaling. The explicit reconstruction-generation gap framing also offers a practical lens for evaluating whether a tokenizer is suited to downstream generation rather than just reconstruction.
Claims
- Text conditioning in the codec decoder, rather than in the language model alone, is a viable lever for achieving extreme compression rates in speech tokenization without adversarial training. (§3.1, Table 4)
- A single end-to-end training objective (flow-matching loss) is sufficient to jointly optimise quantization and reconstruction in a speech codec, eliminating the need for multi-stage pipelines. (§3.1, §4.2.2)
- The reconstruction-generation gap — the degradation in intelligibility when tokens are predicted by a language model rather than encoding reference speech — varies substantially across tokenizer architectures and is not captured by reconstruction metrics alone. (§4.3, Figure 3)
- Lower token rates in speech tokenizers can improve autoregressive TTS intelligibility by shortening prediction sequences and reducing error accumulation, particularly on linguistically challenging inputs. (§4.3, Table 5)
- Binary Spherical Quantization without a commitment loss achieves stable end-to-end training of a speech codec and produces superior representations to standard VQ under equal codebook sizes. (§4.2.2, Table 4)
Limitations and Open Questions
Warning
TaDiCodec’s text-aware decoder is not a general-purpose audio codec: it requires a transcript at both training and inference time. The strong reconstruction and TTS results are conditional on text availability; performance at 6.25 Hz without text conditioning is not competitive (WER exceeds 10% at 12.5 Hz without text, per Table 4). This limits applicability to codec-transmission, speech enhancement, or any scenario where transcriptions are unavailable.
The diffusion decoder introduces multi-step inference latency. At 32 steps, decoding speed is acceptable for generation but higher than GAN vocoders; reducing to 5 steps degrades quality noticeably. The authors propose distillation as future work but have not yet demonstrated single-step performance.
The system has been validated on TTS only; whether TaDiCodec tokens are useful for spoken dialogue modeling, speech understanding, or other downstream tasks remains untested. The prompt mechanism, while effective, means full reconstruction quality depends on a speech prefix being available — similar to zero-shot TTS conditioning, but potentially limiting in some SCA deployment scenarios.
Wiki Connections
Core concepts: neural-codec · flow-matching · autoregressive-codec-tts · zero-shot-tts · spoken-language-model · self-supervised-speech
Related in-corpus papers: 2504.10352