arXiv · 2025 · Preprint
Tao Li et al. (China Mobile Nineverse Artificial Intelligence Technology / Peking University) · → Paper · Demo: ✓ · Code: ?
DisCo-Speech introduces DisCodec, a two-stage disentangled neural speech codec that explicitly factorizes speech into content, prosody, and timbre subspaces, enabling a standard autoregressive language model to perform independent zero-shot prosody and timbre control without specialized architecture modifications.
Problem
Codec-based autoregressive language models excel at voice cloning via in-context learning — replicating timbre and prosody together from a speech prompt — but cannot independently control each attribute. The root cause is that standard acoustic codecs (EnCodec, SoundStream) and even semantic-aware codecs entangle timbre and prosody in their representations, making it impossible for a continuation-based LM to adjust one without affecting the other. Prior disentangled codecs such as FACodec (NaturalSpeech 3), MSR-Codec, and FreeCodec provide factorized tokens but suffer from either insufficient decoupling, reliance on complex pipeline architectures, or the well-known disentanglement-reconstruction trade-off: enforcing strict attribute independence degrades reconstruction fidelity, while relaxing constraints causes information leakage. Additionally, three-stream representations are inconvenient for standard LM prediction that expects a single unified token sequence.
Method
DisCo-Speech consists of two co-designed components: DisCodec and a Text-to-Codec Language Model.
DisCodec is trained in two sequential stages. In Stage 1 (tri-factor disentanglement), three parallel encoders extract frame-level representations for content (a DAC-style convolutional encoder), timbre (ECAPA-TDNN producing frame-level features aggregated into 48 global tokens via cross-attention), and prosody (dilated causal convolutions). Each branch is quantized with Finite Scalar Quantization (FSQ): single FSQ for content and timbre, residual two-layer FSQ for prosody. Decoupling constraints are carefully graduated to match the natural hierarchy of speech attributes: a phone classification loss (from a fine-tuned Wav2Vec model) anchors content tokens; an F0 regression loss supervises the first prosody FSQ layer; a correlation loss guides the second prosody FSQ layer toward prosody residuals; a GRL speaker classifier strips timbre from the prosody branch; and soft orthogonality constraints (adjustable cosine similarity targets β) allow partial overlap between prosody-content (β_c = 0.01, since content and prosody share temporal dynamics) while enforcing near-orthogonality between prosody-timbre (β_t = 1e-4). A speaker classification loss on the timbre branch closes the loop. Stage 1 ends with a reconstruction decoder trained with Mel and waveform losses.
Stage 2 freezes the encoders and trains a new decoder that fuses content and prosody by summing their quantized embeddings and re-quantizing them into a single unified content-prosody token sequence (z_cp). The decoder stacks Transformer blocks followed by a BigVGANv2 generator, trained with GAN losses (multi-scale Mel reconstruction, feature matching, adversarial). This stage resolves both the downstream-usability problem — a single-stream z_cp is directly predictable by a standard LM — and partially mitigates the disentanglement-reconstruction trade-off by allowing the decoder to learn to reconstruct from fused representations.
Text-to-Codec LM: A Qwen2.5-1.5B model is initialized and trained to predict content-prosody tokens z_cp autoregressively given BPE text tokens. Training uses a pretraining phase on 120k hours (Emilia + internal), followed by SFT on 5k hours. At inference, the LM receives a prosody prompt speech (with its corresponding text) and the target text; it performs prosodic continuation to generate z_cp matching the prosodic pattern of the prompt. The DisCodec decoder then synthesizes the waveform from z_cp conditioned on the target speaker’s global timbre tokens q_t — extracted separately from the desired target speaker reference. This clean separation (LM for prosody continuation, decoder for timbre injection) is the key architectural enabler of independent control.
Key Results
Codec reconstruction (LibriSpeech test-clean): DisCodec achieves UTMOS 4.10, WER 2.92%, STOI 0.86, and SSIM 0.81 at 50 tokens/s. Among 50 tokens/s codecs, this is competitive: X-Codec2 leads on WER (2.47%) and SSIM (0.82) and UTMOS (4.13), while DisCodec matches or exceeds most others including Mimi, SpeechTokenizer, and DAC by large margins. The trade-off for disentanglement capability is modest: PESQ WB drops to 1.98 vs. 2.43 for X-Codec2, and STOI is 0.86 vs. 0.92 for X-Codec2. MSR-Codec (the closest disentangled competitor) achieves UTMOS 4.15 but at a lower token rate (62.5/s) and with no content intelligibility metric reported.
Zero-shot voice conversion (disentanglement proxy, 1,680 SEED-TTS-Eval pairs): DisCodec achieves SSIM 0.61 and F0 Correlation 0.59 — the highest SSIM and F0cor among all compared systems (CosyVoice2: SSIM 0.55, F0cor 0.48; Vevo: 0.60 / 0.50; SeedVC: 0.58 / 0.56), at UTMOS 3.98.
Voice cloning (SEED-TTS-Eval): DisCo-Speech reaches WER 3.01% and SSIM 0.597 on English, and CER 2.08% and SSIM 0.677 on Mandarin. Among one-stage AR models, it is comparable to Spark-TTS (WER 1.98%, SSIM 0.584 EN). Multi-stage flow-matching systems outperform on SSIM: F5-TTS (0.647) and CosyVoice2 (0.652). IndexTTS2, which also supports independent control, achieves SSIM 0.706 EN and 0.765 ZH.
Prosody control (AB preference test, 500 samples): In style scenarios (rhythm/speech rate), DisCo-Speech beats Vevo with 51.5% vs. 21.3% timbre preference and 48.9% vs. 20.0% prosody preference. Against IndexTTS2, the result is closer: DisCo-Speech slightly leads on timbre (37.4% vs. 29.0% style) but trails on prosody in emotion scenarios (36.0% vs. 38.5%), where IndexTTS2’s expressive generation — at the cost of timbre leakage — edges ahead.
Novelty Assessment
The contribution is primarily at the codec level: the two-stage training protocol that separates disentanglement learning from high-fidelity reconstruction is a principled response to a well-documented bottleneck. The soft orthogonality constraint with content-specific and timbre-specific relaxation targets is a careful engineering improvement over both the hard-orthogonality (information loss) and no-constraint (leakage) extremes, validated in ablation. The fusion-and-re-quantization step in Stage 2 — summing content and prosody embeddings before re-quantizing into a single stream — is a clean design choice that enables a completely standard LM without multi-codebook prediction complexity.
That said, the architectural novelty is incremental relative to NaturalSpeech 3 (FACodec), Vevo, and MSR-Codec, which explore similar disentangled codec designs. The claimed advantages over these systems are borne out by the VC disentanglement metrics and preference tests, but the margin over Vevo (which achieves SSIM 0.664 in voice cloning) is not decisive. The work is honest in noting that prosody control gains come partly at the cost of voice cloning SSIM relative to flow-matching baselines.
Field Significance
Moderate — DisCodec introduces a principled two-stage training approach for codec disentanglement that sidesteps the hard reconstruction-disentanglement trade-off by decoupling the learning of attribute separation from waveform fidelity. It demonstrates that a standard LM can achieve independent prosody and timbre control without multi-stream prediction or specialized architecture changes, provided the codec resolves entanglement at the token level. The result positions the codec design problem as a key bottleneck for controllable speech generation, rather than the LM architecture.
Claims
- Resolving timbre-prosody entanglement at the codec level enables a standard autoregressive language model to perform independent prosody and timbre control without requiring multi-stream prediction or specialized generation architectures. (§3, §3.1.2)
- A two-stage codec training protocol that separates attribute disentanglement (Stage 1) from fused reconstruction (Stage 2) mitigates the disentanglement-reconstruction trade-off more effectively than single-stage approaches. (§3.1, §3.1.2, Appendix B)
- Soft orthogonality constraints with attribute-specific relaxation targets outperform both unconstrained and hard-orthogonality approaches in balancing speaker similarity and prosody preservation in zero-shot voice conversion. (§3.1.1, Appendix B, Table 6)
- Codec-based one-stage autoregressive TTS achieves competitive but lower speaker similarity than multi-stage flow-matching systems on the SEED-TTS-Eval benchmark, reflecting an inherent tension between AR variability and speaker fidelity. (§4.2.2, Table 4)
- Independent disentanglement of prosody and timbre improves rhythm and speech-rate control in stylistic scenarios but does not fully match the expressiveness of systems that permit partial timbre-prosody entanglement in emotional conditions. (§4.2.2, Table 3)
Limitations and Open Questions
- Speaker similarity gap: DisCo-Speech’s voice cloning SSIM (0.597 EN) trails multi-stage systems like F5-TTS (0.647) and CosyVoice2 (0.652). The paper attributes this to inherent AR variability and compact codec representations — reducing AR sampling variance or increasing timbre token resolution could help.
- Expressive prosody stability: The model shows instability with highly exaggerated prosody, likely a training-data diversity limitation (the 6k fine-tuning set may not cover extreme expressiveness).
- Disentanglement-reconstruction balance: Despite the two-stage design, PESQ scores for DisCodec remain lower than non-disentangled codecs at the same token rate (1.98 vs. 2.43 for X-Codec2), suggesting further headroom remains.
- No streaming or real-time evaluation: Whether the LM’s autoregressive structure and BigVGANv2 decoder can operate in low-latency streaming scenarios is not addressed.
- Evaluation limited to Mandarin and English: Multilingual generalization of the disentangled representations is untested.
Wiki Connections
DisCodec’s FSQ-based factorization builds directly on the codec foundation established by VALL-E and extends the disentangled codec line of work. The prosody-timbre independence design addresses limitations discussed in CosyVoice 2 and CosyVoice. Comparison baselines include IndexTTS2, which is the primary competitor on emotion expressiveness, and DiTAR, another hybrid AR+decoder framework. Seed-TTS provides the SEED-TTS-Eval benchmark used for voice cloning evaluation.
Concept pages: disentanglement, neural-codec, zero-shot-tts, prosody-control, autoregressive-codec-tts, voice-conversion