arXiv · 2025 · Preprint
Yixuan Zhou et al. (Tsinghua University (THUHCSI, THUNLP), ModelBest) · → Paper · Demo: ✓ · Code: ✓
VoxCPM replaces the discrete speech tokenizer entirely with a differentiable scalar-quantization bottleneck inside an end-to-end continuous-latent TTS model, letting a semantic language model and a residual acoustic model specialize without external codec supervision.
Problem
TTS systems built on discrete codec tokens (e.g. EnCodec/RVQ-based LM pipelines) achieve stable autoregressive generation but hit a “quantization ceiling”: the codec irreversibly discards acoustic detail, and state-of-the-art systems compensate with multi-stage LM-then-diffusion pipelines that create a semantic-acoustic divide, since the LM never sees acoustic reality and the diffusion decoder never sees high-level linguistic context. Fully continuous autoregressive alternatives (e.g. MELLE, DiTAR) avoid quantization loss but conflate two different jobs, semantic-prosodic planning and fine-grained acoustic rendering, inside a single continuous objective, which the paper argues is a root cause of error accumulation and instability on long or complex utterances.
Method
VoxCPM is a hierarchical, tokenizer-free architecture that generates continuous speech latents autoregressively over patches, with no pre-trained speech codec or discrete vocabulary anywhere in the pipeline. A local audio encoder (LocEnc) compresses historical VAE latent patches into acoustic embeddings that condition a Text-Semantic Language Model (TSLM), initialized from a pre-trained text LM (MiniCPM-4-0.5B) and operating directly on BPE text tokens rather than phonemes. The TSLM’s continuous hidden states pass through a Finite Scalar Quantization (FSQ) layer, a differentiable bottleneck (256 dimensions, 9 scalar levels) that produces a “semi-discrete” semantic-prosodic skeleton via straight-through estimation. This bottleneck is used purely as a regularizer on the hidden-state space, not as a discrete prediction target the way RVQ codebooks are used in codec-LM pipelines.
A Residual Acoustic Language Model (RALM) then conditions on the TSLM’s text hidden states, the FSQ output, and the historical acoustic embeddings to recover the fine-grained speaker identity and spectral detail that quantization discarded. The FSQ skeleton and RALM residual are summed into a single conditioning signal that drives a Local Diffusion Transformer decoder (LocDiT), a bidirectional transformer following the DiTAR design that denoises the next latent patch conditioned on the previous patch (framing generation as outpainting) and supports classifier-free guidance at inference. The entire stack, LocEnc, TSLM, FSQ, RALM, and LocDiT, is trained end-to-end under a single conditional flow-matching objective plus a stop-token BCE loss, with gradients flowing through the FSQ bottleneck via straight-through estimation. A separately trained causal audio VAE (DAC-style causal convolutional encoder/decoder, 640x downsampling, 16kHz audio to 25Hz latents) supplies the continuous latent space, enabling streaming-compatible encoding and decoding. VoxCPM-0.5B trains for 500K iterations on 40 H100 GPUs over an internal 1.8M-hour bilingual (Chinese/English) corpus, using a two-phase Warmup-Stable-Decay schedule where the decay phase is reported as important for zero-shot speaker similarity; a smaller VoxCPM-Emilia variant trains on the public 95K-hour Emilia dataset for direct comparison with prior work.
Key Results
On SEED-TTS-EVAL, VoxCPM reports an English WER of 1.85% and Chinese CER of 0.93%, with SIM of 72.9% (EN) and 77.2% (ZH), stated as the strongest results among open-source systems and ahead of IndexTTS2 and CosyVoice2 on the reported columns, though several closed-source systems (DiTAR, MiniMax-Speech) report lower WER/CER on the same table. On CV3-EVAL, an in-the-wild expressive voice-cloning benchmark, VoxCPM reaches ZH-CER 3.40% and EN-WER 4.04%, and on the CV3 Hard-Test subset an EN-WER of 7.89%, which the paper notes is better than the closed-source CosyVoice3 baselines on that split. Subjective evaluation (20 raters, 5-point MOS) shows VoxCPM at N-MOS/S-MOS of 4.10/4.11 (ZH) and 4.11/4.18 (EN), trailing IndexTTS2 on Chinese naturalness (4.25 vs. 4.10) but exceeding it on speaker similarity in both languages. The VoxCPM-Emilia variant, trained on 95K hours versus 1.8M hours for the full model, still reaches EN-WER 2.34%/ZH-CER 1.11%, which the authors use to argue the architecture is comparatively data-efficient, though absolute performance is clearly lower than the full-scale model. Reported inference RTF is 0.17 on a single RTX 4090. All baseline numbers are drawn from official implementations or the original papers rather than a fully controlled re-run, so cross-system comparisons should be read with that caveat.
Ablations isolate two architectural claims. Removing the FSQ bottleneck entirely (fully continuous hierarchical model) causes catastrophic degradation on hard Chinese cases (ZH-hard CER rising to 24.92% from 18.19% with FSQ at the chosen dimension), and FSQ dimensionality shows a clear non-monotonic optimum around 128-256 dimensions rather than benefiting from either extreme. Removing the RALM (TSLM output feeding LocDiT directly, architecturally similar to DiTAR) degrades EN-WER from 2.98% to 4.34% and ZH-hard CER from 18.19% to 25.0%, and removing the historical acoustic embedding from the RALM’s input degrades results further, supporting the claim that explicit residual acoustic modeling, not just a bigger TSLM, is doing the work.
Novelty Assessment
The contribution is genuinely architectural: using FSQ as a differentiable regularizing bottleneck on hidden states, rather than as a discrete prediction target the way RVQ/VQ codebooks are used in codec-LM TTS, is a distinct design choice from the dominant “LM predicts discrete tokens, diffusion refines them” pipeline pattern, and the paper backs the design with ablations that isolate the FSQ bottleneck and the RALM residual pathway separately. That said, several individual components are adapted from prior work rather than invented here: the bidirectional local diffusion transformer over patches follows DiTAR directly, the flow-matching training objective is standard, and the causal audio VAE follows a DAC-style design. The paper is transparent about this lineage in its related-work section. The scale of training data (1.8M hours internal, proprietary) is also a meaningful contributor to the reported state-of-the-art numbers among open-source systems, and the VoxCPM-Emilia ablation partially, but not fully, isolates architecture from data scale as the source of the gains.
Field Significance
Tip
High — VoxCPM demonstrates that a differentiable semi-discrete bottleneck can replace an external speech tokenizer entirely while still producing SOTA-among-open-source zero-shot TTS results, offering a concrete alternative to the two-stage discrete-token-then-diffusion pipeline pattern that dominates recent codec-LM TTS systems.
The paper’s ablations give the field a clear, reusable result: an internal FSQ-style regularization bottleneck, not a downstream prediction target, is sufficient to induce semantic/acoustic specialization inside an otherwise fully continuous autoregressive-diffusion hierarchy, and this specialization is measurably responsible for stability on hard cases rather than an artifact of scale alone. The open-source release (weights, code, demo) makes this a directly reproducible reference point for tokenizer-free continuous TTS.
Claims
- supports: A differentiable scalar-quantization bottleneck applied to hidden states, rather than used as a discrete prediction target, can induce semantic/acoustic task separation inside a continuous autoregressive TTS model without an external speech tokenizer.
Evidence: Removing the FSQ bottleneck from an otherwise identical hierarchical architecture increases ZH-hard-case CER from 18.19% to 24.92%, while FSQ dimensionality shows a non-monotonic optimum around 128-256 dimensions rather than monotonic improvement with capacity. (§4.3, Table 6)
- supports: Explicitly separating acoustic detail recovery into a dedicated residual module improves robustness on complex inputs beyond what a single semantic language model plus a diffusion decoder achieves.
Evidence: Removing the RALM (TSLM output feeding the diffusion decoder directly, architecturally close to DiTAR) degrades EN-WER from 2.98% to 4.34% and ZH-hard-case CER from 18.19% to 25.0%; removing the historical acoustic embedding from the RALM input degrades results further. (§4.4, Table 7)
- complicates: Learning-rate schedule design, not just architecture, materially affects zero-shot speaker similarity in large-scale continuous TTS training.
Evidence: A two-phase Warmup-Stable-Decay schedule’s decay phase alone improves ZH-hard-case CER from 13.22% to 8.87% and SIM by 4.4 points over the stable-phase-only checkpoint on an otherwise identical model. (§4.5, Table 8)
- complicates: Classifier-free guidance strength in diffusion-based TTS decoders trades off intelligibility against speaker similarity non-monotonically, with both very low and very high guidance scales degrading both metrics simultaneously.
Evidence: CFG scale 1.0 (no guidance) yields EN-WER 16.32% and SIM 55.1%, while scale 5.0 yields EN-WER 12.78% and SIM 60.7%; the optimum at scale 2.0 achieves EN-WER 1.85% and SIM 72.9%, with degradation on both sides of the optimum. (§4.6, Table 9)
- refines: Removing dependency on a pre-trained discrete speech tokenizer does not require sacrificing zero-shot voice cloning quality relative to discrete-token-based open-source TTS systems.
Evidence: On SEED-TTS-EVAL, the fully continuous VoxCPM reports SIM of 72.9% (EN) and 77.2% (ZH), exceeding the discrete-token-based IndexTTS2 and CosyVoice2 baselines on the same benchmark. (§4.2, Table 3)
Limitations and Open Questions
Multilingual capability is limited to Chinese and English by construction; the paper explicitly reports uncertain generalization to other languages, and prosody/emotion control lacks any intuitive or precise user-facing conditioning mechanism. The causal audio VAE operates at 16kHz, which the authors acknowledge falls short of the 24kHz or 44.1kHz sampling rates typically expected for high-fidelity applications. Baseline comparisons draw on official implementations or numbers reported in prior papers rather than a uniformly controlled re-evaluation, so cross-system rankings on tables that mix reproduced and self-reported numbers should be read cautiously. The training corpus (1.8M hours) is internal and not released, which limits independent reproduction of the full-scale result even though code and weights for the trained model are public.
Wiki Connections
- Flow Matching — VoxCPM trains its Local Diffusion Transformer decoder end-to-end under a conditional flow-matching objective rather than a standard diffusion or regression loss.
- Autoregressive Codec TTS — VoxCPM is explicitly positioned against the dominant discrete-codec-token autoregressive paradigm, replacing the external tokenizer with an internal differentiable FSQ bottleneck.
- Neural Audio Codec — the paper argues that reliance on pre-trained neural codecs like EnCodec creates a “quantization ceiling” and a semantic-acoustic divide that its tokenizer-free design avoids.
- Zero-Shot TTS — VoxCPM is evaluated on zero-shot voice cloning benchmarks (SEED-TTS-EVAL, CV3-EVAL) against a wide range of open- and closed-source zero-shot TTS systems.
- Disentanglement — the FSQ bottleneck and residual acoustic pathway are trained to implicitly separate semantic-prosodic content from acoustic/speaker detail within a single continuous representation space, validated via t-SNE analysis of the learned representations.
- Prosody Control — the paper shows the TSLM backbone infers appropriate prosodic style directly from text content genre (news, poetry, conversation) without an explicit prosody-control mechanism, though the authors note this control is not precise or user-adjustable.
- DiTAR — VoxCPM’s LocDiT decoder directly follows DiTAR’s bidirectional local diffusion transformer design, and the paper’s “w/o RALM” ablation is described as architecturally analogous to DiTAR’s continuous approach.
- IndexTTS2 — used as a primary open-source baseline on both SEED-TTS-EVAL and CV3-EVAL, and is the system VoxCPM trails on Chinese naturalness MOS while exceeding it on speaker similarity.
- FireRedTTS-2 — cited as a discrete-token industrial TTS baseline and compared directly on SEED-TTS-EVAL benchmark tables.
- MELA-TTS — grouped with VoxCPM as a joint transformer-diffusion continuous representation approach in the related-work discussion of hierarchical and residual TTS modeling.
- FELLE — cited as a related continuous latent autoregressive approach using token-wise coarse-to-fine flow matching, contrasted with VoxCPM’s single unified hierarchy.
- CLEAR — cited as a related continuous latent autoregressive modeling approach for high-quality low-latency speech synthesis.
- KALL-E — cited as a related approach combining joint transformer-diffusion with next-distribution prediction, part of the hierarchical/residual TTS modeling lineage VoxCPM positions itself within.