arXiv · 2026 · Preprint
Yixuan Zhou et al. (Tsinghua University (SIGS THUHCSI, THUNLP) / ModelBest) · → Paper · Demo: ✓ · Code: ✓
VoxCPM2 scales the tokenizer-free hierarchical diffusion-autoregressive design of VoxCPM to a 2B-parameter, 48 kHz, 30-language open-weight model that handles basic TTS, natural-language voice design, style-controllable cloning, and continuation cloning through input layout alone, with no discrete speech tokenizer and no mode-specific modules.
Problem
The predecessor, VoxCPM, showed that a single continuous-latent backbone with an internal semi-discrete bottleneck can match token-based zero-shot TTS, but it was a bilingual (Chinese/English), 16 kHz, roughly 0.6B-parameter system that supported only continuation-style cloning. Open TTS foundation models that do cover many languages and natural-language voice control typically rely on discrete codec tokens plus a separate acoustic decoder, or route control through dedicated style encoders, adapters, or per-mode variants. The report asks whether the tokenizer-free hierarchical paradigm can be extended to multilingual coverage, higher output sample rates, and instruction-driven control inside one set of parameters without that fragmentation.
Method
Representation. Speech lives entirely in the continuous latent space of AudioVAE V2, an asymmetric causal-convolutional VAE whose encoder takes 16 kHz audio (downsampling rates [2, 5, 8, 8], 640x reduction) and emits 64-dimensional latents at 25 Hz, while a deeper and wider decoder (upsampling [8, 6, 5, 2, 2, 2]) reconstructs 48 kHz audio (§3.2). Encoding at 16 kHz keeps the latent sequence short, lets the original 16 kHz training corpus be reused, and avoids latent mismatch across source sample rates; the 48 kHz decoder performs implicit super-resolution. The decoder also accepts an optional target-sample-rate condition. Only the rate-related modules change relative to the original AudioVAE.
Backbone. The backbone groups every four latent frames into a patch, giving a 6.25 Hz autoregressive sequence (160 ms per step). As in VoxCPM, a Local Encoder (LocEnc) summarises past patches, a Text-Semantic Language Model (TSLM, initialised from MiniCPM-4-1B, 28 layers, hidden size 2048) produces hidden states that pass through a Finite Scalar Quantization (FSQ) bottleneck to form a semi-discrete “semantic skeleton”, a Residual Acoustic Language Model (RALM) recovers acoustic detail, and a Local Diffusion Transformer (LocDiT, the patch-level design taken from DiTAR) generates the next continuous patch; a stop predictor on the TSLM-FSQ states decides termination (§3.1). VoxCPM2-specific refinements are: a learnable concatenation-projection (replacing summation) when fusing FSQ states with LocEnc embeddings before the RALM; separate prefix tokens for the semantic state, residual state, and timestep in LocDiT (replacing a single summed conditioning vector); FSQ width doubled from 256 to 512 dimensions (9 levels per dimension retained); and removal of rotary position embeddings from the RALM (NoPE) to improve long-utterance stability (§3.3.1). Total backbone size grows from about 0.6B (VoxCPM) to about 2B, with maximum sequence length 8192 (§3.3.3, Table 1).
Reference pathway and unified sequence organization. A new isolated reference-audio segment, delimited by REF_START/REF_END tokens and excluded from the loss, supplies speaker identity without needing a transcript or acting as a temporal prefix (§3.3.2). All generation modes are then expressed as layouts over three building blocks (text, optionally prefixed by a natural-language voice or style description; reference audio; target audio): basic TTS, voice design (description + text), reference cloning, controllable cloning (reference + style description + text), and continuation cloning (prompt transcript + target text, prompt audio as the start of the target segment) (§3.4, Table 2). Descriptions are plain text prefixes to the TSLM; there is no style encoder.
Training. The objective is unchanged from VoxCPM: patch-level conditional flow matching on target latent patches plus a binary stop loss, with 10% dropout of the LM-side conditioning to enable classifier-free guidance (§3.5). A three-stage curriculum moves from multilingual TTS pretraining (60 s segments, 4096 context), to joint TTS and controllable pretraining (rising share of description-annotated and reference-triplet data, 8192 context, up to 3-minute audio), to a high-quality annealing SFT stage with balanced language sampling. Training data exceed 2 million hours, mostly Chinese and English, with the other 28 languages at roughly 1K to 50K hours each (§3.6). Controllability data combine open-source expressive corpora with several thousand hours of internally annotated speech, described by general audio-understanding models (Step-Audio R1, Gemini 2.5 Pro) and checked with gender/age/emotion classifiers. To break the correlation between style and textual content, the model itself re-synthesises annotated voices and styles onto unrelated transcripts, and these self-generated pairs are used mainly in stage 2.
Inference. Generation uses CFG (default scale 2.0), sway sampling (from F5-TTS) and CFG-Zero*; the causal TSLM/RALM and patch-local LocEnc/LocDiT allow chunked streaming with a stateful VAE decoder (§3.7).
Key Results
Seed-TTS-Eval (Table 3). VoxCPM2 scores 1.84 WER / 75.3 SIM on test-EN, 0.97 CER / 79.5 SIM on test-ZH, and 8.13 CER / 75.3 SIM on test-ZH-Hard. It does not lead any column among open-source systems: Fish Audio S2 has the lowest error rates (0.99, 0.54, 5.99) and LongCat-AudioDiT the highest SIM (78.6, 81.8, 79.7). VoxCPM2’s SIM is second among open-source systems on all three subsets. Relative to VoxCPM (1.85/72.9, 0.93/77.2, 8.87/73.0), SIM rises by 2.3 to 2.4 points on every subset while error rates are flat on test-EN, slightly worse on test-ZH, and better on test-ZH-Hard. Against Qwen3-TTS (1.23/71.7, 1.22/77.0, 6.76/74.8) it trades higher English WER and hard-set CER for higher SIM.
Inference recipes (Table 4). Supplying the reference through both the isolated pathway and the continuation prefix gives the highest SIM on all three subsets (79.5 / 75.2 / 74.9), while reference-only gives the lowest Chinese error rates (1.81 on test-ZH, 6.85 on test-ZH-Hard) at a SIM cost (67.0 / 70.0). The report states that the reference + continuation recipe is used for all other zero-shot cloning results, but the Table 4 reference + continuation row (0.99/79.5, 1.94/75.2, 7.44/74.9) does not match the VoxCPM2 row of Table 3 (1.84/75.3, 0.97/79.5, 8.13/75.3), and the report does not explain the difference; both rows were checked against the source table images.
CV3-Eval (Table 5). VoxCPM2 has lower error than Fish Audio S2 only on hard-zh (8.55 vs. 9.1); Fish Audio S2 is lower on the other ten columns (e.g., en 2.43 vs. 5.00, hard-en 4.4 vs. 8.48, fr 6.26 vs. 9.85). Against CosyVoice 3-1.5B, VoxCPM2 is lower on nine columns, ties on ko (5.69), and is marginally higher on en (5.00 vs. 4.99).
MiniMax-MLS-Test, 24 languages (Tables 6 and 7). Speaker similarity is VoxCPM2’s clearest strength: it has the highest SIM in 21 of 24 languages, against MiniMax-Speech, ElevenLabs, Qwen3-TTS, and Fish Audio S2. The exceptions are Czech (78.3 vs. Fish Audio S2 79.8), Romanian (79.7 vs. MiniMax 80.9), and Italian (78.0 vs. Qwen3-TTS 81.7; the source table bolds VoxCPM2’s Italian value even though it is not the highest). The text’s “22 of 24” count includes Italian. Intelligibility is weaker: VoxCPM2 has the lowest WER only on Finnish (2.63), and is far behind on Arabic (13.05), Hindi (19.7), Ukrainian (6.32), Cantonese (38.58), and the two languages the paper says were unseen in training, Czech (24.13) and Romanian (21.58). The authors partly attribute the Arabic and Hindi errors to Whisper-large-v3, the ASR used for this benchmark.
Internal 30-language benchmark (Table 8). With Gemini 3.1 Flash Lite as the ASR, average WER/CER is 1.68% (recomputed from the per-language values). Per the table, 29 languages are under 3% (Greek, 3.17, is the exception) and 7 are under 1%; the prose says 28 and 6. Against Fish Audio S2, VoxCPM2 is lower on 20 languages, higher on 9 (ar, de, es, it, ja, ko, ms, pl, ru), and tied on Turkish. Fish Audio S2 fails on Khmer, Lao, and Burmese (75 to 87% CER), which drives most of the gap. This test set, its ASR choice, and the baseline runs are internal, and no Fish Audio S2 average is reported.
InstructTTSEval (Table 9). On English, VoxCPM2 scores 84.2 / 83.2 / 71.4 (APS / DSD / RP), the highest among the open systems listed (e.g., Qwen3-TTS-VD 82.9 / 82.4 / 68.4; MOSS-VoiceGenerator 68.2 / 82.0 / 68.7). The text says it outperforms “all compared systems”, but the closed Gemini-TTS-Pro row in the same table is higher on APS (87.6) and DSD (86.0); VoxCPM2 leads every listed system only on English RP. On Chinese, APS ties Qwen3-TTS at 85.2 (Gemini-TTS-Pro 89.0), while DSD (71.5) and RP (60.8) trail Qwen3-TTS (81.1 / 65.1), MOSS-VoiceGenerator (80.0 / 74.0), MiMo-Audio (74.3 / 61.5), and Gemini-TTS-Pro (90.1 / 75.5); RP also trails VoiceSculptor (61.5).
Subjective tests (Tables 12 to 14; about 100 samples, 50 listeners, double-blind). For zero-shot cloning, VoxCPM2’s N-MOS of 4.78 ties IndexTTS2 and sits just below Qwen3-TTS (4.80), and its S-MOS of 4.74 is the highest (IndexTTS2 4.71, with ±0.03 intervals that overlap). For multilingual synthesis (8 languages), its N-MOS of 4.78 is nominally highest (others 4.76 to 4.77, ±0.02), and its S-MOS of 4.66 is third, behind OmniVoice (4.72) and Fish Audio S2 (4.70). For voice design, its I-MOS of 4.50 is the highest (Qwen3-TTS-VD 4.41), but its N-MOS of 4.48 is below Qwen3-TTS-VD (4.61). In other words, the subjective results are mixed: S-MOS leads on cloning but not on multilingual synthesis, even though objective SIM leads strongly on MiniMax-MLS.
AudioVAE V2 (Table 10). The asymmetric 16-to-48 kHz VAE is best on only one of six columns (Song Describer MelD-16k, 1.133). VoxCPM1.5’s 44 kHz VAE has better full-band mel distance (VCTK 1.139 vs. 1.335), and the original 16 kHz VoxCPM VAE is marginally better on 16 kHz-band VCTK metrics (MelD 0.801 vs. 0.813, STOI 0.911 vs. 0.907, PESQ 3.94 vs. 3.906). So the design keeps most of the low-band quality while adding a 48 kHz output, rather than beating either predecessor.
Efficiency (Table 11). On one RTX 4090, the RTF is 0.30 in PyTorch (about 8 GB VRAM) and 0.13 with Nano-vLLM. The PyTorch path is slower than VoxCPM1.5 (0.15) and VoxCPM (0.17), so the “competitive or better RTF” statement depends on comparing an optimised VoxCPM2 serving path with unoptimised predecessors. Streaming latency (time to first audio) is not reported.
Novelty Assessment
The contribution is mainly scaling and engineering integration on an existing architecture. The hierarchical TSLM/FSQ/RALM/LocDiT design and its training objective come from VoxCPM, and LocDiT itself comes from DiTAR. The new pieces are small structural refinements (concatenation-projection fusion, multi-token LocDiT conditioning, a wider FSQ, NoPE in the RALM), an isolated reference-audio segment, an asymmetric VAE that decodes at a higher sample rate than it encodes, and a single sequence organization that turns voice design and controllable cloning into plain text-prefix layouts. None of the backbone refinements is ablated, so their individual effect is unknown. The asymmetric VAE and the self-synthesis trick for separating style from content are the most reusable ideas, though neither is evaluated in isolation beyond Table 10. The rest of the advance comes from more parameters (about 0.6B to 2B), more data (over 2M hours, 30 languages), and a controllability-focused data pipeline. The main empirical contribution is evidence that a tokenizer-free continuous-latent autoregressive model at this scale gets strong speaker similarity across many languages and competitive open-model instruction following. On intelligibility it remains behind the strongest token-based systems.
Field Significance
Moderate. VoxCPM2 is an open-weight (Apache 2.0) data point showing that the continuous-latent, tokenizer-free diffusion-autoregressive route can carry multilingual coverage and natural-language voice design inside one backbone. Its clearest objective advantage over token-based multilingual systems is speaker similarity, while intelligibility trails the strongest baselines. As a successor release that mostly scales its predecessor, with unablated refinements, it extends the evidence for an existing paradigm rather than setting a new direction.
Claims
- supports: Continuous-latent autoregressive TTS without a discrete speech tokenizer can achieve higher speaker similarity than discrete-token systems across many languages.
Evidence: On MiniMax-MLS-Test, VoxCPM2 has the highest SIM in 21 of 24 languages against MiniMax, ElevenLabs, Qwen3-TTS, and Fish Audio S2 (e.g., English 85.4 vs. 79.7 for Fish Audio S2); the authors attribute this to the continuous-latent paradigm, though model scale and data also differ. (§4.3, Table 7)
- complicates: High objective speaker similarity in multilingual zero-shot TTS does not imply comparable intelligibility, especially for lower-resource languages.
Evidence: The same model that leads SIM in 21 of 24 MiniMax-MLS languages has the lowest WER in only one (Finnish), with WER of 13.05 on Arabic, 19.7 on Hindi, and over 20 on unseen Czech and Romanian; it also trails Fish Audio S2 on 10 of 11 CV3-Eval columns. (§4.3, Tables 5, 6, 7)
- refines: Providing the speaker reference both as an isolated identity prefix and as a transcribed continuation prompt gives higher speaker similarity than either route alone, while reference-only prompting can improve intelligibility on difficult content.
Evidence: Reference + continuation gives the highest SIM on all three Seed-TTS-Eval subsets (79.5 / 75.2 / 74.9), and reference-only gives the lowest CER on test-ZH (1.81) and test-ZH-Hard (6.85) at reduced SIM (67.0 / 70.0). (§4.2, Table 4)
- supports: Natural-language voice design and style-controlled cloning can be learned as plain text prefixes to a shared TTS backbone, without dedicated style encoders or per-mode modules.
Evidence: VoxCPM2 handles voice design and controllable cloning purely through sequence layout and scores 84.2 / 83.2 / 71.4 on English InstructTTSEval, the highest among listed open systems, with the highest I-MOS (4.50) in the voice-design listening test; Chinese DSD and RP still trail Qwen3-TTS and MOSS-VoiceGenerator. (§3.4, §4.4, §4.7, Tables 9, 14)
- complicates: Speaker-embedding cosine similarity and listener-rated speaker similarity can rank TTS systems differently.
Evidence: VoxCPM2 leads objective SIM in 21 of 24 MiniMax-MLS languages, but in the 8-language listening test its S-MOS (4.66) ranks below OmniVoice (4.72) and Fish Audio S2 (4.70); the authors also note that SIM does not fully reflect perceived cloning quality. (§4.7, Tables 7, 13)
Limitations and Open Questions
Warning
None of the architectural refinements (concatenation-projection fusion, multi-token LocDiT conditioning, wider FSQ, NoPE RALM) or the self-synthesised style-decoupling data is ablated. Parameters, data scale, sample rate, and architecture all change at once relative to VoxCPM, so the report cannot attribute any gain to a specific design change.
- The zero-shot results in Table 3 and Table 4 disagree for what is stated to be the same inference recipe, and the report does not reconcile them. This leaves it unclear which numbers represent the default configuration.
- Several headline comparisons rest on internal choices. The 30-language benchmark, its Gemini-based ASR, and the Fish Audio S2 runs on it are not public, and the ASR differs from the Whisper-large-v3 setup on MiniMax-MLS-Test, which makes cross-benchmark intelligibility hard to compare.
- The text overstates some table results: “best on InstructTTSEval-EN” ignores Gemini-TTS-Pro, “highest SIM on 22 of 24 languages” counts Italian where Qwen3-TTS is higher, and the per-threshold language counts differ from Table 8.
- Intelligibility on Arabic, Hindi, Ukrainian, Cantonese, and the unseen Czech and Romanian is far behind commercial and token-based baselines on MiniMax-MLS-Test.
- The controllability data are partly synthesised by the model itself and annotated by LLM-based audio captioners. The report does not analyse whether this introduces artifacts or annotation biases.
- Streaming is described but not measured (no first-packet latency), and singing is mentioned only qualitatively.
- The listening-test design is small (about 100 samples). The baseline sets differ by test, and most MOS differences fall within or near the reported 95% confidence intervals.
Wiki Connections
- Flow Matching — VoxCPM2’s LocDiT is trained with a patch-level conditional flow-matching objective and sampled with CFG, sway sampling, and CFG-Zero* inside an autoregressive loop.
- Zero-Shot TTS — Evaluates prompt-based cloning of unseen speakers on Seed-TTS-Eval, CV3-Eval, and MiniMax-MLS-Test, and compares isolated-reference, continuation, and combined prompting recipes.
- Multilingual TTS — Trains one backbone on 30 languages and 9 Chinese dialects and reports per-language intelligibility and similarity on 24-language and 30-language test sets.
- Instruction-Conditioned TTS — Supports natural-language voice design and style-controlled cloning as text prefixes, evaluated on InstructTTSEval and with an instruction-following MOS.
- Neural Audio Codec — Introduces AudioVAE V2, an asymmetric continuous-latent VAE that encodes at 16 kHz and decodes at 48 kHz, with reconstruction metrics against two earlier VoxCPM VAEs.
- Subjective Evaluation — Reports N-MOS, S-MOS, and I-MOS from 50 listeners in a double-blind test, and notes cases where objective SIM and perceived similarity diverge.
- VoxCPM — The direct predecessor whose hierarchical TSLM/FSQ/RALM/LocDiT design, AudioVAE, and training objective VoxCPM2 scales and refines.
- DiTAR — Source of the Local Diffusion Transformer over patch-wise continuous latents that VoxCPM2 reuses, and a closed-source Seed-TTS-Eval baseline.
- Fish Audio S2 — The main multilingual baseline, with lower error rates on Seed-TTS-Eval, most of CV3-Eval, and most MiniMax-MLS languages, but lower SIM than VoxCPM2 in almost every MiniMax-MLS language.
- Qwen3-TTS — Baseline for zero-shot cloning, multilingual similarity, and voice design; it ties VoxCPM2 on Chinese APS and leads it on Chinese DSD/RP and voice-design N-MOS.
- LongCat-AudioDiT — Open-source baseline with the highest Seed-TTS-Eval SIM on every subset and lower N-MOS than VoxCPM2 in the cloning listening test.
- CosyVoice 3 — Seed-TTS-Eval baseline and source of the CV3-Eval benchmark, on which VoxCPM2 is lower-error than the 1.5B CosyVoice 3 on most languages.
- MiniMax-Speech — Source of the MiniMax-MLS-Test 24-language benchmark and a commercial baseline on it.
- InstructTTSEval — The benchmark used to measure instruction-following accuracy across the APS, DSD, and RP subtasks.