arXiv · 2026 · Preprint
Shi Lian et al. (Xiaohongshu Inc. / Shanghai Jiao Tong University) · → Paper · Demo: ✓ · Code: ✓
dots.tts is an open-weight 2B continuous autoregressive TTS model. An LLM plans speech on a 6.25 Hz semantic summary of its own output, and a full-history flow-matching head renders 25 Hz AudioVAE latents. Reward-free self-corrective post-training and CFG-aware MeanFlow distillation make the continuous rollout both stable and fast enough for streaming.
Problem
Discrete-token AR TTS (the CosyVoice family, Qwen3-TTS, Seed-TTS, IndexTTS) inherits the LLM training stack, but its tokenizer caps the acoustic detail the model can express. Continuous-latent AR systems (KALL-E, DiTAR, VibeVoice, VoxCPM, ARDiT) remove that cap, but they lose the quantization step that snaps an imperfect sample back to a valid acoustic state. In a continuous rollout, every small prediction error reaches the decoder and feeds back as conditioning for the next step. The authors identify this long-range error accumulation, plus a post-training stack that is less mature than the discrete-token one, as the main barriers to production use of continuous AR TTS.
Method
dots.tts has two decoupled networks: an AudioVAE that defines the continuous representation, and an AR backbone that predicts that representation one patch at a time (§2.1).
AudioVAE. The VAE follows the HoliTok recipe (the authors’ companion tokenizer, accepted record 2605.29948 without a page yet). A fully causal convolutional encoder (strides 2, 2, 2, 4, 6, 10) maps 48 kHz mono audio to a 128-dimensional latent at 25 Hz (1920x downsampling). A causal BigVGAN-v2-style decoder reconstructs the waveform. The latent carries KL plus flow regularization, following KALL-E. Training has two stages (§2.2, §2.6.1). Stage 1 (500K steps) uses reconstruction losses only: multi-scale mel, multi-period plus sub-band CQT adversarial, and feature matching. Stage 2 (200K steps) adds a frame-level cosine alignment to layer 23 of a frozen WavLM, plus a multitask block (small encoder and small LLM head) trained on ASR, emotion and speaker classification. The aim is a latent that is easier for the backbone to learn. The multitask LLM head is discarded; its encoder becomes the backbone’s semantic encoder. The VAE is frozen for all backbone training.
Backbone. Three modules split semantic planning from acoustic rendering (§2.3 to §2.5):
- An LLM initialized from Qwen2.5-1.5B Base reads BPE text (no phonemes) and a 6.25 Hz stream of audio-semantic embeddings. It emits one hidden state per audio step.
- An AR flow-matching head (AR-FM), an 18-layer DiT (hidden 1024, RoPE, QK-norm, adaLN-zero), takes each LLM hidden state and generates the next 4-frame patch of the 25 Hz VAE latent. Its context interleaves all earlier LLM hidden states and all earlier clean patches with the current noisy patch (full-history conditioning). A frozen CAM++ x-vector enters as a global adaLN condition. The LLM stream and the speaker stream are each dropped with probability 0.5 to enable joint CFG over content and timbre.
- A causal semantic encoder (24-layer Transformer, reused from AudioVAE Stage 2) compresses each newly generated patch into one 6.25 Hz embedding for the next LLM step. The LLM therefore never sees raw VAE latents, which the authors found necessary for stable continuous rollouts.
The AR-FM head trains in parallel over all patches. A block-causal mask over a concatenated [clean history | noisy generation] sequence, with RoPE positions reset between the two halves, reproduces the per-step inference context exactly (§2.5.2, Figure 2). Pretraining optimizes a rectified-flow velocity loss, the only loss that reaches the LLM and semantic encoder, plus a balanced BCE stop head on a detached copy of the LLM state (§2.6.2). Pretraining has three stages (§3.2.2): modality alignment with a frozen LLM on Emilia only (100K steps), end-to-end general training on the full 1.5M-hour mixture (700K steps), and annealing on a re-filtered high-quality subset (100K steps). Two sequence layouts are trained: plain mode (full text as prefix) and a 1T1A interleaved mode in which one BPE token alternates with one audio step, so an upstream dialogue LLM can drive synthesis token by token (§2.3.1).

Post-training updates only the DiT inside the AR-FM head (§2.6.3, §3.2.3, §3.2.4):
- Self-corrective alignment adapts SOAR, a reward-free self-correction method for diffusion models, to the AR-FM head. Each step takes one detached CFG-guided Euler rollout from a noised sample and re-noises the result toward the noise endpoint (6 auxiliary states per sample). The DiT then regresses an endpoint-consistent velocity that steers these off-trajectory states back to the clean latent. No reward model, preference model or external teacher is used (50K steps).
- CFG-aware MeanFlow distillation trains a student DiT, with an added interval-duration embedder, to predict the teacher’s CFG-guided mean velocity over an interval. The teacher trajectory comes from a 16-step Euler solver. The student needs one conditional forward pass per step and runs at 2 to 4 NFE without separate CFG passes (50K steps).
The pretrained and SOAR checkpoints use 10 Euler steps with CFG (effective NFE 20, CFG scale 1.2).
Key Results
All baseline numbers come from original papers or official default inference configurations; the authors do not re-run baselines under a shared setup (§3.3.1). TTS evaluation has no human listening test. Expressiveness is judged by a Gemini model.
AudioVAE reconstruction (Table 1, LibriSpeech test-other). At 48 kHz and 25 Hz, the dots.tts VAE reaches PESQ-NB 4.09, PESQ-WB 3.95, STOI 0.973, UTMOS 3.75, SIM 0.969 and WER 4.14. SIM 0.969 and WER 4.14 are the best non-oracle values in the table; Semantic-VAE (Semantic-VAE) is next on both at 0.963 and 4.15. MingTok-Audio from Ming-UniAudio is higher on PESQ-NB (4.23), PESQ-WB (4.12) and STOI (0.981), and SAC (SAC) is higher on UTMOS (3.84). The discrete codecs, including WavTokenizer, XY-Tokenizer and X-codec2 from Llasa, trail the continuous representations on PESQ, STOI, SIM and WER. The baselines run at 16 or 24 kHz while the dots.tts VAE runs at 48 kHz, so the comparison is not matched on bandwidth.
Seed-TTS-Eval (Table 2). The authors report the best average on both metrics. Average SIM: SOAR 79.2, Pretrain 78.8, MF NFE=4 78.2, against Seed-TTS 77.8 and VoxCPM 2 76.7. Average WER: Pretrain 2.92%, MF NFE=4 2.94%, SOAR 2.95%, against CosyVoice 3 at 3.06% and Qwen3-TTS at 3.07%. The averages include only systems that report test-zh-hard; MiniMax-Speech, MegaTTS 3, DiTAR and VibeVoice have no average. Per column, dots.tts does not lead everywhere:
- test-en WER: Qwen3-TTS 1.23 beats dots.tts (best variant MF NFE=4 at 1.29).
- test-zh WER: MiniMax-Speech 0.83 beats dots.tts SOAR at 0.94.
- test-zh-hard WER: CosyVoice 3 5.83 beats dots.tts Pretrain at 6.46.
- test-en SIM: dots.tts SOAR ties MegaTTS 3 at 77.1.
- test-zh SIM (SOAR 81.0) and test-zh-hard SIM (SOAR 79.5) are the highest in the table.
SOAR lifts SIM over Pretrain on all three subsets, but test-zh-hard WER rises from 6.46 to 6.60. MeanFlow at NFE=4 holds WER near SOAR and costs about one SIM point; NFE=3 and NFE=2 degrade further (average WER 3.21 and 3.43).
MiniMax-Speech multilingual, 24 languages (Table 3). SOAR has the highest average SIM (83.9), ahead of VoxCPM 2 (82.3), Fish Audio S2 (78.0), MiniMax (76.6) and ElevenLabs (65.5). The authors report a dots.tts variant leading per-language SIM outright on 19 of 24 languages. Average WER is the weak side: 6.6% (Pretrain) and 6.8% (SOAR, MF4), against MiniMax 2.8%, Fish Audio S2 3.7% and VoxCPM 2 5.7%; only ElevenLabs (7.5%) is higher. Outliers drive the gap: Arabic WER is 36.19% for SOAR against 1.67% for MiniMax, and Hindi, Thai, Turkish and Korean are also well above the best baseline. The authors attribute this to insufficient BPE coverage for script-divergent, low-resource languages.
CV3-Eval (Table 4). On monolingual subsets, MF NFE=4 has the lowest hard-en WER (4.37% vs Fish Audio S2 4.4%). Fish Audio S2 leads zh (2.65%) and en (2.43%), and VoxCPM 2 leads hard-zh (8.55%). On cross-lingual cloning, the only baselines with numbers are CosyVoice 2 and 3. dots.tts SOAR has the highest SIM in both directions (en to zh 75.0, zh to en 72.8, against CosyVoice 3 at 66.9 and 66.4). CosyVoice 3 has lower cross-lingual WER (8.01% vs 10.75% en to zh; 4.32% vs 5.66% zh to en).
EmergentTTS-Eval (Table 5, Gemini-2.5-Pro judge, win-rate vs gpt-4o-mini-tts). Pretrain has the highest overall win-rate among open-source systems (49.2%; MF4 47.9%, SOAR 47.6%, Qwen3-TTS 42.8%, VoxCPM2 41.1%, F5-TTS 15.3%). It still trails Gemini-2.5-Flash-TTS (70.7%), Gemini-2.5-Pro-TTS (69.3%) and gpt-4o-audio-preview (65.2%). SOAR has the top Syntactic Complexity score across all rows (65.7%, next MF4 64.8% and Gemini-2.5-Pro 61.8%). SOAR also has the lowest WER among open-source rows (10.45%), though closed-source MiniMax speech-02-hd is lower (10.02%). Pretrain leads open-source systems on Emotions (72.7%; next Qwen3-TTS with the Ryan voice at 60.5%). Complex Pronunciation (16.4% to 18.0%) and Foreign Words (36.3% to 39.5%) are weak.
Efficiency (§3.4). On one H800 under vllm-omni, with MF at NFE=4, time to first packet is 85.4 ms at RTF 0.231 in plain mode and 54.4 ms at RTF 0.245 in 1T1A interleaved mode. The paper reports no quality numbers for the interleaved mode, so its quality cost is not shown.
Novelty Assessment
Most components are adapted from cited work. The VAE recipe comes from HoliTok, latent flow regularization from KALL-E, the AR diffusion-head formulation from ARDiT and DiTAR, the self-correction objective from SOAR (originally for image diffusion), and few-step distillation from MeanFlow. The new parts are in how these pieces combine and train:
- The LLM only sees a learned 6.25 Hz semantic re-encoding of its own generated latents.
- The flow-matching head conditions on the full history of LLM states and clean patches, trained in one pass with a block-causal mask that reproduces inference exactly.
- The reward-free self-corrective objective is applied to an AR speech flow head.
- CFG is fused into the MeanFlow target, so the distilled student needs no unconditional pass.
The paper has no component ablations. It does not test what full-history conditioning, semantic-only feedback or the Stage 2 VAE losses each contribute. The only ablation-like evidence is the Pretrain, SOAR and MF rows. The scale (1.5M hours, mostly in-house) and the full open release are a large part of the practical contribution.
Field Significance
Tip
High — dots.tts gives a released, fully documented recipe for continuous-latent AR TTS whose zero-shot SIM and WER match or exceed strong discrete-token systems on Seed-TTS-Eval. It shows a reward-free, flow-native post-training stage as an alternative to reward-model RL for stabilizing continuous AR speech generation.
Claims
- supports: Continuous-latent autoregressive TTS can reach speaker similarity and intelligibility at or above strong discrete-token AR systems on standard zero-shot benchmarks.
Evidence: On Seed-TTS-Eval, dots.tts SOAR reaches average SIM 79.2 and average WER 2.95% (Pretrain 2.92%), against CosyVoice 3 at 75.3 / 3.06% and Qwen3-TTS at 74.5 / 3.07%. Per-column WER leads remain with baselines on test-en, test-zh and test-zh-hard. (§3.3.3, Table 2)
- supports: Reward-free self-correction of a flow-matching head, trained on its own off-trajectory rollout states, can improve continuous AR speech generation without a reward model or preference data.
Evidence: Post-training only the DiT with the SOAR-style objective raises Seed-TTS-Eval average SIM from 78.8 to 79.2 and lowers CV3-Eval hard-en WER from 5.99% to 4.49%. Seed-TTS-Eval average WER moves from 2.92% to 2.95%. (§3.2.3, Table 2, Table 4)
- complicates: Post-training that improves text faithfulness in expressive TTS can reduce emotional and paralinguistic expressiveness.
Evidence: On EmergentTTS-Eval, the Pretrain to SOAR step raises Syntactic Complexity win-rate from 58.4% to 65.7% but lowers Emotions from 72.7% to 63.9% and Paralinguistics from 54.7% to 52.7%. Overall win-rate drops from 49.2% to 47.6%. (§3.3.6, Table 5)
- complicates: Feeding raw BPE text to an LLM-initialized TTS backbone leaves large intelligibility gaps on under-represented, script-divergent languages, even when speaker similarity stays high.
Evidence: On the 24-language MiniMax-Speech set, dots.tts has the highest average SIM (83.9) but an average WER of 6.8%, against 2.8% for MiniMax. Arabic WER is 36.19% against 1.67%, while Arabic SIM stays at 79.1. The authors attribute the gap to BPE token coverage. (§3.3.4, Table 3, §5 Limitations)
- refines: Few-step mean-flow distillation of a CFG-guided flow head keeps intelligibility near the multi-step teacher at around four function evaluations, but similarity and WER degrade at two or three.
Evidence: MF at NFE=4 (no separate CFG pass) has a Seed-TTS-Eval average WER of 2.94% against 2.95% for SOAR at effective NFE 20, at a SIM cost of 1.0 (78.2 vs 79.2). NFE=3 and NFE=2 rise to 3.21% and 3.43% WER. NFE=4 gives 85.4 ms first-packet latency at RTF 0.231 on one H800. (§3.3.3, Table 2, §3.4)
Limitations and Open Questions
- No component ablations. The paper credits stability to full-history flow conditioning, semantic-only LLM feedback and the Stage 2 VAE objectives, but no table isolates their effects. The only supporting remarks are qualitative: pilot runs on the full mixture were unstable during the frozen-LLM modality-alignment stage, and that stage alone reached about 42% WER on Seed-TTS-Eval (§3.2.2).
- Baselines are not re-run. All baseline numbers come from original papers or default releases. AudioVAE baselines run at 16 to 24 kHz while the dots.tts VAE runs at 48 kHz.
- No human listening test. Naturalness and expressiveness are judged only by automatic metrics and a Gemini-2.5-Pro judge on EmergentTTS-Eval.
- Low-resource multilingual WER. WER is high on Arabic, Hindi, Thai, Turkish and Vietnamese, and Complex Pronunciation and Foreign Words scores are weak (§5).
- Instruction control is not evaluated. The authors note that the release lacks explicit style or instruction control, although the model is trained on caption-paired data (§5).
- Streaming quality is not reported. The interleaved 1T1A mode is measured only for latency and RTF.
- Mostly in-house data. About 1.2M of the 1.5M training hours are in-house Chinese/English data, which limits reproducibility from scratch, although code and checkpoints are released under Apache 2.0.
- The authors flag voice-cloning misuse risk and recommend consent policies, synthetic-speech detection and watermarking.
Wiki Connections
- Autoregressive Codec TTS — An LLM backbone autoregressively generates continuous latents of an AudioVAE that the paper itself trains and evaluates on reconstruction, the continuous-latent variant of codec-LM TTS.
- Flow Matching — A per-patch rectified-flow DiT head with full-history conditioning, reward-free self-corrective post-training, and CFG-aware MeanFlow distillation to 2 to 4 NFE.
- Zero-Shot TTS — Evaluated on prompt-based cloning of unseen speakers on Seed-TTS-Eval, MiniMax multilingual, CV3-Eval (including cross-lingual) and EmergentTTS-Eval.
- Neural Audio Codec — Introduces a causal 48 kHz, 25 Hz continuous AudioVAE with WavLM alignment and multitask supervision, benchmarked against discrete codecs and continuous tokenizers.
- Multilingual TTS — Trained on a multilingual corpus and evaluated on 24 languages, with leading average SIM but a large low-resource WER gap.
- Streaming TTS — Fully causal design with a 1T1A interleaved dual-streaming layout, reaching 54.4 ms first-packet latency.
- ARDiT — The autoregressive diffusion-transformer formulation that inspires the semantic-planning and acoustic-rendering decomposition.
- DiTAR — The LM-plus-per-patch-DiT design that the AR-FM head follows, and a Seed-TTS-Eval baseline.
- KALL-E — Source of the flow regularization applied to the AudioVAE latent.
- VoxCPM2 — Strongest open continuous-AR baseline across Seed-TTS-Eval, the multilingual set, CV3-Eval and EmergentTTS-Eval.
- CosyVoice 3 — Discrete-token baseline and source of CV3-Eval; leads cross-lingual WER while dots.tts leads cross-lingual SIM.
- Seed-TTS — Source of the Seed-TTS-Eval benchmark and the next-best average SIM baseline.
- MiniMax-Speech — Source of the 24-language multilingual test set and the lowest-WER baseline on it.
- Fish Audio S2 — Baseline that leads several CV3-Eval monolingual subsets and beats dots.tts on multilingual average WER.