arXiv · 2025 · Preprint
Dongya Jia et al. (ByteDance Seed) · → Paper · Demo: ✗ · Code: ✗
DiTAR introduces a patch-based autoregressive framework that combines a causal language model for inter-patch prediction with a bidirectional diffusion transformer (LocDiT) for intra-patch generation of continuous speech representations, achieving state-of-the-art zero-shot TTS robustness and quality at 3–43x lower compute than competing NAR diffusion systems.
Problem
Autoregressively generating continuous speech representations (without discrete tokenization) has been hindered by two failure modes: (1) using a causal AR model with a diffusion head leads to suboptimal performance because the causal unidirectional attention conflicts with the close inter-frame correlations characteristic of continuous tokens; (2) approaches that share parameters between AR and diffusion (e.g., Transfusion, ARDiT) accumulate substantial computational demands as sequence length and model size grow. The standard two-stage coarse-to-fine pipeline with discrete tokens also suffers from error accumulation between stages.
Method
DiTAR uses a continuous VAE tokenizer (encoder follows multiple Conv layers with adversarial training via BigVGAN decoder; the VAE compresses 24 kHz waveforms to 40 Hz latent vectors of dimension 64). There are no discrete codebooks; the model operates entirely on continuous representations.
The architecture has three modules:
-
Aggregation Encoder: A bidirectional transformer that processes P consecutive continuous tokens (a “patch”) and produces a single aggregated embedding via a learnable special token at the beginning of each patch. Both encoder and decoder use Pre-Norm, RMSNorm, and RoPE.
-
Causal Language Model (LM): Processes the sequence of aggregated patch embeddings autoregressively to produce a conditioning embedding h_i for each new patch.
-
LocDiT (Local Diffusion Transformer): A bidirectional attention DiT that generates the next patch of continuous tokens given h_i, using a conditional flow-matching loss. Crucially, historical context patches are provided as prefix inputs to LocDiT, framing the task as outpainting rather than generation from scratch — this is essential (removing history causes WER to degrade from 1.736 to 22.874 for patch size 4).
LM Guidance: During training, LM output h_i is randomly replaced with a zero vector (p=0.1) to train unconditional LocDiT. At inference, classifier-free guidance is applied by computing LocDiT twice (conditional and unconditional), with only one LM computation — reducing CFG overhead relative to discrete LMs.
Temperature Sampling: Temperature τ ∈ [0,1] is defined as the time point to introduce noise in the reverse diffusion ODE. At τ=1, sampling is stochastic (standard); at τ=0, generation is deterministic. At τ∈(0,1), an intermediate x̂_0 is estimated and noise is injected at that point, balancing diversity and determinism.
For zero-shot TTS, the system formats input as: phoneme embeddings concatenated with reference speech embeddings as prefix; the LM autoregressively generates target speech patches; a binary stop classifier at the LM output predicts termination. Training uses AdamW with constant lr=1e-4 on 16 A100 GPUs (0.6B) or 32 A100 GPUs (1B), batch 15k tokens/GPU, 0.5M steps.
Key Results
On LibriSpeech test-clean subset B (F5-TTS protocol, Emilia-trained), DiTAR achieves WER 2.39% and UTMOS 4.22 at 0.6B/NFE=10 — better than MaskGCT (WER 2.72, UTMOS 3.90, 1.1B, NFE=50) and F5-TTS (WER 2.42, UTMOS 3.88, 0.3B, NFE=32), with 2.75 TFLOPs vs. 37.36 (F5-TTS) and 116.66 (MaskGCT). On subset A (NaturalSpeech3 protocol, Librilight-trained), DiTAR achieves WER 1.78%, SIM 0.64, UTMOS 4.15 — best WER among all systems. Subjective evaluation on subset B: DiTAR achieves N-MOS 3.69, Q-MOS 3.87, S-MOS 3.55, CMOS 0.00 (used as reference baseline), outperforming E2TTS (CMOS −0.32) and F5TTS (CMOS −0.04). At 1B on Seed-EN, DiTAR achieves WER 1.685 and SIM 0.735, outperforming Seed-TTS DiT on robustness (1.733 WER).
Scaling is well-behaved: both data scale (20k → 280k hours) and model scale (0.1B → 1B) monotonically improve WER and SIM. The LM and LocDiT benefit more from scaling than the encoder.
DiTAR computes FLOPs 3–43x lower than competing NAR systems (NAR 60–117 TFLOPs vs. DiTAR 2.75 TFLOPs at NFE=10), due to the patch-level design shortening the causal LM’s sequence.
Novelty Assessment
The core insight is the divide-and-conquer patchification: the causal LM handles inter-patch (long-range, low-frequency) structure while the bidirectional DiT handles intra-patch (high-frequency, locally correlated) fine detail. This cleanly separates the problem and avoids both the failure modes of pure diffusion (slow, no AR latency benefit) and pure causal AR with continuous tokens (poor quality due to unidirectional bias on correlated frames). The historical context mechanism for LocDiT (outpainting formulation) and the temperature definition for continuous ODE diffusion are novel. The paper is accepted at ICML 2025, suggesting community validation of the approach’s significance.
Field Significance
Tip
High — DiTAR introduces a structurally novel solution to continuous-token autoregressive speech generation, resolving the long-standing tension between unidirectional causal attention and the bidirectional correlation structure of continuous acoustic frames. Its divide-and-conquer patchification approach enables single-stage generation without discrete tokenization while maintaining AR latency properties, providing a new architectural direction that subsequent work (DiSTAR, Dragon-FM, DualSpeechLM) builds on or benchmarks against.
Claims
- Causal autoregressive modeling with a diffusion head on continuous tokens yields substantially degraded robustness compared to non-causal approaches, because unidirectional attention is misaligned with the inter-frame correlation structure of continuous acoustic representations. (§3.1.2, §4.3)
- Providing historical context patches as prefix inputs to a local diffusion transformer is critical for generation quality: without context, robustness collapses catastrophically, while even one historical patch restores near-full performance. (§3.2, Table 5)
- Classifier-free guidance applied exclusively to the diffusion decoder (LM Guidance), requiring only one LM inference pass, significantly improves generation quality while keeping computational cost substantially lower than guidance methods that double the full model. (§3.3, §4.3, Figure 4)
- Patch-based autoregressive generation with a local diffusion decoder achieves comparable or superior quality to fully non-autoregressive diffusion systems at 3–43x lower compute, because the AR structure shortens the effective sequence over which diffusion must operate. (§4.1.2, Table 1)
- Defining temperature as the noise injection time point in the reverse diffusion ODE provides a smooth diversity-determinism trade-off for continuous-valued language models, without requiring slow SDE solvers. (§3.4, §4.4.1, Table 6)
Limitations and Open Questions
DiTAR generates continuous representations that directly pass to BigVGAN without a separate vocoder; the quality ceiling is thus tied to the VAE reconstruction quality. The system has not been extended to multilingual settings or streaming. NFE=10 is assumed throughout experiments — the optimal NFE vs. quality trade-off for real-time applications needs further study. Patch size 4 is optimal empirically but the reasons for the U-shaped curve (performance degrades at both very small and very large patches) are only partially explained.
Wiki Connections
DiTAR is a foundational paper for diffusion-tts and autoregressive-codec-tts, specifically for the hybrid continuous AR+diffusion paradigm. It directly challenges the assumption that discrete tokenization is necessary for AR speech generation. The LM guidance approach and temperature definition for ODE diffusion are contributions to the general diffusion-tts methodology. 2512.14291 (GLM-TTS) uses DiTAR as a baseline (CER 1.02, SIM 75.3 on Seed-TTS-eval test-zh at 0.6B), confirming external reproduction of DiTAR’s competitive performance.
Citing papers in this corpus (integrated 2026-05-29): 2603.29339 (LongCat-AudioDiT) benchmarks against DiTAR on Seed benchmark; 2510.12210 (DiSTAR) builds on the patch-based AR drafting concept from DiTAR but extends it to discrete RVQ code space. Back-linked by 2507.22746 (Dragon-FM), which compares chunk-wise AR+FM hybrid with DiTAR as a related architecture; and by 2508.08961 (DualSpeechLM), which benchmarks against DiTAR’s continuous AR approach.