arXiv · 2024 · Preprint
Liu et al. (The Chinese University of Hong Kong, Shenzhen) · → Paper · Demo: ✓ · Code: ?
ARDiT generates continuous mel-spectrogram tokens autoregressively using a decoder-only flow-matching transformer, bypassing discrete audio tokenization entirely and achieving competitive zero-shot TTS quality with single-step inference after diffusion distillation.
Problem
Discrete audio tokenization underpins the dominant class of audio language models, but introduces a fundamental rate-distortion tradeoff: achieving high-fidelity reconstruction requires high bitrate, which in turn produces long token sequences that are expensive to model autoregressively. Existing workarounds either sacrifice fidelity by restricting the bitrate to a compressed, task-specific representation, or assume conditional independence among tokens in ways that can degrade generation quality when the underlying quantization method does not support that assumption. A further problem is that discrete tokenization requires VQ-GAN training with auxiliary losses and codebook management, adding engineering complexity. The paper asks whether autoregressively generating speech in continuous space can sidestep these issues while retaining the architectural benefits of language model decoding.
Method
ARDiT replaces discrete audio tokens with continuous mel-spectrogram tokens produced by a VAE-based autoencoder. The autoencoder encodes log mel spectrograms into a latent sequence at a 4x temporal downsampling factor, yielding 23.5 tokens per second at 1.7 kbps theoretical bitrate. The decoder is a conditional flow-matching DiT that reconstructs mel spectrograms from the latent tokens, enabling near-lossless reconstruction compared to discrete codecs at similar bitrates.
The core autoregressive model (ARDiT) is a decoder-only transformer that operates semi-autoregressively: instead of generating one token at a time, it samples blocks of B continuous tokens jointly using flow matching. The training scheme, adapted from SSD-2, uses a specially designed attention mask that allows all blocks in a sequence to be processed in a single forward pass during training (enabling teacher-forcing), while preserving the strictly causal dependency needed for autoregressive inference with KV-cache. During inference, each block is sampled by solving an ODE over multiple steps, then the next block is generated conditioned on all previous blocks.
Duration control is implemented via a manipulation of RoPE position embeddings: speech token indices are scaled by the ratio of phoneme count to total speech length, giving the model implicit information about the target duration without requiring a separate duration predictor. Fill-in-the-middle (FIM) training extends ARDiT to support zero-shot TTS and text-based speech editing, where both a prefix and suffix of known speech frames condition the generation of the middle section.
To overcome the latency cost of multi-step ODE solving at inference time, the paper applies Distribution Matching Distillation (DMD), which distills each block’s flow-matching model into a single-step generator by minimizing an Integral KL divergence between the student and teacher sample distributions. The distilled student model (ARDiT DMD) generates one or more continuous tokens per network evaluation, with one model variant producing 170ms of 24 kHz speech per step. All transformer models share the same DiT architecture with 12 layers, 1024-dimensional embeddings, and 16 attention heads. Waveforms are synthesized from mel spectrograms using the pre-trained BigVGAN vocoder (112M parameters).
Key Results
On the UniCATS Test Set B for zero-shot TTS, ARDiT(DMD, B=4) achieves MUSHRA naturalness of 79.3 compared to 77.2 for StyleTTS2 and 62.1 for VoiceCraft. Speaker similarity MUSHRA reaches 68.4, significantly above all baselines (p < 0.01 against StyleTTS2’s 52.4). WER with Whisper medium is 1.81%, below the ground truth reconstruction WER of 2.39%, showing excellent intelligibility. ARDiT(B=1, no DMD) already achieves strong speaker similarity (MUSHRA 64.0) while the teacher models lag behind their distilled counterparts on naturalness.
For speech editing on Test Set C, ARDiT(DMD, R=1) achieves MUSHRA naturalness of 77.8 overall (77.5 long, 78.1 short), closely approaching ground truth (78.1), and significantly outperforming both VoiceCraft (64.6) and UniCATS (66.2) at p < 0.01. WER drops to 1.46%, below VoiceCraft’s 1.79%.
The block size ablation (§4.4) shows that autoregressive models consistently outperform the fully non-autoregressive limit (B=INF) on both intelligibility and speaker similarity, with the gap widening with more training. This provides direct evidence that the causal dependency structure in autoregressive generation meaningfully improves TTS quality beyond what parallel generation achieves.
Novelty Assessment
The central novelty is operating the autoregressive language model paradigm in continuous spectrogram space rather than on discrete codec tokens. While continuous-space autoregressive models have been studied in other domains and SSD-2 introduced the attention masking scheme that enables efficient training of autoregressive diffusion transformers, this paper is the first to apply the combination to TTS and to pair it with DMD for single-step inference per block. The FIM-based conditioning for zero-shot TTS and speech editing with a continuous-token AR model is also a genuine contribution. The VAE autoencoder design (separate encoder + flow-matching decoder) is competently executed but not novel on its own.
The evaluation uses MUSHRA rather than the more commonly reported MOS, which makes direct numerical comparison with the broader literature harder. The evaluation dataset is limited to LibriTTS test-clean audiobook speech, and all training is on LibriTTS only, so claims about in-the-wild generalization cannot be drawn from these results. The ablation over block sizes is well-designed and provides genuine scientific value.
Field Significance
Tip
High — ARDiT provides the clearest demonstration to date that continuous-space autoregressive generation can match or exceed discrete-token autoregressive TTS on quality while avoiding the bitrate-fidelity tradeoff of VQ codecs. The combination with DMD distillation also demonstrates a practical path to single-step-per-block inference, which reduces the main latency disadvantage of combining diffusion sampling with autoregressive sequence generation. The paper opens a viable alternative design axis to the dominant codec-language-model paradigm and provides evidence that autoregressive structure continues to matter even when tokens are continuous.
Claims
- Autoregressive generation of continuous speech tokens preserves the quality advantages of high-bitrate representations while avoiding the length constraints of high-bitrate discrete codecs. (§1, §3.2)
- Diffusion distillation into single-step generators can improve perceptual naturalness of TTS samples relative to their multi-step teacher models, not only match it. (§4.2, Table 1)
- Autoregressive decoding order materially improves speech intelligibility and speaker similarity in TTS compared to fully parallel non-autoregressive diffusion generation at the same total computation budget. (§4.4, Figure 4)
- Fill-in-the-middle training enables near-ground-truth speech editing quality with continuous-token autoregressive models, bringing MUSHRA scores within statistical confidence intervals of human recordings. (§4.3, Table 2)
Limitations and Open Questions
Warning
All training and evaluation is on LibriTTS (reading-style English audiobook speech from 2,306 speakers). The paper does not test on any in-the-wild, noisy, or spontaneous speech dataset, and explicitly notes this as a limitation. Generalization to more diverse or large-scale data is unverified.
The evaluation uses MUSHRA rather than MOS, which means direct numerical comparison with most of the TTS literature requires caution. The block-size ablation fixes total computation across conditions by fixing inference steps, but this does not reflect real-world latency (smaller B means more sequential ODE solves). Duration prediction relies on a simple heuristic using reference speech alignment rather than a learned predictor, which may limit applicability when no reference speaker is available. Sampling techniques such as beam search, which are standard for improving discrete LM quality, are not available for ARDiT, and the paper notes this as future work. A failure rate of approximately 7% in speech editing (before post-filtering) is acknowledged, reduced to 2.2% with batch post-filtering at N=8, which adds inference-time cost.
Wiki Connections
- autoregressive-codec-tts — ARDiT challenges the premise of codec-based autoregressive TTS by showing that continuous-space autoregressive generation is a competitive alternative to discrete tokenization.
- flow-matching — the core generative model within each AR block is a flow-matching transformer (DiT), and the DMD distillation is framed in terms of flow-matching velocity fields.
- zero-shot-tts — ARDiT achieves zero-shot speaker adaptation through FIM conditioning on a 3-second audio prompt, matching or exceeding state-of-the-art baselines on the UniCATS benchmark.
- diffusion-tts — ARDiT extends the autoregressive diffusion framework (SSD-2) to speech, and DMD distillation resolves the multi-step inference bottleneck common to diffusion TTS systems.
- distillation — Distribution Matching Distillation is the key inference optimization, collapsing multi-step ODE sampling into single-step generation while improving perceptual quality.
- neural-codec — the paper directly critiques discrete neural codec representations and argues that high-bitrate continuous tokens are preferable for generative modeling quality.
- 2301.02111 (VALL-E) — ARDiT positions itself relative to VALL-E as a continuous-token alternative to the discrete codec LM paradigm, using a similar zero-shot TTS framing via FIM conditioning.
- 2403.16973 (VoiceCraft) — VoiceCraft is a primary baseline on both zero-shot TTS and speech editing; ARDiT’s DMD models substantially outperform it on MUSHRA naturalness.
- 2403.03100 (NaturalSpeech 3) — cited as a representative non-autoregressive diffusion baseline for zero-shot TTS; ARDiT’s continuous-token approach contrasts with its factorized codec design.
- 2206.04658 (BigVGAN) — the pre-trained BigVGAN vocoder is used for mel-to-waveform synthesis throughout ARDiT’s pipeline.
- 2308.16692 (SpeechTokenizer) — cited as an example of sophisticated disentangled discrete codec representation, representing the class of approaches ARDiT aims to circumvent.
- 2210.02747 (Flow Matching) — the theoretical foundation for both the per-block generative model and the DMD distillation procedure used in ARDiT.