arXiv · 2025 · Preprint
Xinlu He et al. (Amazon AGI) · → Paper · Demo: ✗ · Code: ✗
Introduces a frame-by-frame autoregressive diffusion head grafted onto a multimodal LLM backbone, generating continuous speech embeddings without quantization while a retained language-model head controls variable-length speech boundaries.
Problem
MLLM-based TTS approaches typically frame speech generation as next-token prediction over discrete codec tokens, inheriting the unified-framework advantages of large language models but inheriting quantization’s information loss along with them: discrete tokenization discards fine-grained acoustic detail, limiting naturalness and fidelity. Continuous representations avoid this bottleneck but are harder to integrate into an autoregressive LLM, which by construction predicts discrete symbols. Prior work combining diffusion with autoregressive generation either routes through intermediate discrete semantic tokens or relaxes strict frame-level causality by predicting multi-frame blocks per autoregressive step, sacrificing some of the fine granularity that motivated continuous representations in the first place. A further practical gap is that continuous-token TTS systems built as MLLMs need a way to determine where speech starts and stops without external classifiers, and joint training of an autoregressive backbone with a continuous-output diffusion module is unstable because the diffusion head must track the backbone’s non-stationary output distribution.
Method
The system extends an autoregressive multimodal LLM (OPT-125M backbone) with a dual-head architecture: a retained LM head that emits control tokens (<speech_bos>, <cont_speech_gen>, <eos>) to mark the start, continuation, and end of speech generation, and a diffusion head that generates the actual frame-level continuous speech embeddings. At each autoregressive step, the LLM produces a hidden state conditioning a lightweight MLP denoiser (implemented with residual blocks, layer norm, SiLU activation, and adaptive layer-norm modulation) that runs a DDPM-style process (T=1000 training steps, 100 inference steps, cosine noise schedule) to predict the next frame’s embedding from Gaussian noise. Because the LM head alone controls sequence boundaries via explicit supervision of the <cont_speech_gen> token during training, no external classifier or fixed-length constraint is needed to determine when speech generation should terminate.
Speech targets are 64-dimensional continuous embeddings at 25 frames per second from a pretrained VAE-based vocoder (five downsampling ResNet blocks plus a VAE bottleneck over 48kHz stereo waveforms); speaker reference conditioning uses a 768-dimensional embedding extracted from a 3-second reference clip via a separate speaker encoder (LAM), projected into the LLM input space. Training combines a standard noise-prediction diffusion loss with cross-entropy loss on the LM head’s control tokens.
Two training techniques address instability specific to this design. First, masked autoregressive learning: during training, ground-truth embedding frames fed into the causal predictor are independently zeroed out with probability p_mask (a Bernoulli mask), simulating the imperfect self-generated histories the model will see at inference and mitigating exposure bias from teacher forcing. Second, a two-stage training scheme: stage 1 jointly trains the LLM backbone and diffusion head end-to-end; stage 2 freezes the entire LLM side (backbone, LM head, and speech projections) and trains only the diffusion head, fixing its input distribution so it can focus on refining the mapping from LLM hidden states to acoustic embeddings rather than chasing a moving target.
Key Results
On LibriSpeech(PC) test-clean, the proposed 160M-parameter system reaches WER 1.95%, speaker similarity (SIM) 0.54, and UTMOS 4.00 (§5.1, Table 1). Against reproduced baselines from the NaturalSpeech3 protocol, this beats VALL-E (discrete tokens, 400M params: WER 6.11%, SIM 0.47, UTMOS 3.68), MegaTTS (continuous tokens, 500M params: WER 2.32%, SIM 0.53, UTMOS 4.02), Voicebox (non-autoregressive, 400M params: WER 2.14%, SIM 0.48, UTMOS 3.73), and StyleTTS2 (non-autoregressive, 700M params: WER 2.49%, SIM 0.38, UTMOS 3.94) (§5.1, Table 1). The comparison uses reported/reproduced numbers from prior papers rather than the authors’ own re-runs of every baseline, and the proposed system has roughly 2.5–4.4x fewer parameters than the compared baselines.
The two-stage training scheme accounts for the largest internal gain: the stage-1-only baseline reaches WER 3.61%, SIM 0.49, UTMOS 3.21, while adding stage-2 fine-tuning of the diffusion head drops WER to 1.95% (46% relative reduction), raises SIM to 0.54, and raises UTMOS to 4.00 (§5.1). Masking rate ablation shows unmasked training (0%) suffers severe exposure bias (WER 15.06%), while 30% masking is best (WER 6.17% at this ablation’s 3-layer-MLP setting; UTMOS 3.21), and 50% masking degrades performance again (WER 8.13%) (§5.2, Table 2). Diffusion head depth ablation shows monotonic gains from 3 to 12 MLP layers before stage-2 training is applied (WER 6.17% → 3.61%) (§5.2, Table 3).
Novelty Assessment
The core architectural claim, a strictly frame-by-frame autoregressive diffusion head operating directly on continuous embeddings within an MLLM (as opposed to block-wise multi-frame diffusion or routing through intermediate discrete semantic tokens), is a genuine design contribution distinguishing this work from prior autoregressive-diffusion hybrids cited in the paper. The dual-head boundary-control mechanism, using the LM head’s own token vocabulary to mark speech start/continuation/end without an external classifier, is a clean integration choice specific to the MLLM setting. Masked autoregressive training for exposure bias is adapted from existing text and audio generation literature rather than novel to this paper, and the two-stage freeze-then-fine-tune scheme is a training-recipe contribution rather than an architectural one, though the paper’s own ablation shows it is responsible for the majority of the reported improvement (WER 3.61% → 1.95%). The comparison set (VALL-E, MegaTTS, Voicebox, StyleTTS2) reflects 2023-era systems reproduced via a third paper’s protocol rather than head-to-head runs against more recent zero-shot TTS baselines, which limits how strongly the “state-of-the-art autoregressive performance” claim in the abstract should be read.
Field Significance
Moderate — this paper demonstrates that a strictly frame-level autoregressive diffusion head can be integrated into a multimodal LLM backbone for continuous-representation TTS, achieving competitive intelligibility and speaker similarity at a fraction of the parameter count of the continuous-token and non-autoregressive baselines it compares against. Its main technical contributions, the dual-head boundary control and the two-stage training scheme for stabilizing joint AR-diffusion optimization, are useful, reusable design choices for future MLLM-based continuous-token TTS systems, but the evaluation is confined to a single benchmark (LibriSpeech(PC) test-clean) against older reproduced baselines.
Claims
- supports: Continuous, frame-level speech representations can be generated autoregressively by a diffusion head conditioned on a causal language model’s hidden state, without an intermediate discrete tokenization step.
Evidence: The proposed dual-head model attains WER 1.95%, SIM 0.54, UTMOS 4.00 on LibriSpeech(PC) test-clean, outperforming a discrete-token AR baseline (VALL-E: WER 6.11%, SIM 0.47) despite using fewer parameters. (§5.1, Table 1)
- supports: Freezing an autoregressive backbone before fine-tuning a downstream continuous-output diffusion module improves training stability and final quality when the two components are otherwise trained jointly.
Evidence: A two-stage scheme (joint training in stage 1, backbone-frozen diffusion-head-only training in stage 2) reduces WER by 46% relative (3.61% to 1.95%) and improves SIM (0.49 to 0.54) and UTMOS (3.21 to 4.00) over the stage-1-only baseline. (§5.1, §3.4)
- complicates: Masked training used to mitigate autoregressive exposure bias in continuous-token generation is sensitive to the masking ratio, with both too little and too much corruption degrading output quality.
Evidence: WER is 15.06% with no masking, improves to a best of 6.17% at 30% masking, and worsens again to 8.13% at 50% masking (3-layer MLP diffusion head setting). (§5.2, Table 2)
- complicates: Diffusion-based boundary control learned end-to-end via a language-model head can be a practical substitute for oracle stopping signals, but does not fully close the gap to oracle-quality stopping.
Evidence: The proposed EOS-token stopping criterion achieves WER 3.61% and UTMOS 3.21, matching (not exceeding) an oracle end-of-speech stopping-point baseline’s WER 3.46% and UTMOS 3.21, while a naive ground-truth-duration baseline is far worse (WER 29.36%). (§5.2, Table 4)
Limitations and Open Questions
The evaluation is limited to a single dataset condition (LibriSpeech(PC) test-clean, 40 speakers) and a single training corpus (a 50k-hour LibriVox/Libri-Light subset), so generalization to noisier, multilingual, or lower-resource settings is untested. Baseline comparisons in the main results table are drawn from a third-party reproduction (NaturalSpeech3’s protocol) rather than the authors’ own re-implementation, and all compared systems (VALL-E, MegaTTS, Voicebox, StyleTTS2) predate this paper by roughly two years, leaving open how the approach compares against more recent zero-shot TTS systems. The backbone used (OPT-125M) is small relative to contemporary LLM-based TTS systems, so it is unclear whether the reported gains from the dual-head design and two-stage training persist at larger scale. No code or demo release is mentioned in the paper.
Wiki Connections
- Autoregressive Codec TTS — proposes an alternative to discrete-codec autoregressive TTS by replacing the codec token vocabulary with a diffusion head over continuous embeddings, while retaining an autoregressive LM backbone.
- Diffusion TTS — applies diffusion in a strictly frame-by-frame autoregressive mode rather than the typical non-autoregressive full-sequence denoising used elsewhere in diffusion TTS.
- Zero-Shot TTS — conditions on a short speaker reference clip to synthesize speech for unseen speakers without speaker-specific training.
- Neural Audio Codec — deliberately avoids discrete neural codec tokenization, instead generating continuous VAE-based vocoder embeddings directly.
- VALL-E — used as the discrete-token autoregressive baseline that this paper’s continuous-token approach is shown to outperform in WER and speaker similarity at lower parameter count.