arXiv · 2025 · Preprint
Zhijun Liu et al. (ByteDance Seed) · → Paper · Demo: ✓ · Code: ✗
Extends Direct Preference Optimization to autoregressive diffusion models (ARDMs), deriving a training objective that aligns the recently proposed zero-shot TTS model DiTAR to human-preferred speech expressiveness and text robustness.
Problem
Autoregressive diffusion models generate speech by autoregressively predicting continuous latent tokens with a diffusion (denoising) process at each step, avoiding the sequence-length and quantization artifacts of discrete-codec autoregressive TTS. Models in this family (DiTAR and related ARDMs) report state-of-the-art zero-shot TTS quality, but pretraining alone does not guarantee outputs match human preferences: prompted with emotional speech, an ARDM can still produce monotone output, and autoregressive TTS models more generally struggle with alignment errors (missing or inserted words) on long or repetitive texts. Preference-alignment methods such as DPO have been applied to LLMs and, via Diffusion-DPO, to standard (non-autoregressive) diffusion models, but no prior work had derived a preference-optimization objective for the autoregressive-diffusion setting, where each generated token is itself produced by an iterative denoising chain conditioned on prior tokens.
Method
The paper formulates ARDM sampling as a Markov chain over states indexed by (token position, diffusion timestep), where each state carries both the already-denoised history tokens and the currently-noising token. Starting from the standard KL-constrained reward maximization used to derive DPO, the authors marginalize the resulting expression over all intermediate diffusion states of an ARDM trajectory to obtain a Bradley-Terry preference model directly over final token sequences. Applying the same approximation used in Diffusion-DPO (approximating the intractable posterior with the forward diffusion posterior) and Jensen’s inequality to move an expectation outside a log-sigmoid, they derive a tractable lower bound that reduces the reward difference between a preferred and dispreferred sample to a pairwise denoising-loss comparison between the fine-tuned model and a frozen reference model, summed with equal weight across randomly sampled tokens and diffusion timesteps. For DiTAR specifically (a v-prediction ARDM), this reduces to the ARDM-DPO training objective: a log-sigmoid of a weighted difference of squared-error denoising losses (fine-tuned model minus reference model) computed separately on the winning and losing trajectories.
The base model is a 0.4B-parameter DiTAR checkpoint (24-block Transformer LM, 4-block diffusion head, 1024 hidden dim, 16 heads) pretrained on approximately 280,000 hours of internal Chinese/English audio, with generation history encoded separately from next-token denoising and LM Guidance (a classifier-free-guidance analogue) applied at inference with a 16-step DDPM sampler. Preference pairs for two tasks are constructed automatically rather than from human labels: for Task A (expressiveness), 32 candidate continuations per LibriTTS prompt-text pair are ranked by fundamental-frequency variance (F0V), with the highest- and lowest-F0V samples forming a pair (256k pairs, ~1,000 hours); for Task B (robustness on hard/repetitive text), 16 candidates per DidiSpeech-2-prompted, repetition-augmented Chinese sentence are ranked by the negative log-likelihood of a separately trained phoneme-level CTC model, with best/worst forming a pair (430k pairs, ~3,500 hours). DPO fine-tuning uses AdamW (lr 2e-6, effective batch size 1024 pairs) with the KL-regularization weight β swept over a range for each task and normalized by the per-token latent dimensionality.
Key Results
On Task A (expressiveness), the best DPO configuration (200 steps, β=200) raises F0V from 14.2 (base model) to 29.2, more than doubling it, while speaker similarity (SIM) drops only marginally (0.77 → 0.765) and WER improves (5.17 → 3.73). This exceeds both a best-of-64 inference-time sampling baseline (F0V 26.6, no additional training) and iterative rejection-sampling fine-tuning (RAFT, F0V 20.1 after 3 iterations), at a much lower training cost (single DPO run vs. repeated 1,000-hour SFT rounds). A best-of-K control experiment establishes that F0V gains are not achieved by trading away speaker similarity, since increasing F0V via sampling alone does not measurably reduce SIM. Subjective pairwise listening tests (20 listeners, 38 test cases) confirm DPO significantly increases perceived expressiveness while slightly reducing naturalness and speaker similarity relative to the base model.
On Task B (text robustness), the best configuration (9,000 steps, β=1600) reduces CER on the Seed-TTS-Eval hard test set by 25% (8.37 → 6.32) while keeping SIM essentially unchanged (0.711 → 0.712). Subjective evaluation on 40 held-out hard cases shows the DPO model is preferred or tied with the base model on naturalness (95.6% tie/win) and speaker similarity (97.9% tie/win), indicating the robustness gain does not come at a perceptible quality cost. All objective metrics use Seed-TTS-Eval (Whisper-large-v3 WER for English, Paraformer-zh CER for Chinese, WavLM-TDCNN speaker cosine similarity), and each number is averaged over 8 random runs.
Novelty Assessment
The contribution is a training-objective derivation, not a new architecture: the paper extends the DPO/Diffusion-DPO derivation to the autoregressive-diffusion setting by treating ARDM sampling as a Markov chain over (token, timestep) states and marginalizing over intermediate diffusion states, which is a genuine and non-trivial mathematical step not present in either the original DPO or Diffusion-DPO derivations. The empirical validation is solid — two distinct tasks, a grid search over the KL-regularization strength, and comparisons against both inference-time (best-of-K) and training-time (RAFT) alternatives — but is conducted entirely on a proprietary 280,000-hour internal corpus and a single 0.4B base model, with no code release, which limits independent reproduction. The preference signals themselves (F0 variance, CTC-model negative log-likelihood) are automatic proxies rather than human preference labels, so the method demonstrates that ARDM-DPO can optimize a chosen scalar objective, not that it captures broader human preference beyond what these proxies encode.
Field Significance
Moderate — this is, by the authors’ own account, the first preference-alignment method derived specifically for autoregressive diffusion models applied to speech, extending a line of DPO-for-TTS work that has so far targeted discrete-token autoregressive-LM systems. It demonstrates that KL-regularized preference optimization is compatible with the ARDM training objective and yields measurable gains in expressiveness and text robustness on a competitive zero-shot TTS base model, but the evaluation is confined to a single internal model and corpus, and the authors themselves flag training instability on one of the two tasks as unresolved.
Claims
- supports: Direct Preference Optimization can be extended beyond discrete-token generative models to continuous-token autoregressive diffusion models for speech, by marginalizing the DPO objective over intermediate diffusion states of the autoregressive sampling trajectory.
Evidence: The derived ARDM-DPO objective reduces to a log-sigmoid of the difference in denoising losses between the fine-tuned and reference model on winning vs. losing trajectories; applied to a 0.4B DiTAR model it nearly doubles F0 variance (14.2 → 29.2) while WER improves (5.17 → 3.73) after 200 training steps. (§3, §4.2, Table 1)
- supports: Automatic proxy signals, rather than human-labeled preference pairs, can be used to construct DPO training data that improves a targeted, measurable aspect of speech generation quality.
Evidence: Task B builds 430k preference pairs by ranking 16 candidate outputs per prompt using a separately trained CTC model’s negative log-likelihood as an intelligibility proxy; this reduces CER by 25% (8.37 → 6.32) on the Seed-TTS-Eval hard test set while preserving speaker similarity and naturalness in subjective tests. (§4.3, Table 2)
- complicates: DPO fine-tuning of autoregressive diffusion speech models is not guaranteed to be stable and can require early stopping to prevent degradation of speaker similarity and naturalness.
Evidence: Across the β ∈ {200, 400, 800} grid search for Task A, speaker similarity (SIM) decreases monotonically throughout training regardless of β, and the diffusion loss for both preferred and dispreferred samples increases during training instead of diverging as the objective predicts, a phenomenon the authors note also occurs in LLM DPO training and leave uninvestigated. (§4.2, Fig. 3, §5)
- refines: Preference-optimizing a generative TTS model via a single fine-tuning run can match or exceed the quality gains obtained by inference-time best-of-K sampling or iterative rejection-sampling fine-tuning, without their added inference-time or multi-round training cost.
Evidence: On Task A, ARDM-DPO (200 steps, β=200) reaches F0V 29.2, higher than best-of-64 sampling (26.6, no training) and three iterations of RAFT rejection-sampling SFT (20.1), while requiring a single training run and no extra inference-time sampling. (§4.2, Table 1)
Limitations and Open Questions
Warning
All experiments fine-tune a single proprietary 0.4B DiTAR checkpoint pretrained on an internal 280,000-hour corpus, with no code or model release; the reported gains cannot be independently reproduced or verified on a public base model.
Training instability on Task A (monotonically decreasing speaker similarity across all tested β values, and diffusion loss increasing for both winning and losing samples contrary to the theoretical objective) is acknowledged as unresolved and requires manual early stopping rather than a principled stopping criterion. The preference signals used (F0 variance, CTC-model negative log-likelihood) are automatic proxies for expressiveness and intelligibility rather than direct human preference judgments, so the method is validated against these specific proxies rather than against a broader notion of listener preference; the paper explicitly leaves investigation of preference-dataset construction quality, and the root cause of the training instability, to future work.
Wiki Connections
- RLHF Speech — derives a Direct Preference Optimization variant specifically for autoregressive diffusion TTS, extending preference-alignment training beyond discrete-token autoregressive-LM and standard diffusion settings.
- Zero-Shot TTS — fine-tunes a zero-shot prompt-conditioned TTS model (DiTAR) via preference optimization, improving expressiveness and text robustness without altering its zero-shot capability.
- Diffusion TTS — extends the Diffusion-DPO derivation to the autoregressive-diffusion setting, where each generated token is itself produced by a diffusion denoising process conditioned on prior tokens.
- Subjective Evaluation — validates objective F0-variance and CER gains with pairwise listening tests (20 listeners) on naturalness, speaker similarity, and expressiveness.
- DiTAR — is the base zero-shot TTS model this paper fine-tunes with ARDM-DPO; no architectural changes are made to DiTAR itself.
- Autoregressive Diffusion Transformer for TTS — is the precursor ARDM architecture (by overlapping authors) that DiTAR builds on and that motivates the autoregressive-diffusion Markov-chain formulation used here.
- Enhancing Zero-shot TTS with Human Feedback — is prior work applying human-feedback-based alignment to autoregressive-LM zero-shot TTS, which this paper extends to the autoregressive-diffusion setting.
- MPO — is concurrent work applying preference optimization to language-model-based TTS, forming part of the same emerging line of DPO-for-TTS methods.
- Next Tokens Denoising for Speech Synthesis — is concurrent work on next-token-diffusion speech generation, part of the broader ARDM paradigm this paper’s alignment method targets.
- Seed-TTS — supplies the Seed-TTS-Eval toolkit (WER/CER and speaker-similarity computation) used for all objective evaluation in this paper.
- CosyVoice 3 — is a comparable large-scale TTS system that likewise incorporates post-training/alignment steps, illustrating the broader trend toward preference-based post-training in production TTS systems.
- Multimodal Latent Language Modeling with Next-Token Diffusion — proposes the general next-token-diffusion paradigm that the ARDM formulation used in this paper’s derivation instantiates for speech.