arXiv · 2025 · Preprint
Jiacheng Shi et al. · → Paper · Demo: ✓ · Code: ?
Introduces stepwise preference optimization for diffusion TTS, replacing endpoint-only DPO-style emotion alignment with dense, time-conditioned reward signals computed at every denoising step.
Problem
Emotional TTS systems typically condition generation on coarse categorical emotion labels or rely on proxy classifiers to judge whether output matches an intended affect. Preference-based alignment methods such as Direct Preference Optimization (DPO) offer an alternative, but when applied to diffusion models the standard practice is to assign a preference label only at the final generated output and propagate it uniformly back across all intermediate denoising steps. The paper argues this endpoint-propagation assumption is often invalid for diffusion trajectories, since not every intermediate state on a path toward a “preferred” final output is itself preferable, and that this yields sparse, temporally coarse supervision for signals like prosody and emotional expressiveness that unfold gradually across the denoising process.
Method
The paper proposes Emotion-Aware Stepwise Preference Optimization (EASPO), which fine-tunes a pretrained Grad-TTS diffusion decoder using preference feedback computed locally at each denoising step rather than at the trajectory endpoint. At a given latent state, the policy samples a small set of candidate next-step mel-spectrograms; a frozen scoring model ranks them, and the highest- and lowest-scoring candidates form a win/lose pair used to update the policy, after which one candidate is chosen at random (not the winner) to continue the rollout, decoupling supervision from sampling and avoiding biased trajectories toward degenerate regions.
The scorer, Emotion-Aware Stepwise Preference Model (EASPM), is built on CLEP, a CLAP-style contrastive audio-language encoder, adapted with a time-aware normalization layer so it can score noisy intermediate mel-spectrograms rather than only clean audio. EASPM is fine-tuned on emotion-labeled preference pairs derived from MSP-Podcast, where a target emotion (e.g. “happy”) is marked preferred over a distractor (e.g. “neutral”) for matched text, with both waveforms perturbed by the same diffusion noise at a sampled timestep to make scoring step-aware.
The alignment objective (EASPO) formulates denoising as a T-step MDP and matches the difference in log-likelihood ratios (relative to a frozen reference policy) between the win and lose candidates to the corresponding difference in EASPM reward scores, via a mean-squared error with a time-dependent weight. Training skips the earliest, highest-noise denoising steps (controlled by a threshold κ) since these carry limited speech structure, and randomly shuffles the optimization step to improve sample efficiency. The base TTS model is Grad-TTS with 80-dim mel-spectrograms; the text encoder and duration predictor are frozen, and only the decoder (score network) is fine-tuned. A pretrained HiFi-GAN vocoder converts mel-spectrograms to waveform at inference.
Key Results
On the English ESD test split, the method outperforms seven emotion-controllable TTS baselines (FG-TTS, PromptTTS, EmoSpeech, EmoDiff, CosyVoice, CosyVoice2, EmoVoice) on all four objective metrics reported: emotion similarity (Emo SIM 99.15 vs. 98.59 for the next-best, EmoVoice), prosody similarity (3.89 vs. 3.78 for CosyVoice2), WER (3.74, second-lowest after PromptTTS’s 3.25), and UTMOS (4.47 vs. 4.43 for CosyVoice2). A 20-rater listening test against four of the strongest baselines shows consistent gains in naturalness (MOS 3.94 vs. 3.63 for CosyVoice2), emotional expressiveness (Emo MOS 4.28 vs. 3.79), emotion-text consistency (MOS EC 4.04 vs. 3.83 for CosyVoice2), and emotion recognition accuracy by raters (85.84% vs. 82.10% for CosyVoice2).
Ablations isolate the source of the gains: comparing EASPO against three prior diffusion-based RL alignment methods (DDPO, D3PO, Diffusion-DPO) applied to the same Grad-TTS backbone shows consistent improvements from stepwise, time-conditioned supervision over endpoint-propagated preference (Table 8). Removing EASPM’s timestep conditioning or its CLEP initialization each degrades all four objective metrics (Table 3). Randomly selecting the next rollout state from the candidate pool, rather than continuing from the top-ranked candidate, improves results by avoiding bias toward dispreferred regions (Table 4/text). A denoising window of roughly [0, 750] out of 1000 steps (skipping the noisiest early steps) gives the best balance of diversity and emotional clarity, and a candidate pool of k=4 outperforms k=2 and k=8.
Novelty Assessment
The contribution is primarily a training-recipe and reward-model innovation rather than a new base architecture: the TTS backbone (Grad-TTS) and the contrastive audio-language scorer (CLAP/CLEP) are both pre-existing, and the reward-difference matching objective is adapted from a prior proximal reward-difference method (PRDP) originally developed for image diffusion alignment. The genuinely new element is combining stepwise, time-conditioned preference scoring with local win/lose candidate selection from a shared latent, replacing the endpoint-propagation assumption used by prior diffusion-DPO variants (Diffusion-DPO, DDPO, D3PO, EmoDPO). The comparison against these RL-alignment baselines under a matched Grad-TTS backbone (Table 8) is a fair, controlled test of the stepwise-vs-endpoint design choice specifically, which strengthens the claim beyond simply reporting better numbers than unrelated systems. The broader comparison against seven emotion-TTS baselines (Table 1/2), by contrast, mixes architectures, training data, and model scale, so those gains cannot be attributed to the stepwise mechanism alone.
Field Significance
Moderate — the paper provides a controlled ablation showing that stepwise, time-conditioned preference supervision outperforms endpoint-propagated DPO-style alignment for diffusion TTS on the specific problem of emotional expressiveness, which sharpens a design question (where in the diffusion trajectory should preference feedback be applied) that recent diffusion-alignment work had left open. The evaluation is limited to a single backbone (Grad-TTS) and a single, small emotion benchmark (English ESD).
Claims
- supports: Applying preference supervision at each denoising step of a diffusion model, rather than only at the final generated output, improves alignment with a fine-grained target attribute compared to endpoint-propagated preference optimization.
Evidence: Under a matched Grad-TTS backbone, the stepwise method outperforms three prior diffusion-DPO variants (DDPO, D3PO, Diffusion-DPO) on emotion similarity, prosody similarity, WER, and UTMOS. (§3.4, Table 8)
- complicates: The assumption underlying trajectory-level preference propagation in diffusion models, that all intermediate states on a path toward a preferred final output are themselves preferred, does not reliably hold.
Evidence: The paper motivates its stepwise design specifically by rejecting this assumption, and shows local win/lose selection at a shared latent outperforms endpoint-propagated variants. (§1, §2.2)
- supports: A contrastive audio-language scoring model can be adapted to evaluate noisy intermediate diffusion states, enabling automatic preference-pair construction without categorical emotion classifiers.
Evidence: Adding a time-aware normalization layer to a CLAP-based encoder (CLEP) and training it with diffusion-perturbed audio pairs lets it score emotional-prompt consistency directly on noisy mel-spectrograms; removing this timestep conditioning degrades Emo SIM, Prosody SIM, WER, and UTMOS. (§2.1, Table 3)
- complicates: In RLHF-style fine-tuning of diffusion generation, continuing rollouts from the highest-reward candidate at each step, rather than a randomly selected candidate, biases training toward degenerate regions and reduces overall quality.
Evidence: The paper reports that random selection of the next rollout state from the candidate pool consistently improves performance relative to continuing from the top-ranked (winning) candidate. (§2.1, §3.4, Table 4)
Limitations and Open Questions
Warning
Evaluation is confined to a single diffusion TTS backbone (Grad-TTS) and a single, small English emotion corpus (ESD: 5 emotions × 10 speakers, 350 utterances/emotion). Generalization to other diffusion or flow-based TTS architectures, other languages, or continuous/non-categorical emotional targets is untested.
The preference model (EASPM) is trained on categorical emotion labels plus a fixed set of acoustic descriptors (pitch, loudness, jitter, shimmer) from MSP-Podcast, so its notion of “emotional preference” is still anchored to discrete emotion categories rather than free-form style description. The paper does not report parameter counts for the fine-tuned decoder or wall-clock/inference overhead introduced by the k-candidate sampling and reward scoring at each training step, which matters for RLHF-style diffusion fine-tuning cost. Code availability is not stated in the paper; only a demo page is mentioned.
Wiki Connections
- Emotion Synthesis — proposes a stepwise preference-optimization framework specifically for aligning diffusion TTS output with fine-grained emotional targets, moving beyond categorical-label or classifier-proxy conditioning.
- Diffusion TTS — fine-tunes a Grad-TTS diffusion decoder post-hoc with a stepwise RLHF-style objective rather than proposing a new diffusion architecture.
- RLHF Speech — adapts DPO-style preference optimization to the denoising trajectory of a diffusion TTS model, replacing endpoint preference propagation with dense per-step reward supervision.
- Subjective Evaluation — reports a 20-rater listening test measuring naturalness, emotional expressiveness, emotion-text consistency, and emotion recognition accuracy against four baselines.
- EmoVoice — used as one of seven emotion-controllable TTS baselines compared against in both objective and subjective evaluation.
- CosyVoice — used as a baseline TTS system in the objective comparison (Table 1).
- CosyVoice 2 — used as the strongest baseline TTS system in both objective and subjective comparisons (Tables 1–2).
- Seamless — its AutoPCP tool is used to compute the Prosody SIM objective metric.
- UTMOS — used as the objective naturalness/perceptual-quality metric throughout evaluation.