ACL · 2026 · Conference

Fangming Feng et al. (Zhejiang University) · → Paper · Demo: ✓ · Code: ✓

A training-free sampling procedure for flow-matching TTS decoders that pre-shifts the initial noise toward the target emotion and replaces constant classifier-free guidance with a likelihood-ratio-driven schedule, aimed at high-arousal emotional speech that stays intelligible.

Problem

Emotion-controllable TTS systems handle neutral and mildly expressive speech well, but high-arousal emotions (shouting, sobbing, extreme surprise) expose a stability-expressiveness trade-off: pushing intensity tends to produce mispronunciations and acoustic artifacts, while stable settings fall short of the target intensity. The authors illustrate this with CosyVoice2 outputs on their high-intensity set, where few samples combine high emotion scores with high overall quality (Figure 1; both scores here come from Gemini 2.5 Pro used as a proxy rater, not from human listeners). They attribute the problem to two sampling-time causes in flow-matching decoders: an isotropic Gaussian starting point that carries a bias toward neutral prosody, and uniform classifier-free guidance (CFG) whose linear extrapolation distorts the acoustic output when applied at full strength at every step. Prior inference-time work on noise selection and guidance intervals comes from image generation, and the one speech-specific training-free emotion method the authors discuss (EmoSteer-TTS, activation steering) is described as unstable at high intensities.

Method

The method has two components, both applied at inference to a frozen pretrained model with no retraining or fine-tuning.

Emotion-Rectified Noise Prior (ERNP). Starting from a standard Gaussian sample, ERNP takes one forward Euler “lookahead” step of size τ using a high guidance scale λ_init, then one backward “calibration” step from that point using a low base scale λ_base. Under the approximation that flow-matching velocities are nearly constant along the short segment, the net effect is to add τ(λ_init − λ_base) times the conditional-minus-unconditional velocity difference to the initial noise, which the authors interpret as a semantic gradient toward the target emotion region. The default setting is λ_init = 30, λ_base = 1 (Appendix A.3).

Likelihood-Inverse Guidance (LIG). Instead of CFG’s linear extrapolation, the learned emotion-conditional density is modelled as a mixture of a neutral fallback distribution and an ideal emotional distribution, weighted by a purity coefficient π (default 0.95). Inverting the mixture and taking log-gradients gives a guided velocity of the usual form v_u + λ(v_c − v_u), but with a state-dependent scale λ = R_t / (R_t − (1 − π)), where R_t is the ratio of conditional to unconditional density at the current state. When R_t is large (the trajectory is already in an emotional region), λ approaches 1; as R_t falls toward 1 − π (drift toward neutral), λ grows, and it is capped at λ_max = 30 for numerical stability. R_t is tracked recursively during sampling: modelling each Euler step’s transition as Gaussian, the log-ratio increment reduces to the difference between the squared distances of the applied velocity from the unconditional and conditional velocities, scaled by Δt²/(2σ²) with σ = 1 − t. This costs no extra network calls beyond the two CFG branches already computed.

Figure 2: Overview of our method. Left: ERNP injects a semantic gradient to align the initial state with the target emotional manifold. Right: LIG dynamically regulates the guidance scale λ t based on the likelihood ratio R t to prevent neutral drift. Middle: The combined approach rectifies the generative flow (purple line), ensuring high-intensity expressiveness while preserving acoustic stability compared to the baseline (gray dashed line).

The method is applied to three frozen pretrained systems: CosyVoice 2 and IndexTTS2 (both autoregressive language models feeding a flow-matching acoustic decoder) and F5-TTS (a non-autoregressive flow-matching model). For CosyVoice2, where emotion enters through a prompt to the language model, a dual-branch setup runs the language model once with and once without the emotion prompt, and the two outputs serve as the conditional and unconditional inputs to the flow-matching decoder under a fixed seed. The paper does not spell out the equivalent wiring for IndexTTS2 or exactly what the conditional and unconditional branches are for F5-TTS, whose emotion comes from the reference prompt. Model sizes are not reported.

Key Results

Evaluation uses ESD (first 100 prompts of speakers 0001 and 0011), EmoVoice, the read-speech subset of Expresso, and HIED, a 400-utterance high-arousal set (Angry, Happy, Sad, Surprise; 37 speakers) that the authors curated from public corpora by ranking candidates with a pretrained arousal predictor. Neutral speech from the target speaker is the reference audio on the standard benchmarks. Objective metrics are WER (FunASR for Chinese, Whisper for English), emotion2vec cosine similarity (ES) and emotion recall, and UTMOSv2; “GMOS” is a Gemini 2.5 Pro automatic score, not a listening test. Human MOS (SMOS, PMOS, QMOS, EMOS) covers only 50 HIED samples per system; rater count is not reported.

Standard benchmarks (Table 1), six base-versus-method pairs. WER falls in five of six pairs, with the largest drops for IndexTTS2 on EmoVoice (2.97 to 2.13) and CosyVoice2 on Expresso (3.13 to 2.27); it rises for IndexTTS2 on ESD (1.71 to 1.90), which the authors attribute to ESD being in IndexTTS2’s training data. ES rises in five of six pairs (CosyVoice2 on Expresso 0.72 to 0.83) and falls for CosyVoice2 on EmoVoice (0.77 to 0.75). Emotion recall improves in only two of six pairs (CosyVoice2 on Expresso 39.7% to 43.1%, IndexTTS2 on EmoVoice 46.3% to 48.5%) and drops slightly in the other four. UTMOSv2 improves in four pairs and drops in two (IndexTTS2 on ESD 3.47 to 3.33, CosyVoice2 on Expresso 3.71 to 3.61). GMOS is equal or higher in all six.

Human MOS on HIED (Table 2). For CosyVoice2 all four scores rise (EMOS 3.53 to 3.82, PMOS 3.62 to 3.87, QMOS 3.79 to 3.90, SMOS 3.85 to 3.91). For IndexTTS2, EMOS rises from 3.94 to 4.18 and QMOS from 3.95 to 4.02, but speaker similarity (SMOS) falls from 3.92 to 3.77.

F5-TTS (Tables 3 to 5). On HIED, WER drops from 4.41 to 2.53 and recall rises from 52.1% to 61.7%, while ES falls from 0.78 to 0.73. Human EMOS rises from 3.63 to 3.89, with SMOS (3.32 to 3.33) and PMOS (3.08 to 3.10) essentially unchanged. On ESD, EmoVoice and Expresso, F5-TTS with the method improves WER, ES and UTMOSv2 on every dataset (e.g. EmoVoice WER 3.91 to 2.67, ES 0.77 to 0.83).

Ablations (IndexTTS2 on HIED, Tables 6, 7, 10). Replacing rectified noise with standard Gaussian noise raises WER from 3.05 to 3.25 and lowers ES from 0.84 to 0.73. Among guidance schedules, constant CFG gives WER 4.08 / ES 0.79 / GMOS 3.65, limited-interval CFG gives 3.86 / 0.74 / 3.91, and LIG gives 3.05 / 0.84 / 4.22, the lowest WER and highest ES and GMOS of the three. At reduced step counts the WER gain holds (NFE=10: 7.48 to 4.31; NFE=5: 15.29 to 7.98), but recall drops below baseline (NFE=10: 58.8% to 49.2%; NFE=5: 55.7% to 49.5%) and ES is lower at NFE=5 (0.74 to 0.71). A trajectory analysis (Table 9) reports reduced cumulative angular deviation (34.26 to 15.83) and lower log straightness (2.28 to 1.04) versus the baseline.

Comparisons are fair in the sense that each base model is compared against itself with only the sampler changed, but there is no comparison against other emotion-control methods such as EmoSteer-TTS, and the ablations are run on a single model and dataset.

Novelty Assessment

Both components are inference-procedure contributions. ERNP is close in spirit to image-domain noise-optimization and pre-inference correction work the authors cite; its specific form (a guided lookahead step followed by a low-guidance retraction) is a simple, cheap way to inject a conditional direction into the starting noise. LIG is the more original piece: deriving a state-dependent guidance scale from a mixture model of neutral and emotional densities, and estimating the needed likelihood ratio recursively from the velocity differences already computed for CFG, is a principled alternative to hand-tuned guidance intervals. Several derivation steps rely on approximations (near-constant velocity over the lookahead, Gaussian transition kernels, treating the score-velocity proportionality loosely) that are stated but not tested on their own. The empirical case is real but uneven: WER gains are consistent, while emotion recall and speaker similarity sometimes drop, and much of the headline improvement is reported via an LLM-judge score (GMOS).

Field Significance

Moderate. The paper shows that sampling-time changes alone (where the noise starts and how guidance strength varies along the trajectory) can reduce intelligibility failures in high-arousal emotional TTS across three different pretrained flow-matching decoders, without retraining. Its contribution is primarily an inference recipe and supporting analysis rather than a new model, and its evidence is limited by small human tests and reliance on automated emotion and quality scorers.

Claims

  • supports: Training-free changes to the sampling procedure of a frozen flow-matching TTS decoder can improve intelligibility of high-arousal emotional speech while maintaining or raising perceived emotional intensity.

    Evidence: Applying ERNP and LIG lowers WER for F5-TTS on HIED from 4.41 to 2.53 with human EMOS rising from 3.63 to 3.89, and lowers WER in five of six base/method pairs across CosyVoice2 and IndexTTS2 on ESD, EmoVoice and Expresso. (§4.2.1, §4.2.2, Tables 1, 3, 4)

  • refines: Classifier-free guidance strength in emotional TTS works better as a state-dependent quantity than as a constant or a fixed time window.

    Evidence: On IndexTTS2/HIED, a likelihood-ratio-driven schedule reaches WER 3.05 and ES 0.84, versus 4.08 and 0.79 for constant CFG and 3.86 and 0.74 for limited-interval CFG; the derived scale stays near 1 when the trajectory is already in an emotional region and grows only on drift toward neutral. (§3.2.2, §4.3.2, Table 7)

  • supports: The initial noise of a flow-matching TTS sampler carries a bias that affects whether the output reaches a target emotion, and shifting it toward the condition improves emotional expressiveness.

    Evidence: Replacing standard Gaussian initialization with the emotion-rectified prior raises ES from 0.73 to 0.84 and lowers WER from 3.25 to 3.05; with plain Gaussian noise the adaptive guidance scale spikes above 18 early in sampling, versus a peak around 7 to 8 with the rectified prior. (§4.3.1, Table 6, Figure 4)

  • complicates: Gains in intelligibility and emotion-embedding similarity from inference-time emotion steering do not carry over uniformly to emotion classification recall or speaker similarity.

    Evidence: Emotion recall drops in four of six base/method pairs on standard benchmarks and falls well below baseline at NFE=10 (58.8% to 49.2%) and NFE=5 (55.7% to 49.5%); IndexTTS2’s human SMOS on HIED falls from 3.92 to 3.77. (§4.2.1, §4.5, Tables 1, 2, 10)

  • complicates: Inference-time headroom for emotional TTS shrinks when the evaluation data overlaps the base model’s training distribution.

    Evidence: For IndexTTS2 on ESD, which is part of IndexTTS2’s training data, WER rises from 1.71 to 1.90 and UTMOSv2 falls from 3.47 to 3.33 while ES only moves from 0.76 to 0.79. (§4.2.1, Table 1)

Limitations and Open Questions

Thin human evaluation

Human MOS covers only 50 HIED samples per system with an unreported number of raters, and Table 4’s caption mentions 95% confidence intervals that the table does not show. Many headline comparisons instead rest on GMOS (a Gemini 2.5 Pro judge) and emotion2vec-based ES/recall, which are automated proxies.

  • By the authors’ own account, the method depends on continuous, iterative ODE sampling and does not apply to discrete autoregressive models or single-step generators. In CosyVoice2 and IndexTTS2 it only affects the flow-matching acoustic stage, so errors made by the autoregressive token stage are outside its reach (§Limitations).
  • HIED is small (400 utterances, skewed toward male speakers at 295 vs 105 utterances) and was built by ranking clips with an automated arousal predictor; the paper does not state whether it is released.
  • All ablations and hyperparameter sweeps (π, λ_max, λ_init/λ_base, NFE) use a single model (IndexTTS2) on HIED, so the defaults may not transfer to other backbones.
  • The baseline in the NFE study (WER 4.36 at NFE=25) differs from the constant-CFG row in the guidance ablation (4.08), and the exact baseline sampler settings are not stated.
  • No comparison against other emotion-intensity control or steering methods.
  • The derivation relies on approximations (straight local trajectories, Gaussian transition kernels with σ = 1 − t) that are not validated separately.

Wiki Connections

  • Flow Matching — Contributes a training-free sampler modification (rectified initial noise and likelihood-ratio guidance) for frozen flow-matching TTS decoders, with trajectory-straightness analysis.
  • Emotion Synthesis — Targets high-arousal emotional speech and reports emotion similarity, recall and human emotion MOS for three emotion-capable systems.
  • Subjective Evaluation — Reports human SMOS, PMOS, QMOS and EMOS on a 50-sample HIED subset, alongside an LLM-judge score used as a cheaper substitute.
  • CosyVoice 2 — One of the three frozen base models the method is applied to, via a dual-branch language-model setup that supplies conditional and unconditional inputs to its flow-matching decoder.
  • IndexTTS2 — Frozen base model used for the main comparisons and all ablations; ESD overlap with its training data explains its weakest results.
  • F5-TTS — Frozen non-autoregressive flow-matching base model used to show the method works without a language-model stage.
  • EmoSteer-TTS — Prior training-free emotion-control method via activation steering, cited as unstable at high intensities but not compared against experimentally.
  • EmoVoice — Source of one of the three standard emotional benchmarks used for evaluation.
  • EXPRESSO — Its read-speech subset is used as an evaluation benchmark.
  • Classifier-Free Diffusion Guidance — The constant-scale guidance that LIG replaces with a state-dependent schedule.
  • emotion2vec — Used only as the evaluation scorer for emotion similarity and recall.