arXiv · 2026 · Preprint

Guanrou Yang et al. (Shanghai Jiao Tong University / Shanghai Innovation Institute / Tencent / Zhejiang University) · → Paper · Demo: ? · Code: ✗

GROW applies on-policy reinforcement learning to an autoregressive-diffusion TTS model by reweighting the ordinary flow-matching loss with a signed, group-normalized advantage, so the deterministic ODE sampler never has to be converted into an SDE.

Problem

Policy-gradient RL for flow-matching generators (Flow-GRPO and its variants) needs per-step likelihood ratios. To get them, it converts the deterministic sampling ODE into an SDE, stores trajectories, and evaluates per-step divergences to a reference model. In an autoregressive-diffusion model such as DiTAR, an ODE is solved for every latent patch, so this cost recurs across the whole utterance. Prior RL for TTS mostly targets discrete codec LMs (Seed-TTS, DiffRO). Continuous-generator alternatives use offline preference pairs (ARDM-DPO), a learned value model (VGPO), or a probabilistic reformulation of the flow model (F5R-TTS).

A cheaper route exists: reward-weighted flow-matching regression from image generation (ORW-CFM-W2, AWM). The authors argue that it fails on strong pretrained TTS models. Speaker-similarity and intelligibility rewards cluster tightly within a prompt group, so positive exponential weights reduce mostly to self-imitation of the model’s own rollouts.

Method

The base model is a faithful reproduction of DiTAR. A Semantic-VAE encodes 16 kHz audio into 40 Hz, 64-dimensional continuous latents. A six-layer aggregation encoder compresses each patch of P=4 latents into one token. A causal Qwen3-0.6B backbone (28 layers, hidden size 1024, initialized from released weights) models dependencies across patches. A six-layer, 120M-parameter Local Diffusion Transformer (LocDiT) generates the frames within each patch under a linear flow-matching objective. A linear stop head ends the utterance. The model is pretrained for 200K steps on about 95K hours of English and Chinese Emilia. That checkpoint is both the RL initialization and the frozen reference.

GROW changes only the training objective. For each (prompt waveform, prompt text, target text) example, the current policy produces G=8 rollouts with Euler sampling, CFG (γ=1.5) and F5-TTS sway sampling. Rollouts use 10 NFE; evaluation uses 32. Each rollout gets two frozen-model rewards: speaker similarity (WavLM-large speaker-embedding cosine to the prompt) and 1 − WER (Whisper-large-v3 for English, Paraformer-zh for Chinese). Each reward stream is z-normalized within the group and the two are summed with equal weights. The result is a zero-mean, signed weight, the GRPO advantage estimator applied to flow-matching regression. Groups with zero reward variance are discarded and redrawn, following DAPO.

The derivation starts from reward-weighted regression, w = exp(τr). A group-mean baseline turns it into exp(τA). The loss is then split into a shared self-imitation term (mean weight times the plain flow-matching loss) and a reward-contrast term (weight deviations). When advantages are small, the contrast-to-self-imitation ratio is about τσ, where σ is the within-group advantage spread. The authors therefore keep only the linear term of the Taylor expansion, drop the constant 1, fold τ into the learning rate, and divide by σ.

Each rollout is prepended with its prompt and re-encoded by the VAE. It is then trained with the advantage-weighted flow-matching loss on the conditional velocity only, without CFG. A squared velocity discrepancy to the frozen reference (β_W2=0.025) serves as a tractable Wasserstein-2 upper bound. This term prevents negative weights from inflating regression errors without bound. Training uses AdamW at a constant 2×10⁻⁶, batch 128, for several hundred updates. Gradients flow through the full AR backbone as well as the LocDiT.

Key Results

All systems start from the same pretrained DiTAR. All are evaluated at 32 NFE on LibriSpeech-PC test-clean and on Seed-TTS test-en and test-zh. “Avg. ∆” is the pretrained value minus the system’s value for WER, averaged over the three sets, so a positive number means a reduction. For SIM and UTMOS it is the system minus the pretrained value. (Table 1)

  • Against the pretrained model: GROW (rollout NFE 10) cuts the average WER from 2.016 to 1.558 (∆ 0.458). It raises the average SIM from 0.676 to 0.715 (∆ 0.039). UTMOS rises slightly (∆ +0.02). (Table 1)
  • Against DiTAR-GRPO: the authors’ own Flow-GRPO implementation on the same model also uses MixGRPO windowed SDE steps and CPS sampling. At matched rollout NFE, GROW’s average WER reduction is 0.477 vs. 0.246 (NFE 32), 0.458 vs. 0.269 (NFE 10) and 0.413 vs. 0.196 (NFE 5). GROW’s SIM gain is 0.040/0.039/0.038 against 0.023/0.030/0.018. DiTAR-GRPO lowers average UTMOS at every NFE (−0.07, −0.04, −0.11), while GROW raises average UTMOS (+0.03, +0.02, +0.06). (Table 1)
  • Per test set: across all seven rows, a GROW configuration gives the highest SIM on each set: 0.701 on LibriSpeech-PC (NFE 32 and 10 tie), 0.703 on Seed-EN (NFE 32) and 0.743 on Seed-ZH (NFE 32). The lowest WER on LibriSpeech-PC is GROW at NFE 32 (1.868), and on Seed-ZH GROW at NFE 10 (0.983). On Seed-EN, GROW at NFE 5 is lowest (1.721), just below DiTAR-GRPO at NFE 10 (1.728). The default GROW at NFE 10 (1.763) is not the best Seed-EN WER. (Table 1)
  • Training speed: relative to DiTAR-GRPO at NFE 32, GROW trains 1.1×, 2.9× and 4.3× faster at NFE 32/10/5. At matched NFE the advantage is about 1.1×, because rollout dominates step time. (Table 1, §Experimental Result)
  • Weighting ablation: the signed A/σ weight gives ∆WER 0.458 and ∆SIM 0.039. The exp(τA) weight gives 0.199 and −0.003, and exp(τr) gives 0.095 and 0.000. The authors report that sweeping τ together with β_W2 never lifted SIM above the pretrained model under exponential weighting. (Table 2)
  • Training velocity and CFG: applying the loss to the guided velocity instead of the conditional one nearly cancels the gains (∆WER 0.094, ∆SIM 0.007). Rollouts without CFG (γ=0) give a slightly larger WER reduction (0.465) and a smaller SIM gain (0.033). (Table 3)
  • Anchor and learning rate: without the W2 anchor, ∆WER drops to 0.134 and SIM falls (∆ −0.003). With β_W2=1, the policy barely moves (∆WER 0.068, ∆SIM 0.004). β_W2=0.025 at 2×10⁻⁶ gives the largest gains on both metrics among the eight configurations. (Table 4)
  • Reward composition: a WER-only reward gives ∆WER 0.452 with no SIM change (0.001). A SIM-only reward gives ∆SIM 0.043 and still cuts WER by 0.249. The combined reward gives 0.458 and 0.039. (Table 5)

The comparison is controlled: same base model, prompts and evaluation NFE. However, the only RL baseline is the authors’ own reimplementation. All metrics are automatic, and the WER and SIM evaluators are the same models used as rewards.

Novelty Assessment

The contribution is a training objective, not a new network. The architecture is DiTAR. Advantage-weighted flow matching (AWM), reward-weighted flow matching with a W2 anchor (ORW-CFM-W2) and the GRPO advantage estimator all exist already, mainly in image generation. What is new is the combination for an autoregressive-diffusion TTS model. The paper also gives an explicit analysis of why positive exponential weights collapse into self-imitation when rewards are saturated. Per-stream within-group normalization lets two rewards with different scales be fused, and the conditional-velocity choice turns out to be decisive (Table 3). The self-imitation argument is a first-order heuristic, not a proof, but Table 2 supports it empirically. The paper says the method also applies to non-autoregressive flow-matching TTS such as F5-TTS, but it does not test this.

Field Significance

Moderate — The paper shows that RL post-training of a continuous autoregressive-diffusion TTS model can act directly through the pretraining flow-matching loss, keeping the deterministic ODE sampler. In this setting it beats a Flow-GRPO-style SDE baseline on intelligibility, speaker similarity and training cost. Its most transferable finding is diagnostic: with tightly clustered rewards from a strong pretrained model, zero-mean signed advantage weights work where positive exponential reward weights barely move the model.

Claims

  • supports: Reward-driven post-training of flow-matching TTS can act directly on the standard flow-matching regression loss, without converting the deterministic sampler into a stochastic one or estimating per-step likelihoods.

    Evidence: Reweighting DiTAR’s flow-matching loss with a group-normalized advantage lowers the three-set average WER by 0.458 and raises SIM by 0.039. A Flow-GRPO-style SDE baseline on the same model gains 0.269 and 0.030 at matched 10-NFE rollouts. (§GROW, §Experimental Result, Table 1)

  • refines: When a strong pretrained TTS model already produces tightly clustered rewards, reward weights must be centered and signed to carry useful credit assignment; strictly positive exponential weights reduce mostly to self-imitation.

    Evidence: Signed A/σ weighting gives ∆WER 0.458 and ∆SIM 0.039. exp(τA) and exp(τr) give 0.199/−0.003 and 0.095/0.000, and no τ/β_W2 setting lifted SIM above the pretrained model under exponential weighting. (§From Exponential to Linear Group-Relative Advantage, Table 2)

  • complicates: Signed advantage weighting on a squared regression loss is unstable without a reference anchor, and the anchor strength trades stability against the size of the policy update.

    Evidence: Removing the W2 anchor reduces ∆WER to 0.134 and leaves SIM below the pretrained level (∆ −0.003). β_W2=1 nearly freezes the policy (∆WER 0.068). The best result comes at β_W2=0.025. (§Experimental Result, Table 4)

  • supports: On-policy RL rollouts for flow-matching TTS can use far fewer sampling steps than final inference with little loss of post-training quality, which cuts training cost.

    Evidence: Cutting GROW’s rollout NFE from 32 to 10 shrinks the average WER gain only from 0.477 to 0.458, with SIM gains of 0.040 and 0.039, at 2.9× the training speed of 32-NFE DiTAR-GRPO. The SDE baseline degrades more clearly at 5 NFE (∆WER 0.196). (§Experimental Result, Table 1)

  • complicates: Intelligibility and speaker-similarity rewards in TTS post-training are asymmetrically coupled: optimizing speaker similarity also improves intelligibility, but optimizing intelligibility does not improve speaker similarity.

    Evidence: A SIM-only reward raises SIM by 0.043 and also cuts WER by 0.249. A WER-only reward cuts WER by 0.452 but changes SIM by only 0.001. (§Experimental Result, Table 5)

Limitations and Open Questions

Warning

All results are automatic metrics. The ASR and speaker models used to score WER and SIM are the same frozen models used as RL rewards. Reward over-optimization against these evaluators cannot be ruled out, and no human listening test checks the gains. UTMOS is the only held-out quality measure.

  • The method is shown on one base model, the authors’ DiTAR reproduction. Applicability to non-autoregressive flow-matching TTS is asserted but not tested.
  • The only RL baseline is the authors’ own Flow-GRPO implementation (with MixGRPO and CPS). There are no comparisons against published post-trained systems or other reward-weighted flow-matching variants (AWM, DiffusionNFT, ORW-CFM-W2) on the same model.
  • Language coverage is English and Chinese only. Rewards cover intelligibility and speaker similarity only, so prosody, expressiveness and naturalness are not optimized or measured beyond UTMOS.
  • Optimization is sensitive to the learning rate and anchor weight (Table 4), and the RL stage runs for only “several hundred updates”. Longer-horizon stability is not reported.
  • Code, the DiTAR reproduction and checkpoints are promised but not released at the time of this version.

Wiki Connections

  • RLHF Speech — Introduces a group-relative, advantage-weighted on-policy RL objective for TTS that avoids ODE-to-SDE conversion, with WER and speaker-similarity rewards.
  • Flow Matching — The RL update acts directly on the flow-matching regression loss of the LocDiT head, with a Wasserstein-2 velocity anchor to the frozen reference.
  • Zero-Shot TTS — Post-training targets prompt-based voice cloning and is evaluated on the standard zero-shot LibriSpeech-PC and Seed-TTS test sets.
  • Multilingual TTS — The model is pretrained and RL-tuned on English and Chinese prompts and gains on both English and Chinese test sets.
  • DiTAR — The autoregressive-diffusion architecture GROW fine-tunes. The authors reproduce it with a Qwen3-0.6B backbone and a Semantic-VAE latent space.
  • ARDM-DPO — An earlier offline preference-optimization approach for DiTAR, which GROW contrasts with its on-policy reward weighting.
  • F5R-TTS — A prior GRPO route for flow-matching TTS that reformulates the flow model probabilistically, the trajectory-likelihood approach GROW avoids.
  • DiffRO — Representative codec-LM reward optimization, cited as the discrete-token RL line GROW departs from.
  • Seed-TTS — Supplies two of the three evaluation sets and an earlier comparison of REINFORCE, PPO and DPO for TTS.
  • F5-TTS — Source of the sway-sampling schedule and the LibriSpeech-PC test-clean protocol, and named as a non-autoregressive model GROW could also apply to.
  • Semantic-VAE — Provides the continuous 40 Hz latent space the DiTAR reproduction models.
  • DeepSeekMath — Origin of the GRPO group-normalized advantage estimator that GROW applies to flow-matching regression.