arXiv · 2025 · Preprint
Li et al. (Columbia University) · → Paper · Demo: ✓ · Code: ✓
Extends metric optimization to the duration predictor in a distilled flow-matching TTS system using GRPO-based reinforcement learning, and introduces teacher-guided sampling to recover prosodic diversity lost during distillation.
Problem
Non-autoregressive TTS systems trained via distribution matching distillation achieve efficient inference, but the duration predictor sits outside the differentiable optimization loop: its categorical output cannot receive gradient signal from the downstream synthesis model. Duration prediction quality directly constrains intelligibility and speaker similarity, yet prior work either trains it separately from speech generation or uses heuristic approaches based on prompt speaking rate. A second limitation appears during distillation: compressing 32 denoising steps to 4 causes mode shrinkage, reducing prosodic diversity without proportionately degrading spectral quality.
Method
DMOSpeech 2 builds on F5-TTS as a 300M-parameter teacher model and trains a student via improved distribution matching distillation (DMD2), matching the teacher’s output distribution in 4 denoising steps. Adversarial training with a conformer discriminator improves perceptual quality, and direct metric optimization with differentiable CTC and speaker verification losses optimizes WER and speaker similarity end-to-end through the student generator.
The duration predictor is an encoder-decoder transformer with 4 encoder layers processing text, 4 decoder layers with cross-attention consuming the mel-spectrogram prefix, and a softmax output over 300 duration classes in 100ms bins (up to 30 seconds). It is first trained with cross-entropy on ground truth remaining lengths, then fine-tuned with GRPO for 1.5K steps. During GRPO training, K=16 duration samples are drawn per input; each sample is synthesized by the 4-step student model; rewards are computed as the sum of CTC log-likelihood (intelligibility) and speaker embedding cosine similarity scaled by lambda=3 (to equalize the magnitude of the two terms). The clipped surrogate objective with KL regularization against a frozen reference policy prevents reward over-optimization.
To address mode shrinkage, teacher-guided sampling routes the initial denoising steps through the teacher model (establishing prosodic structure and text-speech alignment), then switches to the student model for the remaining 2-3 acoustic refinement steps at a configurable transition point t_switch. The teacher handles approximately 6-14 steps rather than its full 32; the student still performs only 2-3 steps. A higher t_switch allocates more steps to the teacher and recovers more diversity, at the cost of computation.
Training uses ~95k hours of English and Chinese data from Emilia. The speaker verification model is distilled from WeSpeaker SimAMResNet34, and the CTC ASR model is a 6-layer transformer encoder trained on mel-spectrograms. All experiments run on 8 NVIDIA H100 GPUs.
Key Results
On Seed-TTS-en, the 4-step student-only DMOSpeech 2 achieves WER 1.752 and SIM 0.698, compared to 1.947 and 0.662 for the F5-TTS teacher (32 steps), and 3.750 and 0.672 for the same system without RL optimization of the duration predictor. RTF improves to 0.032 versus 0.1671 for the teacher, a 5.3x speedup. On Seed-TTS-zh, CER improves from 1.695 (F5-TTS) to 1.527 with SIM rising from 0.750 to 0.760 (Table 1).
Subjective CMOS evaluations show DMOSpeech 2 significantly outperforms the variant without duration RL: CMOS-N -0.43 and CMOS-S -0.48 in English (p<0.01), and CMOS-N -0.26 and CMOS-S -0.31 in Chinese (p<0.05). Naturalness scores are statistically indistinguishable from ground truth recordings in both languages.
Against state-of-the-art baselines (Table 2), the student-only model outperforms CosyVoice 2 (0.5B, WER 3.358), Spark-TTS (0.5B, WER 2.308), LLaSA-8B (WER 3.994), and matches or exceeds MaskGCT (0.7B) on intelligibility, with RTF 0.032 versus MaskGCT’s 2.397 (75x faster). MaskGCT achieves higher SIM (0.772 vs 0.698) but at substantially higher compute and worse intelligibility.
Teacher-guided sampling (16 steps, t_switch=0.25) recovers pitch CV-f0 to 0.593, representing 89.1% of the teacher’s diversity (0.666), versus 0.464 for the student alone. It further improves WER to 1.738 and CER to 1.468 while remaining 1.8x faster than F5-TTS.
An ablation over duration prediction methods (Table 3) shows performance ranging from WER 1.723 (best-of-8 oracle) down to 3.750 (unoptimized predictor), confirming duration as the primary quality bottleneck independent of the synthesis model.
Novelty Assessment
The central novelty is applying GRPO specifically to the duration predictor rather than to the full synthesis pipeline. This is well-motivated: the duration predictor produces discrete categorical outputs that cannot receive upstream gradients, and its position upstream of the synthesis model means it acts as an unconstrained source of error in otherwise metric-optimized systems. Targeting RL at this component rather than the full TTS loop reduces per-training-step compute by orders of magnitude, since reward samples can be generated by the cheap 4-step student rather than a full 32-step inference.
The teacher-guided sampling mechanism is straightforward but addresses a real phenomenon (mode shrinkage in distilled flow-matching models) that had not been explicitly characterized in prior work. The observation that diversity loss concentrates in prosodic and temporal dimensions while acoustic quality remains stable is a useful diagnostic finding.
Both contributions are incremental relative to the same group’s DMOSpeech. GRPO itself comes from DeepSeekMath and has been applied to TTS by at least one concurrent paper. The evaluation compares against strong baselines on standard benchmarks and the gains are real and statistically supported, but the paper does not introduce a new paradigm.
Field Significance
Moderate — The paper demonstrates that RL applied to a discrete upstream subcomponent (duration prediction) in a distilled TTS pipeline delivers meaningful intelligibility and speaker similarity gains at low computational cost. The teacher-guided sampling contribution identifies and offers a remedy for a previously undercharacterized diversity reduction in distilled flow-matching TTS. Both contributions are technically sound and the evaluation is thorough, but the work extends an existing framework by the same authors rather than opening a new direction.
Claims
-
supports: Applying GRPO to a discrete upstream subcomponent of a distilled TTS pipeline achieves perceptual metric gains at substantially lower computational cost than whole-system RL.
Evidence: Duration predictor GRPO fine-tuning uses 4-step student inference for reward computation and converges in 1.5K steps; on Seed-TTS-en, WER improves from 3.750 (no RL) to 1.752 and SIM from 0.672 to 0.698 without modifying the synthesis model. (§3.2.2, §4.1, Table 1)
-
supports: Duration prediction quality independently constrains intelligibility and speaker similarity in non-autoregressive TTS regardless of synthesis model capacity.
Evidence: With the same flow-matching generator, WER on Seed-TTS-en ranges from 1.723 (best-of-8 oracle duration sampling) to 3.750 (unoptimized predictor) to 1.752 (RL-optimized), establishing duration as the primary bottleneck. (§A.1, Table 3)
-
complicates: Distribution matching distillation in flow-matching TTS reduces prosodic diversity disproportionately relative to acoustic quality, concentrating degradation in temporal and structural dimensions.
Evidence: Step reduction from 32 to 4 lowers pitch CV-f0 by 30.3% (from 0.666 to 0.464) while mean-centered UTMOS distributions remain consistent across teacher and student, showing diversity loss is prosodic rather than spectral. (§3.3.1, §4.3.3, Table 1, Figure 3)
-
refines: Best-of-N sampling over discrete duration predictions establishes a practical performance ceiling for RL-based duration optimization in non-autoregressive TTS.
Evidence: Best-of-8 duration sampling achieves WER 1.723 and SIM 0.724 on Seed-TTS-en; the RL-optimized system reaches WER 1.752 and SIM 0.698 in a single pass, approaching but not fully closing the gap to this oracle upper bound. (§A.1, Table 3)
-
supports: A hybrid inference strategy using a high-diversity teacher for early denoising and a distilled student for acoustic refinement can restore prosodic diversity without sacrificing metric optimization gains.
Evidence: Teacher-guided sampling (t_switch=0.25, 16 steps) recovers CV-f0 to 0.593 (89.1% of teacher diversity) while achieving WER 1.738 and SIM 0.699, outperforming the teacher on both objective metrics at 1.8x faster RTF. (§3.3.2, §4.3.3, Table 1)
Limitations and Open Questions
The RL training is sensitive to the number of steps: performance degrades significantly beyond 1.5K steps due to reward over-optimization, constraining total optimization capacity. Best-of-8 sampling still slightly outperforms the RL approach on speaker similarity (0.724 vs 0.698), indicating the GRPO optimization does not fully close the gap to the theoretical ceiling. Rewards are limited to WER and speaker similarity; explicit naturalness or prosody rewards are not explored. The teacher-guided variant requires holding both teacher (300M) and student (300M) in memory simultaneously, which may be prohibitive in constrained deployments. Evaluation covers only English and Chinese from the Emilia distribution; generalization to other languages or out-of-domain speakers is untested.
Wiki Connections
- Flow Matching — F5-TTS serves as the teacher model; the student is distilled from it via distribution matching distillation adapted for the flow-matching objective.
- RLHF Speech — the paper’s central contribution applies GRPO to the duration predictor, using WER and speaker similarity as reward signals rather than learned preference models.
- Zero-Shot TTS — DMOSpeech 2 synthesizes speech from unseen speakers conditioned on a short audio prompt, evaluated under the cross-sentence task.
- Evaluation Metrics — the paper employs WER, CER, SIM, RTF, and CV-f0 jointly, and the CV-f0 metric is introduced here specifically to measure prosodic diversity.
- Subjective Evaluation — CMOS tests against ground truth recordings confirm naturalness and similarity claims not fully captured by objective metrics.
- F5-TTS — serves as both the teacher model for distribution matching distillation and the primary baseline for comparison.
- Seed-TTS (Seed-TTS) — provides the evaluation test sets (Seed-TTS-en and Seed-TTS-zh) and evaluation protocol used throughout the paper.
- CosyVoice (CosyVoice) — compared as a state-of-the-art baseline in the system-level evaluation; DMOSpeech 2 outperforms it on intelligibility with a fraction of the computational cost.
- CosyVoice2 (CosyVoice 2) — compared as a state-of-the-art baseline; DMOSpeech 2 achieves lower WER and 16.5x faster RTF.
- VALL-E (VALL-E) — cited as a foundational autoregressive codec approach, representing the alternative paradigm to diffusion-based non-autoregressive TTS.