arXiv · 2025 · Preprint
Wei Wang et al. (ByteDance) · → Paper · Demo: ✓ · Code: ?
IntMeanFlow is a distillation framework that adapts the MeanFlow average-velocity paradigm to TTS by replacing Jacobian-vector products with numerically integrated teacher displacements, enabling 1–3 NFE inference on flow-based TTS models without sacrificing synthesis quality.
Problem
Flow-based TTS models achieve high synthesis quality but require many function evaluations (NFE) at inference, limiting real-time usability. MeanFlow, a recent image-generation approach that models average rather than instantaneous velocity, offers a path to 1-step generation but poses two critical obstacles when applied to TTS: (1) its self-bootstrap training mechanism requires guidance mixing similar to flow matching and is prone to collapse without careful tuning, and (2) it relies on Jacobian-vector products (JVPs) which are memory-intensive, incompatible with common custom CUDA operators such as flash attention, and impractical for large-scale TTS training. Prior distillation methods for TTS such as DMOSpeech depend on auxiliary models and fixed step sampling schedules, adding training complexity and reducing inference flexibility.
Method
IntMeanFlow reframes average-velocity learning as a distillation problem. A pretrained flow-matching teacher model computes instantaneous velocities at multiple intermediate states within a time interval [t, r]. The accumulated displacement over that interval — a discrete approximation of the continuous integral of instantaneous velocity — is divided by the interval length to yield a target averaged velocity. The student model is trained to minimize the mean-squared error between its predicted averaged velocity and this teacher-derived target (Equation 8 in the paper). This formulation eliminates the JVP computation entirely and removes the need for a self-bootstrap mechanism, since the student is guided by fixed teacher outputs rather than its own evolving predictions.
The student architecture is initialized from a pretrained flow-matching model. To incorporate the additional conditioning variable r alongside t, the authors add a linear projection W that maps the concatenated time embeddings [E(t), E(r)] back to the original feature dimension of t. W is initialized as a partial identity matrix so the model behaves identically to the pretrained teacher at the start of distillation, enabling stable warm-start.
To further improve inference quality, the authors propose the Optimal Step Sampling Search (O3S) algorithm. Rather than using a uniform or heuristically fixed step schedule, O3S applies coordinate-wise ternary search over step positions: it fixes all but one sampling step and searches for the placement that maximises a quality metric (speaker similarity in this work) on a development set, cycling through all steps until no further improvement is observed. The insight motivating O3S is that speech quality as a function of step position exhibits near-convex behaviour, making ternary search applicable and efficient.
Experiments cover two flow-based TTS architectures: F5-TTS (text-to-mel, text2mel), where the flow model operates directly on text embeddings and learns alignment implicitly; and CosyVoice2 (token-to-mel, token2mel), where a language model generates time-aligned semantic tokens that a separate flow model converts to mel spectrograms. For F5-TTS students, models ranging from 158M (small) to 336M (base) and 592M (medium) parameters are trained on LibriTTS (585 hours) or the Emilia dataset (95K hours). For CosyVoice2 distillation, LibriTTS is used as a substitute for the unavailable proprietary training data. The teacher uses classifier-free guidance (CFG scale 3.0) during distillation but the student is trained and evaluated without CFG, reducing inference cost.
Key Results
On the Seed-TTS test-en benchmark (text2mel, F5-TTS base + IMF + O3S, 3 NFE vs. teacher 32 NFE): WER drops from 4.48% (2-NFE without O3S) to 1.60%, and speaker similarity (SIM-o) reaches 0.65, approaching the teacher’s 0.67. CMOS is −0.61 relative to human reference. RTF improves from 0.243 (teacher, 32 NFE) to 0.021 (3-NFE student), approximately an 11x speedup. With O3S at only 2 NFE, WER falls from 5.97% to 1.73% on test-zh, demonstrating that O3S provides large gains without extra inference cost.
On the LibriSpeech-PC benchmark (token2mel, CosyVoice2 + IMF, 1 NFE): WER is 2.18% and SIM-o is 0.63, with CMOS of −0.03 versus the full 32-NFE CosyVoice2 system — near-lossless quality at 1 NFE. For the flow module alone, RTF falls from 0.510 to 0.026. Vanilla MeanFlow applied directly to CosyVoice2 (line 3 in Table 2) achieves competitive WER and SIM-o but substantially worse UTMOS (3.96 → 3.96 vs. IMF’s 4.28) and CMOS (−0.73 vs. −0.03), confirming that the instability of direct MeanFlow application degrades subjective quality.
Ablation over teacher NFE during distillation (Table 3) shows that reducing teacher NFE from 16 to 2 reduces training time per step from 5.17s to 1.12s, with moderate objective metric degradation and noticeable subjective quality loss at the lowest (2-step) teacher.
Novelty Assessment
The core contribution is methodological: replacing MeanFlow’s Jacobian-vector product with a finite-sum approximation of the integral velocity from teacher rollouts. This makes MeanFlow-style average-velocity distillation practical for large-scale TTS without requiring JVP support from underlying operators. The initialization strategy ensuring warm-start from pretrained flow models is a useful but minor engineering contribution. O3S is presented as a general post-hoc step schedule optimizer; the ternary search idea is well-established, but its application to TTS inference step selection is novel and practically impactful. The overall contribution is primarily a training-recipe improvement enabling a known paradigm (MeanFlow) to transfer to TTS; it is not an architectural innovation.
Field Significance
Moderate — IntMeanFlow solves a concrete compatibility problem that blocked MeanFlow-style average-velocity distillation from scaling to large TTS models, by replacing Jacobian-vector products with a numerically integrated teacher displacement. This makes 1–3 NFE inference accessible for existing flow-based TTS architectures without retraining from scratch, and the O3S step-schedule search provides an inference-time quality improvement that generalises across models. The contribution is primarily a training-recipe advancement rather than a fundamental architectural change.
Claims
- Approximating average velocity via discrete numerical integration of teacher displacements is a viable alternative to Jacobian-vector products for MeanFlow-style distillation, avoiding GPU memory overhead and flash attention incompatibility. (§3.1)
- Flow-matching TTS models can be distilled to 1–3 NFE with near-lossless quality when the student is initialised from the pretrained teacher with a warm-start strategy. (§3.3, §4.2, §4.3, Table 1, Table 2)
- Coordinate-wise ternary search over sampling step positions consistently improves both objective and subjective quality metrics without increasing inference NFE. (§3.2, §4.2.1, Table 1)
- Training instability in direct MeanFlow application to TTS manifests as degraded subjective quality even when objective metrics (WER, SIM-o) appear competitive. (§4.3, Table 2)
- A larger teacher model during distillation improves student quality even when the student is significantly smaller, enabling efficient deployment of smaller models trained with stronger teacher guidance. (§4.2.3, Table 1)
Limitations and Open Questions
The distillation framework inherits the ceiling of the teacher model — it cannot exceed teacher quality. The paper evaluates only mel-spectrogram-level flow models; the vocoder and language model components of a full TTS pipeline are untouched. For token2mel distillation (CosyVoice2), training data mismatch (LibriTTS vs. proprietary 170K-hour corpus) may understate the method’s potential. O3S uses speaker similarity as its search metric, which may not be optimal for all TTS tasks; the sensitivity of results to metric choice is not explored. Finally, the paper does not study streaming or low-latency inference scenarios, where per-chunk NFE budgets may differ.
Wiki Connections
- flow-matching — this paper advances few-step inference for flow-matching TTS by introducing a distillation mechanism that reduces NFE from 32 to 1–3 while preserving quality, addressing one of the primary practical limitations of iterative flow sampling
- F5-TTS — primary base model used for the text2mel distillation experiments
- Seed-TTS — provides the Seed-TTS test-en and test-zh benchmarks used for evaluation
- zero-shot-tts — both F5-TTS and CosyVoice2 baselines use zero-shot speaker conditioning, preserved through distillation
- evaluation-metrics — evaluation uses WER, SIM-o, UTMOS, and UV.MOS alongside human CMOS/SMOS ratings
- subjective-evaluation — CMOS and SMOS listening tests are central to validating near-lossless quality at reduced NFE