ACL · 2025 · Conference
Yushen Chen et al. (Shanghai Jiao Tong University) · → Paper · Demo: ✓ · Code: ✓
F5-TTS is a fully non-autoregressive TTS system that resolves the alignment-robustness failure of E2 TTS by inserting ConvNeXt V2 blocks to refine the character sequence before acoustic concatenation, achieving an RTF of 0.15 at 16 NFE with competitive zero-shot quality across English and Mandarin benchmarks.
Problem
Alignment between text and speech is the central challenge for alignment-free non-autoregressive TTS. E2 TTS showed that padding raw character sequences with filler tokens to match mel-frame length could replace phoneme alignment, duration predictors, and text encoders — but the resulting model suffered slow convergence and a systematic 7% failure rate on Mandarin test samples where WER exceeded 50%, a failure that re-ranking could not rescue. The paper diagnoses this as a structural problem: deeply entangling semantic character tokens with acoustic mel frames in a flat transformer forces the model to reconcile two modalities with a large length gap and incompatible information densities, producing instability that persists across training. Alternative approaches (Voicebox, NaturalSpeech 3, Matcha-TTS) had addressed alignment through phoneme-level force alignment or monotonic alignment search, reintroducing the complexity that E2 TTS sought to remove. The question the paper pursues is whether the simplicity of E2 TTS’s pipeline can be preserved while fixing its robustness.
Method
F5-TTS frames TTS as a text-guided speech-infilling task under conditional flow matching with the optimal-transport path. The acoustic input is 100-dimensional log mel-filterbank features at 24 kHz (hop length 256). Text is converted to a character sequence — raw alphabets for English, full pinyin for Chinese — and padded with filler tokens to match the mel frame count.
The key architectural departure from E2 TTS is a pre-concatenation text refinement stage: the padded character sequence first passes through ConvNeXt V2 blocks (4 layers, 512 embedding / 1024 FFN dim) before being concatenated with the speech input in the feature dimension. This gives the text representation an individual modeling space to adapt toward the acoustic modality prior to in-context learning, rather than arriving at concatenation with raw character embeddings that carry no acoustic priming. The main denoising backbone is a Diffusion Transformer (DiT) with adaLN-zero adaptive Layer Norm blocks — 22 layers, 16 attention heads, 1024/2048 embedding/FFN dim — replacing the flat U-Net Transformer used by E2 TTS, which reduces GFLOPs (173 vs. 293 for 158M small models). The flow step t conditions adaLN via sinusoidal embedding rather than being appended to the input sequence. RoPE is used for self-attention; a convolutional position embedding is applied to the concatenated input.
Training uses the Emilia multilingual dataset filtered to approximately 95K hours of English and Chinese. AdamW with a peak learning rate of 7.5e-5, linear warmup for 20K updates, linear decay over 1.2M total updates, batch size of 307,200 audio frames, on 8 NVIDIA A100 80G GPUs for over one week. Classifier-free guidance is applied in two stages: masked-speech input is dropped with probability 0.3, and text input together with masked speech is additionally dropped with probability 0.2. The base model has 335.8M parameters total.
At inference, duration is estimated from the character-count ratio of reference to generation transcript — no learned duration predictor. The paper introduces Sway Sampling: a monotonic transform applied to uniformly sampled flow steps that biases integration toward smaller t values (s < 0). Early flow steps determine text-speech alignment (sketching the speech silhouette); later steps handle fine acoustic detail. Densifying the early phase improves faithfulness without increasing total NFE. Sway Sampling is plug-and-play across any CFM-based TTS model without retraining. The Euler ODE solver with 16 NFE and s = -1 achieves RTF 0.15 on an NVIDIA RTX 3090. Vocos converts generated mel spectrograms to waveform.
Key Results
On LibriSpeech-PC test-clean (the standardised 1127-sample benchmark released by this paper), F5-TTS at 32 NFE reaches WER 2.42% and SPK-SIM 0.66, versus E2 TTS (32 NFE) WER 2.95% / SPK-SIM 0.69 and CosyVoice WER 3.59% / SPK-SIM 0.66. At 16 NFE the RTF is 0.15, compared to Voicebox 0.64 and DiTTo-TTS 0.162, with WER degrading only to 2.53%.
On Seed-TTS test-en, F5-TTS (32 NFE) achieves WER 1.83%, SPK-SIM 0.67, and CMOS +0.31, outperforming all open-source baselines on naturalness. On Seed-TTS test-zh, WER 1.56%, SPK-SIM 0.76, CMOS +0.21. Seed-TTS DiT achieves WER 1.733% / SPK-SIM 0.790 on test-en, with a gap attributable to orders-of-magnitude larger training data.
Ablations (155M small models, Mandarin) show the ConvNeXt branch is load-bearing: removing it (F5-TTS -Conv2Text) causes alignment failure entirely; the long-skip connection variant does not help; and applying ConvNeXt to E2 TTS’s U-Net backbone (E2 TTS + Conv2Text) degrades to WER 18.1%, showing the improvement is DiT-specific.
Sway Sampling ablations confirm its benefit: on LibriSpeech-PC test-clean, WER improves from 2.84% (no SS, 32 NFE) to 2.41% (SS, 32 NFE), and UTMOS from 3.70 to 3.89. The strategy also improves E2 TTS, from WER 2.95% to 2.84%, confirming transferability.
Note
Comparisons with VALL-E 2 and MELLE use scores from Meng et al. (2024) on a different LibriSpeech test-clean subset; these cannot be directly compared to LibriSpeech-PC results. The Seed-TTS DiT baseline is proprietary and evaluated on its own reported scores, not re-evaluated under controlled conditions.
Novelty Assessment
The contribution has two distinct components. The ConvNeXt text refinement is architecturally novel in a targeted sense: it is a principled diagnostic-driven design change that addresses a specific, documented failure mode in E2 TTS. It is not a paradigm shift — the overall pipeline remains E2 TTS’s — but the ablation evidence that the fix is backbone-specific (cannot transfer to U-Net) demonstrates that it is more than a generic improvement recipe.
Sway Sampling is genuinely new as an inference-time tool for CFM-based TTS with a mechanistically motivated rationale (early-step alignment priority), and its portability without retraining gives it field-level utility beyond this paper. The “leak and override” experiment provides unusually direct causal evidence for the hypothesis.
The LibriSpeech-PC benchmark release addresses a real methodological gap in zero-shot TTS evaluation where prior systems used incompatible and unreleased test subsets.
What is not novel: the CFM objective, DiT backbone, infilling training task, and CFG strategy are all established. The overall architecture is a well-motivated combination rather than a structural invention.
Field Significance
Tip
High — F5-TTS establishes that alignment robustness in alignment-free TTS is achievable through targeted representation engineering — ConvNeXt text refinement within a DiT backbone — rather than reintroducing phoneme supervision, advancing the E2/Voicebox design lineage. Sway Sampling extends as a portable inference-time improvement for the broader flow-matching TTS community. The full open-source release of code, checkpoints, and LibriSpeech-PC benchmark enables reproducible comparisons that have been absent from the zero-shot TTS literature.
Claims
- Explicit phoneme-level alignment is not required for robust non-autoregressive zero-shot TTS; modality-specific representation refinement before feature concatenation can resolve text-speech alignment failures in alignment-free architectures. (§3.2, §5.1)
- Deep entanglement of semantic and acoustic features in a shared transformer produces systematic alignment failures in non-autoregressive TTS that persist across training scale and cannot be addressed by re-ranking. (§3.2, §5.1, Table 4)
- Inference-time flow step scheduling that densifies early integration steps improves faithfulness and speaker similarity in flow-matching TTS without any retraining. (§3.2, §5.2, Table 5)
- Non-autoregressive flow-matching TTS trained at 100K-hour scale can match autoregressive systems on intelligibility while achieving substantially lower real-time factor. (§5, Table 1)
- Absence of standardised benchmark subsets with released prompt lists is a structural barrier to fair cross-paper comparison in zero-shot TTS; community-released evaluation sets directly reduce this problem. (§4)
Limitations and Open Questions
Warning
Speaker similarity (SPK-SIM 0.66–0.67 on English test sets) remains below ground truth (0.69–0.73) and well below Seed-TTS DiT (0.790 EN, 0.809 ZH), which operates at unknown but much larger training scale. Whether this gap is architectural or purely a function of training data volume is not resolved by this paper.
The mel spectrogram representation creates a sequence-length bottleneck: mel frames are significantly longer than the text token sequence, making computation and attention cost scale unfavourably with utterance length. The paper identifies more efficient continuous representations as the primary remaining efficiency direction.
Duration estimation via character-count ratio introduces uncontrolled variance at inference, particularly for prosodically complex or phonetically irregular text. A lightweight learned duration predictor could improve rhythm and naturalness at moderate complexity cost.
Fine-grained paralinguistic control — emotion, speaking style, prosodic emphasis — is absent. The infilling framing naturally mirrors the reference prompt’s style, but there is no mechanism to override or interpolate paralinguistic attributes. Broader multilingual coverage beyond English and Chinese is also unevaluated.
Wiki Connections
This paper directly advances flow-matching by demonstrating a robustness-oriented architecture for the alignment-free variant of the paradigm. It is a core reference for zero-shot-tts as a fully open-weight, multilingual system with competitive zero-shot performance. The training on Emilia and code-switching inference connect to multilingual-tts. The LibriSpeech-PC benchmark release is relevant to evaluation-metrics and subjective-evaluation.