arXiv · 2025 · Preprint
Siratish Sakpiboonchit (National University of Singapore) · → Paper · Demo: ✓ · Code: ?
Applies SmoothCache, an existing inference acceleration technique for diffusion transformers, to F5-TTS without any retraining, achieving up to a 43% reduction in real-time factor at high NFE settings while preserving intelligibility and speaker similarity.
Problem
Diffusion transformer (DiT)-based TTS models achieve high synthesis quality but remain computationally expensive because inference requires many iterative denoising steps. Prior acceleration strategies such as improved ODE solvers, knowledge distillation, pruning, and quantisation each require either retraining the model or modifying its architecture. A zero-training, inference-only acceleration path that does not compromise model integrity or require access to training pipelines had not been demonstrated for DiT-based TTS.
Method
The paper integrates SmoothCache into the F5-TTS inference pipeline, where F5-TTS generates log mel spectrograms via flow matching on a DiT backbone and converts them to waveforms using the Vocos vocoder.
SmoothCache exploits the observation that hidden representations in transformer layers change little between consecutive denoising timesteps. A calibration phase processes ten randomly sampled utterances from LibriSpeech-PC test-clean and records the L1 relative error between adjacent timesteps for each self-attention (Attn) and feed-forward (FFN) sublayer. When the error for a layer at a given timestep falls below a threshold alpha, that layer’s output is marked safe to cache; at inference, cached outputs are retrieved instead of recomputed.
A key finding during calibration is that independently derived per-layer schedules cause audible artefacts, because caching one sublayer while the other is recomputed disrupts residual connections. To resolve this, the paper adopts a unified schedule where the cache pattern derived from the attention layer is applied identically to both attention and FFN sublayers. Experiments vary two parameters: the number of denoising function evaluations (NFE, 32 or 16) and the caching threshold alpha (0.15, 0.25, 0.30, or 0.50), resulting in schedules that cache between one-quarter and one-half of all timesteps. No model weights are modified.
Evaluation uses the publicly available F5-TTS v1 base model with exponential moving averaged weights, the Euler ODE solver, CFG strength 2, and Sway Sampling coefficient -1, matching the original F5-TTS configuration exactly.
Key Results
On LibriSpeech-PC test-clean and Seed-TTS test-en, the main results (Table 1) show that the 32 NFE baseline (no cache) achieves RTF 0.46, WER 2.03%, SIM-o 0.67, and UTMOS 3.86 on LibriSpeech-PC. Caching approximately one-quarter of steps (alpha=0.15) reduces RTF to 0.32 with effectively no quality change. Caching approximately half of steps (alpha=0.25) reduces RTF further to 0.26 (a 43% reduction, approaching 2x speedup) with only marginal degradation in NISQA (4.23 to 4.19).
At 16 NFE, moderate caching (alpha=0.30) reduces RTF from 0.23 to 0.18 with negligible quality impact, but aggressive caching (alpha=0.50) begins to degrade output in ways that parallel the effect of simply reducing NFE: UTMOS drops from 3.90 to 3.79 and NISQA from 4.14 to 3.97 on LibriSpeech-PC.
A direct ablation (Table 2) compares cached inference against equivalent step-reduction without caching, matched to the same number of compute steps. A user study with 37 participants found 53% preference for the 32 NFE cached variant over a 24 NFE no-cache baseline with the same compute budget, suggesting a marginal perceptual advantage for caching over step reduction at higher NFE. At 16 NFE the difference was negligible (49% vs. 51%).
All RTF measurements use an NVIDIA T4 GPU for 10-second speech segments.
Novelty Assessment
The contribution is engineering integration applied to TTS. SmoothCache is an existing acceleration method developed for image, video, and general audio diffusion models; F5-TTS is an existing published TTS system. This paper adapts SmoothCache to the TTS setting and identifies a practical fix (the unified caching schedule) for an inter-layer dependency issue that emerges when both Attn and FFN layers are cached simultaneously. The unified schedule is a heuristic rather than a principled derivation, and the authors acknowledge this.
The scope of experiments is explicitly limited by computational resources (the paper reads as a course project, noting constraints “due to computational cost and time constraints”), covers a single GPU type, uses a calibration set of only ten utterances, and tests a single TTS model. The results are nonetheless internally consistent and the ablation against step-reduction baselines is well-designed. The finding that caching is most effective at high NFE settings (where redundancy is greater) is a useful characterisation of the trade-off.
Field Significance
Low — This paper provides confirmatory evidence that zero-training inference caching transfers from image and video diffusion to TTS, and quantifies the speed-quality trade-off for one model under one GPU configuration. The contribution is a narrow application of an existing method with a practical workaround for an inter-layer dependency problem. It does not introduce new architecture, training procedure, or evaluation methodology, and the experimental scope is limited to a single system.
Claims
-
supports: Zero-training transformer layer caching can accelerate diffusion-based TTS inference without retraining or architectural changes.
Evidence: Applying SmoothCache to F5-TTS with alpha=0.25 (caching approximately half of 32 NFE steps) reduces RTF from 0.46 to 0.26 while keeping WER, SIM-o, and UTMOS within measurement noise on LibriSpeech-PC test-clean, requiring no modification to model weights. (§3.2, Table 1)
-
complicates: The inference-time quality benefit of layer caching in diffusion TTS depends on the length of the denoising trajectory.
Evidence: At 32 NFE, caching 50% of steps (alpha=0.25) preserves quality; at 16 NFE, the same proportional caching (alpha=0.50) degrades UTMOS from 3.90 to 3.79 and NISQA from 4.14 to 3.97 on LibriSpeech-PC test-clean, comparable to reducing NFE directly. (§4.1, Table 1)
-
complicates: Per-layer cache schedules derived independently for attention and feed-forward sublayers introduce inter-layer dependency artefacts in diffusion transformer TTS.
Evidence: Applying separate Attn-only and FFN-only cache schedules to F5-TTS caused audible artefacts in informal listening tests; adopting a unified schedule derived from the attention layer pattern resolved this issue while maintaining cache utilisation. (§3.1)
-
supports: Calibrated layer caching provides a marginally superior quality-compute trade-off compared to equivalent step reduction at high NFE settings.
Evidence: A user study with 37 participants found 53% preference for 32 NFE with caching (24 effective compute steps) over a 24 NFE no-cache baseline using the same compute budget; the difference collapsed to 49% vs. 51% at the lower 16 NFE setting. (§4.1, Table 2)
Limitations and Open Questions
Warning
RTF measurements use a single GPU (NVIDIA T4) and a single TTS model (F5-TTS). Generalisation to other GPU architectures, other DiT-based TTS systems, or different vocoder configurations is untested.
The calibration set contains only ten utterances, which may not capture the full distribution of synthesis conditions. The unified caching schedule is a heuristic fix for inter-layer dependency; a more principled joint optimisation of attention and FFN cache schedules is unexplored. Subjective evaluation relies on A/B preference tests rather than CMOS or SMOS ratings, limiting the quantitative characterisation of quality degradation. The interaction of SmoothCache with CFG strength, Sway Sampling coefficient, and alternative ODE solvers (midpoint, Heun-3) is not investigated.
Wiki Connections
- Flow Matching — the base F5-TTS system uses flow matching on a DiT backbone; this paper demonstrates that flow-matching TTS inference is amenable to zero-training caching acceleration.
- Diffusion TTS — the caching technique targets the iterative denoising loop central to diffusion and flow-matching TTS, and the analysis of redundancy across timesteps is specific to this inference paradigm.
- Zero-Shot TTS — F5-TTS, the target model, performs zero-shot voice cloning from a reference audio prompt; the paper evaluates caching fidelity under this conditioning regime.
- Evaluation Metrics — the paper reports WER, SIM-o, UTMOS, and NISQA across multiple caching configurations, providing a multi-metric view of the speed-quality trade-off.
- Subjective Evaluation — a user study with 37 participants conducts A/B preference tests on speaker similarity and overall quality, directly comparing cached and step-reduced inference configurations.