arXiv · 2025 · Preprint
Huo et al. · → Paper · Demo: ? · Code: ?
DiTReducio is a training-free inference acceleration framework for DiT-based TTS that exploits temporal and branch redundancy across diffusion transformer layers through a three-phase progressive calibration process, reducing FLOPs by over 50% at an optimal operating point while preserving speaker similarity and intelligibility.
Problem
DiT-based non-autoregressive TTS systems achieve strong quality and parallelism but incur substantial compute cost at inference: the self-attention mechanism scales quadratically with sequence length, and classifier-free guidance (CFG) doubles the forward pass by running conditional and unconditional branches in parallel. Existing acceleration approaches for diffusion-based TTS rely primarily on distillation, which requires additional training and teacher-model dependency. Training-free alternatives developed for image and video generation (DeepCache, Learning-to-Cache) have not been adapted to the speech domain, and token-wise attention optimizations conflict with FlashAttention compatibility. A plug-and-play, quality-controllable acceleration technique suitable for deployed TTS inference remains absent from the literature.
Method
DiTReducio operates as a wrapper around any DiT-based TTS model that uses CFG. The framework identifies two structural redundancies in the denoising process. Temporal redundancy refers to high output similarity between adjacent timesteps in a given layer: outputs at step t closely approximate those at step t-1, particularly in shallow layers. Branch redundancy refers to similarity between the conditional and unconditional branch outputs at certain layer-timestep combinations under CFG. To exploit these, the framework introduces two compression strategies. Temporal Skipping (TS) caches the module output from the preceding timestep and reuses it, skipping computation entirely. Branch Skipping (BS) runs only the conditional branch and reconstructs the unconditional branch output using a cached branch residual (the difference between unconditional and conditional outputs from the previous step).
A key insight is that temporal redundancy is correlated with the self-attention pattern of a layer at a given timestep: layers showing a diagonal-like pattern (tokens attending primarily to neighbours) are consistently more temporally redundant than layers showing a striped pattern. The framework uses cosine similarity between the attention heatmap and an identity matrix as a cheap proxy to identify high-redundancy layer-step pairs without computing the output loss directly.
The calibration proceeds in three phases over only three full inference passes. In the Check Phase, the top 10% of layer-step pairs ranked by diagonal-pattern similarity are flagged as candidate temporal-redundancy pairs. In the Pre-Calibration Phase, TS is selectively applied to only those flagged pairs, and pairs whose output loss exceeds a dynamic threshold are removed from the candidate set. In the Calibration Phase, all layer-step pairs (including those not flagged) are re-evaluated under both TS and BS, with a depth-dependent threshold: layer l receives threshold (l/L) * delta, so shallower layers are compressed more aggressively than deeper ones. The resulting strategy table (T x L binary mask over timesteps and layers) is fixed and reused for all subsequent inference calls without any recomputation.
The framework is evaluated on F5-TTS and MegaTTS 3, both of which use conditional flow matching within DiT. For F5-TTS, conditional and unconditional branch inputs are concatenated into a single batch to enable BS while maintaining FlashAttention compatibility. For MegaTTS 3, which uses multi-condition CFG with two conditional branches, separate residuals are maintained for each branch pair.
Key Results
At the recommended T4 threshold on LibriSpeech-PC test-clean (1,127 samples), F5-TTS with DiTReducio achieves a real-time factor (RTF) of 0.129 versus a baseline of 0.178 (27.5% reduction) with an operations ratio of 45.58% (54.4% fewer FLOPs). Speaker similarity (SIM-o) drops from 0.640 to 0.618 and WER is nearly unchanged (2.634% vs 2.636%). At the maximum T6 threshold, RTF reaches 0.112 (37.1% reduction) and FLOPs drop to 34.42% of baseline, but SIM-o degrades to 0.59 and WER rises slightly to 2.9%. MegaTTS 3 shows a more pronounced RTF improvement at T4: RTF falls from 0.396 to 0.224 (43.4% reduction) with SIM-o declining from 0.750 to 0.734. Quality degradation for MegaTTS 3 remains minimal until T5, suggesting the model is more amenable to this style of compression.
Ablation confirms that combining TS and BS is necessary: TS alone causes WER to reach 23.06% at maximum threshold due to loss of conditional information, while BS alone provides limited additional speed beyond T3. The Check Phase and Pre-Calibration Phase together improve both quality retention and final acceleration compared to the single-phase Calibration Phase alone.
Novelty Assessment
The core ideas in DiTReducio are adapted from image and video diffusion acceleration (DeepCache, Learning-to-Cache), where caching intermediate activations across timesteps to skip redundant computation is well established. The genuine contributions are (1) the observation that attention pattern geometry (diagonal vs striped) reliably predicts temporal redundancy in speech DiT layers, enabling a cheap proxy to guide calibration; (2) the Branch Skipping formulation, which handles the conditional/unconditional branch redundancy in CFG without simply reusing the unconditional output directly; and (3) the three-phase progressive calibration that converts these observations into a model-specific, per-pair strategy. The framework does not require any training data or retraining, and it is compatible with FlashAttention, which is a practical constraint that prior token-wise methods fail to satisfy.
The evaluation scope is narrow: two models, one dataset (LibriSpeech-PC test-clean), no human listening test, and no comparison against other TTS-specific distillation methods on equivalent hardware. The claim of a 75.4% FLOPs reduction cited in the abstract appears to refer to a different accounting than Table 1’s Ops Ratio column and is not directly reconcilable from the paper.
Field Significance
Moderate — DiTReducio provides a practical, training-free path to reducing inference cost in DiT-based flow-matching TTS models, one of the main continuous-output decoder choices in current zero-shot synthesis systems. The pattern-guided calibration approach is a genuine adaptation of image-domain caching ideas to the structural properties of speech diffusion, and the FlashAttention compatibility constraint distinguishes it from prior token-wise approaches. The contribution is primarily methodological rather than foundational; its relevance depends on adoption in deployed DiT-based TTS pipelines.
Claims
-
supports: Training-free caching of redundant DiT computations can substantially reduce FLOPs in TTS without retraining and without significant intelligibility degradation.
Evidence: At the optimal T4 threshold, DiTReducio reduces F5-TTS FLOPs by 54.4% while keeping WER within 0.002 percentage points of the uncompressed baseline on LibriSpeech-PC test-clean. (§4.2, Table 1)
-
supports: Diagonal-like self-attention patterns in DiT layers are a reliable proxy for identifying temporally redundant layer-timestep pairs during diffusion inference.
Evidence: Cosine similarity between an attention heatmap and the identity matrix is strongly correlated with temporal redundancy (defined as output deviation when caching the previous timestep’s output); layer-step pairs scoring above 0.35 in diagonal similarity approach 100% redundancy. (§3.3, Figure 4)
-
complicates: Temporal skipping alone in DiT TTS inference causes severe conditional information loss, making branch skipping a necessary complement for quality-preserving acceleration.
Evidence: Ablation on F5-TTS shows that TS-only achieves greater speedup but reaches WER of 23.06% at maximum threshold; full DiTReducio (TS + BS) holds WER at 2.9% under the same threshold. (§4.3, Figure 6)
-
complicates: Training-free DiT acceleration exhibits a threshold-dependent quality cliff: beyond a moderate compression ratio, marginal speed gains decrease while speaker similarity degradation accelerates.
Evidence: For F5-TTS, moving from T4 to T6 yields only an additional 0.017 RTF improvement (0.129 to 0.112) but drops SIM-o from 0.618 to 0.590; the paper identifies T4 as the practical optimum. (§4.2, Table 1)
-
refines: Inference-time acceleration techniques from image and video diffusion transfer to speech synthesis but require domain-specific adaptation, particularly for classifier-free guidance structures.
Evidence: Branch skipping is introduced to handle speech CFG’s conditional/unconditional branch redundancy, and the multi-condition CFG in MegaTTS 3 requires separate residuals for each branch pair, illustrating that speech-specific CFG structures need targeted handling. (§3.2, §Appendix A.1)
Limitations and Open Questions
Warning
DiTReducio applies only to DiT-based models; it does not generalise to autoregressive or non-DiT non-autoregressive architectures. Additionally, the paper reports no subjective listening test: quality claims rest entirely on SIM-o and WER, which may not capture perceptual artefacts introduced by aggressive caching.
Calibration requires high-quality reference audio whose acoustic properties match the target deployment context; the paper does not specify how much calibration data is needed or how sensitive results are to calibration set choice. The framework’s compatibility with streaming or chunk-based inference is not discussed. The FLOPs accounting in the abstract (75.4% reduction) is inconsistent with Table 1’s Ops Ratio values, making the headline figure difficult to verify.
Wiki Connections
- Flow Matching — DiTReducio targets flow-matching DiT models (F5-TTS and MegaTTS 3 both use conditional flow matching) and the redundancy patterns it exploits arise specifically in multi-step flow-matching inference.
- Diffusion TTS — the framework is designed for DiT-based TTS, which sits at the intersection of diffusion-model inference and transformer architecture.
- Zero-Shot TTS — both systems evaluated (F5-TTS and MegaTTS 3) are zero-shot TTS models; DiTReducio accelerates inference in zero-shot deployment contexts.
- Evaluation Metrics — the paper introduces RTF and FLOPs ratio alongside SIM-o and WER as the primary evaluation axes, foregrounding computational efficiency metrics alongside quality metrics.
- F5-TTS — one of the two models directly evaluated; DiTReducio is implemented on F5-TTS and achieves 27.5% RTF reduction at the recommended threshold.
- MegaTTS 3 — the other primary evaluation target; MegaTTS 3’s multi-condition CFG required a specific BS adaptation for two conditional branch pairs.
- NaturalSpeech 3 — cited as a representative DiT-based TTS system within the class of models DiTReducio targets.
- MaskGCT — cited as a zero-shot non-autoregressive TTS system, situating DiTReducio within the landscape of efficient NAR TTS inference.