arXiv · 2025 · Preprint
Tao Zhu et al. (Xinjiang University) · → Paper · Demo: ? · Code: ?
Applies Easy Consistency Tuning (ECT), a technique developed for image generation, to a diffusion-based TTS acoustic model, enabling one-step speech synthesis by fine-tuning a pretrained diffusion model directly rather than distilling a separate student from a teacher.
Problem
Diffusion-based acoustic models produce high-quality mel-spectrograms but require many iterative sampling steps, which slows inference. Consistency distillation (as in CoMoSpeech) addresses this by training a separate one-step student model from a pretrained diffusion teacher, but this requires maintaining and training two models and makes the student’s quality dependent on the teacher’s performance, adding training complexity. Easy Consistency Tuning (ECT) was proposed in the image domain as a way to reach one-step generation by progressively tightening a consistency constraint on a single pretrained diffusion model, without a separate student, but had not been applied to speech synthesis before this paper.
Method
ECTSpeech follows a Grad-TTS-style pipeline: a text encoder (six Feed-Forward Transformer blocks, matching CoMoSpeech’s encoder) converts phonemes into hidden representations, a duration predictor estimates per-phoneme durations, and a length regulator expands these into frame-level features that condition a U-Net-based denoising network via a prior mel-spectrogram mean. To improve multi-scale feature fusion at the U-Net’s skip connections, the authors introduce a Multi-Scale Gate module (MSGate) with four parallel branches (1×1, 3×3, 5×5 convolutions and a global-pooling branch), whose outputs are concatenated, fused via a 1×1 convolution, and passed through a sigmoid to produce gating weights applied element-wise to the input features.
Training proceeds in two stages. First, the full model (text encoder, duration predictor, length regulator, denoiser) is pretrained end-to-end using the Elucidated Diffusion Model (EDM) formulation, with duration loss, prior mel-spectrogram loss, and an EDM reconstruction loss. Second, the text encoder, duration predictor, and length regulator are frozen, and only the denoising network is fine-tuned with an ECT loss: two correlated noise levels t and r (r ≤ t) are sampled, the network’s output at the higher noise level t is pushed toward its own (stop-gradient) output at the lower noise level r, and r is annealed from near 0 toward t over training to progressively tighten the consistency constraint and smoothly transition the model from a diffusion model to a consistency model. Because longer utterances otherwise dominate the loss, the ECT loss is normalized by a binary validity mask over mel-spectrogram frames. An exponential moving average (EMA) of the denoiser’s weights is maintained during consistency tuning and used exclusively at inference. A pretrained HiFi-GAN vocoder converts the generated mel-spectrogram into waveform. Model size is not reported. No neural audio codec is used; the pipeline operates on mel-spectrograms with a conventional GAN vocoder.
Key Results
On LJSpeech (Table 1), the pretrained diffusion model (50-step sampling) achieves the best MOS (4.35 ± 0.09) and best FD (0.6328) among all compared systems, including CoMoSpeech’s teacher. After consistency tuning, ECTSpeech at 1-step sampling reaches MOS 4.16 ± 0.08, FAD 0.5246 (best among all systems, including 50-step baselines), and RTF 0.0077, while requiring only 170k consistency-tuning steps on top of the pretrained model, versus training a separate CoMoSpeech student (the authors state ECTSpeech reaches comparable one-step quality to CoMoSpeech with about 10% of the latter’s total training iterations). ECTSpeech’s one-step output also outperforms FastSpeech2 and DiffGAN-TTS on both MOS and distributional metrics (FAD, FD).
Ablations (Table 2) show that removing MSGate raises FAD from 0.5246 to 0.6621 and FD from 0.6976 to 0.7453 and lowers MOS from 4.16 to 4.09; removing masked normalization similarly degrades all three metrics. Removing consistency tuning entirely (using the pretrained diffusion model directly at 1-step) causes a much larger collapse: FD rises to 7.9731, FAD to 1.6863, and MOS drops to 3.41, indicating that consistency tuning, not architecture alone, is responsible for most of the one-step quality.
All comparisons are on a single dataset (LJSpeech, single speaker) with a modest MOS panel (10 utterances, 10 raters per system), and all baselines are several years older than the paper (FastSpeech2, DiffGAN-TTS, Grad-TTS, CoMoSpeech); there is no comparison against more recent flow-matching-based one-step or few-step TTS systems.
Novelty Assessment
The core technical move, applying Easy Consistency Tuning to a diffusion TTS acoustic model, is an adaptation of an existing image-generation technique to a new domain rather than a new training principle; the paper’s own contribution is explicitly framed as being “the first time” this strategy has been used in speech synthesis. The genuinely novel architectural piece is MSGate, a lightweight multi-branch gated fusion module added to U-Net skip connections; the ablation shows it provides a real but modest quality gain (MOS 4.09 → 4.16, FAD 0.66 → 0.52) on top of consistency tuning. The claimed practical benefit, avoiding a separate teacher-student distillation pipeline while matching CoMoSpeech’s one-step quality with roughly 10% of its training steps, is a meaningful efficiency claim, but it is evaluated only on a single, small, single-speaker dataset against baselines that predate the current wave of flow-matching-based fast TTS systems.
Field Significance
Moderate — this paper demonstrates that a consistency-tuning strategy developed for image generation transfers to diffusion-based TTS acoustic modeling, providing evidence that one-step generation quality comparable to teacher-student distillation is achievable by fine-tuning a single pretrained model. Its contribution is primarily an engineering adaptation paired with a small, genuinely new fusion module (MSGate), evaluated only on a single-speaker English dataset against older baselines.
Claims
- supports: Consistency tuning applied directly to a pretrained diffusion model, without a separate teacher-student distillation pipeline, can reach one-step speech synthesis quality comparable to consistency-distillation approaches while requiring substantially fewer total training steps.
Evidence: ECTSpeech reaches MOS 4.16 (1-step) versus CoMoSpeech’s 4.19 (1-step) using 170k consistency-tuning steps atop a 1.7M-step pretrained diffusion model, described by the authors as roughly 10% of CoMoSpeech’s total training iterations. (§4.4, Table 1)
- supports: Multi-scale gated feature fusion in a diffusion denoiser’s skip connections improves one-step generation quality over standard skip connections, particularly when sampling is limited to a single step.
Evidence: Removing MSGate degrades FAD from 0.5246 to 0.6621, FD from 0.6976 to 0.7453, and MOS from 4.16 to 4.09 under 1-step inference on LJSpeech. (§4.5, Table 2)
- complicates: The bulk of the quality recovery in one-step consistency-based TTS comes from the consistency-tuning objective itself rather than from architectural improvements to the denoiser.
Evidence: Disabling consistency tuning (using the pretrained diffusion model directly at 1-step) collapses FD from 0.6976 to 7.9731, FAD from 0.5246 to 1.6863, and MOS from 4.16 to 3.41, a far larger drop than removing MSGate or masked normalization individually. (§4.5, Table 2)
- complicates: Loss normalization by valid-frame count is needed to prevent longer utterances from dominating the consistency-tuning objective and degrading quality on shorter utterances.
Evidence: Removing masked normalization raises FAD from 0.5246 to 0.6947 and FD from 0.6976 to 0.7031, with MOS dropping from 4.16 to 4.11. (§3.3, §4.5, Table 2)
Limitations and Open Questions
Warning
All experiments are conducted on a single single-speaker English dataset (LJSpeech), and comparisons are limited to older diffusion- and GAN-based baselines (FastSpeech2, DiffGAN-TTS, Grad-TTS, CoMoSpeech); no comparison is made against more recent flow-matching-based fast TTS systems, and generalization to multi-speaker or multilingual settings is untested.
The MOS evaluation uses only 10 utterances rated by 10 listeners per system, a small panel relative to standard TTS evaluation practice. Model parameter counts and computational cost of the MSGate module itself are not reported. The authors state future work will explore lighter-weight designs and extension to multi-speaker and emotional speech synthesis, which remain open in this paper.
Wiki Connections
- Diffusion TTS — extends a diffusion-based acoustic model (EDM-style pretraining) with a consistency-tuning fine-tuning stage to enable one-step sampling without a separate teacher-student pipeline.
- Transformer Encoder-Decoder TTS — reuses a Feed-Forward Transformer phoneme encoder and duration-predictor pipeline in the style of FastSpeech2/Grad-TTS as the conditioning front end for its diffusion denoiser.
- Evaluation Metrics — reports Fréchet Distance and Fréchet Audio Distance alongside MOS and RTF/NFE to characterize distributional similarity and inference cost.
- Subjective Evaluation — validates synthesis quality with a human MOS listening test (10 raters, 10 utterances per system) alongside objective distributional metrics.
- Grad-TTS — adopts Grad-TTS’s diffusion-based acoustic modeling pipeline as the pretraining backbone before applying consistency tuning.
- FastSpeech2 — compares against FastSpeech2 as a fast non-autoregressive TTS baseline in the main results table.
- HiFi-GAN — uses a pretrained HiFi-GAN vocoder to convert generated mel-spectrograms into waveforms for all compared systems.