arXiv · 2025 · Preprint
Xiaozhou Tan et al. (University of Sheffield) · → Paper · Demo: ? · Code: ?
Replaces the continuous-time noising and discretized-inference pipeline of diffusion and flow-matching TTS with a fully discrete-time formulation, and compares four discrete-time noising processes, including two previously unexplored in speech synthesis, against a continuous-time baseline.
Problem
Continuous-time diffusion and flow-matching TTS models are trained with a noising process defined in continuous time (typically additive Gaussian noise, chosen mainly because it keeps the score or velocity field analytically tractable) but sampled with a discretized solver at inference. This creates two mismatches: training assumes noise levels the model is never actually exposed to as the timestep approaches the fully corrupted extreme, and training/inference operate in different time representations (continuous vs. discretized). Restricting noising to additive Gaussian noise also implicitly assumes the data (e.g. mel-spectrogram) is locally isotropic Gaussian, which does not hold for real speech signals whose covariance depends on signal energy. Discrete-time diffusion-like processes avoid both problems by construction, but at the time of writing had not been evaluated for TTS, and more diverse discrete-time noising types (multiplicative noise, deterministic blurring, noise mixtures) had only been explored in image generation.
Method
The paper builds four discrete-time noising processes on top of a Grad-TTS-style pipeline: a text encoder that produces a prior U (with a Monotonic-Alignment-Search-trained duration predictor, following Glow-TTS/Grad-TTS), a non-causal residual convolutional decoder adapted from Grad-TTS, and a fixed pretrained HiFi-GAN vocoder. Unlike the original Grad-TTS, the decoder does not predict scores; it directly predicts the clean mel-spectrogram X₀ at every discrete step n ∈ {0,…,N}, trained with an MSE loss between the predicted and true clean data. Because inference reuses exactly this clean-data-prediction target at each discrete step, training and inference conditions are identical, removing the continuous/discrete mismatch of DM/FM models.
Four noising processes are compared, all computable in closed form without simulation: (1) “Grad-TTS-DT,” a discretized version of Grad-TTS’s original additive-Gaussian SDE noising; (2) “RFAG,” a discretized rectified-flow-style straight path between clean data and (Gaussian noise + prior); (3) “RFMG,” the same straight-path formulation but with multiplicative Gaussian noise (ε ~ N(I, σ²I)) applied to the prior instead of additive noise, intended to better match the signal-dependent, non-isotropic covariance of real speech; (4) “Blurring,” a fully deterministic noising process that applies a heat-equation-based spectral smoothing operator (cosine-basis projection) to the clean data, interpolated toward the prior; and (5) “Mixture,” which draws noise from a Gaussian centered on the blurring output, combining deterministic structure with stochastic variability. Inference for all systems except Blurring follows a first-sampling algorithm that alternates clean-data prediction and re-noising to n-1; Blurring uses a second algorithm with a first-order correction term suited to smooth, deterministic degradation. All decoders are initialized from a frozen Grad-TTS encoder checkpoint, trained for 500 epochs with the encoder frozen and then 100 more epochs with the encoder unfrozen.
Key Results
On LJSpeech with 10 inference steps, the fully discrete Grad-TTS-DT nearly matches or slightly beats the continuous-time Grad-TTS baseline on objective metrics (MCD 5.53 vs. 5.71, log F0 error 0.31 vs. 0.33, UTMOSv2 3.95 vs. 4.03) and edges it out on subjective MOS (4.07 vs. 4.02, ground truth 4.15). RFAG and RFMG (additive vs. multiplicative Gaussian noise within the same rectified-flow-style path) perform comparably to each other and to Grad-TTS-DT across MCD, log F0, and UTMOSv2, at both 10 and 5 inference steps, indicating multiplicative noise is not obviously worse despite its non-isotropic structure. The Mixture and Blurring systems achieve the best log F0 scores, but Blurring has the lowest UTMOSv2 and joint-lowest MOS (3.86) of all systems, suggesting the fully deterministic noising process smooths out acoustic detail without enough randomness to support naturalness. Reducing inference steps from 10 to 5 leaves objective metrics essentially unchanged for every discrete-time system, unlike the discretization-sensitive continuous baseline, which the authors attribute to the exact match between training and inference conditions in the discrete-time formulation.
Novelty Assessment
The contribution is architectural/methodological rather than a new state-of-the-art result: no discrete-time variant clearly outperforms the continuous-time Grad-TTS baseline, and the paper is explicit that its shared encoder is suboptimal. What is new is the systematic exploration, within a single controlled discrete-time framework, of noising processes beyond additive Gaussian noise for speech: multiplicative Gaussian noise, deterministic blurring, and a blurring/Gaussian mixture had previously only been explored for image generation, and this paper is the first to bring them to TTS. The comparison is a fair like-for-like replacement of Grad-TTS’s decoder objective (score prediction to clean-data prediction) while holding the encoder and vocoder fixed, which makes the reported parity between fully discrete and partially discretized training a genuine empirical finding rather than an artifact of a stronger backbone.
Field Significance
Moderate — this paper demonstrates that a fully discrete-time training/inference formulation can match a popular continuous-time diffusion TTS baseline while removing the training/inference discretization mismatch, and it is the first to test multiplicative Gaussian noise, blurring, and a blurring/Gaussian mixture as noising processes in speech synthesis. The evaluation is narrow (single speaker, single dataset, a suboptimal shared encoder), so the result reads as a proof of concept for discrete-time noising diversity rather than a new best-performing system.
Claims
- supports: A fully discrete-time training and inference formulation can match the subjective and objective speech quality of a continuous-time diffusion baseline while eliminating the continuous/discrete mismatch inherent to score-based samplers.
Evidence: Grad-TTS-DT (discretized additive-Gaussian noising, clean-data prediction) achieves MCD 5.53, log F0 error 0.31, UTMOSv2 3.95 versus the continuous Grad-TTS baseline’s 5.71 / 0.33 / 4.03 at 10 inference steps, and a higher subjective MOS (4.07 vs. 4.02). (§4.1, Table 1; §4.2, Table 3)
- supports: Multiplicative Gaussian noise is a viable alternative to additive Gaussian noise as a diffusion-like noising process for speech, despite its non-isotropic covariance structure.
Evidence: RFMG (multiplicative noise) achieves MCD, log F0, and UTMOSv2 close to RFAG (additive noise) across matched σ values at both 10-step and 5-step inference. (§4.1, Table 1, Table 2)
- complicates: Fully deterministic noising processes are insufficient on their own for naturalness in discrete-time diffusion-like speech synthesis; some stochasticity in the noising process appears necessary.
Evidence: The fully deterministic Blurring system records the lowest UTMOSv2 of all compared systems (3.71 at 10 steps, 3.66 at 5 steps) and ties for the lowest subjective MOS (3.86), which the authors attribute to blurring smoothing critical acoustic detail without enough variability. (§4.1, Table 1, Table 2; §4.2, Table 3)
- refines: Consistency between training and inference conditions, not just the number of inference steps, determines how well a diffusion-like TTS model generalizes to fewer sampling steps.
Evidence: All discrete-time systems retain near-identical MCD and log F0 error when inference steps are reduced from 10 to 5, which the authors attribute to the discrete-time formulation exposing the model to exactly the same conditions at training and inference. (§4.1, Table 1, Table 2)
Limitations and Open Questions
Warning
All systems are trained and evaluated on a single English, single-speaker corpus (LJSpeech) with a text encoder inherited unmodified from a pretrained Grad-TTS checkpoint that the authors themselves describe as suboptimal; the reported parity between discrete-time and continuous-time systems has not been tested on multi-speaker, multilingual, or zero-shot settings.
The mixture noise ratio between blurring and Gaussian components is fixed at a constant borrowed from an image-generation paper rather than tuned for speech, and the authors note the Mixture system’s UTMOSv2 might improve with a different ratio. The paper also does not compare against more recent flow-matching TTS systems (e.g. rectified-flow or optimal-transport conditional flow matching models cited in related work), only against the original score-based Grad-TTS, so it is unclear how the discrete-time variants compare to the current flow-matching state of the art rather than to an older diffusion baseline.
Wiki Connections
- Diffusion TTS — replaces Grad-TTS’s continuous-time score-based noising and discretized SDE sampling with a fully discrete-time, clean-data-prediction formulation, and tests deterministic and mixture noising processes not previously used in diffusion TTS.
- Flow Matching — discretizes the rectified-flow straight-path formulation into a discrete-time process (RFAG, RFMG) and compares it directly against a discretized DDPM-style additive-Gaussian process within the same framework.
- Subjective Evaluation — reports Mechanical-Turk MOS ratings from 20 native listeners per sample across all six compared systems plus ground truth, alongside objective metrics.
- Evaluation Metrics — evaluates each noising variant with MCD, log F0 error, and UTMOSv2 at two different inference-step counts to test both quality and step-count robustness.