COLING · 2025 · Conference
Jiaxuan Liu et al. (University of Science and Technology of China) · → Paper · Demo: ✓ · Code: ?
DiffStyleTTS replaces the variance adaptor in a FastSpeech2-style acoustic model with a conditional diffusion module that hierarchically models prosody at both coarse-grained (implicit style) and fine-grained (explicit pitch/energy/duration) levels, using an improved classifier-free guidance with dynamic thresholding to enable controllable and transferable prosody at inference.
Problem
Non-autoregressive acoustic models such as FastSpeech2 optimize a simple regression objective, which causes over-smoothing of prosodic predictions and limits the diversity of generated speech. Traditional prosodic transfer methods like Global Style Tokens (GST) lack fine-grained controllability over transfer intensity. Prior diffusion-based TTS work (Grad-TTS, Guided-TTS) targets the full Mel-spectrogram rather than prosodic features specifically, and DiffProsody — the most directly related predecessor — constrains prosodic features through a discriminator but still offers no controllability over prosody during inference. The one-to-many mapping from text to prosody therefore remains inadequately addressed.
Method
DiffStyleTTS is a multi-speaker acoustic model built on the FastSpeech2 backbone. The architecture retains the feed-forward Transformer (FFT) encoder (4 blocks, 256-D embeddings) and decoder (6 blocks), a 5-layer convolutional PostNet, speaker embedding lookup, and a HiFiGAN vocoder. The key change is replacing FastSpeech2’s variance adaptor with a conditional diffusion module for hierarchical prosody modeling.
Hierarchical prosody is captured at two levels. Coarse-grained implicit style conditions (ISC) are extracted from reference Mel-spectrograms via a GST module (10 tokens, 256-D token embeddings, 4-head attention). Fine-grained explicit prosodic features (phoneme-wise log-pitch, log-energy, and duration) are predicted by the conditional diffusion module.
The conditional diffusion module implements classifier-free guidance (CFG) using two jointly trained denoisers sharing the same bidirectional dilated convolution architecture (12 residual layers, WaveNet-style): one conditioned on both text embeddings and implicit style conditions (Ψ_θ1), one conditioned on text embeddings only (Ψ_θ2). During inference a guiding scale η linearly interpolates their noise predictions, balancing diversity and quality. A dynamic thresholding improvement corrects the standard deviation of the guided noise output to prevent phoneme distortion (elongation/noise artefacts) that arises at high guiding scales (η ≥ 7.0). A correction scale γ controls the strength of this fix.
Three inference modes are supported: (1) diversified controllable inference (tune η and γ to vary prosodic diversity without a reference); (2) prosodic transfer (transfer prosodic style from a reference utterance to a target speaker, with intensity adjustable via η); (3) prosodic control (select individual GST tokens or blend their weights to synthesize speech with a specified prosodic style, and apply pitch/energy/duration scaling factors).
Training uses a composite loss: two DDPM diffusion losses (one conditional, one unconditional), a decoder Mel-spectrogram reconstruction loss, and a PostNet residual loss. All diffusion models use 200 denoising steps (T = 200) with a cosine noise schedule. The optimal inference setting reported is η = 1.0, γ = 0.7.
Key Results
Evaluated on a private 54-hour Mandarin dataset with 9 male speakers. Listening tests used 14 raters (18 utterances per model, 2 per speaker). DiffStyleTTS achieves MOS 4.18 ± 0.06, outperforming all baselines: FastSpeech2 (3.85), Grad-TTS (4.08), Guided-TTS (4.15), and DiffProsody (4.10). Ground truth scores 4.55.
On prosodic distribution accuracy (JS Divergence), DiffStyleTTS obtains the best pitch (0.065), ties DiffProsody on energy (0.030), and nearly ties DiffProsody on duration (0.045 vs. 0.046).
On synthesis speed (RTF), DiffStyleTTS (0.048) is faster than Grad-TTS (0.250) and Guided-TTS (0.479) because it runs diffusion only over low-dimensional prosodic features rather than full Mel-spectrograms. It is also faster than DiffProsody (0.063), though slower than FastSpeech2 (0.019).
Prosodic transfer preference tests (12 raters, 16 sentences): DiffStyleTTS is preferred over FastSpeech2+GST (52.5% vs. 33.1%, p = 0.008) and over DiffProsody (50.0% vs. 16.7%, p = 0.013) for Reference A, with similar margins for Reference B.
Ablation confirms all three components are necessary: removing ISC from the diffusion condition degrades MOS to 3.92; removing ISC from text embeddings degrades to 3.80; removing text embedding conditions entirely collapses quality to MOS 2.05.
Novelty Assessment
The core contribution is applying diffusion-based CFG to prosodic feature prediction specifically (pitch, energy, duration) rather than to the full Mel-spectrogram, enabling a meaningful speed advantage over Mel-diffusion models while retaining diversity benefits over regression-based models. The dynamic thresholding fix for CFG-induced phoneme distortion is a modest but practical engineering improvement borrowed from image diffusion. The hierarchical two-level prosody conditioning (implicit via GST + explicit via CFG-guided diffusion) is a clear architectural contribution within the non-autoregressive TTS paradigm. Overall the contribution is incremental relative to DiffProsody, but the explicit inference-time controllability (η, γ, GST token blending) and the speed-quality trade-off analysis add genuine value. Evaluation is limited to a private dataset, which restricts reproducibility.
Field Significance
Moderate — DiffStyleTTS demonstrates that applying diffusion and classifier-free guidance directly to low-dimensional prosodic features (rather than the full Mel-spectrogram) yields a favorable speed-quality trade-off, contributing evidence that targeted diffusion is a viable path for controllable prosody in non-autoregressive TTS. The hierarchical ISC+CFG design provides a concrete template for combining coarse-style transfer with fine-grained diversity control, though the evaluation is restricted to a private Mandarin dataset which limits generalizability claims.
Claims
- Applying diffusion-based classifier-free guidance to low-dimensional prosodic features (pitch, energy, duration) rather than the full Mel-spectrogram achieves faster inference than Mel-diffusion baselines while retaining prosodic diversity benefits over regression-based models. (§3.2, Table 1)
- Hierarchical conditioning combining coarse-grained implicit style tokens with fine-grained explicit prosodic diffusion outperforms both purely regression-based and discriminator-constrained diffusion baselines on naturalness and prosodic distribution accuracy. (§3.2, Table 1)
- Classifier-free guidance scale controls a trade-off between prosodic diversity and naturalness quality in diffusion-based prosody prediction, with distortion artefacts emerging at high guiding scales. (§3.3, Table 2)
- Dynamic thresholding applied to CFG noise outputs effectively mitigates phoneme distortion caused by excessive guiding scales in prosodic diffusion models. (§2.2, §3.3)
- GST-based implicit style conditioning enables meaningful prosodic transfer across speakers, outperforming both regression-based style transfer and discriminator-constrained diffusion in subjective preference evaluations. (§3.4, Table 3)
Limitations and Open Questions
Speaker timbre and prosody are not disentangled — the model can transfer prosodic style but cannot fully separate it from speaker identity. The GST tokens show partial overlap in t-SNE visualizations, indicating shared prosodic modes and incomplete independence. The evaluation is conducted on a private Mandarin Chinese dataset (9 male speakers from similar genres), limiting generalization claims across languages, genders, or more varied speaking styles. The 200-step diffusion process at inference is still far slower than FastSpeech2, and no acceleration scheme (consistency models, flow-matching) is explored. Whether the CFG approach generalises to cross-lingual or highly emotional speech remains untested.
Wiki Connections
- diffusion-tts — this paper advances the concept by targeting explicit prosodic features (pitch, energy, duration) with diffusion rather than the full Mel-spectrogram, achieving faster RTF while preserving diversity
- prosody-control — the hierarchical ISC + explicit feature prediction design and the three inference modes directly address controllable prosody
- disentanglement — the failure to fully separate timbre from style motivates future work in this area; the GST token visualization reveals incomplete independence
- transformer-enc-dec-tts — the FastSpeech2 backbone places this in the non-autoregressive transformer lineage
- subjective-evaluation — evaluation methodology (MOS listening tests, subjective preference tests, coefficient of variation for diversity) is relevant to this concept