Speech Generation Research Wiki

Diffusion Text-to-Speech

Properties1
aliasesdiffusion speech synthesis, score-based TTS, latent diffusion TTS

4 min read

Abstract

Diffusion TTS generates speech by iteratively refining noise or latent variables under text, speaker, and style conditioning. It is established for high-fidelity, zero-shot, and expressive synthesis, but representation choice, solver cost, acceleration, guidance, and fair comparison with flow or consistency models determine its practical value.

Current State

Iterative diffusion can produce high-fidelity and natural speech across TTS and adjacent speech-generation tasks. This is strongly supported within the reviewed graph; representative evidence includes DiffStyleTTS: Diffusion-based Hierarchical Prosody Modeling for Text-to-Speech with Diverse and Controllable Styles, WaveFM: A High-Fidelity and Efficient Vocoder Based on Flow Matching. Comparisons span different datasets, vocoders, and sampling budgets, so quality gains are not attributable to the diffusion objective alone.

Iterative denoising remains a central latency and compute bottleneck for diffusion speech synthesis. This is strongly supported within the reviewed graph; representative evidence includes DiffStyleTTS: Diffusion-based Hierarchical Prosody Modeling for Text-to-Speech with Diverse and Controllable Styles, Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech. Reported speed depends on solver, hardware, target length, and whether encoder and vocoder costs are included.

Distillation, caching, and progressive calibration can reduce diffusion-TTS inference steps while retaining useful quality. This is strongly supported within the reviewed graph; representative evidence includes WaveFM: A High-Fidelity and Efficient Vocoder Based on Flow Matching, Seed-TTS: A Family of High-Quality Versatile Speech Generation Models. Hard-case intelligibility, diversity, and speaker similarity can degrade before aggregate naturalness metrics expose the loss.

Compressed latent targets reduce diffusion generation complexity without necessarily sacrificing speech quality. This is strongly supported within the reviewed graph; representative evidence includes AudioLDM: Text-to-Audio Generation with Latent Diffusion Models, NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers. Benefits depend on the latent encoder; reconstruction loss and semantic predictability can become new bottlenecks.

Guidance strength provides a direct but non-monotonic control over diffusion quality, fidelity, and conditioning adherence. This is strongly supported within the reviewed graph; representative evidence includes DiffStyleTTS: Diffusion-based Hierarchical Prosody Modeling for Text-to-Speech with Diverse and Controllable Styles, MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis. Guidance optima vary by condition type and evaluation metric, and classifier-free guidance increases per-step compute.

Transformer backbones support scalable diffusion modeling of speech and long-range conditioning. This is strongly supported within the reviewed graph; representative evidence includes WaveFM: A High-Fidelity and Efficient Vocoder Based on Flow Matching, StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusion. Scaling evidence is confounded by data volume and target representation, while transformer depth increases inference cost.

Method Landscape

  • Score-based diffusion synthesis. 14 memberships demonstrate a shared practical pattern, not matched superiority.
  • Flow, consistency, and bridge alternatives. 7 memberships demonstrate a shared practical pattern, not matched superiority.
  • Adversarial and distilled diffusion. 5 memberships demonstrate a shared practical pattern, not matched superiority.
  • Autoregressive-diffusion hybrids. 5 memberships demonstrate a shared practical pattern, not matched superiority.
  • Diffusion-transformer acoustic models. 4 memberships demonstrate a shared practical pattern, not matched superiority.

Key Trade-offs

  • Diffusion generators can transfer unseen-speaker identity from reference speech without speaker-specific fine-tuning. Speaker-similarity results depend on prompt quality, language match, and the embedding model used for evaluation.
  • Factorizing content, speaker, and style conditioning improves controllability in diffusion speech generation. Representation separation is often inferred from downstream metrics rather than verified causally.
  • Diffusion models can support explicit style, emotion, prosody, and singing control. Control accuracy, naturalness, and speaker preservation are rarely evaluated jointly under matched conditions.
  • Robust training and representation design improve diffusion speech generation under noisy or mismatched conditions. Noise types and recording channels vary substantially, limiting generalization from individual robustness tests.

Open Questions

  • Which combinations of target representation, solver, and conditioning yield the best quality-latency frontier under matched compute?
  • Can one-step or cached diffusion systems retain diversity, robustness, and speaker identity on difficult out-of-domain prompts?
  • When should diffusion be preferred to flow matching or consistency objectives once architecture and inference budget are controlled?
  • How should human preference supervision be applied to noisy intermediate diffusion states without reward-model bias?
  • Which evaluation suite jointly captures naturalness, intelligibility, identity, expressiveness, diversity, and deployment latency?

Go Deeper

Read Diffusion Text-to-Speech: In Depth for detailed evidence, method relationships, disagreements, implications, and a representative reading path.

Scope

This synthesis covers 46 papers through Q3 2025. Evidence spans multiple architectures, datasets, tasks, and evaluation protocols. Strong status applies within the encoded graph; shared lineages, infrastructure evidence, emerging findings, and unmatched speed or quality comparisons retain narrower interpretations.

This page is generated from wiki/_claims/diffusion-tts.yaml (digest date: 2026-07-26). For complete structured claims and paper-level provenance, use the source YAML.

Backlinks

Graph View