arXiv · 2021 · Preprint
Tan et al. (Microsoft Research Asia) · → Paper · Demo: ? · Code: ?
A comprehensive 63-page survey of neural TTS that organises the field into key pipeline components (text analysis, acoustic models, vocoders, end-to-end models) and five advanced topic areas (fast, low-resource, robust, expressive, and adaptive TTS), serving as a consolidated reference and taxonomy for the neural speech synthesis literature through mid-2021.
Problem
By 2021, the neural TTS literature had grown rapidly across a wide range of architectures, conditioning strategies, and application settings. Researchers entering the field faced a fragmented landscape of over 450 papers spanning acoustic models, neural vocoders, generative frameworks (flow, GAN, VAE, diffusion), and practical concerns like latency, low-resource languages, and speaker adaptation. No single resource organised these contributions into a coherent taxonomy or identified the dominant trends and open problems. This survey addresses that gap.
Method
The survey is structured around two complementary framings. The first covers the three canonical pipeline stages of a TTS system: text analysis (normalisation, G2P, prosody prediction), acoustic models (mel-spectrogram generation from phonemes or characters), and neural vocoders (waveform synthesis from acoustic features). A secondary taxonomy classifies works as autoregressive or non-autoregressive, and by generative family: normalising flows, GANs, VAEs, and diffusion probabilistic models. A timeline of representative models (Table 18 and continuations) situates each system within the progression from statistical parametric synthesis through early neural models (WaveNet, Tacotron) to the parallel and fully end-to-end systems dominant by 2021.
The five advanced topic chapters cover: (1) fast TTS, contrasting parallel non-autoregressive generation with lightweight model design and domain-knowledge speedup techniques; (2) low-resource TTS, covering self-supervised pre-training, cross-lingual transfer, and data mining; (3) robust TTS, analysing alignment learning failures and the shift from attention mechanisms to explicit duration prediction; (4) expressive TTS, cataloguing explicit label conditioning, reference encoder approaches, VAE and flow-based variation modeling, and disentanglement methods across information granularities from frame to utterance level; and (5) adaptive TTS, covering few-shot, zero-shot, and untranscribed data adaptation scenarios.
The survey also compiles resources including open-source implementations (ESPnet, NeMo, Mozilla TTS, Coqui, and model-specific repositories for Tacotron, FastSpeech, HiFi-GAN, etc.), benchmark corpora, and TTS challenges (Blizzard, Voice Conversion Challenge, Zero Resource Speech Challenge).
Key Results
As a survey, the paper does not report original experimental results. Its primary empirical contribution is the comparative characterisation of model families through structured tables. Table 8 summarises training and inference time complexity across architectural paradigms: RNN-based autoregressive models carry O(N) complexity in both training and inference, while feed-forward Transformer-based systems (FastSpeech family) achieve O(1) by replacing attention-based alignment with explicit duration prediction. GAN-based vocoders are non-autoregressive with O(1) inference; flow and diffusion models require O(T) iterations proportional to the number of sampling steps.
Table 6 compares generative model families along four properties (simplicity, parallel generation, latent manipulation, likelihood estimation): autoregressive models are the simplest but do not support parallel inference; GANs support parallel inference but cannot estimate likelihood; flows and diffusion-based models support both parallel inference and likelihood estimation but require iterative sampling. The paper catalogues alignment strategies in Table 11, showing that predicting duration explicitly is the only mechanism that simultaneously satisfies the local, monotonic, and completeness properties for robust alignment.
Novelty Assessment
The paper’s contribution is organisational and conceptual rather than empirical. It introduces a coherent dual taxonomy (pipeline components and generative model family), the staged end-to-end progression framework (Stages 0-4 from statistical parametric synthesis to fully end-to-end models), and structured comparative tables that did not exist in consolidated form at the time. Its novelty lies in synthesis and framing, not in new architectures or training methods. The survey is notably thorough for 2021 standards, covering the diffusion model vocoder literature (DiffWave, WaveGrad, Grad-TTS) and flow-based acoustic models (Glow-TTS, Flowtron) that had only recently appeared. Coverage of codec-based and language model-based TTS (VALL-E and subsequent work) is necessarily absent, as that paradigm emerged after the survey was written.
Field Significance
Tip
High, conceptual contribution. This survey provides the taxonomy and reference framing that the neural TTS community used through the mid-2021 to mid-2023 period, before the LLM-based codec paradigm reshaped the field. It introduces a staged end-to-end progression model and a generative-family comparison framework that can serve as a useful baseline when characterising how dramatically the dominant paradigm shifted after VALL-E. Its citation density (36 in-corpus citations concentrated in Q3-Q4 2025) indicates that subsequent TTS system papers used it as a standard field framing reference.
Claims
- The transition from attention-based alignment to explicit duration prediction in non-autoregressive acoustic models eliminates word skipping and repeating artefacts while enabling parallel mel-spectrogram generation. (§3.4, Table 11)
- Modelling variation information (prosody, style, speaker) at multiple granularities from frame to utterance level improves expressiveness while fine-grained control remains difficult when relying on implicit latent representations. (§3.5.1, §3.5.3)
- Diffusion-based vocoders achieve high perceptual quality but suffer from slow inference due to iterative reverse diffusion, motivating step-reduction methods. (§2.4, Table 6)
- Zero-shot speaker adaptation that relies solely on a speaker encoder without any fine-tuning produces noticeably lower voice quality than few-shot adaptation, particularly for speakers distant from the training distribution. (§3.6.2)
- Low-resource TTS benefits from cross-lingual transfer because human languages share vocal organs and phonetic structures, though phoneme set mismatches require explicit bridging. (§3.3)
Limitations and Open Questions
The coverage horizon is mid-2021, which means the survey predates several paradigm shifts: neural codec-based discrete speech tokenisation, large language model-based TTS (VALL-E, NaturalSpeech 2), flow-matching acoustic models, and the instruction-conditioned and zero-shot capabilities that came with LLM-scale training. The survey’s taxonomy (acoustic model + vocoder pipeline) is also less applicable to fully codec-native systems where this separation dissolves.
The survey focuses on English and Mandarin and, while noting multilingual coverage, does not deeply address the orthographic diversity or tonal complexity of the majority of the world’s languages. Evaluation methodology is discussed mainly by cataloguing metrics rather than analysing their reliability or mutual disagreement, which limits the survey’s utility for assessing evaluation practices.
Open questions identified include: better variation information modeling that reconciles controllability and expressiveness, casual and spontaneous speech synthesis, parameter-efficient adaptation at scale, and energy-efficient training and inference.
Wiki Connections
Concepts: transformer-enc-dec-tts, gan-vocoder, diffusion-tts, flow-matching, evaluation-metrics, autoregressive-codec-tts
In-corpus papers: 1609.03499 (WaveNet, the first modern neural vocoder catalogued), 2105.06337 (Grad-TTS, a diffusion-based acoustic model covered in the survey), 2104.00355 (speech resynthesis from discrete disentangled self-supervised representations)
2406.04904 — XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model 2407.05361 — Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation