EMNLP · 2025 · Conference
Tianxin Xie et al. (HKUST-GZ / University of Surrey) · → Paper · Demo: ? · Code: ?
The first comprehensive survey of controllable TTS, organizing ~80 methods (2018–2025) into a four-strategy control taxonomy (style tagging, reference speech, natural language description, instruction-guided) and surveying architectures, feature representations, datasets, and evaluation metrics.
Problem
Despite rapid growth in TTS quality and controllability, there is no unified survey of controllable TTS specifically. Prior surveys (e.g., Tan et al. 2021) cover neural TTS broadly but not the fine-grained control dimension that has become central to industrial deployment. The gap is particularly acute for LLM-driven control via natural language prompts, which has emerged as a separate research thread without a systematic treatment. Researchers lack a shared taxonomy for comparing control strategies, feature representations, and architectural choices.
Method
The survey is organized around three axes: control strategies, model architectures, and feature representations.
Control strategies (Table 2): Four categories form the core taxonomy:
- Style tagging — predefined attribute labels (pitch, speed, emotion) directly condition the acoustic model. Simple but limited in expressive diversity.
- Reference speech prompts — a short audio clip provides speaker identity or style at inference, enabling zero-shot TTS and voice cloning. High personalization but relies on clean reference audio.
- Natural language descriptions — text prompts (e.g., “a calm female voice”) steer generation. User-friendly; limited by description precision and model comprehension.
- Instruction-guided control — LLMs interpret free-form instructions combining content and paralinguistic directives. Highest expressivity and precision; highest system complexity.
Architectures: Two broad families are covered — non-autoregressive (NAR: transformer enc-dec, flow-matching, diffusion, GAN) and autoregressive (AR: token-level LMs over discrete codec representations). The survey traces the transition from Tacotron/FastSpeech-family NAR models to VALL-E-family AR codec LMs and hybrid systems.
Feature representations: Six controllable attributes are reviewed: prosody (pitch, energy, speed), timbre (speaker identity, voice quality), emotion (categorical and continuous), style (speaking style, register), language (multilingual, code-switching), and environment (acoustic scene, room acoustics). For each, the survey covers how the attribute is represented (e.g., F0 contours, speaker embeddings, emotion category labels, SSL features) and how it is injected into the model (conditioning, adapter, style tokens).
Datasets (Table 3): 19 publicly available datasets are catalogued with annotation coverage across pitch, energy, speed, age, gender, emotion, emphasis, topic, description, and dialogue dimensions. Ranges from IEMOCAP (12h, 10 speakers, 2008) to Parler-TTS (50,000h, natural language descriptions, 2024) and SpeechCraft (2,391h, 3,200 speakers, multi-attribute, 2024).
Evaluation (Table 4): Objective metrics — MCD, FDSD, WER, cosine speaker similarity, PESQ — and subjective metrics — MOS, CMOS, AB test, ABX test. The survey notes that no standard benchmark unifies controllability evaluation across control types.
Table 1 compiles ~80 representative methods from 2018–2025 with binary controllability checkmarks across the six feature dimensions.
Key Results
This is a survey; it reports no new system numbers. Key synthesized findings:
- Transition from NAR to AR: The field has shifted from NAR acoustic models (FastSpeech, Tacotron variants) toward AR codec LMs (VALL-E family) and flow-/diffusion-based models, with the latter offering better zero-shot speaker generalization and natural language controllability.
- Instruction-guided control is the frontier: Natural language prompts via LLMs (e.g., PromptTTS, InstructTTS, VoxInstruct, ControlSpeech) represent the most flexible control paradigm but remain the least mature, with no standardized evaluation.
- Emotion and prosody remain open problems: Fine-grained, independent control of emotion, prosody, and timbre simultaneously is not reliably achieved by any single system. Most systems handle one or two dimensions well.
- Dataset gap: Datasets with comprehensive multi-attribute annotations are sparse; most existing datasets cover only emotion or only speaker identity, not both.
- Evaluation is fragmented: No benchmark combines naturalness (MOS), intelligibility (WER), speaker similarity (cosine), and controllability accuracy in a single standardized protocol.
Novelty Assessment
This is explicitly the first comprehensive survey on controllable TTS specifically. The four-strategy taxonomy is the primary organizational contribution; it cleanly separates approaches that differ in user-facing interface (labels vs. audio vs. text vs. LLM) rather than only in model architecture. The Table 1 compilation (~80 methods with controllability attribute checkmarks) is a practical reference tool. The appendix provides historical context (HMM-era → neural → LLM-era) and clarifies the quantization/tokenization distinction that is often conflated in the codec TTS literature. The survey is limited to TTS; VC controllability and SCA control interfaces are mentioned but not surveyed in depth.
Field Significance
Moderate — This survey introduces a useful four-strategy taxonomy (style tagging, reference speech, natural language description, instruction-guided) that provides a shared vocabulary for comparing the growing body of controllable TTS work. Its primary contribution is conceptual organization: it maps a fragmented literature onto a coherent framework and identifies evaluation fragmentation and multi-attribute disentanglement as the field’s most pressing open problems. It does not propose new methods or benchmarks, so its impact depends on whether the taxonomy gains traction as a reference framing.
Claims
- Instruction-guided control via natural language, while the most expressive TTS control paradigm, remains the least mature, with no standardized evaluation protocol across systems. (§3.2.4, §4.2)
- Fine-grained simultaneous control over multiple speech attributes (emotion, prosody, timbre) is not reliably achieved by any existing single system, representing a fundamental open problem in controllable TTS. (§5.1)
- The absence of a benchmark that jointly measures naturalness, intelligibility, speaker similarity, and controllability accuracy prevents systematic progress comparison across control strategies. (§4.2, §5.1)
- The field has undergone a structural shift from non-autoregressive acoustic models toward autoregressive codec language models and flow-matching systems, driven by improvements in zero-shot speaker generalization and natural language control. (§3.1.3)
- Datasets with comprehensive multi-attribute annotations covering emotion, speaker identity, prosody, and style simultaneously remain scarce, constituting a practical bottleneck for developing fully controllable TTS systems. (§4.1, §5.1)
Limitations and Open Questions
Coverage cuts off around late 2024/early 2025; systems from mid-2025 onward (Qwen3-TTS, Marco-Voice, DiSTAR, etc.) are not included. The survey focuses on single-attribute controllability; multi-attribute simultaneous control is noted as an open problem but not quantitatively evaluated. Robustness to out-of-distribution prompts and long-form synthesis are identified as challenges without proposed solutions. The awesome-controllable-speech-synthesis GitHub list is maintained as a living update mechanism.
Wiki Connections
As a survey, this paper provides the broadest structural view of controllable TTS in the corpus. It serves as an entry point to the instruction-conditioned-tts, emotion-synthesis, prosody-control, zero-shot-tts, and evaluation-metrics concept pages. The four-strategy taxonomy maps directly onto concept pages: reference-speech-prompt strategy → zero-shot-tts; natural language description → instruction-conditioned-tts; emotion control → emotion-synthesis; prosody control → prosody-control.
In-corpus papers referenced by this survey: VALL-E (foundational AR codec LM), CosyVoice (supervised semantic tokens), F5-TTS (flow-matching zero-shot), ControlSpeech (zero-shot speaker and style control), StyleTTS-ZS (distilled time-varying style diffusion), MELLE (autoregressive speech synthesis without VQ), EmoVoice (LLM-based emotional TTS).