Abstract
Instruction-conditioned TTS turns natural-language requests into controllable speech attributes such as emotion, accent, rate, prosody, and identity. Rich prompting and modular control plans are effective, while local composition, speaker preservation, synthetic supervision, bias, and judge reliability remain uneven.
Current State
Natural-language instructions can control multiple speech attributes within one synthesis system, including emotion, accent, speaking rate, prosody, and vocal identity. This is strongly supported within the reviewed graph; representative evidence includes Natural language guidance of high-fidelity text-to-speech with synthetic annotations, CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens. Control accuracy varies sharply by attribute and instruction abstraction; success on emotion or rate does not imply reliable control of age, timbre, non-verbal events, or local emphasis.
Instruction fine-tuning can add useful controllability to a pretrained speech generator with substantially less data than full pretraining. This is strongly supported within the reviewed graph; representative evidence includes CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens, FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications. Small-data adaptation is demonstrated for bounded domains or attribute sets; diverse instruction tuning can reduce performance on foundational tasks.
Diverse, fine-grained instruction supervision generalizes better than narrow templates or single-task instruction corpora. This is strongly supported within the reviewed graph; representative evidence includes SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities, SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning. Instruction diversity is often confounded with dataset scale, source-domain diversity, and filtering quality.
Automatic labeling, LLM captioning or rewriting, and synthetic-speech pipelines can scale instruction-speech supervision beyond manual annotation. This is strongly supported within the reviewed graph; representative evidence includes SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities, Natural language guidance of high-fidelity text-to-speech with synthetic annotations. Automatically generated labels inherit classifier, LLM, TTS, and voice-conversion errors; several studies do not isolate scale from annotation quality.
Explicit intermediate representations can decouple instruction interpretation from speech rendering and make controllability more modular. This is strongly supported within the reviewed graph; representative evidence includes SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities, Chain-Talker: Chain Understanding and Rendering for Empathetic Conversational Speech Synthesis. The best representation is unsettled: free-form descriptions are accessible, while categorical, SSML, and numerical plans are usually more precise.
Global style embeddings are insufficient for instructions that specify local timing, linguistic position, or independently composable acoustic attributes. This is emerging within the reviewed graph; representative evidence includes Multi-interaction TTS toward professional recording reproduction, BatonVoice: An Operationalist Framework for Enhancing Controllable Speech Synthesis with Linguistic Intelligence from LLMs. Evidence comes from different tasks and representation families, so the boundary between embedding limitations and training-data limitations is not yet isolated.
Method Landscape
- Direct prompt-conditioned speech language models. 11 memberships demonstrate a shared practical pattern, not matched superiority.
- Description-embedding style control. 9 memberships demonstrate a shared practical pattern, not matched superiority.
- Instruction-data and annotation pipelines. 8 memberships demonstrate a shared practical pattern, not matched superiority.
- Instruction-following benchmarks and audits. 8 memberships demonstrate a shared practical pattern, not matched superiority.
- Multimodal spoken-instruction models. 7 memberships demonstrate a shared practical pattern, not matched superiority.
Key Trade-offs
- Instruction-driven style changes can unintentionally alter speaker identity unless style and timbre are explicitly separated or corrected. Explicit separation improves controllability, but its necessity has not been established across all prompt-conditioned architectures.
- Open-source instruction-conditioned speech systems lag closed commercial systems most clearly on abstract, compositional, and degree-scaled control. Commercial systems are opaque and change over time, while evaluations cover different languages and prompt taxonomies.
- Fine-grained instruction following remains attribute-dependent: models that control emotion or speaking rate may fail on age, timbre, phonemic length, emphasis, or non-verbal vocal events. Most evidence is diagnostic or complicating rather than a controlled comparison over a shared attribute inventory.
- Automatic judges can approximate human instruction-adherence ratings, but they do not reliably capture all perceptual alignment, naturalness, or domain-specific failure modes. Agreement depends on task subjectivity, judge model, prompt design, and whether the target domain resembles the judge’s calibration data.
Open Questions
- Which instruction representation provides the best balance of free-form usability, local temporal precision, and compositional control?
- How can instruction-conditioned systems preserve speaker identity while independently changing style, emotion, and non-verbal events?
- Which automatic evaluation protocol predicts human judgments across abstract prompts, atypical speech, and multilingual settings?
- How much instruction diversity is needed before gains saturate, and can it be separated from raw data scale?
- Can open systems close the commercial-system gap on abstract and degree-scaled control without relying on proprietary data or judge models?
Go Deeper
Read Instruction-Conditioned Text-to-Speech: In Depth for detailed evidence, method relationships, disagreements, implications, and a representative reading path.
Scope
This synthesis covers 44 papers through Q3 2025. Evidence spans multiple architectures, datasets, tasks, and evaluation protocols. Strong status applies within the encoded graph; shared lineages, infrastructure evidence, emerging findings, and unmatched speed or quality comparisons retain narrower interpretations.
This page is generated from wiki/_claims/instruction-conditioned-tts.yaml (digest date: 2026-07-25). For complete structured claims and paper-level provenance, use the source YAML.