Abstract
Instruction-conditioned TTS turns natural-language requests into controllable speech attributes such as emotion, accent, rate, prosody, and identity. Rich prompting and modular control plans are effective, while local composition, speaker preservation, synthetic supervision, bias, and judge reliability remain uneven.
Current State
Natural-language instructions can control multiple speech attributes within one synthesis system, including emotion, accent, speaking rate, prosody, and vocal identity. The evidence is strong. Examples include Natural language guidance of high-fidelity text-to-speech with synthetic annotations, CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens. Control accuracy varies sharply by attribute and instruction abstraction; success on emotion or rate does not imply reliable control of age, timbre, non-verbal events, or local emphasis.
Instruction fine-tuning can add useful controllability to a pretrained speech generator with substantially less data than full pretraining. The evidence is strong. Examples include CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens, FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications. Small-data adaptation is demonstrated for bounded domains or attribute sets; diverse instruction tuning can reduce performance on foundational tasks.
Diverse, fine-grained instruction supervision generalizes better than narrow templates or single-task instruction corpora. The evidence is strong. Examples include SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities, SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning. Instruction diversity is often confounded with dataset scale, source-domain diversity, and filtering quality.
Automatic labeling, LLM captioning or rewriting, and synthetic-speech pipelines can scale instruction-speech supervision beyond manual annotation. The evidence is strong. Examples include SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities, Natural language guidance of high-fidelity text-to-speech with synthetic annotations. Automatically generated labels inherit classifier, LLM, TTS, and voice-conversion errors; several studies do not isolate scale from annotation quality.
Explicit intermediate representations can decouple instruction interpretation from speech rendering and make controllability more modular. The evidence is strong. Examples include SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities, Chain-Talker: Chain Understanding and Rendering for Empathetic Conversational Speech Synthesis. The best representation is unsettled: free-form descriptions are accessible, while categorical, SSML, and numerical plans are usually more precise.
Global style embeddings are insufficient for instructions that specify local timing, linguistic position, or independently composable acoustic attributes. This is emerging within the reviewed graph; representative evidence includes Multi-interaction TTS toward professional recording reproduction, BatonVoice: An Operationalist Framework for Enhancing Controllable Speech Synthesis with Linguistic Intelligence from LLMs. Evidence comes from different tasks and representation families, so the boundary between embedding limitations and training-data limitations is not yet isolated.
Q4 makes guidance strength and text-only voice control explicit design boundaries. Cross-modal, chained, and iterative guidance can improve adherence to emotion, style, or editing instructions, but stronger guidance can reduce naturalness, intelligibility, or preservation of untargeted content (Cross-modal Consistency Guidance, Step-Audio-EditX, DMP-TTS). Text descriptions and multimodal identity prompts remove the need for reference audio and support broad creative control (InstructAudio, UniSpeaker), yet their one-to-many ambiguity and demographic priors make exact identity fidelity inherently underconstrained.
Method Landscape
- Direct prompt-conditioned speech language models. 15 memberships demonstrate a shared practical pattern, not matched superiority.
- Description-embedding style control. 14 memberships demonstrate a shared practical pattern, not matched superiority.
- Multimodal spoken-instruction models. 11 memberships combine text, speech, scene, or identity inputs.
- Instruction-data pipelines and audits. These families contain 11 and 10 memberships and expose how supervision quality and evaluation determine apparent instruction following.
- Structured and guidance-based control. Eight structured-intermediate and three guidance-based systems trade free-form flexibility for more explicit planning or stronger inference-time adherence.
Key Trade-offs
- Instruction-driven style changes can unintentionally alter speaker identity unless style and timbre are explicitly separated or corrected. Explicit separation improves controllability, but its necessity has not been established across all prompt-conditioned architectures.
- Open-source instruction-conditioned speech systems lag closed commercial systems most clearly on abstract, compositional, and degree-scaled control. Commercial systems are opaque and change over time, while evaluations cover different languages and prompt taxonomies.
- Fine-grained instruction following remains attribute-dependent: models that control emotion or speaking rate may fail on age, timbre, phonemic length, emphasis, or non-verbal vocal events. Most evidence is diagnostic or complicating rather than a controlled comparison over a shared attribute inventory.
- Automatic judges can approximate human instruction-adherence ratings, but they do not reliably capture all perceptual alignment, naturalness, or domain-specific failure modes. Agreement depends on task subjectivity, judge model, prompt design, and whether the target domain resembles the judge’s calibration data.
- Adherence versus fidelity. Increasing guidance or repeating edits can strengthen the requested attribute while degrading untargeted content, identity, or acoustic quality.
Open Questions
- Which instruction representation provides the best balance of free-form usability, local temporal precision, and compositional control?
- How can instruction-conditioned systems preserve speaker identity while independently changing style, emotion, and non-verbal events?
- Which automatic evaluation protocol predicts human judgments across abstract prompts, atypical speech, and multilingual settings?
- How much instruction diversity is needed before gains saturate, and can it be separated from raw data scale?
- Can open systems close the commercial-system gap on abstract and degree-scaled control without relying on proprietary data or judge models?
- How should systems express uncertainty when a text-only voice description admits many perceptually plausible realizations?
Go Deeper
Read Instruction-Conditioned Text-to-Speech: In Depth for detailed evidence, method relationships, disagreements, implications, and a representative reading path.
Scope
This synthesis covers 57 papers through Q4 2025. The graph contains 10 strongly supported, 5 emerging, and 2 contested clusters. Evidence spans multiple architectures, datasets, tasks, and evaluation protocols. Contested automatic judging and interpreter–renderer conclusions are retained rather than averaged into broader convergence.
This page is generated from wiki/_claims/instruction-conditioned-tts.yaml (digest date: 2026-09-26). For complete structured claims and paper-level provenance, use the source YAML.