Abstract
Emotion synthesis controls affect, style, intensity, and paralinguistic behavior in generated speech. Explicit conditioning is effective, but the practical frontier has shifted toward compositional control, within-utterance dynamics, contextual appropriateness, reward robustness, and preserving naturalness, intelligibility, and identity together.
Current State
Explicit emotion conditioning reliably improves recognizability and perceived expression. Categorical labels, continuous affect dimensions, references, and learned style embeddings all provide useful control (Prompt-Guided Emotive TTS, TTS-CtrlNet). Results depend on taxonomy, class balance, and the neutral baseline; acted categories do not establish natural emotional behavior in conversation.
Natural-language control broadens the expressive vocabulary beyond fixed labels. Descriptions can specify mixed emotions, speaking manner, intensity, or pragmatic intent, while audio references often retain a naturalness advantage (CosyVoice, InstructTTSEval). Understanding an instruction and realizing it acoustically remain distinct failure points, and open-ended prompts lack standardized evaluation.
Emotion is increasingly controlled within the utterance rather than once per sentence. Word-, phoneme-, frame-, and sub-sentence conditioning improves emphasis, transitions, and emotional dynamics (TTS-CtrlNet, EMORL-TTS). Finer control requires reliable alignment and transition smoothing. The late non-verbal study supports dedicated event modeling, but does not objectively verify requested event counts or durations (Fine-Grained Non-Verbal Expression Control).
Identity and emotion can be controlled more independently, but not perfectly separated. Predictors, contrastive losses, perturbation, and factorized pathways reduce leakage between speaker and affect (Marco-Voice, ZSDEVC). Habitual expression is part of perceived identity, so complete removal can make a target voice less recognizable or natural.
Stronger emotion does not monotonically improve perceived quality. Across guidance, reference transfer, and composed controls, greater intensity can improve recognition while reducing intelligibility, naturalness, or identity. The late non-verbal study makes the distinction explicit: fine-grained events improve emotion recognition and expressiveness but slightly lower naturalness. Evaluations should therefore report operating-point curves and separate perceptual axes.
Context is becoming a primary affect signal. Dialogue history, narrative, semantic content, visual scenes, faces, and retrieved exemplars provide complementary information about appropriate delivery (Chain-Talker, Emotion-Aware Audio-Visual Modeling). More context is not automatically better: irrelevant retrieval and upstream recognition errors can reduce affect fidelity.
Preference optimization improves targeted emotional alignment while creating reward-model risk. DPO, GRPO, token-level rewards, and robust policy optimization extend supervised expressive training (EMORL-TTS, RRPO). Reward models may favor exaggerated classifier cues or degenerate trajectories; held-out listener tests across emotion, naturalness, identity, and intelligibility remain necessary.
Dedicated treatment of laughter, sighs, fillers, and emphasis is now well supported. Ordinary TTS objectives and lexical metrics often treat intended vocal events as errors. Specialized data and rewards improve control, but can worsen WER or identity scores even when listeners prefer the output. This is a measurement mismatch, not automatically a synthesis failure.
Method Landscape
- Autoregressive emotion-aware speech models condition token generation on labels, instructions, dialogue, or paralinguistic tokens and increasingly use preference post-training.
- Flow-matching systems inject global, local, reference, or instruction-derived affect into parallel acoustic generation and low-step editing.
- Transformer encoder–decoder systems place emotion and style controls at sentence, word, phoneme, or frame level through attention and hierarchical conditioning.
- Hybrid semantic–acoustic systems separate affect interpretation, planning, acoustic rendering, speaker identity, and dialogue context.
- GAN, diffusion, and VAE systems support expressive transfer, iterative refinement, and continuous latent control, each with different stability and leakage trade-offs.
Key Trade-offs
- Control strength versus fidelity. Stronger emotion can improve recognizability while reducing naturalness, identity, or intelligibility.
- Labels versus open-ended instructions. Categories are easier to supervise; descriptions are richer but harder to interpret and evaluate consistently.
- Temporal precision versus stability. Local control enables transitions and emphasis but increases alignment and accumulation errors.
- Reference expressiveness versus leakage. Audio prompts carry rich prosody but may leak lexical content, speaker traits, or corpus artifacts.
- Reward scale versus reward robustness. Preference optimization can improve alignment, then saturate or exploit an imperfect evaluator.
Open Questions
- How can emotion reward models resist acoustic reward hacking without suppressing subtle listener-preferred expression?
- Can emotion, dialect, speaker identity, rate, and paralinguistic events compose without cross-attribute degradation?
- What temporal granularity supports natural transitions without fragile alignment?
- Which protocol jointly evaluates emotional appropriateness, trajectory, naturalness, intelligibility, and identity?
- How can emotional coherence persist across long multi-turn dialogue rather than isolated utterances?
Go Deeper
Read Emotional and Expressive Speech Synthesis: In Depth for the detailed research landscape, disagreements, evaluation implications, and representative reading path.
Scope
This synthesis covers 98 reviewed papers published through the Q4 2025 evidence window. It spans TTS, emotional voice conversion, spoken dialogue, multimodal generation, preference alignment, and paralinguistic control. Much of the evidence uses acted or automatically annotated emotion and short utterances, so strong control results do not by themselves establish culturally robust, spontaneous, or long-horizon emotional coherence.
This page is generated from wiki/_claims/emotion-synthesis.yaml (digest date: 2026-10-31). For complete structured claims and paper-level provenance, use the source YAML.