arXiv · 2025 · Preprint
Xie et al. (HKUST (Guangzhou) / Tencent AI Lab) · → Paper · Demo: ✓ · Code: ?
EmoSteer-TTS applies activation steering to the internal DiT layers of existing flow-matching TTS systems at inference time without any training, enabling fine-grained emotion conversion, interpolation, erasure, and multi-emotion blending on models like F5-TTS, E2-TTS, and CosyVoice2.
Problem
Emotion control in TTS typically requires training dedicated models with labeled emotional data, limiting applicability to black-box or pre-trained systems. Existing emotion-conditioned TTS systems require reference audio or explicit emotion labels at training time, and most support only discrete emotion categories. Systems trained with emotion conditioning often struggle to independently control emotional expression without degrading speaker similarity. The problem is compounded by the lack of large-scale, clean emotional speech data for training.
Method
EmoSteer-TTS is a plug-in inference-time method for adding emotion control to any DiT-based flow-matching TTS model. The pipeline has three stages:
Stage 1: Steering vector construction (difference-in-means). For each target emotion class, the system collects activations from a specific intermediate DiT layer for emotional and neutral speech samples from labeled corpora. For each training sample i, the emotion direction vector is computed as the normalized difference: v_i^e = (act_emotional - act_neutral) / ||act_emotional - act_neutral||. These are averaged over N samples to produce a mean steering vector e_emotion per emotion class. This difference-in-means method is borrowed from activation engineering research in LLMs.
Stage 2: Topk emotional token selection. Not all tokens in the sequence are equally emotion-bearing. An emotion2vec model scores each token position by its emotional relevance. The top-k positions (where emotion is concentrated) are identified; only these positions are targeted for steering during inference. This prevents steering from distorting prosodically neutral segments like silences or unstressed phonemes.
Stage 3: Inference-time activation steering. During flow-matching inference, the steering vector is added with strength α to the activations of selected tokens at every k-th DiT layer. For F5-TTS (22 layers), steering is applied every 5 layers; for E2-TTS (8 layers), every 3 layers; for CosyVoice2 (56 layers), every 5 layers. The steering strength α controls the emotional intensity: α=0 gives the base model output; positive α adds the target emotion; β in the erasure direction (negative projection) removes the source emotion.
This framework naturally supports:
- Emotion conversion: α > 0 for target emotion
- Emotion interpolation: linearly ramping α from 0 to target
- Emotion erasure: project out the emotion vector with β > 0
- Emotion replacement: erasure followed by conversion
- Multi-emotion blending: weighted combination of emotion vectors
The steering vectors are constructed offline from an 11-corpus dataset (6,900 utterances, 6 basic emotions: anger, disgust, fear, happiness, sadness, surprise) covering MSP-Podcast, ESD, RAVDESS, CREMA-D, and 7 other corpora, both English and Chinese.
Key Results
Table 3 compares EmoSteer-TTS against label-based (EmoSphere++, EmoDubber) and description-based (EmoVoice) emotion TTS systems, plus unsteered CosyVoice2 as a description-based baseline.
In-distribution (MSP-Podcast + ESD): EmoSteer-TTS + F5-TTS achieves E-SIM (emotion similarity) 0.280 average, N-MOS 3.31, WER 2.79, S-SIM 0.59. EmoSteer-TTS + E2-TTS: E-SIM 0.275, WER 3.28 (slightly higher). EmoSteer-TTS + CosyVoice2: E-SIM 0.275, WER 2.83. These results compare favorably against CosyVoice2 baseline (E-SIM 0.240, WER 2.71) and EmoVoice (E-SIM 0.205, WER 5.31) while being training-free.
For emotion interpolation, EI-MOS 4.00 (F5-TTS variant); for erasure, EE-MOS 4.02 (E2-TTS variant). Label-based baselines (EmoSphere++, EmoDubber) cannot perform interpolation or erasure tasks.
OOD evaluation (EMNS + SeedTTS): EmoSteer-TTS + F5-TTS: WER 2.65, S-SIM 0.65, E-SIM 0.260, EI-MOS 3.46, EE-MOS 3.92. Performance is broadly consistent across in-distribution and OOD splits, supporting generalization claims.
F0 contour visualizations (Appendix F) confirm that emotion interpolation produces monotonically increasing pitch for anger, happiness, and surprise, while fear and sadness show non-monotonic patterns (attributed to multi-modal acoustic expression).
Novelty Assessment
The paper’s core novelty is the application of activation steering — a technique from LLM interpretability research — to DiT-based TTS models for training-free emotion control. This is a natural but non-obvious transfer. The topk emotional token selection to focus steering on emotion-bearing positions is a practical improvement over global steering. The unified treatment of conversion, interpolation, erasure, and multi-emotion blending under a single vector arithmetic framework is clean and well-motivated. The training-free nature is valuable for production systems where model retraining is costly. The limitation is that the method assumes internal representations of emotion in pre-trained models, which may not generalize to models trained without emotional data exposure.
Field Significance
Tip
High — EmoSteer-TTS introduces activation steering as a viable mechanism for emotion control in pre-trained TTS systems, transferring a technique from LLM interpretability research to the speech synthesis domain. It demonstrates that emotion is encoded as manipulable linear directions in DiT activations, enabling inference-time control without any training, which provides a new paradigm for adding expressiveness to already-deployed models. The finding that topk emotion-bearing tokens can be isolated and steered independently opens a direction for interpretable control of continuous acoustic attributes in generative speech models.
Claims
- Training-free inference-time activation steering can achieve emotion control in flow-matching TTS models that is competitive with or superior to systems explicitly trained for emotion conditioning. (§4.2, Table 1)
- Pre-trained flow-matching TTS models encode emotion as approximately linear directions in DiT activation space, enabling vector arithmetic over multiple emotion categories. (§3, §4.6)
- Emotion-bearing information in TTS DiT activations is concentrated in a subset of token positions identifiable via speech emotion recognition scores, and selective steering of these positions outperforms global activation modification. (§3.3, §4.6)
- Activation steering for emotion control generalises across model architectures and scales without retuning, producing consistent results on F5-TTS (22 layers), E2-TTS (8 layers), and CosyVoice2 (56 layers). (§4.2, §4.5)
- Training-free emotion manipulation via activation steering incurs a moderate speaker similarity cost relative to the unsteered baseline, indicating residual speaker-emotion entanglement in DiT representations. (§4.2, Table 1)
Limitations and Open Questions
- The steering vectors are derived from difference-in-means, which assumes linearity of emotional representations in DiT activations — a plausible but unverified assumption.
- Speaker similarity drops slightly in EmoSteer+F5 (S-SIM 0.59) vs unsteered F5 (0.66), suggesting some speaker-emotion coupling in the activations.
- The method requires access to DiT intermediate activations, ruling out fully black-box deployment.
- The topk selection method depends on emotion2vec quality; errors in emotion token detection propagate to steering performance.
- Evaluated on 6 basic emotions only; nuanced or blended emotions outside the training distribution are not assessed.
- Laughter, whispering, and other non-categorical paralinguistic phenomena are not covered.
Wiki Connections
EmoSteer-TTS applies to F5-TTS, E2-TTS, and CosyVoice2 (CosyVoice 2). The difference-in-means steering vector construction shares principles with Marco-Voice, which uses the same technique for constructing emotion embeddings. The flow-matching DiT architecture connects to flow-matching. The emotion control capability connects to emotion-synthesis and prosody-control concept pages. The in-corpus reference Controllable TTS Survey and EmoVoice provide context for the broader emotion TTS landscape. The training-free paradigm contrasts with the supervised RLHF post-training approach in Vevo2. Back-linked by EMORL-TTS, which uses EmoSteer-TTS as a comparison baseline for RL-based emotion control.