arXiv · 2025 · Preprint
Wang et al. (Harbin Institute of Technology) · → Paper · Demo: ? · Code: ?
Emo-FiLM introduces word-level emotion modulation into LLM-based TTS by aligning self-supervised emotion features to word boundaries and applying Feature-wise Linear Modulation to text embeddings, enabling dynamic intra-utterance emotional transitions that sentence-level control cannot capture.
Problem
Most emotional TTS systems assign a single emotion label, reference audio clip, or natural language prompt to an entire utterance. This sentence-level conditioning is effective for uniform emotion but fails when the emotional trajectory shifts within a sentence, such as a phrase that begins with surprise and resolves into relief. No existing benchmark adequately measures this capability either, since existing datasets annotate emotion at the utterance level only.
Method
Emo-FiLM has two components: a Fine-grained Emotion Annotation stage and an Emotion-modulated Generation stage.
In the annotation stage, emotion2vec — a self-supervised speech emotion model — extracts frame-level emotion features from training speech. The Montreal Forced Aligner (MFA) provides frame-to-word alignment, after which masked average pooling aggregates each word’s frame sequence into a fixed-dimensional vector. A lightweight Transformer jointly trained with classification (discrete emotion category) and regression (continuous emotion intensity) heads produces word-level emotion labels for each training utterance.
In the generation stage, the pretrained CosyVoice2 LLM-TTS backbone is frozen. An Emotion Encoder maps discrete category and continuous intensity labels into dense embeddings, which are fused with text embeddings. An E-FiLM layer then computes dimension-wise scale (gamma) and shift (beta) parameters from these fused emotion features and applies an affine transformation to the text hidden representations before they are fed into the autoregressive decoder. The decoder generates speech tokens autoregressively, with a vocoder (HiFi-GAN) and flow-matching decoder inherited from CosyVoice2. Training optimises a joint objective combining label-smoothed cross-entropy for speech token prediction and stepwise cross-entropy for emotion classification.
To support evaluation of intra-utterance dynamics, the authors construct the Fine-grained Emotion Dynamics Dataset (FEDD): 1,000 utterances across 5 speakers and 5 emotions, with 500 mild transitions generated via natural language instructions and 500 strong transitions created by concatenating emotionally distinct segments from the same speaker.
Key Results
On the ESD global emotion task (1,500 samples, 10 speakers, 5 emotions), Emo-FiLM achieves Emo SIM of 98.78%, DTW of 23.98, and WER of 3.12%, outperforming CosyVoice2 (DTW 27.48, WER 6.21%) and EmoSpeech on all objective metrics (Table 1). On the more challenging FEDD fine-grained task, Emo-FiLM achieves Emo SIM of 99.32%, DTW of 49.62, WER of 7.32%, EMOS of 4.19, and NMOS of 4.23, beating CosyVoice2 on DTW by 9.1% relative (Table 1).
Ablation results (Table 2) are informative: removing word-level fine-grained data causes the largest degradation (FEDD DTW rises from 49.62 to 133.97), demonstrating that sub-sentence supervision is the core driver of performance. Removing the FiLM layer (replaced by simple addition) yields DTW 70.5, and removing the auxiliary emotion classification loss yields DTW 73.9 — confirming that both the modulation mechanism and the explicit emotion supervision contribute meaningfully.
Novelty Assessment
The paper’s primary technical contribution is applying FiLM conditioning to modulate LLM-TTS text embeddings at the word level using self-supervised emotion features. FiLM is an established conditioning technique from computer vision (visual question answering); the novelty lies in adapting it to sub-sentence emotional control in LLM-TTS. The backbone (CosyVoice2) is used frozen, so the architectural contribution is the E-FiLM module and its emotion annotation pipeline. The FEDD dataset is a genuine gap-filling contribution, as no public benchmark provides sentence-level emotional transition annotations. The ablations are thorough and the baselines (EmoSpeech, GenerSpeech, CosyVoice2) represent the three main E-TTS conditioning paradigms, making the comparison credible. The evaluation introduces DTW as a dynamic complement to frame-averaged Emo SIM, which is a useful methodological step.
Field Significance
Moderate — Emo-FiLM demonstrates that word-level FiLM conditioning is a practical mechanism for capturing emotional dynamics that global E-TTS methods cannot model, and the FEDD dataset provides a reproducible benchmark for evaluating this capability. The contribution is primarily an engineering integration of self-supervised emotion features and an established modulation technique onto a frozen LLM-TTS backbone, rather than a fundamental architectural change. It advances the sub-field of fine-grained expressive TTS and provides a reusable evaluation resource.
Claims
-
supports: Sub-sentence emotion conditioning produces more accurate emotional dynamics in TTS than sentence-level conditioning.
Evidence: Emo-FiLM outperforms CosyVoice2 by 9.1% on DTW and 12.7% on ESD DTW, with higher EMOS on both test sets, when comparing word-level emotion labels against global prompt conditioning. (§4.1, Table 1)
-
supports: Feature-wise Linear Modulation is an effective mechanism for injecting word-level conditioning signals into the text representations of autoregressive LLM-TTS systems.
Evidence: Ablation replacing the FiLM layer with simple addition increases FEDD DTW from 49.62 to 70.5, demonstrating that the non-linear affine modulation is essential beyond mere feature fusion. (§4.2, Table 2)
-
supports: Self-supervised speech emotion models provide useful word-aligned supervision for fine-grained emotional TTS when combined with forced alignment.
Evidence: emotion2vec frame-level features aligned via MFA and pooled to word boundaries form the annotation backbone; removing word-level data tuning causes the most severe degradation (FEDD DTW 49.62 to 133.97). (§2.1, §4.2, Table 2)
-
complicates: Frame-averaged emotion similarity metrics are insufficient for evaluating intra-utterance emotional dynamics in TTS output.
Evidence: The authors note that Emo SIM averages frame-level emotion vectors, potentially obscuring dynamic information, and introduce DTW as a complementary metric that more sensitively distinguishes the models’ ability to track emotional transitions. (§3.4)
-
supports: Explicit emotion classification as an auxiliary training objective improves fine-grained emotion control in multi-task TTS training.
Evidence: Removing the emotion classification loss increases FEDD DTW from 49.62 to 73.96, a degradation comparable to removing the FiLM layer entirely. (§4.2, Table 2)
Limitations and Open Questions
The FEDD dataset is small (1,000 utterances, 5 speakers) and constructed partly via concatenation of emotionally distinct segments, which may not capture naturally occurring emotional transitions. The method is not evaluated for spontaneous or conversational speech, where emotion boundaries are less well-defined. The backbone (CosyVoice2) is frozen, so the approach inherits any limitations of that system in speaker diversity or prosodic range. Model size and speaker generalization are not reported.
Wiki Connections
- Emotion Synthesis — introduces word-level FiLM conditioning to capture intra-utterance emotional transitions, extending beyond the sentence-level control that dominates this area.
- Autoregressive Codec TTS — the system is built on the frozen CosyVoice2 LLM-TTS backbone, adapting an autoregressive codec TTS architecture for fine-grained emotion control.
- Self-Supervised Speech — uses emotion2vec, a self-supervised speech emotion model, as the core feature extractor for generating word-level emotion annotations.
- Evaluation Metrics — introduces DTW as a complementary dynamic evaluation metric alongside frame-averaged Emo SIM for assessing intra-utterance emotional fidelity.
- Subjective Evaluation — reports EMOS and NMOS with human raters to validate both emotion fidelity and naturalness of the synthesized speech.