NAACL · 2025 · Conference
Yejin Jeon et al. (POSTECH) · → Paper · Demo: ? · Code: ?
Uses an LLM to predict both a holistic dialogue emotion tag and the specific words within the target utterance that should carry that emotion, then trains a dedicated acoustic embedding with a selective masking loss that exposes the TTS model only to the audio of those emotion-salient words.
Problem
Emotional dialogue speech synthesis (EDSS) combines two strands of prior work: emotive speech synthesis, which conditions a TTS model on a discrete emotion label or reference audio that a user must manually supply, and dialogue speech synthesis, which derives a holistic stylistic condition from prior conversational turns (typically a BERT-encoded concatenation of dialogue history) but otherwise treats the target utterance as undifferentiated text. Both lines lack a mechanism for the model to reason about which words in the target utterance should carry the emotional or prosodic emphasis, even though human speech concentrates affective cues on a small number of words or phrases rather than distributing them evenly across an utterance. Prior LLM-assisted approaches to stylistic TTS either require curating large text-description datasets per utterance or predict numeric prosodic attributes (pitch/energy deltas per word), which the paper argues is a less natural use of an LLM’s text-reasoning strengths than directly identifying emotion-bearing words.
Method
The backbone is a non-autoregressive FastSpeech2 TTS model, left largely unmodified in its variance adaptor and mel-spectrogram decoder. The novel components sit at the input side. A dual-stage prompting procedure queries GPT-3.5 with the dialogue history and the upcoming target utterance: Stage 1 predicts a single holistic emotion tag (one of five categories: anger, happiness, neutral, sadness, surprise) for the target utterance; Stage 2, conditioned on that tag, asks the LLM to identify the specific words or phrases in the target utterance that should be emphasized to convey it. These selected words are converted into a phoneme-level binary rationale sequence (1 for phonemes belonging to a selected word, 0 otherwise), embedded via a lookup table, and concatenated with the phonemic encoder output before the variance adaptor.
To incorporate the acoustic side of these rationales (not just their textual identity), the paper introduces a selective audio masking procedure: the ground-truth audio is zeroed out everywhere except the time spans (from forced-alignment timestamps) corresponding to the LLM-selected words; for neutral-tagged utterances the entire mask is zero. This masked audio is passed through a frozen pretrained HuBERT model, and the resulting embedding becomes the training target for a separate, learnable acoustic embedding that is initialized from the phoneme-level rationale representation but trained independently (gradients are blocked from flowing back into the phonemic encoder). This acoustic embedding is then concatenated with the rationale-augmented phonemic representation and the holistic emotion tag embedding before the variance adaptor. The total loss combines the standard FastSpeech2 reconstruction and variance-adaptor losses with the new selective audio masking loss, weighted by a tuned coefficient λ (best value found empirically at 0.5). HiFi-GAN converts predicted mel-spectrograms to waveform.
Key Results
On the DailyTalk dataset (5 emotion categories, 15,578 utterances after filtering for LLM-correct emotion prediction), the proposed model is compared against two baselines: the original DailyTalk context-encoder model (BERT-encoded dialogue history, no LLM rationale), and an emotion-label-only variant (ExE-DT) that concatenates a discrete emotion ID without any dialogue history. The proposed model outperforms both baselines on MOS across all five emotion categories (e.g., 4.04 vs. 3.79 for happiness, 4.11 vs. 3.65 for surprise, both relative to DailyTalk) and on ABX preference share (e.g., 58% vs. 26% on the neutral category relative to DailyTalk). On the objective Speech Emotion Accuracy (SEA) metric, the proposed model reaches 46.16%, a gain of +12.98 points over ExE-DT and +16.47 points over DailyTalk. Word error rate (10.9%) is comparable to the two baselines (9.0–10.4%), indicating the added emotion machinery does not measurably harm intelligibility (§5.1, Table 1).
Ablations isolate the two architectural contributions. Replacing the boolean phoneme-level rationale labels with a degenerate all-zero (“singular”) labelling drops SEA from 44.77% to 37.99%, confirming that the binary word-selection signal itself carries useful information (§5.2, Table 3). Removing the separate acoustic embedding (AE) entirely costs 6.94 points of SEA relative to the full model, and forcing the masking target to include full unmasked (non-neutral-excluded) audio rather than the selectively masked spans costs 9.59 points, showing both that a modality-independent acoustic representation matters and that the masking itself (rather than just using HuBERT features) is doing useful work (§5.3, Table 4). A secondary analysis compares GPT-3.5 word selections against a single human annotator and finds 73.5% overlap and similar average word counts (2.5 vs. 3.2) (§5.2, Table 2), and an appendix experiment swapping GPT-3.5 for Llama-3-8B shows a modest SEA drop (41.11% vs. 46.16%), attributed via POS-tag analysis to GPT-3.5 selecting more adjectives and nouns (Appendix B, Table 6).
Novelty Assessment
The architectural contribution is a genuine, if narrowly scoped, novelty: using an LLM not to generate a holistic style embedding (as in prior ChatGPT-tag approaches) but to select specific words for differential acoustic supervision via a masking loss is a new mechanism not previously applied to EDSS. The backbone TTS model (FastSpeech2) and vocoder (HiFi-GAN) are unmodified off-the-shelf components, and the use of HuBERT as a frozen feature extractor for the masking target is a standard self-supervised representation but is used here only as a fixed loss target, not as a core architectural input to the synthesis pipeline itself. The evaluation is conducted on a single dataset (DailyTalk) with a fairly small emotional-utterance subset (only 20.4% of utterances carry a non-neutral label, with happiness alone making up 85.8% of those), which limits the strength of the emotion-by-emotion conclusions, particularly for anger and sadness, which the paper’s own appendix shows have the lowest LLM tag-prediction accuracy (57%). The baseline comparisons are reasonable (one dialogue-context baseline, one emotion-label-only baseline) but both are weaker historical systems; no comparison is made against more recent diffusion- or LLM-conditioned EDSS systems.
Field Significance
Moderate — the paper demonstrates a specific mechanism (LLM-selected word rationales combined with a selective audio masking loss) for injecting localized, word-level emotional emphasis into a TTS pipeline, going beyond the holistic, utterance-level style conditioning common in prior dialogue-aware TTS work. The empirical gains are consistent but evaluated on one dataset with a notably emotion-imbalanced training set, and the underlying architecture and vocoder are unmodified prior components.
Claims
- supports: Selecting and acoustically emphasizing a small subset of emotion-salient words, rather than conditioning on a single holistic style embedding, improves emotion recognizability of synthesized dialogue speech.
Evidence: The proposed model improves Speech Emotion Accuracy by 12.98 points over an emotion-label-only baseline and 16.47 points over a dialogue-context-only baseline, with consistent MOS and ABX gains across all five emotion categories. (§5.1, Table 1)
- supports: Large language models can identify text spans that carry emotional salience with moderate agreement to human judgment, without being trained on acoustic or prosodic data.
Evidence: GPT-3.5’s word selections for emotion-bearing rationales overlap with a single human annotator’s selections at a rate of 0.735, with comparable average word counts (2.5 vs. 3.2 words per utterance) across four non-neutral emotion categories. (§5.2, Table 2)
- complicates: Training a separate, modality-independent representation for masked acoustic cues is necessary; forcing a single representation to jointly encode textual rationale and acoustic detail degrades performance.
Evidence: Removing the dedicated acoustic embedding and relying only on the phoneme-level binary rationale label drops SEA by 6.94 points relative to the full model; using full (unmasked) audio as the embedding target rather than selectively masked audio drops SEA by a further 9.59 points relative to the full model. (§5.3, Table 4)
- complicates: LLM-based emotion tag prediction accuracy varies substantially across emotion categories and is markedly worse for emotions underrepresented in the training distribution.
Evidence: Stage 1 prompting holistic emotion tag accuracy is 84% for happiness and 74% for surprise but only 57% for anger and sadness, the two emotion categories that together make up under 7% of the labelled training utterances. (§5.2, Appendix A, Figure 4)
Limitations and Open Questions
Warning
The training and evaluation data are heavily emotion-imbalanced: only 20.4% of DailyTalk utterances carry a non-neutral emotion label, and within that subset happiness accounts for 85.8% while anger is only 2.55%. Results for the lowest-accuracy, lowest-frequency emotion categories (anger, sadness) should be treated with caution.
The paper’s own limitations section flags that the approach assumes GPT-3.5 can reliably correlate text-based word selection with the acoustic features a human speaker would naturally emphasize when reading the same text aloud, a capability the authors acknowledge is not independently verified beyond the single-annotator overlap study and the Llama-3 generalizability check. The evaluation uses only one dataset (DailyTalk) and one TTS backbone (FastSpeech2); whether the selective masking loss transfers to autoregressive or diffusion-based backbones, or to languages other than English, is untested. No code or demo availability is reported in the paper.
Wiki Connections
- Emotion Synthesis — proposes a word-level rationale and selective audio masking mechanism as an alternative to holistic emotion-label or reference-audio conditioning for emotive TTS.
- Prosody Control — implements emphasis at the level of specific emotion-bearing words rather than predicting numeric pitch/energy/duration targets, contrasting with prior LLM-prompted prosody-attribute approaches.
- Instruction-Conditioned TTS — uses an LLM’s text-reasoning capability to derive synthesis conditioning signals (emotion tag and word rationale) automatically from dialogue context rather than requiring manual style selection at inference time.