SIGDIAL 2025 · 2025 · Workshop
Matsuura et al. (Carnegie Mellon University) · → Paper · Demo: ? · Code: ✓
EmoNews integrates a sentiment-guided PromptTTS module into a retrieval-augmented news dialogue system and proposes a new subjective evaluation scale for emotional spoken dialogue systems.
Problem
Emotional TTS has matured considerably, yet task-oriented spoken dialogue systems have been slow to adopt expressive speech synthesis. The gap arises partly because SDS and emotional TTS research have developed in isolation and partly because there are no standardised metrics for evaluating the social goals (such as empathy and emotional appropriateness) that emotionally expressive speech is meant to serve. Prior news-focused dialogue systems synthesise responses in a single, neutral voice regardless of the emotional valence of the reported content.
Method
EmoNews follows a fully cascaded architecture: Whisper Large handles ASR, LLaMA 3.2 1B with retrieval-augmented generation (RAG) and dynamic in-context prompting handles language understanding and response generation, a fine-tuned distilled RoBERTa model acts as the sentiment analyser, and PromptTTS (Guo et al., 2022) fine-tuned on the ESD dataset provides emotional speech synthesis. The sentiment analyser maps the LLM’s text response to one of five emotion categories (neutral, happy, sad, angry, surprised) and passes the emotion tag as a conditioning signal to PromptTTS. The RAG component retrieves the most relevant news article from a Chroma-indexed Free News database using Sentence Transformers, and grounds LLM replies via the retrieved snippet.
PromptTTS was pre-trained on LJSpeech and fine-tuned on 17,500 English utterances from the ESD dataset. A preliminary quality check comparing UTMOS, DNSMOS, PLCMOS, and WER confirmed that the fine-tuned PromptTTS was comparable to FastSpeech and VITS, with qualitative evaluation confirming distinct emotional variation across categories. The baseline system used the same ASR and RAG-LLM modules but replaced PromptTTS and the sentiment analyser with a VITS model pre-trained on LJSpeech in a neutral configuration.
Key Results
Subjective evaluation was conducted using a seven-item Likert-scale questionnaire (1—5) covering RAG relevance, task achievement, speech emotion appropriateness, and user engagement, collected across 10 dialogues per system (n = 20 conversation samples total). Mann-Whitney U tests and Cohen’s d effect sizes were reported due to the small sample size.
The proposed system significantly outperformed the baseline on speech emotion appropriateness (4.10 vs. 1.70; d = 3.07, p < 0.001), confirming that sentiment-guided PromptTTS reliably controls emotion according to news context. Engagement scores and conversation turn counts both showed large effect sizes favouring the proposed system (d = 0.824 and d = 0.831 respectively) but did not reach statistical significance (p = 0.09 and p = 0.07). RAG performance and task achievement scores were not significantly different between systems and both plateaued around 3.0, indicating room for improvement in retrieval quality and response coherence regardless of TTS emotion.
The authors attribute the moderate (not strong) engagement gains to abrupt, discrete emotion transitions that do not account for prior conversational context.
Novelty Assessment
The contribution is primarily engineering integration: PromptTTS emotion conditioning, LLM sentiment classification, and RAG-based retrieval are all established techniques combined here into a news dialogue pipeline. The main technical novelty is the application combination rather than any new method. The evaluation questionnaire (seven Likert items adapted from empathy and engagement scales in prior work) is a modest evaluation-contribution that may be reusable for similar systems. The sample size of 20 conversations limits the reliability of the engagement and turn-count results, which only approach significance. Overall this is a proof-of-concept workshop paper demonstrating feasibility rather than advancing the state of emotional TTS or dialogue system design.
Field Significance
Low — EmoNews provides a concrete example of integrating emotion-conditioned TTS into a task-oriented dialogue pipeline, but the core finding (that emotional speech improves perceived emotion appropriateness) is unsurprising and the evaluation is small-scale. Its primary contribution to the field is the proposed evaluation questionnaire for emotional dialogue systems, which addresses a real gap in standardised assessment methodology for socio-conversational agents.
Claims
-
supports: Integrating emotion-conditioned TTS into a spoken dialogue system produces measurably more appropriate emotional speech than a neutral TTS baseline.
Evidence: Speech emotion appropriateness scores were 4.10 (proposed) vs. 1.70 (baseline) with a very large effect size (Cohen’s d = 3.07, p < 0.001) across 20 human-judged conversations. (§3.5, Table 2)
-
complicates: Sentiment-driven emotional speech control does not straightforwardly translate into significantly higher user engagement in short task-oriented interactions.
Evidence: Engagement scores and number of SDS turns showed large effect sizes favouring the proposed system (d = 0.824, d = 0.831) but failed to reach statistical significance (p = 0.09, p = 0.07) with n = 20 conversations; the authors attribute this partly to abrupt discrete emotion shifts that ignore conversational history. (§3.5, Table 2)
-
complicates: Discrete, context-free emotion label assignment produces step-change tonal shifts that may undermine conversational naturalness even when individual utterance emotion is appropriate.
Evidence: Mean engagement remained around 3.0 (on a 5-point scale), and the authors explicitly note that abrupt emotion transitions without considering prior context may have suppressed engagement gains. (§3.5)
-
supports: Subjective evaluation of emotional spoken dialogue systems requires task-specific metrics beyond standard TTS naturalness; generic MOS-type ratings do not capture emotion appropriateness or conversational engagement.
Evidence: The paper proposes a seven-item questionnaire covering RAG quality, task achievement, speech emotion appropriateness, and engagement, arguing that existing evaluation frameworks lack appropriate metrics for social goals in emotional SDSs. (§3.3)
Limitations and Open Questions
Warning
The evaluation uses only 20 conversation samples (10 per system) conducted by the authors themselves, which is underpowered for the engagement and turn-count analyses and does not include independent users. Significance could not be established for engagement despite large effect sizes, and the results may not generalise beyond news summarisation.
The system uses discrete, context-free emotion tagging: sentiment is predicted independently for each LLM response without tracking emotional trajectory across turns. This produces abrupt shifts and likely suppresses engagement. Future work would benefit from emotion-aware dialogue state tracking and smoother emotion interpolation across conversational turns. Task achievement scores plateaued around 3.0 for both systems, suggesting that retrieval quality and response coherence are the binding constraints for this application, not TTS expressiveness.
Wiki Connections
- Emotion Synthesis — EmoNews applies emotion-conditioned TTS within a cascaded dialogue system, demonstrating how emotional speech synthesis can be adapted to task-oriented contexts.
- Prosody Control — PromptTTS conditions on discrete emotion labels to modulate prosodic features including cadence, intensity, and pitch in the synthesised responses.
- Evaluation Metrics — the paper proposes a seven-item Likert-scale questionnaire for evaluating emotional spoken dialogue systems, targeting emotion appropriateness and engagement dimensions absent from standard TTS metrics.
- Subjective Evaluation — human judges score all system comparisons; the paper adapts existing empathy and engagement scales from dialogue systems research to the emotional SDS setting.