arXiv · 2025 · Preprint
Liu et al. (UC Berkeley) · → Paper · Demo: ✓ · Code: ?
EMO-Reasoning introduces a benchmark and metric suite for assessing emotional coherence in spoken dialogue systems, revealing that all evaluated systems fall significantly short of human-level affective reasoning, especially across multi-turn exchanges.
Problem
Existing spoken dialogue benchmarks focus on linguistic content — question answering, instruction following, turn-taking — but do not systematically evaluate whether a system detects, interprets, and appropriately responds to a user’s emotional state. Evaluation of emotional reasoning has been fragmented: some work examines emotion recognition from speech, other work examines conversational style, but no unified framework captures whether a system can track affective state over a multi-turn conversation and produce contextually appropriate emotional responses. The scarcity of emotionally annotated conversational speech further limits data-driven progress.
Method
EMO-Reasoning evaluates emotional reasoning through three complementary lenses: continuous emotion metrics, categorical emotion metrics, and human perceptual ratings.
For continuous evaluation, the framework relies on a Continuous Speech Emotion Recognition (CSER) model that produces frame-level predictions of valence, arousal, and dominance (VAD). The CSER model is a BiLSTM trained on the MSP-Conversation dataset (224 conversations, 59.6 hours) using concordance correlation coefficient (CCC) loss, with WavLM-Large as the acoustic feature extractor. From the CSER output, four scores are derived: Emotional Contagion Score (ECS, measuring whether the system mirrors positive user affect via dynamic time warping), Emotional Balancing Score (EBS, assessing moderation of extreme negative states), Emotional Stability Score (ESS, penalising abrupt emotional swings), and Emotion Reasoning Score (ERS, their average). Cross-turn extensions (CT-ECS, CT-EBS, CT-ESS, CT-ERS) aggregate these across dialogue turns to evaluate multi-turn coherence.
For categorical evaluation, four high-reliability emotion classes (neutral, happy, angry, sad) are scored using a human-rated transition matrix — a 4x4 table of rationality scores collected from 20 annotators that encodes contextual appropriateness of emotion pairs (e.g., mirroring happiness scores 1.0; responding to happiness with sadness scores 0.2). SenseVoice provides automatic categorical labels for inference.
Human perceptual ratings cover Emotional Rationality (ER), Emotional Naturalness (EN), and Response Relevance (RR), scored on a 1–5 scale by 20 annotators (10 native, 10 non-native English speakers) across 20 sampled dialogues per system.
The benchmark dataset is synthesised using GPT-4 to generate dialogue scripts and emotional descriptors, then rendered via CosyVoice (see 2407.05407) into high-quality emotional speech. This covers 500 utterances each of neutral, happy, angry, and sad, plus 1,251 natural language description prompts, totalling approximately 7.4 hours. Real human speech from DailyTalk (2,541 dialogues, 21.7 hours) serves as an upper-bound reference. Seven spoken dialogue systems are evaluated: LLaMA-Omni, mini-omni, mini-omni2, Freeze-Omni, GLM-4-Voice, Moshi, and dGSLM, spanning LLM-centric streaming, codec-based autoregressive, and purely speech-driven architectures.
Key Results
Human dialogues from DailyTalk consistently outperform all evaluated SDMs across all three metric types (continuous ERS: 0.884; categorical Avg: 0.844; perceptual ERS: 0.928 in single-turn settings), establishing a clear ceiling that no system approaches (Table IV, Table V).
Among evaluated SDMs, Freeze-Omni achieves the strongest categorical-based performance (Avg: 0.738), while GLM-4-Voice leads on continuous-based scores (ERS: 0.803) in single-turn settings — the two do not co-rank, suggesting complementary failure modes. LLaMA-Omni and mini-omni cluster near the bottom on perceptual ERS (0.584 and 0.560 respectively), indicating that strong continuous scores do not guarantee human-perceived emotional quality.
Multi-turn evaluation (Table V) reveals a pronounced degradation for Moshi and dGSLM: Moshi’s perceptual ERS drops from 0.284 to 0.271, and dGSLM’s from 0.296 to 0.291, while their categorical averages fall to 0.470 and 0.461 respectively — well below the human baseline of 0.925. The multi-turn gap is larger than any difference between single-turn systems, underscoring that emotional consistency over extended dialogue is a harder problem than per-utterance emotional appropriateness.
Automatic and human metrics correlate moderately but diverge on systems with less conventional output styles: Moshi and dGSLM are overestimated by automatic metrics relative to human ratings, signalling that current automatic emotion metrics do not fully capture human perception of extended emotional coherence (§V, Figures 3 and 4).
Note
Comparisons between SDMs and the human reference are not on exactly the same dialogues — the human data comes from DailyTalk while system outputs are elicited via synthetic prompts in some conditions. Direct score comparisons should account for this distributional difference.
Novelty Assessment
The paper’s core contribution is the benchmark and metric suite, not any architectural or modelling advance. The individual metric components (DTW-based alignment, CCC training, categorical rationality scoring) are standard signal-processing and machine learning tools assembled into a principled evaluation protocol. The novelty lies in combining continuous VAD trajectories, discrete categorical rationality, and human perceptual ratings within a single unified scoring framework, and in introducing cross-turn aggregations (CT-ECS, CT-EBS, CT-ESS, CT-ERS) that explicitly measure affective consistency across dialogue turns. The use of a TTS-generated synthetic dataset to overcome emotional speech scarcity is practical and replicable. The benchmark reveals a consistent and significant gap between human and machine emotional coherence that prior work had not quantified systematically, which is a concrete empirical contribution.
Field Significance
Moderate — EMO-Reasoning provides the first unified benchmark for emotional reasoning in spoken dialogue systems covering continuous, categorical, and perceptual dimensions simultaneously. Its primary value is in exposing a consistent and undercharacterised failure mode across the current generation of SDMs: while systems handle linguistic content adequately, they do not sustain emotionally coherent trajectories over multi-turn exchanges. This evidence is relevant to the design of next-generation dialogue systems aiming for affective alignment.
Claims
- Current spoken dialogue systems reliably fail to sustain emotionally coherent responses across multi-turn conversations, with degradation compounding across turns. (§V, Table V)
- Automatic continuous and categorical emotion metrics correlate moderately with human perceptual judgements on most SDMs, but overestimate emotional coherence on systems with atypical output characteristics. (§V, Figure 3, Figure 4)
- Optimising for categorical emotion appropriateness and optimising for continuous affect trajectory alignment represent distinct objectives that current SDMs trade off against each other rather than jointly satisfying. (§V, Table IV)
- Synthetic emotional speech generated by controllable TTS systems provides sufficient variety and quality to serve as evaluation stimulus material when real annotated data is scarce. (§IV-A, Table III)
Limitations and Open Questions
Warning
Human perceptual evaluation covers only 20 randomly sampled dialogues per system evaluated by 20 annotators; at this sample size, inter-rater agreement statistics are not reported, limiting confidence in the perceptual ERS rankings. The authors acknowledge this and commit to releasing agreement analyses upon acceptance.
The benchmark currently covers English only, constraining applicability to multilingual SDMs. The categorical metric relies on four high-reliability emotion classes (neutral, happy, angry, sad); more nuanced affective states (contempt, fear, surprise) are captured only indirectly through continuous VAD, leaving a gap between the benchmark’s coverage and the full affective spectrum. Metric thresholds are derived from MSP-Conversation statistics; their transferability to other languages or domains is untested, though the authors show robustness to small percentile shifts. The CSER model underperforms the baseline on arousal (CCC 0.554 vs. 0.594) while outperforming on valence and dominance; whether the arousal shortfall materially affects benchmark rankings is not examined.
Wiki Connections
This paper evaluates spoken dialogue systems including 2503.04721 (Full-Duplex-Bench, another SDM benchmark from overlapping authors) and uses 2407.05407 (CosyVoice) for synthetic data generation.
Relevant concepts: spoken-language-model, emotion-synthesis, evaluation-metrics, subjective-evaluation, speech-to-speech.