arXiv · 2026 · Preprint
Zhuoyue Gao et al. (Northeastern University, China) · → Paper · Demo: ? · Code: ?
A three-stage empathetic spoken dialogue framework that explicitly models affective context from speech before generic speech encoding, rather than relying on emotion labels or letting the encoder learn affect implicitly, then uses that context to drive both response generation and prosody-adapted speech synthesis.
Problem
Empathetic spoken dialogue requires perceiving paralinguistic cues (prosody, tone, emotional intensity), not just linguistic content. Existing speech-to-speech LLMs handle this poorly in two ways: cascaded ASR-then-LLM pipelines discard paralinguistic information during transcription, while end-to-end systems that use general-purpose speech encoders (Whisper, HuBERT) compress and abstract speech early, before any affect-specific processing, which the paper argues weakens fine-grained emotional information. Systems that do target empathetic speech typically rely on explicit emotion annotations or discrete emotion-category supervision, which is costly to collect and cannot capture continuous, subtle emotional variation. Multi-turn dialogue compounds this: existing SLLM architectures (e.g., SALMONN’s independent per-window segmentation, OpenS2S’s sequential embedding concatenation) lack explicit mechanisms for tracking how affect evolves across turns.
Method
ES4R is a three-stage framework. In the empathetic understanding stage, each dialogue turn’s speech is first passed through a Whisper feature extractor and downsampler, then through a dual-level attention mechanism applied before the generic speech encoder: intra-turn attention (multi-head self-attention within a turn) captures turn-level affective state, and the resulting per-turn representations are concatenated in chronological order and passed through inter-turn attention to capture how affect evolves across the dialogue history. Only after this prepositive affective modeling are the enhanced representations fed into a generic speech encoder and a modality adapter that maps them into the LLM’s embedding space.
In the empathetic generation stage, cross-modal attention uses the speech-side affective representation as the query against text-history keys/values, letting speech affect determine which historical semantic content the model attends to. Training uses a two-path strategy: a text-only path and a speech-fused path (with PLoRA applied only to the fused-speech branch to avoid disrupting the LLM’s original text capabilities), optimized jointly with a cross-entropy loss on the speech path and a KL-divergence distillation loss that uses the text path’s output distribution as a teacher for the speech path.
In the speech synthesis stage, ES4R extracts an energy trajectory (L2 norm of the mel-spectrogram, averaged per turn) across the dialogue history as a lightweight, interpretable proxy for affective intensity, without explicit emotion classification. The trend of this trajectory (decreasing, increasing, or flat) selects one of three empathetic strategies (comfort, encourage, neutral), each mapping to a fixed pair of StyleTTS2 prosody-control parameters (α for phoneme duration/speaking rate, β for diffusion-sampling expressiveness). An additional empathy-memory weighting mechanism fuses per-utterance StyleTTS2 style vectors using inverse-energy weighting, so that lower-energy (more subdued/negative) turns contribute more to the final synthesis style, preventing subtle negative affect from being averaged out by louder or more neutral turns.
Key Results
On AvaMERG (fine-tuned), ES4R with a Qwen3-8B backbone achieves the best or near-best scores across BLEU-1/2/3/4, BERTScore, and ROUGE-1/2/L among all compared baselines (ASR+LLM, Whisper+LLM, BLSP-Emo, SALMONN, OpenS2S, LLaMA-Omni 2, Qwen2.5-Omni-7B), and results hold with a Llama-3.1-8B-Instruct backbone. In human A/B evaluation (100 sampled conversations, 3 annotators), ES4R wins across all five text-response dimensions (Topic, Emotion, Specific, Action, Empathy) against both OpenS2S (56-61% win rate) and LLaMA-Omni 2 (67-70% win rate). On speech-response DMOS evaluation, ES4R shows strong empathy-expressiveness and emotional-consistency win rates (e.g., 68.4%/61.2% vs. OpenS2S on DMOS-E/DMOS-C) though streaming baselines retain an edge on raw fluency/latency-related quality perception. On zero-shot transfer to MELD (a structurally different multi-party dataset, no fine-tuning), ES4R outperforms OpenS2S and LLaMA-Omni 2 on most metrics (ROUGE-L 0.099, BERTScore 0.841, Empathy 7.93 vs. 6.59 and 5.42).
Ablations show the dual-level (prepositive) attention module is the largest single contributor: removing it collapses BLEU-4 from 0.0758 to 0.0078 and drops the human-evaluated Empathy win rate to 76.3% (vs. the full model). Removing the cross-modal fusion module produces a smaller degradation. Removing the energy-based empathetic TTS module drops DMOS-E win rate to 76.2% and DMOS-C to 67.3%, confirming the synthesis-side contribution is separable from and additive to the generation-side gains.
Novelty Assessment
The core architectural idea, applying affect-specific attention before the generic speech encoder rather than relying on the encoder or a post-hoc fusion step to preserve affective cues, is a genuine, testable design choice, and the ablations isolate its contribution cleanly (the largest degradation of any component removed). The energy-trajectory-as-affect-proxy synthesis mechanism is a lightweight, interpretable alternative to explicit emotion classification, though it maps to only three discrete speaking strategies via fixed hyperparameter presets rather than continuous control. The overall system integrates existing components (Whisper encoder, an off-the-shelf LLM via PLoRA, StyleTTS2 for synthesis) into a new pipeline rather than introducing new generative or encoding mechanisms at the component level; the contribution is the specific ordering and structuring of affective modeling relative to those components, validated with a notably broad evaluation protocol (automatic, human A/B, LLM-judge, cross-dataset zero-shot, and per-component ablation).
Field Significance
Field Significance
high — This paper isolates a specific architectural choice, where to place affective context modeling relative to speech encoding, and demonstrates with clean ablations that doing it before encoding matters substantially for empathetic response quality, while also showing the resulting affective representations transfer zero-shot to a structurally different dialogue dataset.
Claims
- supports: Modeling structured affective context via dedicated attention before generic speech encoding preserves more emotion-relevant information for downstream empathetic response generation than encoding speech first and fusing affect afterward.
Evidence: Removing the dual-level (intra-turn/inter-turn) prepositive attention module collapses BLEU-4 from 0.0758 to 0.0078 and reduces the human-evaluated Empathy win rate to 76.3% relative to the full model (§4.6.1, Table 1, Table 2).
- supports: A lightweight, interpretable proxy for affective intensity derived from acoustic energy trajectories, rather than explicit emotion classification or supervision, can drive effective empathetic prosody adaptation in speech synthesis.
Evidence: Removing the energy-based strategy-selection and style-fusion synthesis module drops empathy-expressiveness (DMOS-E) win rate to 76.2% and emotional-consistency (DMOS-C) win rate to 67.3% relative to the full ES4R system (§4.6.1, Table 3).
- complicates: End-to-end SLLM systems fine-tuned to maximize semantic and affective accuracy on a target dataset can exhibit lower lexical response diversity than cascaded ASR+LLM pipelines evaluated on the same task.
Evidence: ES4R scores lower on Dist-1/Dist-2 diversity metrics than several baselines despite outperforming them on BLEU/ROUGE/BERTScore and empathy dimensions, attributed to ASR-error-induced output randomness in cascaded systems and to some baselines’ additional external pretraining data (§4.5.1, Table 1).
- supports: Affective context representations learned via prepositive modeling on one empathetic dialogue dataset can transfer zero-shot to a structurally different dialogue dataset without fine-tuning.
Evidence: Zero-shot transfer from AvaMERG (counseling dialogues) to MELD (multi-party TV-show dialogue) outperforms OpenS2S and LLaMA-Omni 2 on ROUGE-L, BERTScore, and LLM-judged Empathy despite the domain shift (§4.6, Table 5).
Limitations and Open Questions
All human-evaluation results in the paper (text A/B comparisons and speech DMOS scores, spanning all reported win/tie/loss percentages) come from only three annotators, a small and non-diverse rater pool for claims about empathy and emotional-consistency perception across the full evaluation.
The empathetic speech synthesis stage maps affect to only three fixed speaking strategies (comfort, encourage, neutral) via preset hyperparameter pairs, rather than continuous, fine-grained style control; the authors identify extending this as future work. The framework does not model additional paralinguistic cues (e.g., voice quality) or speaker-aware personalization. No end-to-end streaming or low-latency inference is implemented, limiting applicability to real-time interactive settings; the authors note this as a direction for future work. Speech-generation quality itself is assessed only via human evaluation (DMOS), with no objective acoustic quality metric reported, since the paper judges standard objective TTS metrics inapplicable when responses across systems are not lexically matched.
Wiki Connections
- Spoken Language Models — proposes a three-stage SLLM framework with prepositive affective attention and PLoRA-based speech/text dual-path training, evaluated against several other end-to-end SLLM baselines.
- Emotional and Expressive Speech Synthesis — drives empathetic prosody adaptation from an energy-trajectory-based affect proxy rather than explicit emotion classification, selecting among discrete empathetic speaking strategies.
- Prosody Control — adjusts StyleTTS2’s phoneme-duration and diffusion-expressiveness parameters based on dialogue-history energy trends, and fuses per-utterance style vectors via inverse-energy weighting to avoid averaging out subtle negative affect.
- OpenS2S — a compared open-source end-to-end empathetic speech-to-speech baseline, outperformed by ES4R on text-response dimensions (56-61% win rate) and speech DMOS-E/DMOS-C.
- LLaMA-Omni 2 — a compared low-latency streaming end-to-end spoken chatbot baseline, outperformed by ES4R on text-response dimensions (67-70% win rate) though it retains an edge on speech fluency/quality perception.
- Qwen2.5-Omni — a compared fully multimodal understanding-and-generation baseline in the automatic and LLM-based evaluation tables.
- SALMONN — a compared multimodal LLM baseline using dual-encoder and window-level Q-Former speech-text alignment, cited as an example of the shallow per-window fusion strategy ES4R’s inter-turn attention is designed to improve on.