arXiv · 2026 · Preprint
Yuhang Jia et al. (Nankai University / Meituan LongCat Interaction Team) · → Paper · Demo: ? · Code: ?
Introduces EmpathyEval, a descriptive natural-language automatic evaluator of spoken-dialogue empathy, and ReEmpathy, an end-to-end spoken language model that interleaves spoken response generation with free-form empathetic reflective reasoning at inference time via a novel alternating-inference mechanism, rather than relying on rigid supervised targets or scalar preference scores.
Problem
End-to-end spoken language models (SLMs) can directly perceive paralinguistic cues (emotion, tone, speaker state) that cascaded ASR-LLM-TTS pipelines discard, making them well-suited for empathetic dialogue. But current approaches to improving SLM empathy rely on rigid supervision: a single ground-truth response in supervised fine-tuning, or a scalar preference score in reinforcement learning. Both are fundamentally limited for modeling empathy, since there is no single “correct” empathetic response and a numerical score cannot capture the nuance of appropriate emotional expression. The paper asks whether a richer, descriptive feedback signal, and a mechanism for the model to reason about that feedback during generation rather than only during training, can better capture and improve empathetic behavior.
Method
The work has two parts. First, EmpathyEval is a descriptive, natural-language automatic evaluator for spoken-dialogue empathy, built by adapting Qwen3-Omni-30B-A3B-Captioner. To construct its training data, the authors build an automatic pipeline (GPT-4 + the zero-shot TTS system CosyVoice2 + the emotion-captioned EmotionTalk seed-audio dataset) that generates a short grounding story for a social scenario, a user query expressing a specific emotional state, three system responses of differing attitude (Positive/empathetic, Neutral/objective, Negative), a GPT-4-generated descriptive empathy assessment scoring four dimensions (Need Support, Wording Appropriateness, Emotion Understanding, Emotional Support), and finally zero-shot-synthesized speech for both query and response. This yields 18,000 annotated Mandarin spoken dialogues. EmpathyEval is then trained in stages: first adapting Qwen3-Omni to the emotion domain via speech emotion recognition and emotion-captioning fine-tuning on EmotionTalk, then supervised fine-tuning on the constructed empathy dataset to jointly assess linguistic content and paralinguistic cues and produce a free-form descriptive assessment; four regression heads are additionally attached to predict continuous scores along the four empathy dimensions.
Second, ReEmpathy is an end-to-end SLM (fine-tuned from GLM-4-Voice) that incorporates an Empathetic Self-Reflective Alternating Inference mechanism. Rather than generating a response and then separately evaluating or revising it, ReEmpathy generates output as a sequence of alternating fixed-length chunks: response chunks (containing spoken tokens and their text transcripts) and reflection chunks (free-form, unspoken reasoning tokens assessing empathetic quality). The response and reflection streams are jointly conditioned on the evolving dialogue context, so reflective reasoning can influence subsequent responses and newly generated responses continuously reshape the reflective process, exploiting SLMs’ ability to decode extra unspoken tokens alongside speech tokens without added inference latency. Training data for this alternating pattern is constructed off-policy: EmpathyEval scores a large empathetic dialogue dataset, and the resulting descriptive assessments are converted into reflection text tokens interleaved with the corresponding response audio and transcript tokens, which are used to fine-tune GLM-4-Voice on the full interleaved sequence. A further self-consistency property emerges from this design: the model’s own generated reflection sequences closely match assessments obtained by directly re-running EmpathyEval on the model’s output, without EmpathyEval being called at inference time.
Key Results
EmpathyEval shows strong agreement with both a GPT-4-based text pipeline and human raters: on its own in-domain test set it reaches an average Linear Correlation Coefficient (LCC) of 0.86 against the GPT-4 pipeline (MSE 0.12 on a 1-5 scale), and on a 300-sample human-annotated gold-standard set (150 in-domain, 150 out-of-domain from OpenS2S) it maintains LCC 0.71 against human MOS ratings, notably retaining substantial agreement out-of-domain. For ReEmpathy, comparing against GLM-4-Voice baselines optimized with supervised fine-tuning (SFT), Direct Preference Optimization (DPO, using EmpathyEval-scored preference pairs), and a single-step Chain-of-Thought-before-Speaking (CoTBS) variant, ReEmpathy achieves the best score on every EmpathyEval dimension (NS, WA, EU, ES) and on GPT-4-judge AB-testing and MOS-style ratings, reaching a GPT-4 MOS of 3.818, matching the human ground-truth response’s own GPT-4 MOS (3.81). Notably, plain SFT outperforms both DPO and CoTBS despite the latter two representing more sophisticated optimization strategies; CoTBS in particular, despite explicitly generating empathy-related reasoning as a pre-response step, underperforms plain SFT, indicating that the specific alternating structure of the reflection mechanism (not merely the presence of reasoning content) is responsible for ReEmpathy’s gains. Disabling the reflective mechanism via instruction while keeping ReEmpathy’s training otherwise identical (“wo reflect”) eliminates most of the improvement over SFT, isolating the reflective-alternation mechanism itself as the source of the gain. Two further ablations probe the mechanism directly: reducing the response/reflection chunk size (increasing alternation frequency) improves empathetic quality down to a chunk size of roughly 15, below which performance degrades from excessive fragmentation; and increasing the cross-attention weight between recently generated reflection and response content improves quality up to a peak around 1.1-1.2x the default weight, beyond which quality declines.
Novelty Assessment
The core novelty is the alternating response/reflection inference structure itself: interleaving fixed-length spoken-response chunks with free-form reflective-reasoning chunks, trained via off-policy supervision from a purpose-built descriptive evaluator, is a genuinely new inference-time mechanism for empathy optimization distinct from both SFT-on-ground-truth and RL/DPO-on-scalar-preference approaches that dominate prior work. The comparison against CoTBS is a well-targeted control that isolates alternation (not merely the presence of reasoning) as the active ingredient. EmpathyEval’s descriptive, multi-dimensional evaluation approach is also a substantive contribution independent of ReEmpathy, giving future empathy-optimization work a reusable, cheaper-than-GPT-4 scoring tool. The underlying backbone (GLM-4-Voice) and evaluator backbone (Qwen3-Omni) are not modified architecturally; the contribution is in the training-data construction and the alternating-generation training/inference procedure layered on top of them.
Field Significance
high — this paper demonstrates that free-form, natural-language reflective reasoning interleaved with response generation at inference time outperforms both conventional end-to-end supervised fine-tuning and RL/DPO-style scalar preference optimization for empathetic spoken dialogue, with a well-controlled ablation (against single-step Chain-of-Thought and a reflection-disabled variant) isolating the alternating structure itself, rather than the mere presence of reasoning tokens or additional training signal, as the source of the improvement.
Claims
- supports: Interleaving spoken response generation with explicit, free-form natural-language reflective reasoning at inference time, rather than relying purely on end-to-end supervised or scalar preference-based training signals, improves an end-to-end spoken language model’s empathetic response quality.
Evidence: ReEmpathy outperforms SFT, DPO, and single-step CoTBS baselines on every EmpathyEval dimension and on GPT-4-judge AB-score/MOS, and disabling the reflective mechanism while holding training data constant eliminates most of the gain. (§4.2, Table 5)
- complicates: Generating explicit empathy-related reasoning as a single upfront step before response generation does not reliably improve, and can actively degrade, empathetic dialogue quality relative to direct supervised fine-tuning without any explicit reasoning step.
Evidence: The CoTBS baseline underperforms plain SFT on every EmpathyEval dimension and GPT-4 judge score despite explicitly generating empathy-related reasoning before the response, indicating that alternating structure, not the mere presence of reasoning, drives improvement. (§4.2, Table 5)
- supports: A descriptive, natural-language automatic evaluator of empathy, obtained by fine-tuning a general audio-language model on curated dialogue-empathy annotations, can achieve human-rating correlation comparable to a large closed-model text pipeline while requiring no online API calls at evaluation time.
Evidence: EmpathyEval reaches LCC 0.71 against 300 human-annotated gold-standard ratings spanning in-domain and out-of-domain samples, close to the GPT-4 text-pipeline baseline’s 0.86 LCC on the in-domain test set. (§4.1, Table 4)
- supports: The frequency of alternation between response-generation and reflection segments, and the relative attention weight a model places on recently generated reflection versus response content, both have measurable, non-monotonic effects on the empathetic quality of an alternating-inference dialogue system.
Evidence: Reducing chunk size (increasing alternation frequency) improves empathetic quality down to roughly chunk size 15, below which performance degrades; increasing cross-attention weight between reflection and response improves quality up to a peak around 1.1-1.2x before declining. (§4.3, Figure 6)
- complicates: Preference optimization using scalar scores from a well-calibrated automatic evaluator is insufficient to reliably improve empathetic dialogue quality, even when that same evaluator’s descriptive assessments are effective as reflective-reasoning training signal in a different mechanism.
Evidence: Direct Preference Optimization, using preference pairs scored by the same EmpathyEval model that supplies ReEmpathy’s reflective supervision, underperforms both plain SFT and ReEmpathy on every empathy-quality metric reported. (§4.2, Table 5)
Limitations and Open Questions
The authors’ own limitations note two open directions: the coupling between response and reflection streams is currently supervised only at the global dialogue level (constructed off-policy from EmpathyEval scores on existing dialogues), so individual response-reflection chunk pairs cannot yet be trained on-policy, limiting fine-grained interpretability and control; and reflective-reasoning supervision is currently limited to supervised fine-tuning, with reinforcement learning or other optimization strategies for the reflective stream left unexplored. The dataset construction, evaluation, and ReEmpathy training are all conducted in Mandarin Chinese (EmotionTalk, the self-constructed empathy dataset, and the OpenS2S Chinese subset); generalization of the approach to other languages is untested in this paper.
Wiki Connections
- Spoken Language Model — fine-tunes GLM-4-Voice, an end-to-end spoken language model, with a novel alternating response/reflection inference mechanism for empathetic dialogue.
- Emotion Synthesis — targets empathy-aware response generation conditioned on the user’s perceived emotional state and needs, evaluated along explicit emotion-understanding and emotional-support dimensions.
- Speech-to-Speech — operates end-to-end on spoken user queries and spoken system responses within a single dialogue model, without intermediate cascaded ASR/TTS stages.
- Subjective Evaluation — validates EmpathyEval’s automatic scores against a 300-sample human-annotated gold-standard set rated independently by three professional annotators in a blind setting.
- Evaluation Metrics — introduces EmpathyEval, a new descriptive, multi-dimensional automatic evaluation model for spoken-dialogue empathy, as a reusable alternative to GPT-4-pipeline or scalar-score evaluation.
- GLM-4-Voice — serves as the backbone end-to-end SLM that ReEmpathy fine-tunes with the alternating-inference empathy mechanism.
- OSUM-EChat — a directly comparable prior end-to-end empathetic spoken chatbot, cited as part of the broader trend of implicit empathy training via supervised speech-to-speech dialogue data.
- Kimi-Audio — used as a comparison baseline SLM in the main empathetic-dialogue evaluation table.
- Qwen2.5-Omni — used as a comparison baseline SLM in the main empathetic-dialogue evaluation table, and as a reference model during EmpathyEval’s adaptation stages.
- CosyVoice 2 — used as the zero-shot TTS system to synthesize query and response speech for the constructed empathy evaluation dataset.
- Qwen3-Omni — the base audio-language model (30B-A3B-Captioner variant) that EmpathyEval is built on top of via staged fine-tuning.