arXiv · 2026 · Preprint
Dindar et al. (Columbia University) · → Paper · Demo: ✓ · Code: ✓
LoRA-fine-tunes an end-to-end spoken dialogue model so that an optional continuous valence-arousal coordinate in the system prompt steers both the content and the vocal delivery of its spoken reply, giving external emotion sensors (face, EEG, ECG, text) a single shared interface to the dialogue model.
Problem
Empathetic spoken dialogue systems such as OpenS2S and OSUM-EChat mostly infer the user’s affect from the spoken query itself. In everyday interaction that speech is often neutral, subtle, or ambiguous, so a model that relies only on the acoustic and lexical cues of the query can produce responses that are semantically fine but emotionally misaligned. Existing systems also tend to represent emotion as a small closed set of discrete labels, which cannot express graded or mixed affect, and they offer no principled way to bring in affect evidence from other channels (facial expression, physiological signals, self-report). The paper also notes a data gap: empathetic dialogue corpora are largely text-only, and the existing synthetic spoken corpora do not pair the same query with multiple emotion-conditioned responses.
Method
The speech-to-speech backbone is GLM-4-Voice: a WhisperVQ tokenizer (a vector-quantized bottleneck inside a Whisper encoder, see Whisper) turns user audio into single-codebook discrete tokens at 12.5 Hz; GLM-4-Voice-9B autoregressively generates response speech tokens; and a streaming flow-matching decoder renders them to waveform. Sympatheia changes only the language model: affect enters as text in the system prompt (“User emotion (valence=v, arousal=a)”, with v, a in [-1, 1]), and GLM-4-Voice-9B is fine-tuned with LoRA (rank 32, alpha 32, on the fused QKV, attention output, and both feed-forward projections) under a standard autoregressive negative log-likelihood. Training runs 5 epochs on 4 L40 GPUs (about 8 hours); the tokenizer and speech decoder are not retrained.
Twelve named emotions (happy, sad, angry, excited, frustrated, anxious, relaxed, surprised, disgusted, tired, content, neutral) serve as fixed anchor coordinates in the valence-arousal (VA) plane; the authors describe the exact values as heuristic design choices, not universal psychological locations (Appendix B.3, Table 4). The VA condition is dropped for one third of non-neutral training samples so the model also learns to infer affect from speech when no cue is given.
Training data is Sympatheia-18k, a fully synthetic corpus. Qwen3-32B writes query-response pairs under emotion-specific empathetic response strategies (for example, gentle reassurance for sad users rather than emotion mirroring), and Qwen3-TTS voices them with emotion-specific style instructions. An Emotional split (about 12k examples, about 1k per emotion) pairs affect-rich queries with appropriate responses; a Neutral split takes 500 neutral queries and pairs each with 12 responses, one per target emotion (6k examples), so the response style can only be learned from the VA condition. Splits are made at the unique-query level (70/30), with near-duplicate queries removed by sentence-embedding similarity.
External sensing modules (an HSEmotion EfficientNet face classifier, the MAET transformer for EEG and eye tracking on SEED-VII, 1D ResNets for ECG and GSR on YAAD, and a DistilRoBERTa text classifier on ISEAR) output softmax distributions over their own label sets; an adaptation layer maps these to the Sympatheia anchors and takes the probability-weighted mean anchor coordinate as the VA input. This preserves uncertainty and lets estimates fall between anchors.
Key Results
Empathy is scored mainly by a Qwen3-Omni-30B audio judge (1-5) and by a human Emotion MOS study (20 Prolific raters, 840 ratings in total, neutral-query setting). Against six spoken dialogue baselines (Table 1), Sympatheia has the highest score in every higher-is-better column: Sympatheia-Neutral 4.37 (next best Kimi-Audio 3.64), Sympatheia-Emotional 4.74 (next best Qwen3-Omni 4.69, a narrow margin), VoiceBench CommonEval real-speech queries 4.22 (Kimi-Audio 3.75), and Emotion MOS 3.86 (Qwen3-Omni 3.32). Standard deviations are large relative to some gaps (Emotion MOS 3.86 ± 1.19 vs. 3.32 ± 1.35; Table 13). Sympatheia also has the lowest cross-emotion response similarity for the same query (BERTScore 0.801, ROUGE-L 0.223, against 0.835 to 0.919 and 0.381 to 0.650 for baselines), meaning its replies change most when only the target emotion changes. The baselines are not fine-tuned on Sympatheia-18k and, in the Neutral and VoiceBench settings, receive the target emotion as a discrete label in their system prompt (“The user is angry.”) rather than VA coordinates.
The VA prompt drives most of the gain on neutral speech (Table 11): without the cue, the fine-tuned model scores 1.88 on Sympatheia-Neutral and 1.69 on VoiceBench, close to the base GLM-4-Voice (1.76 and 1.51). On the expressive Emotional split, however, the fine-tuned model without a cue (4.74) is well above the base model (3.80), and adding the VA cue raises it only to 4.85. A cascaded baseline (Qwen3-Omni writing text plus a prosody descriptor, voiced by Qwen3-TTS) reaches 3.13 / 4.52 / 2.67 on the three settings, above direct Qwen3-Omni on Neutral and VoiceBench but below Sympatheia on all three (Table 12).
For prosody (Table 2), Spearman correlations between target VA and acoustic features of the generated speech are generally stronger for Sympatheia, especially for arousal (F0 mean 0.40, F0 standard deviation 0.46, speaking rate 0.29, spectral centroid 0.28; the best baseline arousal correlation on F0 mean is OpenS2S at 0.18). Sympatheia does not lead every cell: on valence, speaking rate correlates at only 0.01 (Qwen3-Omni 0.04, OSUM-EChat -0.23 in magnitude), spectral centroid at 0.08 (Qwen2.5-Omni 0.17), and on arousal energy standard deviation at 0.06 (OpenS2S 0.09, Qwen2.5-Omni 0.08).
Every external sensing module raises the judge score over the no-cue condition on neutral queries (Table 3): offline face 3.64 vs. 1.92, live face recording 3.39 vs. 1.98, text self-description 3.57 vs. 1.63, EEG 3.14 vs. 1.75, eye tracking 3.05 vs. 1.75, ECG 2.76 vs. 1.84, GSR 2.74 vs. 1.84. Gains track the standalone accuracy of the sensors (text 64.7%, face 62.9%, physiological channels 41.7% to 48.4%; Table 19). Fine-tuning does not degrade general spoken QA relative to base GLM-4-Voice (UTMOS 4.18 vs. 4.02, BERTScore F1 0.627 vs. 0.569, ROUGE-L 0.228 vs. 0.223, text-speech consistency WER 5.42% vs. 5.73%; 100 samples). Adding Gaussian noise to the VA input lowers the score gradually from 4.32 (sigma 0) to 3.30 (sigma 0.5). Per-emotion Emotion MOS is lowest for neutral (2.71) and surprised (3.14) targets (Table 10).
Novelty Assessment
The contribution is primarily engineering integration plus a dataset. The model is an unchanged GLM-4-Voice pipeline with LoRA adapters on the LM and affect injected as plain text in the system prompt; there is no new conditioning module, loss, or decoder. The genuinely useful design ideas are (1) a contrastive Neutral split that holds the query fixed while varying the target emotion, which isolates explicit affect control as a learnable behavior, (2) treating VA as a sensor-agnostic interface so heterogeneous emotion recognizers can be plugged in without retraining, and (3) the probability-weighted anchor mapping that turns classifier uncertainty into graded affect intensity. The comparison is favorable to Sympatheia by construction in the two Sympatheia-18k settings, since only Sympatheia is trained on that synthetic distribution, so the VoiceBench real-speech and human MOS results carry more weight.
Field Significance
Moderate. Sympatheia shows that a modest LoRA fine-tune on contrastive emotion-conditioned synthetic dialogue is enough to make an end-to-end spoken dialogue model follow an explicit continuous affect signal in both content and prosody, and it releases the Sympatheia-18k corpus, adapter weights, and code. It is mainly an engineering step toward multimodal affect-aware voice assistants rather than an architectural advance, and its evidence rests largely on a same-family automated judge.
Claims
- supports: Explicit affect conditioning lets an end-to-end spoken dialogue model produce emotionally appropriate responses when the user’s speech carries little affective evidence, a capability that prompting general speech LMs with discrete emotion labels does not reliably deliver.
Evidence: On neutral spoken queries with the target emotion supplied, the LoRA-tuned model scores 4.37 (judge) and 3.86 Emotion MOS against at most 3.64 and 3.32 for six baselines given a discrete label in the prompt; removing the VA cue drops it to 1.88, near the 1.76 base model. (§4.1, Table 1; Appendix C.2, Table 11)
- supports: Fine-tuning a speech LM on contrastive data that pairs one query with multiple emotion-conditioned spoken responses makes its output prosody track the target affect, particularly along the arousal dimension.
Evidence: Spearman correlations between target arousal and F0 mean, F0 standard deviation, and F0 range of generated speech reach 0.40, 0.46, and 0.45, against at most 0.18, 0.14, and 0.16 for baselines; valence correlations with speaking rate and spectral centroid remain weak (0.01, 0.08). (§4.2, Table 2)
- supports: A continuous valence-arousal interface allows emotion estimates from heterogeneous non-speech sensors to steer spoken response generation without retraining the dialogue model, with gains that scale with sensor reliability.
Evidence: Face, text, EEG, eye-tracking, ECG, and GSR modules all raise the judge score over the no-cue condition (3.64 vs. 1.92 for offline face, 2.74 vs. 1.84 for GSR), and the ordering follows standalone classifier accuracy (64.7% text, 62.9% face, 41.7% to 48.4% physiological). (§4.3, Table 3; Appendix C.7, Table 19)
- refines: Emotion-targeted LoRA fine-tuning of a spoken dialogue model need not degrade its general conversational quality or speech naturalness.
Evidence: On 100 neutral VoiceAssistant-400K QA queries without emotion conditioning, the fine-tuned model matches or slightly exceeds base GLM-4-Voice (UTMOS 4.18 vs. 4.02, BERTScore F1 0.627 vs. 0.569, WER 5.42% vs. 5.73%); the sample is small and no significance test is reported. (§4.4, Table 3)
- complicates: Empathy evaluations of spoken dialogue models that rely on an audio LLM judge from the same model family as the data-generation pipeline may be biased toward that family’s style.
Evidence: Training audio is synthesized by Qwen3-TTS from Qwen3-32B text and the main metric is a Qwen3-Omni judge; the authors flag model-family bias as a risk, and only one human study (20 raters, neutral queries) cross-checks the automated scores. (§Limitations; Appendix C.4, Table 13)
Limitations and Open Questions
Warning
Training data and two of the three automated test settings are synthetic (Qwen3-32B text voiced by Qwen3-TTS), and the main metric is a Qwen3-Omni judge from the same model family. Only Sympatheia is trained on this distribution, and baselines receive a discrete label rather than VA coordinates, so the in-domain margins overstate what a fair comparison would show.
- The VA anchor coordinates are heuristic; the authors note there is no universal mapping from emotion labels to valence-arousal across datasets, cultures, or individuals, and a single VA point cannot represent blended or rapidly shifting affect.
- Affect is conditioned per turn; temporal emotion tracking across a dialogue is left to future work.
- The human evaluation covers only the neutral-query setting with 20 raters and six queries each; perceived empathy, trust, and sincerity in spontaneous long-horizon conversation are untested.
- Sensing-integration experiments pair a sensor’s VA estimate with an unrelated neutral spoken query (except the 10-subject live study), so they measure the interface rather than real co-occurring multimodal behavior; without-cue baselines differ per module because queries are sampled independently.
- The system inherits GLM-4-Voice’s tokenizer and decoder; neither speaker identity nor latency is evaluated, and the approach is demonstrated only on one backbone.
- Privacy, consent, and manipulation risks of affect sensing are discussed (Appendix A) but not technically addressed.
Wiki Connections
- Speech-to-Speech — Sympatheia is an end-to-end spoken dialogue system whose own spoken replies are evaluated for empathy and emotional delivery.
- Spoken Language Model — adapts the GLM-4-Voice-9B speech LM, which consumes user speech tokens, with LoRA to accept an explicit affect condition.
- Emotion Synthesis — the model’s spoken output is steered by continuous valence-arousal targets and evaluated with Emotion MOS and affect-prosody correlations.
- Subjective Evaluation — reports a 20-rater Prolific Emotion MOS study (840 ratings) alongside the automated audio-LLM judge.
- GLM-4-Voice — the backbone Sympatheia fine-tunes (WhisperVQ tokenizer, 9B LM, flow-matching decoder) and its main ablation reference.
- Qwen3-Omni — the strongest baseline on the Emotional split and Emotion MOS, and also the 30B audio judge used for empathy scoring.
- Kimi-Audio — the strongest baseline on Sympatheia-Neutral and VoiceBench CommonEval.
- OpenS2S — a prior empathetic speech LM evaluated as a baseline; it learns affect only from the user’s speech.
- OSUM-EChat — a prior empathetic spoken chatbot evaluated as a baseline.
- Qwen2.5-Omni — a general omni-modal baseline with the highest cross-emotion response similarity in the comparison.
- Qwen3-TTS — synthesizes all Sympatheia-18k query and response audio with emotion-specific style instructions, and voices the cascaded baseline.
- VoiceBench — its CommonEval split supplies the real-speech neutral queries for out-of-domain evaluation.