ACL 2026 (Findings) · 2026 · Conference
Yun Hong et al. (Institute of Computing Technology, Chinese Academy of Sciences / University of Chinese Academy of Sciences) · → Paper · → Preprint · Demo: ? · Code: ✓
Builds an English and Chinese empathetic spoken chatbot without any empathetic speech dialogue data and without updating the LLM: separate semantic and emotion features are aligned to a frozen LLM, which then writes its own pseudo-empathetic training targets, and a streaming speech decoder is trained on emotion-prompted synthetic speech tokens.
Problem
Empathetic spoken chatbots need to hear the emotional tone of the user’s speech and answer with both empathetic content and emotionally fitting speech. Prior systems in this line (OpenS2S, OSUM-EChat, and cascaded designs such as Spoken-LLM) train on purpose-built empathetic speech-to-speech instruction data, typically produced by having a strong LLM write empathetic dialogues and then voicing them with an emotion-controllable TTS model. The authors identify three costs of that approach: the data pipeline is expensive and its dialogues lack content diversity; fine-tuning the LLM on such cross-modal data risks catastrophic forgetting of general ability; and the generated speech often carries little emotional expressiveness. Their premise is that a text LLM already responds empathetically when explicitly told the speaker’s emotional tone (Figure 1), so the problem reduces to delivering speech content and emotion to a frozen LLM in a form it already understands.
Method
FreezeEmpath has three parts: a speech understanding module, a frozen base LLM (Qwen2.5-7B-Instruct), and a speech generation module.
Decoupled speech understanding. A shared Whisper-large-v3 encoder feeds two branches. A speech adapter (downsampling layer plus 2-layer FFN, as in LLaMA-Omni) maps the encoder output into the LLM embedding space as a semantic feature sequence. A new emotion extractor produces a single emotion vector in two steps: layer-level pooling, where a gating network scores each encoder layer and a softmax-weighted sum combines all layers’ hidden states; and frame-level pooling, where a learnable query attends over frames with 4-head cross-attention, followed by a 2-layer FFN (hidden size 2048) into the LLM embedding space. The LLM input is the semantic sequence, then the emotion vector wrapped between fixed text embeddings of connecting words, mimicking a text prompt of the form “[transcript] with a [emotion] tone”. The matching all-text “alignment sequence” substitutes the transcript and the emotion label’s text embeddings for the two speech-derived features.
Training, with the LLM frozen throughout (§3.2).
- Semantic alignment. Following BLSP, the adapter is trained by self-distillation: the frozen LLM’s response to the text instruction is the target when it receives the speech instruction instead. Only the adapter is trained.
- Emotional alignment. Only the emotion extractor is trained, in two steps. SER pretraining uses instruction-formatted data from 10 SER corpora (about 110k utterances, restricted to neutral, happy, sad, angry, surprised), with the LLM cross-entropy loss plus an auxiliary linear classification loss on the emotion vector (weight 0.8). Empathetic-instruction (EI) fine-tuning then uses pseudo-empathetic data built by self-instruct: each speech instruction from InstructS2S-200K (and Chinese instruction data from the authors’ CSLM work) is assigned a randomly sampled emotion label, and the frozen LLM, given the text alignment sequence and an “empathetic spoken chatbot” system prompt, writes the target response. During training the emotion feature comes from a separate SER utterance with that label, paired with the instruction’s speech. SER data are mixed in to protect SER accuracy.
- Speech generation. The empathetic text responses are grouped into predefined emotion categories (such as excitement or comfort) and converted into target speech tokens by the IndexTTS2 text-to-semantic model, prompted with emotional speech from ESD for each category. A streaming speech decoder in the LLaMA-Omni 2 style (a gate-fusion module over LLM hidden states and decoded text-token embeddings, feeding a decoder-only Transformer initialized from Qwen2.5-0.5B) emits W=15 speech tokens for every R=3 input embeddings. It is pretrained on response-to-token pairs, then trained with the frozen LLM in the loop using text inputs only, since the earlier stages already aligned speech and text inputs. Speech tokens (8,192 vocabulary, 50 Hz) are rendered by IndexTTS2’s pretrained flow-matching model and vocoder, which are not trained.
The design means no stage needs real empathetic spoken dialogue: emotional speech enters only through SER corpora (input side) and through IndexTTS2’s emotion prompting (output targets).
Key Results
Evaluation covers empathetic dialogue, SER, and spoken QA (SpokenQA) against roughly 8B-scale systems; some baseline values in Table 1 are quoted from the Fun-Audio-Chat report rather than re-run. All quality, empathy, acoustic, and VStyle scores are produced by LLM judges (GPT-4o on transcripts; Gemini-2.5-Pro on audio).
Empathetic instruction following (SpeechAlpaca, Table 1). FreezeEmpath has the highest empathy score in both speech-to-text (7.63 vs. 6.58 for OpenS2S, the next best) and speech-to-speech (7.27 vs. 6.16) evaluation, and the highest S2T quality (8.76 vs. 8.62 for Kimi-Audio). On S2S quality it is second (7.52) behind Fun-Audio-Chat-8B (7.61). Its Gemini acoustic score, which rates whether the response voice’s emotional style fits the user and whether it sounds natural, is 7.24 against 6.15 (Fun-Audio-Chat-8B), 5.78 (OpenS2S), 4.68 (Kimi-Audio), and 4.53 (Step-Audio2-Mini). Its ASR-WER between speech and text response is 5.13, slightly below Fun-Audio-Chat-8B (5.46) and well below Step-Audio2-Mini (11.46) and Kimi-Audio (14.74), which the authors link to those models’ overly long responses.
Daily empathetic conversation (VStyle-Empathy, Table 1). FreezeEmpath has the highest average (4.52 vs. 4.38 for Step-Audio2-Mini) and leads on six of eight emotion-language cells. It does not lead on Chinese anger (4.18, tied with OpenS2S, behind Step-Audio2-Mini at 4.20) and is weak on Chinese joy (3.94, behind Step-Audio2-Mini 4.69, Kimi-Audio 4.57, and OpenS2S 4.31; only Fun-Audio-Chat-8B is lower at 3.77).
Human preference (Figure 4). In pairwise tests (20 samples, 5 participants, 100 votes per comparison, holistic judgment of helpfulness, empathy, acoustic quality, and emotional prosody of the speech responses), FreezeEmpath wins 34, ties 49, and loses 17 against Step-Audio2-Mini, and wins 33, ties 47, and loses 20 against OpenS2S. Ties are the most common outcome in both comparisons.
SER (Table 3). Average accuracy across six test sets is 70.1%, above BLSP-Emo (63.3%), C2SER (57.6%), Qwen2-Audio (56.3%), and Kimi-Audio (52.9%). On IEMOCAP it ties BLSP-Emo (76.0) and on MELD it is marginally ahead (57.5 vs. 57.3); both datasets’ training splits are in both models’ training data. The larger margins are on CASIA (72.4 vs. 58.2 for C2SER), CaFE (79.3 vs. 75.3), and RESD (55.1 vs. 54.4 for Kimi-Audio). On RAVDESS, Qwen2-Audio scores 94.2 against 80.0, which the authors attribute to likely data leakage.
Spoken QA (Table 2). FreezeEmpath has the highest average S2T accuracy (57.79 vs. 57.50 for Kimi-Audio and 57.12 for Fun-Audio-Chat-8B) and leads on Llama Questions in both S2T (79.33) and S2S (74.67). It is second on TriviaQA S2T (49.71 vs. 50.49 Kimi-Audio) and on Web Questions S2T (44.34 vs. 44.59 Fun-Audio-Chat-8B), and in S2S it trails Fun-Audio-Chat-8B on TriviaQA (46.39 vs. 46.58), Web Questions (39.42 vs. 42.47), and the S2S average (53.49 vs. 53.68). It exceeds LLaMA-Omni2-7B on every column (average S2T 57.79 vs. 49.43; S2S 53.49 vs. 45.09); the authors state the two share structure and speech-to-text training data and credit freezing the LLM for the gap; this is a comparison against a separately trained published model, not an ablation run within the paper.
Ablations (Table 4) and component analysis (Tables 5 and 6). Replacing attention-based frame pooling with average pooling causes the largest drop (SER 65.1, empathy 7.12 vs. 70.1 and 7.63). Removing SER pretraining gives 66.5 / 7.22; removing the auxiliary loss gives 68.4 / 7.47; removing layer-level pooling gives 69.8 / 7.29. Removing EI fine-tuning raises SER accuracy to 71.8 but lowers empathy to 6.64, a trade-off between recognition and empathetic use of the emotion feature. Feeding the LLM ground-truth transcript plus ground-truth emotion label gives an empathy score of 8.21, versus 7.63 for the full speech-feature model and 6.03 when the emotion is a random label, so emotion features remain the weaker link. Against a cascade in which IndexTTS2 voices FreezeEmpath’s text with a neutral emotion prompt, the trained speech decoder has higher WER (5.13 vs. 3.39) but a higher acoustic score (7.24 vs. 5.91).
Novelty Assessment
The contribution is primarily a training recipe assembled from known parts. Frozen-LLM speech chatbots exist (Freeze-Omni), behaviour-alignment self-distillation comes from BLSP, emotion-aware alignment of a speech LLM was explored in BLSP-Emo, the streaming decoder follows LLaMA-Omni 2, and the output tokens, flow-matching model, and vocoder are taken from IndexTTS2. What is new is the combination: a decoupled emotion vector inserted into a text-like prompt template, pseudo-empathetic training targets that the frozen LLM writes for itself from randomly assigned emotion labels, and emotion-prompted IndexTTS2 targets that give the decoder expressive supervision without any recorded empathetic dialogue. The emotion extractor’s gated layer pooling plus query-attention pooling is a standard pooling design, and its ablation shows modest individual gains. The frozen-LLM argument rests on comparisons with separately trained published systems (LLaMA-Omni 2, OpenS2S); no ablation in the paper unfreezes the LLM under the same data and pipeline.
Field Significance
Moderate — The paper shows that empathetic speech responses can be obtained without empathetic speech dialogue data or LLM fine-tuning, by giving a frozen LLM an aligned emotion signal and supervising the speech decoder with emotion-prompted TTS targets. It is a cost-saving recipe for spoken conversational agents rather than a new architecture, and its expressiveness evidence rests mainly on LLM-judge scores and a small human preference test.
Claims
- supports: Keeping the text LLM frozen while aligning speech inputs to its embedding space can preserve its knowledge in a speech LM better than fine-tuning it on cross-modal instruction data.
Evidence: FreezeEmpath reaches 57.79 average S2T SpokenQA accuracy versus 45.97 for OpenS2S (LLM fine-tuned on empathetic speech data) and 49.43 for LLaMA-Omni2-7B, which the authors describe as sharing FreezeEmpath’s structure and speech-to-text training data; these are comparisons with published models, not an in-paper freeze/unfreeze ablation. (§5.1.3, Table 2)
- supports: Emotion-aware spoken dialogue can be trained without recorded empathetic speech dialogues by pairing ordinary speech instructions with emotion features from SER corpora and letting the frozen LLM generate its own empathetic targets.
Evidence: Pseudo-empathetic data built from randomly assigned emotion labels gives the highest SpeechAlpaca empathy scores (7.63 S2T, 7.27 S2S) among the compared systems; removing EI fine-tuning drops empathy to 6.64. (§3.2.2, §5.2.2, Tables 1 and 4)
- complicates: Training a speech-emotion pathway for empathetic response generation trades off against its raw emotion-recognition accuracy.
Evidence: Removing EI fine-tuning raises average SER accuracy from 70.1 to 71.8 while empathy falls from 7.63 to 6.64. (§5.2.2, Table 4)
- supports: Training a speech decoder on emotion-prompted synthetic targets can add emotional expressiveness to dialogue speech output relative to voicing the same text with a neutral TTS prompt, at some cost in intelligibility.
Evidence: Against IndexTTS2 voicing FreezeEmpath’s text with a neutral prompt, the trained decoder scores higher on the Gemini acoustic score (7.24 vs. 5.91) but has higher ASR-WER (5.13 vs. 3.39). (§Appendix D.2, Table 6)
- complicates: Learned emotion features remain a weaker channel into an LLM than explicit emotion labels.
Evidence: With ground-truth transcript and ground-truth emotion label as text, empathy is 8.21, versus 7.63 with the learned speech and emotion features; transcripts alone with the learned features nearly match quality (8.74 vs. 8.76). (§Appendix D.1, Table 5)
Limitations and Open Questions
Warning
Evidence for emotional expressiveness and empathy comes almost entirely from LLM judges: GPT-4o scores transcripts for quality and empathy, and Gemini-2.5-Pro scores the audio for emotional fit and naturalness and produces the VStyle scores. The only human test is a pairwise preference on 20 samples with 5 participants per comparison, in which ties are the most common outcome. No emotion classifier on the output audio and no MOS-style naturalness test are reported.
- The authors’ stated limitation is that only semantic content and emotional tone are modelled; other paralinguistic cues such as speaker gender or age are left out (§Limitations).
- The emotion inventory is restricted to five SER classes (neutral, happy, sad, angry, surprised), and speech output style comes from a set of predefined response emotion categories voiced with ESD prompts, so the range of expressive output is bounded by those categories and by IndexTTS2.
- Some Table 1 baseline values are quoted from the Fun-Audio-Chat report rather than re-measured, and the page cannot tell which cells from the extracted text alone.
- Weak spots remain: Chinese joy on VStyle-Empathy (3.94) is below three of four baselines, and the trained decoder’s WER is higher than the neutral IndexTTS2 cascade.
- The streaming decoder’s latency is not reported, so the streaming design is not evaluated as such.
- Random emotion-label assignment in pseudo-empathetic data can produce emotion-content mismatches; the authors argue this has no significant impact (§Appendix B.1), but no ablation against content-matched labels is shown in the paper.
Wiki Connections
- Spoken Language Model — A frozen Qwen2.5-7B-Instruct consumes Whisper-derived semantic features plus a pooled emotion vector, a decoupled design for emotion-aware speech LMs.
- Speech-to-Speech — An end-to-end empathetic spoken dialogue system whose speech responses are evaluated on SpeechAlpaca and VStyle-Empathy.
- Emotion Synthesis — The speech decoder is trained on emotion-prompted IndexTTS2 targets, and its output’s emotional fit is scored by an audio LLM judge and in human preference tests.
- Autoregressive Codec TTS — Trains a decoder-only Transformer that streams discrete speech tokens from LLM hidden states and text tokens.
- Multilingual TTS — Trains on English and Chinese data and evaluates spoken responses in both languages on VStyle-Empathy.
- Subjective Evaluation — Uses a small pairwise human preference test on the speech responses alongside LLM-judge scores.
- LLaMA-Omni 2 — Source of the streaming speech decoder design and a SpokenQA baseline that FreezeEmpath exceeds on every column.
- IndexTTS2 — Supplies the emotion-prompted target speech tokens and the frozen flow-matching model and vocoder.
- OpenS2S — Empathetic speech LM trained on curated empathetic speech data, used as the main empathetic baseline and in human preference tests.
- Freeze-Omni — Earlier frozen-LLM speech-to-speech dialogue model cited as related work; FreezeEmpath shares its frozen-backbone premise and adds an emotion pathway.
- VStyle — Provides the implicit-empathy test set and evaluation protocol for daily empathetic conversation.
- Step-Audio 2 — Baseline in empathetic dialogue, SpokenQA, and human preference; strongest competitor on several Chinese VStyle cells.
- Fun-Audio-Chat — Baseline that leads on S2S quality and S2S SpokenQA average; some of its reported values are quoted in Table 1.
- Efficient Training for Cross-lingual Speech LMs — Same group’s CSLM work, whose Chinese instruction data FreezeEmpath reuses.