arXiv · 2025 · Preprint
Wang et al. (A*STAR Institute for Infocomm Research) · → Paper · Demo: ? · Code: ?
Two training strategies — explicit emotion metadata injection and implicit contextual-paralinguistic QA data generation — improve Speech-LLM performance on empathetic reasoning tasks by up to 46% over a standard baseline.
Problem
Speech-LLMs trained on conventional question-answering datasets perform well on content and isolated paralinguistic tasks (e.g., “what is the speaker’s emotion?”) but struggle with contextual-paralinguistic reasoning — questions that require jointly understanding emotional cues and conversational content to infer meaning or speaker intent. Prior work generated QA training data from categorical emotion labels alone, leaving dimensional emotion signals (valence, arousal, dominance) unused and limiting the nuance of training supervision.
Method
The paper builds on the MERaLiON AudioLLM architecture, which combines a Whisper large-v3 speech encoder with a Gemma2 9B text decoder bridged by a lightweight MLP adapter. Only the adapter is trained; both encoder and decoder remain frozen. The input representation passes speech through the encoder to produce 1,500-length sequences, which the adapter compresses into 100 token embeddings before concatenation with text instructions for the decoder.
Two complementary training strategies are evaluated. The explicit method augments question prompts at both training and inference time with time-stamped emotion labels derived from a pre-processing speech emotion recognition (SER) module, giving the model direct access to affective context. The implicit method generates a large-scale contextual-paralinguistic question-answering (CPQA) training corpus by prompting GPT-4o with word-level transcripts alongside both categorical emotion labels (Emotion2Vec) and dimensional scores (valence, arousal, dominance from a continuous emotion recognition model), producing 32,960 QA pairs from 4,740 emotion-rich speech clips. The QA generation prompt explicitly instructs the model to draw on dimensional signals and to combine simple and complex reasoning questions, diversifying the training signal beyond template-based paralinguistic QA.
A supplementary PQA* dataset (generated from MSP-Podcast with a similar LLM prompt but without contextual reasoning instructions) is also created as an intermediate training resource. The training data totals approximately 542,609 ASR QA pairs, 277,362 gender-PQA pairs, 219,048 emotion-PQA pairs, and 32,960 CPQA pairs.
Key Results
Performance is measured using GPT-4o-as-judge scores (scaled 0–100) on human-annotated and LLM-generated CPQA evaluation sets, and with LLM-judged scores plus estimated weighted accuracy and F1 on emotion-PQA benchmarks.
On the human-annotated CPQA set, the baseline system (S10) scores 41.0. Adding the PQA* dataset (S20) raises this to 52.06, and further adding CPQA data (S30) reaches 56.75 — a 38.41% relative improvement over the baseline. Injecting explicit emotion metadata at both training and inference (S32) reaches 59.87, a 46.02% gain. Gains are largest for contextual+emotion questions (CE: 37.56 → 53.86 for S30), though contextual-only (C) and contextual+gender (CG) questions also improve substantially (44.86 → 60.29 and 49.77 → 63.86, respectively).
On the LLM-generated CPQA set, improvements are even larger but partly confounded: the authors note that explicit emotion metadata injection may inadvertently expose answers to direct emotion questions embedded in the evaluation set, inflating those gains. For PQA tasks, CPQA training modestly reduces performance on IEMOCAP (50.76 → 45.7 LLM score) while slightly improving MSP-Podcast, suggesting domain mismatch and SER label noise carry over into direct classification queries.
The paper also validates LLM judge reliability by showing consistent rank correlation between GPT-4o judge scores and estimated weighted accuracy and F1 scores derived via embedding-based label extraction. The ratio of correct to incorrect predictions rises monotonically with judge score, supporting the judge’s reliability for classification-type questions.
Novelty Assessment
The contribution is primarily engineering and data-centric. The model architecture (MERaLiON AudioLLM: Whisper + MLP adapter + Gemma2) is adopted wholesale from prior work. The genuine novelty is in the QA generation pipeline: incorporating dimensional emotion annotations (valence, arousal, dominance) into GPT-4o prompts for CPQA data generation, and validating this pipeline at training scale. The LLM judge validation methodology — converting open-ended answers to classification labels via keyword and embedding matching to compute accuracy/F1 — is a useful practical contribution to evaluation methodology. The gains are real and reproducible on human-annotated data, but the reliance on in-house proprietary speech data for CPQA training limits external reproducibility.
Field Significance
Moderate — This paper provides concrete evidence that contextual-paralinguistic training data is a meaningful bottleneck for Speech-LLM empathetic reasoning, and demonstrates a scalable pipeline for generating such data from existing emotion-annotated corpora. The data generation methodology is directly applicable to other Speech-LLM training setups, and the LLM judge validation procedure addresses a practical evaluation gap. Neither the architecture nor the training paradigm is novel; the value lies in the data recipe and its empirical validation.
Claims
- Training Speech-LLMs on data that jointly encodes contextual and paralinguistic reasoning substantially outperforms training on isolated paralinguistic QA templates, even when the underlying speech encoder is frozen. (§V.B.1, Table III)
- Incorporating dimensional emotion annotations (valence, arousal, dominance) alongside categorical labels into LLM-generated QA data diversifies training supervision and improves generalisation to complex emotional states. (§II.D, §V.B.1)
- Explicitly injecting emotion metadata into inference prompts can partially compensate for a model’s limited intrinsic paralinguistic understanding, but training on contextual-paralinguistic data yields more robust generalisation across question types. (§V.B.3, Fig. 2)
- LLM judge scores for open-ended speech-language model evaluation correlate reliably with classification-based accuracy and F1 metrics on questions with deterministic answers, supporting their use as a proxy metric. (§IV, §V.B.4, Table V)
Limitations and Open Questions
Warning
The CPQA training data is derived from a proprietary in-house movie and TV dataset that is not publicly released, making exact replication of the training setup impossible for external researchers.
Emotion labels used both in training and inference prompts come from SER models rather than ground-truth annotations, introducing noise that may suppress performance on direct classification tasks while still benefiting contextual reasoning. The LLM-generated CPQA evaluation set contains evaluation confounds — direct emotion questions that benefit disproportionately from explicit metadata injection — which the authors flag but do not resolve in the current work, requiring stricter QA generation controls in follow-up.
The evaluation scope is narrow: CPQA and emotion-PQA benchmarks only. Performance on other SCA capabilities (dialogue coherence, turn-taking, ASR) is not assessed, so it is unclear whether the CPQA training data causes regression elsewhere. All experiments use a single base architecture (MERaLiON); generalisability to other Speech-LLM frameworks remains untested.
Wiki Connections
Core method: spoken-language-model — this paper extends Speech-LLMs toward empathetic reasoning through data-centric training.
Related capabilities: emotion-synthesis — emotional understanding in generation and comprehension. instruction-conditioned-tts — explicit metadata injection as a conditioning approach. self-supervised-speech — Whisper encoder used as frozen speech representation backbone.
Evaluation: evaluation-metrics — LLM judge validation with classification metric correlation. subjective-evaluation — human-annotated QA benchmark for contextual paralinguistic assessment.
In-corpus reference: 2410.01162 — frozen LLM paralinguistic perception, a related approach to paralinguistic understanding in speech-language models.