EMNLP Findings · 2025 · Conference
Wang et al. · → Paper · Demo: ? · Code: ?
Introduces CP-Bench, a benchmark that evaluates speech-LLMs on contextual paralinguistic reasoning, requiring models to jointly interpret verbal content and non-verbal cues such as emotion, gender, and speaker identity across two in-the-wild QA datasets.
Problem
Existing speech-LLM benchmarks either assess linguistic comprehension in isolation (ASR, translation, spoken QA) or address paralinguistic attributes as direct, single-attribute queries (emotion recognition, gender identification). Neither category measures contextual paralinguistic reasoning, which requires a model to simultaneously interpret what is said and how it is said in order to answer questions about speaker intent, inferred emotion, or multi-speaker dynamics. This gap means evaluations can reward models for narrow competence while failing to expose weaknesses in the integrative reasoning that is necessary for empathetic, socially intelligent speech interaction.
Method
CP-Bench is constructed from YouTube in-the-wild speech, chosen for its emotional richness and natural conversational diversity. Two sets are curated: a long set of 479 samples (30-60 seconds) yielding 1,567 QA pairs, and a short set of 468 samples (10-30 seconds) yielding 1,180 QA pairs.
Questions are categorised into three top-level types: Content-only (C), Direct Paralinguistic (DP), and Contextual Paralinguistic (CP). DP and CP are further decomposed by attribute, producing seven types in total: C, DE (direct emotion), DG (direct gender), DS (direct speaker count), CE (contextual emotion), CG (contextual gender), and CS (contextual speaker count). Annotation combines automatic estimation using Emotion2Vec for emotion labels and a fine-tuned WavLM-ECAPA model for gender, followed by human correction and QA generation by three annotators following a standardised protocol.
Model responses are scored by a LLaMA-3-70B-Instruct judge using a rubric that rewards both content accuracy and paralinguistic relevance. Scores on a 0-100 scale are aggregated via an Unweighted Average Score (UAS) that weights each question type equally regardless of its frequency, preventing dominant types from distorting the overall metric. Results are averaged over five independent judge runs to reduce variance.
Six models are evaluated: closed-source GPT-4o and Gemini Flash 2.0; open-source MERaLiON-AudioLLM and Qwen2-Audio-7B-Instruct; and two cascaded baselines pairing Whisper large-v3 with Gemma2 9B variants.
Key Results
GPT-4o leads on C and CP categories (69.68 on C-long, 67.48 on CP-long; 72.57 on C-short, 68.74 on CP-short) but collapses on DP questions (30.34 long, 29.33 short). This drop is largely explained by Azure content-filtering restrictions that block responses about speaker characteristics. MERaLiON-AudioLLM and Gemini Flash 2.0 show the most balanced profiles: Gemini leads overall on the long set; MERaLiON outperforms Gemini on the short set (73.69 C-short vs. 66.69). Qwen2-Audio leads DP on the long set (62.29) but degrades substantially on the short set (52.39) and scores weakest among end-to-end models on C and CP categories.
Cascaded models consistently underperform on DP and CP. Cascade 1 scores 33.3 on DP-long compared to 60+ for end-to-end speech-LLMs, confirming that text-only pipelines lose access to paralinguistic signal present in the acoustic stream.
Temperature analysis on GPT-4o and MERaLiON reveals no universally optimal setting: lower temperatures improve C-type performance for GPT-4o on long speech, while MERaLiON on shorter speech benefits from higher values; CE-type questions are relatively temperature-insensitive for both models.
A cross-modal conflict ablation (§A.1.1) shows that GPT-4o grounds its answers in audio emotional cues over conflicting text queries in 17 of 27 cases, suggesting genuine multimodal integration in a significant fraction of CP questions.
Novelty Assessment
The benchmark addresses a real and well-motivated gap: prior evaluations treat paralinguistic tasks as isolated attribute queries rather than integrated reasoning problems. The three-tier taxonomy (C/DP/CP) with attribute-level decomposition is a practical contribution that allows failure modes to be attributed to specific modalities. The UAS aggregation scheme and the LLM-judge adaptation for paralinguistic QA are methodologically considered choices beyond importing standard evaluation protocols.
The contribution is entirely in evaluation design and empirical analysis. No new model training is involved. The dataset is limited in scale (under 500 samples per set) and skews toward Singaporean English based on the example content, which constrains the benchmark’s breadth. QA pairs were generated by non-overlapping annotators without cross-verification, introducing potential annotation bias. These limitations are acknowledged by the authors.
Field Significance
Moderate — CP-Bench provides a concrete evaluation framework for a capability gap that is widely acknowledged in the speech-LLM community but poorly quantified. The benchmark fills a methodological hole for spoken conversational agent evaluation and enables direct, structured comparison of leading models on contextual paralinguistic reasoning. Its value depends on uptake by the broader evaluation community.
Claims
-
supports: Speech-LLMs that excel at content understanding do not reliably excel at direct paralinguistic recognition, revealing a systematic capability gap across model families.
Evidence: GPT-4o achieves 69.68 UAS on C and 67.48 on CP (long set) but only 30.34 on DP, while Qwen2-Audio reverses this pattern with 62.29 on DP-long but substantially lower scores on C and CP. (§4.2.1, Table 3)
-
supports: Cascaded ASR-LLM pipelines lose critical paralinguistic signal compared to end-to-end speech-LLMs on tasks requiring direct or contextual paralinguistic reasoning.
Evidence: Cascade models score 33.3-44.18 on DP-long versus 60+ for end-to-end speech-LLMs; the gap narrows on DE-type questions where emotion cues are partly recoverable from transcripts. (§4.2.1, Table 3)
-
complicates: LLM-as-judge evaluation of paralinguistic reasoning requires multi-run averaging to achieve stable estimates, with single-run scores varying by up to 2.57 points per question type.
Evidence: LLaMA-3-70B judge scores showed std values of 0.44-1.73 (long set) and 0.35-2.57 (short set) across five independent runs; human-LLM correlation was 0.71 on a 27-pair subset of CP questions. (§A.1.2, Table 5)
-
complicates: Inference temperature has task-dependent and input-length-dependent effects on speech-LLM performance, with no single optimal value across question types.
Evidence: GPT-4o on CG-type questions peaks at mid-range temperatures; MERaLiON consistently favors lower values; MERaLiON on CS-type shows the largest gains with higher temperature on the short set. (§4.2.3, Figure 2)
-
refines: Shorter speech inputs are generally easier for speech-LLMs across most question types, but direct gender inference does not follow this trend.
Evidence: Models consistently score higher on the short set (10-30s) than the long set (30-60s) for C, CP, CE, CS, DE, and DS types, but DG-type performance does not show the same length-dependent improvement. (§4.2.2)
Limitations and Open Questions
The benchmark is small in scale (under 500 samples per set), reducing statistical reliability for low-frequency question types, particularly CS in the short set. The speech data is drawn from YouTube and contains a notable proportion of Singaporean English, which may limit generalisability across dialects and languages. QA pairs were generated by non-overlapping annotators without a cross-verification round, introducing potential individual bias in both question formulation and emotion-label assignment. The evaluation pipeline relies on a single LLM judge; future work plans to incorporate multiple judges or human evaluation to improve robustness.
Wiki Connections
- Evaluation Metrics — introduces the Unweighted Average Score (UAS) and an LLM-judge evaluation protocol adapted specifically for paralinguistic QA, with scoring criteria that reward both content accuracy and paralinguistic relevance.
- Subjective Evaluation — validates the LLM-judge by comparing its averaged scores against three human annotators on a 27-pair CP subset, finding a 0.71 Pearson correlation and low within-run variance.
- Spoken Language Model — benchmarks GPT-4o, Gemini Flash 2.0, MERaLiON-AudioLLM, and Qwen2-Audio-7B-Instruct on contextual paralinguistic reasoning, exposing systematic capability disparities across model families and input conditions.