arXiv · 2026 · Preprint

Kevin Miller et al. (Boston University) · → Paper · Demo: ✗ · Code: ✗

Counterfactual audits that hold the transcript fixed and vary only affect, prosody, or the timing of an affective shift show that audio-language-model judges for spoken assistants often recover the paralinguistic contrast when both audio versions are shown, but fall near chance when judging one audio context at a time.

Problem

Audio-language models (ALMs) are used more and more as automatic judges and reward models for speech-to-speech assistants. These uses assume that a judge which receives audio also uses the paralinguistic evidence in it (tone, rhythm, pitch, timing). A judge that relies on the transcript, response style, or generic conversational preferences cannot evaluate whether a response fits how the user spoke. Existing benchmarks measure paralinguistic instruction following or emotion perception, but they do not show why a judge fails when it picks the wrong response: it may not hear the cue, may not map the cue to a response, or may not combine the two in one judgment.

Method

Each audit item is a counterfactual tuple: a shared transcript or conversation history, two audio realizations of the same words, and the two responses that fit each realization. Because the words are identical across branches, a text-only or lexically biased judge cannot solve the items consistently. The ground-truth label comes from the construction, not from annotators.

The audit runs as a diagnostic cascade:

  • Native judgment (POINTWISE). The judge hears one audio context and two candidate responses and picks the one that fits. This is the deployment setting.
  • Contrastive recoverability (PAIRWISE). The judge hears both audio contexts and both responses and must match them. The protocol gap (PAIRWISE accuracy minus POINTWISE accuracy; positive means the judge degrades in the one-context setting) measures protocol dependence.
  • Component attribution. Each item gets three binary outcomes: a perception probe P (identify the user’s emotion, or the full per-turn affective trajectory), an oracle response-mapping probe O (choose the response in text when the true emotion is supplied), and native judgment J. The eight (P, O, J) states separate perception bottlenecks, response-mapping bottlenecks, shortcut-like successes, full-stack failures, and “Potemkin” failures (P = 1, O = 1, J = 0), where both component skills succeed but the integrated audio judgment fails.
  • Prompt cueing. Prompts carry no cue, a hard cue (focus only on tone of voice), or, for positional items, a transition cue that tells the judge the user’s affect changes during the conversation.

Two task families vary interaction complexity. SINGLE-TURN-EMOTIONS (189 pairs, 378 pointwise instances, 4 emotion pairings) uses lexically neutral user inputs from the EmoCF subset of CAVA. User audio is synthesized with gpt-4o-mini-tts, responses are written by gemini-2.5-flash and synthesized with gemini-2.5-flash-preview-tts, followed by a manual quality check. POSITIONAL-EMOTION (500 pairs, 1,000 pointwise instances, about 9.9 turns on average) seeds task-oriented dialogues from OD3 and injects goal failures at different turns, so the onset of negative affect moves while the final transcript stays fixed. The right response depends on when the user became frustrated and which failure caused it. Turns are rendered with gpt-4o-mini-tts and filtered by an automatic scorer that combines emotion2vec scores with pitch, speech-rate, and pause heuristics (up to 10 attempts per turn). An earlier multi-turn prototype (EMOTIONAL-CONVERSATIONS) is reported in the appendix because its items are solvable from the last turn alone.

Eleven judges are audited: Gemini-2.5-Flash, Gemini-2.5-Pro, Gemini-3-Flash, Gemini-3-Pro, GPT-4o-mini, GPT-4o, Nova-2-Pro, Phi-4-Multimodal-6B, Qwen-2.5-Omni-7B, DeSTA2.5-Audio-8B, and Voxtral-Small-24B. No model is trained.

Key Results

Contrastive success overstates native reliability. On SINGLE-TURN-EMOTIONS with the hard cue, Gemini-3-Pro reaches 91.0% in PAIRWISE but 65.3% in POINTWISE, and Gemini-2.5-Pro drops from 86.0% to 60.1% (Table 3). On POSITIONAL-EMOTION with the transition cue, Gemini-3-Flash reaches 79.4% in PAIRWISE but 53.0% in POINTWISE. Across positional POINTWISE conditions in Table 3, no judge exceeds 54.5% (Gemini-2.5-Pro, transition cue). The paired-bootstrap protocol gap is significantly positive for all four Gemini judges on single-turn items under both cue settings and on positional items with the transition cue (Table 6). Without the transition cue, Gemini PAIRWISE accuracy on positional items stays between 49.0% and 57.6%, so the Gemini judges need both an explicit transition hint and counterfactual inputs to rise clearly above chance.

Most non-Gemini judges sit near chance everywhere. GPT-4o, GPT-4o-mini, Nova-2-Pro, and the four open models stay between 45.2% and 59.8% in every Table 3 cell, so their small protocol gaps reflect chance-level PAIRWISE accuracy rather than robustness.

Similar accuracy hides different failures. On single-turn items, Gemini-2.5-Pro, Gemini-3-Flash, and Gemini-3-Pro place 26%, 25%, and 19% of their mass in the Potemkin state (Figure 3). Conditional on passing both probes, Gemini-3-Pro still fails the native judgment 26% of the time, and the conditional Potemkin rate reaches 51% for DeSTA2.5-Audio (Figure 4). GPT-4o-mini and GPT-4o have lower reliable-integration mass (38% and 37%) and larger perception bottlenecks, so they fail before integration matters. On positional items, reliable mass is at most 31% (Gemini-3-Pro), and shortcuts and full-stack failures dominate. Component probes match this: positional oracle-mapping accuracy is at most 70.3% even with the transition cue (Table 18), against 78.1% to 96.6% with the hard cue on single-turn items (Table 13).

Validity checks. Five annotators per task, each judging 50 items in the POINTWISE format, score 62.0% to 100.0% on single-turn items and 88.0% to 92.0% on positional items (Table 21). Replacing synthetic user audio with the original CAVA human recordings yields (P, O, J) state profiles with Spearman ρ > 0.9 for all models (Figure 6). Swapping Gemini-generated positional annotations for GPT-generated ones leaves the qualitative judge ranking unchanged (Figure 5). DNSMOS and an LLM judge (claude-haiku-4-5) show small absolute differences between paired responses (mean DNSMOS OVRL difference 0.19; Table 4), which the authors read as balanced response quality.

Novelty Assessment

The contribution is an evaluation design, not a model. Counterfactual response evaluation for voice assistants already exists in CAVA, which supplies the single-turn inputs, and contrastive matching follows prior test-time matching work. What is new is the cascade that separates native judgment from contrastive recoverability and the (P, O, J) decomposition, which turns one accuracy number into an attribution of where the judge fails. The positional task is also new: it isolates when an affective shift happens rather than whether it happens. The evidence base is modest. The main sets use synthesized English speech, with all user audio from one TTS model (gpt-4o-mini-tts) and three voices for the multi-turn set. Human validation is small and serves only to check that items are solvable. The real-speech check covers the single-turn task only.

Field Significance

Moderate — The audit provides a reusable protocol for testing whether an audio judge actually uses paralinguistic evidence before it serves as an evaluator or reward model for spoken conversational agents. Its central finding, that PAIRWISE accuracy can overstate one-context reliability by more than 20 points, is a direct caution for benchmarks that report judge agreement from contrastive formats. Scope limits (English, synthetic stimuli, two emotion-focused tasks) keep it below a broad benchmark.

Claims

  • complicates: Access to audio does not ensure that audio-language-model judges use paralinguistic cues when choosing among responses to a single spoken context.

    Evidence: With the transcript held fixed, all eleven judges stay at or below 65.3% POINTWISE accuracy on single-turn items and at or below 54.5% on positional items, against 50% chance. (§5.1, Table 3)

  • complicates: Contrastive evaluation formats that present both counterfactual audio contexts overstate how reliably an audio judge performs in a one-context deployment setting.

    Evidence: Gemini-3-Pro scores 91.0% in PAIRWISE but 65.3% in POINTWISE on single-turn items with a hard cue, and Gemini-3-Flash scores 79.4% versus 53.0% on positional items with a transition cue; paired-bootstrap protocol gaps are significantly positive for all Gemini judges in these settings. (§5.1, Table 3; §D.1, Table 6)

  • refines: Aggregate judge accuracy is insufficient to characterise audio-judge reliability, because judges with similar accuracy fail at different stages (perception, response mapping, or integration).

    Evidence: Gemini judges carry 19% to 26% Potemkin mass on single-turn items (both probes pass, native judgment fails), whereas GPT-4o and GPT-4o-mini fail earlier through perception bottlenecks; DeSTA2.5-Audio reaches a 51% conditional Potemkin rate. (§5.2, Figures 3 and 4)

  • complicates: Paralinguistic judgment that depends on when an affective shift occurs in a multi-turn dialogue is substantially harder for audio judges than static single-utterance emotion judgment.

    Evidence: Reliable integrated mass on POSITIONAL-EMOTION is at most 31% (Gemini-3-Pro), oracle response-mapping accuracy is at most 70.3% even with a transition cue, and the earlier multi-turn prototype was solvable from the last turn alone. (§5.2, Figure 3; §D.2, Tables 9, 10 and 18)

  • supports: Judge-failure profiles measured on synthesized counterfactual speech can transfer to real human speech when the lexical content is held fixed.

    Evidence: Replacing synthesized single-turn user audio with the original CAVA human recordings gives (P, O, J) state proportions with Spearman ρ > 0.9 across all models. (§5.3, Figure 6)

Limitations and Open Questions

Warning

The main audit sets are built from synthesized English speech (gpt-4o-mini-tts with three voices for the multi-turn set), and the real-speech check covers only the single-turn task. Whether the positional failure profile holds for real multi-turn human dialogue is untested.

  • Human validation uses five annotators per task on 50 items each and only checks that items are solvable. The single-turn range (62% to 100%) is wide; the authors attribute the low end to non-native listeners, so this is not a human ceiling.
  • Item counts behind each accuracy vary by judge and condition (for example, 103 of 189 single-turn PAIRWISE items for Gemini-2.5-Pro without a cue; Table 7), and the paper does not state how unparseable or missing judge outputs were handled.
  • The audit covers emotion and affective timing only. Other paralinguistic dimensions (speaking rate as intent, emphasis, sarcasm, non-verbal vocalizations) are not tested.
  • Proprietary components (Gemini and OpenAI TTS and LLMs) build the items and Gemini models are also judged. The generator-swap check covers positional annotations, not the TTS rendering. Construction code is promised but not yet released.
  • The paper diagnoses failures but leaves fixes (judge training, training-data design) to future work; the only mitigation-like experiment is the appendix staging probe, which feeds a judge its own predicted emotions in text.

Wiki Connections

  • Spoken Language Model — Audits eleven existing audio-language models (Gemini, GPT, Nova, Qwen2.5-Omni, Phi-4-Multimodal, DeSTA2.5-Audio, Voxtral) by feeding them counterfactual speech and measuring whether their judgments track paralinguistic cues.
  • Evaluation Metrics — Tests the validity of audio-language models as automatic evaluators against construction-defined ground truth, with a small human check of item solvability, and proposes protocol-gap and (P, O, J) diagnostics beyond aggregate accuracy.
  • TRACE — Earlier work by the same group on LLM reasoning over audio cues for speech evaluation, cited as evidence that audio access does not imply paralinguistic reasoning; this paper turns that motivation into a counterfactual audit of ALM judges.
  • AudioJudge — Prior study of large audio models as speech evaluators that this audit questions by testing whether such judges actually use paralinguistic evidence.
  • Gemini 2.5 — Gemini-2.5-Flash and Gemini-2.5-Pro are audited judges; they recover contrasts in PAIRWISE but drop toward chance in POINTWISE.
  • GPT-4o — GPT-4o is an audited judge that stays near chance in every protocol, with perception bottlenecks rather than Potemkin failures on single-turn items.
  • Qwen2.5-Omni — Qwen-2.5-Omni-7B is one of the open audited judges and stays near chance on both tasks.
  • Phi-4-Multimodal — Phi-4-Multimodal-6B is an open audited judge that stays near chance on both tasks.
  • emotion2vec — Used as the speech emotion recognizer in the automatic scorer that accepts or rejects synthesized multi-turn renderings.
  • S2S-Arena — Paralinguistic instruction-following benchmark for speech-to-speech systems, cited as measuring what systems can do without diagnosing how judges use paralinguistic cues.