arXiv · 2026 · Preprint

Jiacheng Pang et al. (University of Southern California) · → Paper · Demo: ? · Code: ?

Builds a 2,000-item adversarial benchmark in which synthesized speech contradicts its own transcript on a paralinguistic attribute, shows that audio LLMs overwhelmingly follow the transcript, traces the failure to encoder-depth and LLM-utilization bottlenecks, and largely repairs it with a prompt-conditioned multi-layer audio mixer plus DPO.

Problem

Speech LMs (audio LLMs) score well on speech understanding benchmarks, but existing benchmarks such as MMAU, MMAU-Pro and MMSU do not decouple lexical evidence from acoustic evidence. A model can therefore answer a “paralinguistic” question correctly by reading the words rather than listening to how they are said. The paper asks whether these models actually use acoustic cues (age, gender, emotion, pitch, volume, speed, vocal range, intonation, speaker identity, speaker count) when the transcript asserts something different, and if not, where in the model the information is lost. The work is accepted at ICML 2026; this page follows the arXiv version.

Method

VoxParadox benchmark. Each of 10 tasks has 200 multiple-choice items (2,000 total, average clip 7.81 s). GPT-4o writes a transcript that explicitly asserts an adversarial label (for example, an elderly-sounding voice saying “I am a child”), and the audio is synthesized so that it conveys the true label. Acoustic attributes are enforced by deterministic mechanisms rather than style prompts: ElevenLabs voice metadata for age and gender, Azure SSML pitch contours for rising or falling intonation, GPT-4o TTS plus time-stretching, gain, pitch shifting and range scaling for the three-segment signal-comparison tasks, and concatenation of per-turn clips with known voices for speaker counting and identity. Emotion uses GPT-4o TTS with two high-contrast pairs and is filtered by a SpeechBrain wav2vec 2.0 emotion classifier applied to reversed audio. Every clip must be transcribed exactly by Whisper large-v3 (WER = 0). Two metrics are reported: GT accuracy against the acoustic label, and adversarial-label agreement (ALA), the fraction of predictions that match the transcript-implied label; Appendix B.2.2 also reports the gap defined as ALA minus GT, where larger positive values mean stronger transcript following.

Diagnostics. A reversed-audio condition removes intelligible words while preserving pitch, loudness and timbre. Layer-wise probing freezes Audio Flamingo 3 (AF3), mean-pools encoder and LLM hidden states every two layers, and trains 3-layer MLP probes with 10-fold cross-validation per task; robustness checks repeat this with linear and 5-layer probes, on Qwen2-Audio, HuBERT and CLAP encoders, and on a VoxCeleb2-derived task suite. A pilot intervention concatenates projected tokens from AF-Whisper layers 5 and 15 after the final-layer tokens, training only the new projectors.

Prompt-Conditioned Layer Mixer (PCLM). The text question is encoded with BERT-small, and an MLP with softmax produces weights over five audio-encoder layers (layers 5, 15, 25, 30 and the final layer). Each layer has its own projector into the LLM space (initialized by cloning the pretrained final-layer projector), and the weighted sum of projected tokens replaces the usual final-layer audio tokens. The audio encoder stays frozen throughout. Training has three stages: (1) projector and mixer alignment with the LLM frozen, (2) full-LLM supervised fine-tuning with PCLM enabled, and (3) DPO on the LLM only (beta = 0.1, learning rate 5e-7), using preference pairs in which the correct option is preferred over a randomly sampled incorrect option from paralinguistic MCQ sources. Training data mix SpeechCraft, an AudioSkills-XL subset, VoxCeleb2-derived paralinguistic MCQs and five emotion corpora; VoxParadox is never used for training. PCLM is applied to both AF3 and Qwen2-Audio-7B-Instruct.

Figure 5. Prompt-Conditioned Layer Mixer (PCLM). PCLM taps multiple audio-encoder layers, projects each into the LLM space, and uses prompt-conditioned weights (BERT → MLP+softmax) to form a weighted mixture that is fed to the LLM as audio conditioning. The audio encoder is frozen; we first align projectors and the PCLM module with the LLM frozen, then fine-tune the LLM with PCLM enabled. Finally, we perform DPO on the LLM while freezing all other components

Key Results

Benchmarking (Table 1, Table 6). Across 12 audio LLMs, VoxParadox macro-average GT accuracy ranges from 6.10% (SALMONN-7B) to 30.85% (Audio Flamingo 2); the paper reports that ALA averages 64.34% against 15.33% GT accuracy across models. GPT-4o Audio has GT 8.60% and ALA 81.55% (gap +72.95, the largest in Table 6), and Qwen3-Omni has GT 10.60% and ALA 80.65%. Strong scores on the MMSU paralinguistic subset do not carry over: Step-Audio-R1 scores 54.51% there (the highest MMSU paralinguistic score among the 12 baselines) but 17.45% on VoxParadox, and Gemini 2.5 Flash drops from 51.05% to 24.70%. Audio Flamingo 2, whose audio encoder is CLAP-based rather than ASR-pretrained, is the only model with a negative gap (GT 30.85%, ALA 29.80%) despite a modest MMSU paralinguistic score (27.44%). Per-task results are uneven: Step-Audio-R1 reaches 93.00% and Gemini 2.5 Flash 92.50% on speaker counting, and Kimi-Audio reaches 79.00% on emotion, so the collapse is not uniform across tasks. SALMONN’s low ALA (22.45%) reflects parse failures rather than acoustic grounding, per the authors.

Diagnostics. Reversing the audio raises AF3 GT accuracy from 17.40% to 38.80% and lowers ALA from 68.50% to 28.90% (Figure 3). Probes on frozen AF3 states consistently exceed AF3’s end-to-end task accuracy (utilization gap), and for signal-level tasks probe accuracy is often higher in early or middle encoder layers than in the final layer, with a drop at the encoder-LLM interface (Figure 4). Qwen2-Audio and HuBERT show the same depth-wise degradation; CLAP does not (Figure 8). The static concatenation pilot gives only small gains (17.40% to 19.75%, or 20.80% with layer-5 tokens scaled by 10; Table 7), and per-task results are mixed (age falls from 10.00% to 5.00%).

PCLM and DPO (Table 2, Table 8). On VoxParadox, AF3 rises from 17.40% to 60.00% with PCLM and 65.20% with PCLM + DPO; Qwen2-Audio rises from 14.85% to 68.65% and 72.30%. The SFT-without-PCLM controls trained on the same data reach 34.80% (AF3) and 45.70% (Qwen2-Audio), so the mixer accounts for a large share of the gain beyond fine-tuning alone. ALA falls from 68.50% to 22.60% (AF3) and from 70.25% to 15.95% (Qwen2-Audio) with PCLM + DPO (Appendix C.3). On the MMSU paralinguistic subset, AF3 improves from 37.74% to 54.78% with PCLM + DPO, and Qwen2-Audio from 34.37% to 65.18% with PCLM alone and 63.26% with PCLM + DPO.

Not every comparison favours the full recipe. DPO slightly lowers Qwen2-Audio’s MMSU paralinguistic score relative to PCLM alone (65.18% to 63.26%) and lowers AF3 intonation (49.50% to 47.00%). The SFT-without-PCLM control has the highest AF3 speed accuracy (81.00% vs 73.00% for PCLM + DPO). The paper says PCLM improves all 10 tasks for both base models, but in Table 2 Qwen2-Audio speaker counting goes from 12.50% (base) to 11.50% with PCLM (checked against the table image); it reaches 32.50% only after DPO. On full MMSU, AF3 slips from 51.43% to 50.62% while Qwen2-Audio improves from 50.82% to 55.43%. Without PCLM, adding DPO to SFT lowers Qwen2-Audio’s VoxParadox average (45.70% to 44.50%) but raises AF3’s (34.80% to 40.30%).

Human validation (Table 3). Prolific annotators (60 per item, 200 items) identify the transcript-asserted label with 88.7% per-response accuracy and the acoustic label with 80.9%, far above the 30.85% best model score. Agreement on the acoustic label is weaker for some tasks: majority-vote accuracy is 50.0% on speaker identity and 60.0% on emotion.

Novelty Assessment

The benchmark idea, contradicting lexical and acoustic cues by construction, has a close precedent in LISTEN (EACL 2026), which decorrelates lexical and acoustic emotion cues, and in vision-language contradiction benchmarks; VoxParadox’s addition is breadth (10 tasks), deterministic control of the acoustic attribute through signal processing and SSML rather than style prompts, and the ALA metric that directly measures transcript following. The probing analysis adapts the “VLMs overlook their visual representations” methodology to audio LLMs, and the depth-wise degradation in ASR-pretrained encoders echoes earlier layer-wise analyses of speech encoders; its value is in tying these to an end-to-end failure and confirming it across encoders. PCLM is a modest architectural contribution: prompt-conditioned layer weighting is close to prior learned layer aggregation (VARAN) and prompt-aware encoder mixtures (PaM), but applied within a single encoder at the audio-LLM interface. DPO is used off the shelf. The strongest element is the experimental design that separates the contribution of the mixer from fine-tuning via SFT-only and SFT + DPO controls on two base models, with gains transferring to an external benchmark (MMSU) that was not used for training.

Field Significance

Moderate. The paper provides a controlled demonstration that current speech LMs rely on the transcript rather than the acoustics for paralinguistic questions, localizes the loss to both encoder depth and LLM utilization, and shows that intermediate-layer access plus preference tuning recovers much of the gap on two open models. The benchmark and the ALA metric can serve as a stress test that complements naturalistic paralinguistic benchmarks, but the mitigation is a post-hoc fix evaluated mostly on synthetic, adversarially constructed speech.

Claims

  • supports: Speech LMs built on ASR-pretrained audio encoders tend to answer paralinguistic questions from lexical content rather than acoustic evidence when the two conflict.

    Evidence: Across 12 audio LLMs, VoxParadox GT accuracy averages 15.33% while agreement with the transcript-implied label averages 64.34%; GPT-4o Audio reaches 81.55% ALA at 8.60% GT, and reversing the audio raises AF3 GT accuracy from 17.40% to 38.80%. (§4.1.1, Table 1, Figure 2, Figure 3, Appendix B.2.2, Table 6)

  • complicates: Benchmarks in which lexical and acoustic cues are correlated can overstate the paralinguistic perception of speech LMs.

    Evidence: Step-Audio-R1 scores 54.51% on the MMSU paralinguistic subset but 17.45% on VoxParadox; Gemini 2.5 Flash drops from 51.05% to 24.70%; Audio Flamingo 2 shows the opposite ranking (27.44% MMSU, 30.85% VoxParadox). (§4.1.1, Table 1)

  • supports: Paralinguistic information is often better preserved in intermediate layers of ASR-pretrained speech encoders than in the final layer typically passed to an LLM, and can be further attenuated at the encoder-LLM projection.

    Evidence: Layer-wise probes on frozen AF3 peak in early or middle encoder layers for pitch, volume, speed and range comparisons and drop at the interface; Qwen2-Audio and HuBERT show the same degradation, while a contrastive CLAP encoder does not. (§4.2.2, Figure 4, Appendix B.5, Figure 8)

  • supports: Prompt-conditioned mixing of multiple audio-encoder layers at the audio-LLM interface improves paralinguistic understanding beyond what fine-tuning on the same data achieves with the final layer alone.

    Evidence: With PCLM, AF3 reaches 60.00% and Qwen2-Audio 68.65% VoxParadox GT accuracy versus 34.80% and 45.70% for SFT without PCLM; static concatenation of intermediate layers without LLM tuning gives only 19.75% to 20.80%. (§5.1.2, Table 2, Appendix B.6, Table 7)

  • refines: Preference optimization that prefers correct options over sampled distractors further reduces transcript following in speech LMs, but its benefit depends on the audio interface and is not uniform across tasks or benchmarks.

    Evidence: DPO on top of PCLM raises VoxParadox averages (AF3 60.00% to 65.20%, Qwen2-Audio 68.65% to 72.30%) and cuts ALA to 22.60% and 15.95%, yet lowers Qwen2-Audio’s MMSU paralinguistic score (65.18% to 63.26%) and, without PCLM, slightly lowers Qwen2-Audio’s VoxParadox average (45.70% to 44.50%). (§5.2, Table 2, Appendix C.2, Table 8, Appendix C.3)

Limitations and Open Questions

  • Synthetic, adversarial-only benchmark. All VoxParadox audio is TTS output (ElevenLabs, GPT-4o TTS, Azure) with signal-processing edits, and items are built to contradict the transcript. The authors acknowledge it complements rather than replaces naturalistic benchmarks. The VoxCeleb2 probing replication addresses representations, not end-to-end accuracy on natural speech.
  • Label reliability varies by task. Human majority-vote agreement with the acoustic label is 50.0% for speaker identity, 60.0% for emotion and 66.7% for volume (Table 3), so low model accuracy on those tasks mixes model failure with label ambiguity. Emotion items are filtered by a single wav2vec 2.0 classifier.
  • Training-data proximity. VoxParadox is excluded from training, but the VoxCeleb2-derived training tasks reuse the same task taxonomy, MCQ format and signal-processing post-processing as VoxParadox (Appendix D.1), so part of the VoxParadox gain may reflect format familiarity. The MMSU paralinguistic gains (+17.04 for AF3, +28.89 for Qwen2-Audio) are the cleaner transfer evidence.
  • Post-hoc fix. PCLM is retrofitted to pretrained models with a frozen encoder; the authors expect larger gains from multi-layer access during pretraining, which is untested. Effects on general ability are mixed (AF3 full MMSU 51.43% to 50.62%) and were measured only on MMSU, not on spoken dialogue or generation.
  • Parsing-based scoring. Model outputs are matched with regular expressions; SALMONN’s low scores are attributed to parse failures, which also affects comparability for other verbose models.
  • Code and demo availability are not stated beyond a project page.

Wiki Connections

  • Spoken Language Model — Benchmarks 12 speech LMs on adversarial real-audio inputs and trains a multi-layer audio interface plus DPO on two of them.
  • Evaluation Metrics — Introduces the VoxParadox benchmark and the adversarial-label agreement metric that quantifies transcript following in paralinguistic QA.
  • RLHF Speech — Applies DPO to a speech LM to prefer acoustically supported answers, with ablations showing the gain depends on the audio interface.
  • Audio Flamingo 3 — Primary base model for probing, the concatenation pilot and PCLM + DPO.
  • Qwen2-Audio — Second base model for PCLM + DPO and a probing target with a Whisper encoder.
  • Step-Audio-R1 — Benchmarked baseline with the highest MMSU paralinguistic score yet low VoxParadox accuracy, the paper’s key evidence that standard benchmarks miss transcript following.
  • GPT-4o — Closed-source baseline with the largest ALA-minus-GT gap; its TTS also synthesizes many benchmark items.
  • Qwen3-Omni — Omni baseline showing 80.65% transcript agreement at 10.60% GT accuracy.
  • CP-Bench — Prior in-the-wild contextual and paralinguistic benchmark that VoxParadox complements with controlled contradiction.
  • Frozen LLMs Perceive Paralinguistics — Prior evidence that LLMs can encode paralinguistic information yet fail to use it, consistent with the utilization gap found here.
  • Whisper — Used to verify transcript fidelity of every synthesized clip and the source of the ASR-pretrained encoders whose deep layers lose paralinguistic cues.