arXiv · 2026 · Preprint
Linkai Peng et al. (University of Connecticut / The University of Texas at Austin) · → Paper · Demo: ✗ · Code: ?
Splits the distance between a speech LM’s prompted emotion answer and what a linear probe can decode from the same answer-position hidden state into three additive terms (endpoint validity, a decision-rule gap over the option logits, and a readout-coverage gap), and shows that the two explanatory gaps call for different repairs.
Problem
Speech LMs are increasingly scored on paralinguistic tasks such as emotion recognition by the accuracy of their prompted answers. A low score does not say where the failure happens. The emotion evidence may never reach the language model; it may be in the hidden state but not exposed through the answer readout; or it may reach the option logits and be misread by a poorly calibrated decision rule. Existing benchmarks (Dynamic-SUPERB, AIR-Bench, SD-Eval) and lexical-versus-acoustic cue studies establish behavioral gaps but do not localize them, and probing studies of speech encoders show decodability without testing whether the model uses that information.
Method
The paper builds a “generation-aligned diagnostic ladder” anchored to a single event: the first answer token in a four-option multiple-choice emotion prompt (happy, sad, angry, neutral). At that step it records the emitted token, the four option logits, and the post-normalization hidden state, and computes four accuracies on identical rows:
- A_gen: the model’s actual greedy generation over the full vocabulary, with off-format answers scored wrong.
- A_opt: the argmax over the four option logits only.
- A_aff: a learned affine reader on the three reference-relative option-logit contrasts.
- A_state: a regularized affine reader on the full answer-position state.
The successive differences telescope exactly: A_state − A_gen = Δendpoint (A_opt − A_gen) + Δdecision (A_aff − A_opt) + Δcoverage (A_state − A_aff). All three gaps are defined so that a positive value means the higher rung beats the lower one. Learned readers use five speaker-disjoint folds, and intervals are speaker-clustered bootstrap 95% CIs without family-wise correction. The authors treat Δcoverage as a performance gap, not a pure geometric quantity, because the state reader sees far more input dimensions than the contrast reader.
Three follow-up analyses test the two explanatory gaps:
- Label-free logit correction (decision rule). The paper reuses the existing batch-calibration estimator. It averages the four-option probabilities over the unlabeled evaluation batch, converts them into centered offsets, adds them to the option logits at the first answer step only, and then resumes ordinary full-vocabulary generation. The emitted answer is re-scored. The procedure is label-free but transductive.
- Readout-coverage localization. The rows of the output head for the four option tokens define a native readout space (rank 3 for the relative contrasts, rank 4 for the full option span). At an intermediate layer L*, chosen on training speakers by logit-lens accuracy, the paper compares held-out emotion decoding from the native rank-3 space, from random rank-3 subspaces in its orthogonal complement (20 draws), and from a supervised rank-3 subspace S_decoding selected in that complement.
- Causal interventions and acoustic controls. On held-out minimal pairs (same speaker and transcript, different emotion), the paper replaces the receiver’s component in a rank-4 subspace with the donor’s. It measures answer change and donor following relative to a random rank-4 arm. Acoustic controls decode from 10 or 20 clip-level descriptors, residualize S_decoding on those descriptors linearly or with gradient-boosted trees, and RMS-equalize the input clips before re-extracting states.
The evaluated systems are Qwen2.5-Omni-7B, Qwen2-Audio-7B-Instruct, Audio-Flamingo-3, Kimi-Audio-7B-Instruct and Phi-4-multimodal-instruct (5.6B to 9.8B parameters by their model cards, Table S1). Each is queried with its official chat template under four Latin-square prompt variants that rotate the emotions through the option letters. No model is trained or fine-tuned.
Key Results
Across the ten system-corpus conditions, mean generated accuracy is 0.474 and mean full-state probe accuracy is 0.752, a 27.8-point gap (§I). Table I gives the ladder per condition:
- Δendpoint is effectively zero everywhere. A_gen equals A_opt to four decimals in nine conditions and differs by 0.0001 on Phi-4-MM × VESUS. The strict parser rejects no uncorrected generation (Table S4), so the option-restricted view is a faithful anchor.
- Both explanatory gaps are positive in all ten conditions, with CIs excluding zero. Δdecision ranges from +0.0264 (Phi-4-MM × VESUS) to +0.2067 (Qwen2.5-Omni × CREMA-D). Δcoverage ranges from +0.0062 (Audio-Flamingo-3 × CREMA-D) to +0.3679 (Phi-4-MM × CREMA-D).
- The dominant gap varies by condition. Qwen2-Audio × CREMA-D is decision-dominant (+0.197 vs. +0.060). Phi-4-MM × CREMA-D is coverage-dominant: generation reaches 0.279, the best affine rule over its option logits only 0.354, and the full state 0.722. Qwen2.5-Omni × CREMA-D loses heavily at both steps (+0.207 and +0.138).
- Robustness. A multilayer perceptron on the same three contrasts still leaves a state advantage of +0.0053 to +0.3464. A matched nested-CV budget for both readers changes Δcoverage by at most 0.0164, and delete-one-speaker jackknifing keeps both gaps positive in every replicate (§S4, Tables S5-S6).
The label-free logit correction raises emitted accuracy in all ten conditions, by +0.0098 (Audio-Flamingo-3 × CREMA-D, from a 0.906 baseline) to +0.1392 (Qwen2.5-Omni × CREMA-D, 0.5265 to 0.6657). It recovers 0.27 to 0.87 of the supervised Δdecision (§V-B, Table S7). The corrected option is emitted on 95.53% to 100% of rows. Parseability is unchanged except for a 0.0019 loss on Qwen2.5-Omni × VESUS. According to the discussion, the three largest decision-rule gaps also give the three largest correction gains (§VI-A); Table S7 matches this (Qwen2.5-Omni, Qwen2-Audio and Kimi-Audio on CREMA-D).
Emotion information remains decodable outside the native readout. Held-out S_decoding accuracy is 0.481 to 0.949 and beats the random rank-3 reference in every condition (Table S8, Table S10). The native readout V3 beats random by +0.037 to +0.272 in nine conditions. Phi-4-MM is the exception: +0.022 on CREMA-D and −0.018 [−0.033, +0.001] on VESUS, so its option contrasts carry little more emotion information than a random slice.
That information is rarely used. Replacing the native rank-4 option span raises answer change over random by +0.0181 to +0.3631 in all ten conditions. Replacing the supervised readout-external subspace yields significant answer change in five conditions, never above +0.0206 (§V-D, Table S11). Only Qwen2-Audio × CREMA-D shows significant donor following (+0.0106). In that one condition, a depth sweep finds the external effect peaks at layer 20 and fades toward the output, while the readout-aligned effect grows to +0.4983 (Fig. 5). Under acoustic controls, linear residualization cuts S_decoding accuracy by 0.039 to 0.255. After nonlinear residualization on 20 descriptors, accuracy stays at 0.347 to 0.752, above the 0.25 chance level. Loudness equalization changes it by at most 0.022 (§V-E).
The comparisons are internally fair: every rung uses the same rows, the same parser and speaker-disjoint splits, and the readout-external comparisons are rank-matched with random controls. Coverage is narrow: one English four-way emotion task, two acted corpora, and single-token multiple-choice answers.
Novelty Assessment
The contribution is a diagnostic framework, not a new model. The exact telescoping split of the behavior-to-decodability distance into endpoint, decision-rule and readout-coverage terms, anchored to one verified answer token, is the new element for speech LMs. The paper notes that a related knowledge-prediction gap has been reported for text LLMs on multiple-choice questions. The logit correction is an existing estimator (batch calibration) applied as a generation-time behavioral test rather than offline rescoring, so it is not a new inference procedure. The availability-versus-use framing and the minimal-pair subspace replacement follow standard probing and activation-patching practice. Their contribution here is careful controls: a fixed, label-free native readout space, rank-matched random subspaces, speaker-disjoint evaluation and acoustic residualization. Breadth is modest: five systems, four of them on Qwen-family backbones, and one task.
Field Significance
Low — The paper provides a reusable protocol for evaluating speech LMs on paralinguistic multiple-choice tasks: measure generated answers, option logits and hidden states at the same answer event. It shows that low emitted emotion accuracy can hide substantial decodable evidence, and that part of the loss can be fixed cheaply by calibrating the option logits while the rest cannot. Its evidence is limited to one task, two acted English corpora and five systems from two backbone families.
Claims
- supports: Prompted answer accuracy of speech LMs on paralinguistic tasks substantially underestimates how much task information is linearly decodable at the answer position.
Evidence: Across five speech LMs on CREMA-D and VESUS four-class emotion, generated accuracy averages 0.474 while a speaker-disjoint linear probe on the answer-position state averages 0.752. Phi-4-MM on CREMA-D generates at 0.2793 against a 0.7223 probe. (§V-A, Table I)
- supports: Part of the gap between speech LM answers and their internal paralinguistic evidence comes from miscalibrated decisions over the answer-option logits, which label-free calibration can partly repair at generation time.
Evidence: The decision-rule gap (affine reader on option-logit contrasts minus option argmax) is positive in all ten conditions (+0.0264 to +0.2067). A batch-estimated, label-free option offset applied at the first answer step raises emitted accuracy in all ten conditions (+0.0098 to +0.1392), recovering 0.27 to 0.87 of that gap with parseability loss only on Qwen2.5-Omni × VESUS (0.0019). (§V-B, Table I, Table S7)
- complicates: Paralinguistic information that remains decodable in a speech LM’s hidden state is not necessarily used by the model to form its answer.
Evidence: Supervised emotion directions outside the option-token readout span decode held-out speakers at 0.481 to 0.949, above same-rank random subspaces in every condition. Replacing them with a minimal-pair donor’s components changes answers by at most +0.0206 over random, against +0.0181 to +0.3631 for the native readout span. Donor following is significant in only one condition. (§V-C, §V-D, Table S8, Table S11)
- refines: Whether a speech LM’s paralinguistic failure lies mainly at the decision rule or at the readout depends on the model and corpus, so a single accuracy score cannot select the right repair.
Evidence: Qwen2-Audio on CREMA-D is decision-dominant (+0.197 decision vs. +0.060 coverage), while Phi-4-MM on CREMA-D is coverage-dominant (+0.075 vs. +0.368). Phi-4-MM’s native option contrasts decode emotion barely better than a random subspace (+0.022 on CREMA-D, −0.018 n.s. on VESUS). (§V-A, §V-C, Figure 2, Table S8)
- complicates: Probe-based evidence that speech LMs retain paralinguistic information can partly reflect surface acoustic statistics such as recording level, so such evidence needs acoustic controls.
Evidence: Linear residualization of 10 or 20 clip-level acoustic-prosodic descriptors reduces held-out readout-external decoding by 0.039 to 0.255, and recording level alone separates several class pairs. Decoding stays above chance (0.347 to 0.752) after nonlinear residualization, and loudness equalization changes it by at most 0.022. (§V-E, Figure 6)
Limitations and Open Questions
- Narrow task and model coverage. Only five systems, covering two language-model families (four on Qwen-family backbones), are tested. The task is English four-way acted emotion classification with single-token multiple-choice answers. Spontaneous speech, other languages, other paralinguistic attributes and free-form answers are untested. VESUS has only 10 speakers (5 clusters in the intervention analysis), which widens its intervals.
- Δcoverage conflates information and reader capacity. The full-state reader sees the whole d-dimensional state while the contrast reader sees three numbers. A larger Δcoverage can therefore reflect the bigger input space as well as unexposed information. The rank-matched analyses address this only in part.
- Transductive correction. The logit offsets are estimated from the unlabeled evaluation batch itself. They need a batch of target-domain data, must be re-estimated per domain, and cannot correct a single isolated query.
- Bounded causal conclusions. The weak external-intervention effects apply to the selected rank-4 linear directions. Distributed or nonlinear use of readout-external information is not ruled out. The mid-stack routing pattern comes from one condition (Qwen2-Audio × CREMA-D).
- Unmeasured acoustics. The controls cover recording level and up to 20 descriptors, so the remaining decodable signal may still include unmeasured acoustic cues rather than an abstract emotion representation.
- No repair is tested for the coverage gap. Readout adapters, output-mapping fine-tuning and auxiliary supervision are proposed but not evaluated.
Wiki Connections
- Spoken Language Model — Gives a diagnostic protocol for speech LMs that feed real emotional speech into five existing systems and localize where paralinguistic evidence stops affecting the emitted answer.
- Qwen2-Audio — Evaluated system; the most decision-dominant condition on CREMA-D and the only one with significant readout-external donor following, which motivates the depth sweep.
- Qwen2.5-Omni — Evaluated system; has the largest decision-rule gap (+0.2067) and the largest logit-correction gain (+0.1392) on CREMA-D.
- Audio Flamingo 3 — Evaluated system; the highest generated accuracy (0.906 on CREMA-D) and the smallest readout-coverage gap.
- Kimi-Audio — Evaluated system; shows a large separation between generated behavior and state decodability, with a weakly informative native readout on VESUS.
- Phi-4-Multimodal — Evaluated system (Phi-4-multimodal-instruct; the paper’s reference list cites the text-only Phi-4 report); the most coverage-dominant system, whose option contrasts carry almost no emotion information beyond a random subspace.
- AIR-Bench — Cited as a behavioral benchmark with paralinguistic tasks that establishes accuracy gaps without localizing them, the limitation this paper’s ladder targets.
- emotion2vec — Cited as encoder-side evidence that emotion is linearly recoverable from speech representations; the paper extends this kind of probing to the answer position inside speech LMs and tests whether the information is used.