arXiv · 2026 · Preprint
Hao Yang et al. (Monash University / University College London) · → Paper · Demo: ✗ · Code: ✗
Identifies where paralinguistic signals (age, gender, emotion) and semantic understanding are encoded across the transformer layers of large audio language models, then uses that diagnosis to design a selective-layer fine-tuning protocol that resurfaces paralinguistic-aware responding without full-model retraining.
Problem
Large audio language models (LALMs) extend LLM-style interaction to the speech modality, but they largely inherit a content-centred interaction protocol from text LLMs: responses are conditioned on what a query says, not on paralinguistic cues (age, gender, emotional state) implicit in how it is spoken. The paper frames this gap as more than a missed empathy opportunity: it identifies a child-safety failure mode in which LALMs give adult-appropriate step-by-step guidance for physically risky activities (e.g., electrical or kitchen tasks) regardless of whether the speaker is a child, because paralinguistic cues are not integrated into the model’s semantic understanding. Prior paralinguistic-aware LALM work (ParaS2S, GOAT-SLM, ReEmpathy) either focuses narrowly on emotion, relies on audio-centred evaluation that cannot generalize to LALMs with non-trainable audio synthesizers, or lacks a discriminative way to measure whether a response is actually attribute-conditioned rather than generically acceptable.
Method
The paper first runs five layer-wise diagnostic analyses on two open LALMs, Qwen2.5-Omni and Kimi-Audio, to localize where paralinguistic and semantic information live in the model’s 28 transformer layers. Layer representations are obtained by mean-pooling audio hidden states at each layer. (1) A paralinguistic probe trains linear classifiers per layer on age/gender/emotion attribute prediction. (2) An intent-classification (IC) probe on Fluent Speech Commands measures layer-wise semantic understanding. (3) An IC cosine-similarity analysis compares within-intent versus cross-intent representation similarity to isolate semantic content independent of lexical overlap. (4) An age-aware cosine-similarity analysis uses age-declaration-augmented queries and a purpose-built child-safety dataset (7 risk scenarios, 70 samples, synthesized with commercial TTS across 11 child and 11 adult voices) to test whether models form distinct semantic understanding conditioned on speaker age. (5) A logit-lens analysis probes intermediate-layer predictions against the final output to rule out layers whose apparent “semantic” signal is actually next-token generation machinery rather than paralinguistic-semantic integration. Across all five analyses, both models show a consistent pattern: layers 0-6 carry strong paralinguistic signal that sharply degrades from layer 7, while layers 7-14 show a corresponding rise in semantic/intent signal that plateaus by layer 14.
Building on this diagnosis, the paper proposes a paralinguistic-enhanced fine-tuning (PE-FT) protocol with two components. First, selective-layer fine-tuning applies LoRA-based supervised fine-tuning (standard cross-entropy loss) only to layers 0-14 (the identified paralinguistic and semantic-understanding layers), freezing the remainder and the audio encoder, rather than tuning all layers. Second, an auxiliary dual-level classification head (ADCH) is attached to the layer-14 representation during training only (discarded at inference): a category head predicts which paralinguistic category (age, gender, emotion) a query pertains to, and three category-specific attribute heads predict the corresponding speaker attribute (e.g., child vs. adult). The combined training objective is L = L_SFT + λ(L_cate + L_attr), with λ = 0.5. Training data is a self-constructed corpus of 9,000 audio recordings (1,500 GPT-4.1-generated text samples per category, each rendered as paired audio under contrasting attributes using Typecast TTS for age, Google TTS for gender, and gpt-4o-mini-tts for emotion), built by refining the (unreleased) ParaS2S data pipeline. Because some existing LALMs (e.g., Qwen2.5-Omni) have non-trainable audio synthesizers, the paper also introduces two new evaluation metrics computed from an LLM judge (GPT-4.1): PA-score (mean of per-response judgments in {-1, 0, 1} for wrong/absent/correct attribute-conditioning) and PA-rate (percentage of responses judged attribute-correct).
Key Results
On the self-constructed evaluation set (Table 2), vanilla Qwen2.5-Omni and Kimi-Audio sit near chance on paralinguistic awareness (PA-score close to 0, PA-rate ~50% on age, ~13-15% on gender). Full-layer fine-tuning (layers 0-27) substantially improves all three categories but under-performs on emotion (PA-score 0.393 for Qwen2.5-Omni). Selective-layer fine-tuning (layers 0-14, no ADCH) matches or exceeds full-layer tuning with fewer trainable parameters, e.g., age PA-score 0.96 vs. 0.915 for Qwen2.5-Omni, and emotion PA-score 0.46 vs. 0.393. Adding ADCH (the full PE-FT setting) further improves gender and emotion for both models (e.g., Qwen2.5-Omni emotion PA-score 0.503, Kimi-Audio age PA-rate 97.0%) and best preserves general capability on VoiceBench’s helpfulness score relative to the other two fine-tuning settings. A layer-range ablation (Table 3) confirms that fine-tuning only deep layers (15-27) fails, particularly on emotion (near-zero or negative PA-score), while the identified layers 0-14 give the best overall trade-off. On the child-safety dataset, held out of PE-FT training entirely, PA-rate rises from 7.14%/4.29% (vanilla Qwen2.5-Omni/Kimi-Audio) to 97.14%/98.57% after PE-FT, indicating the learned paralinguistic sensitivity transfers to unseen topics. A generalization test to unseen speakers (Table 6) shows Qwen2.5-Omni maintaining >90% gender PA-rate across seen- and unseen-speaker TTS settings, while Kimi-Audio drops to 68.5%/75.0%.
Novelty Assessment
The contribution is primarily an evaluation and training-recipe advance rather than a new model architecture. The five-way layer-wise probing pipeline (combining classification probes, cosine-similarity contrasts, and logit lens) is a genuine interpretability contribution that converges on a specific layer boundary (0-6 paralinguistic, 7-14 semantic) across two different LALM families, which is more rigorous than typical single-probe layer analyses. PE-FT itself combines known ingredients (LoRA fine-tuning, an auxiliary classification loss) rather than introducing a new architecture; its novelty is in using the layer-wise diagnosis to motivate which layers to tune and where to place the auxiliary head, and in showing this beats brute-force all-layer fine-tuning. The PA-score/PA-rate metrics and the child-safety dataset are useful because they let the paper measure fine-grained attribute-conditioning where the prior ParaS2S score, restricted to audio-centred judgment and full-model-tunable synthesizers, could not discriminate between fine-tuning settings that looked similar on it.
Field Significance
moderate — this paper contributes a layer-localization methodology and a lightweight fine-tuning recipe for a specific, underexplored failure mode (paralinguistic neglect, including a concrete child-safety risk) in large audio language models, evaluated on two current open LALMs. It provides evidence that where you fine-tune matters more than how much you fine-tune for recovering an omitted capability, and it introduces evaluation tooling (PA-score, PA-rate, a child-safety benchmark) that other paralinguistic-awareness work could reuse. The scope is narrow (two models, a self-constructed and partly unreleased dataset) and the layer boundaries are explicitly noted by the authors as model-specific rather than universal.
Claims
- supports: In large audio-conditioned language models, paralinguistic attribute information and content-semantic understanding are localized to distinct, identifiable layer ranges rather than being uniformly distributed across depth.
Evidence: Linear probing across 28 layers on Qwen2.5-Omni and Kimi-Audio found stable, high classification accuracy for age/gender/emotion in layers 0-6, followed by a sharp drop at layer 7 toward near-random accuracy, while intent-classification probing accuracy rose sharply at layer 7 and plateaued through layer 14; the pattern was corroborated by cosine-similarity and logit-lens analyses. (§4.1-§4.3, Figures 1-5)
- supports: Fine-tuning only the layers identified as jointly encoding paralinguistic and semantic information can recover a suppressed model capability more effectively than fine-tuning the entire model, while using fewer trainable parameters.
Evidence: Selective-layer tuning (layers 0-14) outperformed all-layer fine-tuning (layers 0-27) on both LALMs, e.g., age PA-score of 0.96 vs. 0.915 for Qwen2.5-Omni and gender PA-score of 0.97 vs. 0.95 for Kimi-Audio; deep-layer-only tuning (15-27) failed on emotion for both models. (§6.2-§6.3, Tables 2-3)
- complicates: Generic audio-centred response-quality scores can fail to discriminate between fine-tuning strategies that differ substantially in whether they actually condition responses on paralinguistic attributes.
Evidence: The ParaS2S quality score improved similarly across full-layer, selective-layer, and PE-FT settings for both models, while PA-score and PA-rate showed large differences between the same settings, motivating the paper’s introduction of the latter two metrics. (§6.2)
- complicates: Paralinguistic-aware fine-tuning gains do not generalize uniformly to unseen speakers across model families, suggesting learned attribute sensitivity can partly reflect speaker-specific training cues rather than fully generalized attribute understanding.
Evidence: On a cross-category unseen-speaker gender test, PE-FT Qwen2.5-Omni maintained PA-rate above 90%, while PE-FT Kimi-Audio dropped to 68.5% (Typecast voices) and 75.0% (GPT-TTS voices), a decline the authors attribute to weaker underlying gender probing signal in vanilla Kimi-Audio. (§6.3, Table 6, Figure 1)
Limitations and Open Questions
The paper's gender analysis treats gender as a binary category inferred solely from vocal characteristics, which the authors acknowledge does not capture self-identified gender and may not generalize across individuals with atypical vocal characteristics for their identified gender; the gender results should be read as findings about a simplified experimental abstraction, not a general claim about gender perception.
Additional open questions: the identified layer boundaries (0-6 paralinguistic, 7-14 semantic) are derived from two specific LALMs and the authors explicitly note boundaries may vary across other model families, so the PE-FT recipe as specified is not guaranteed to transfer without re-running the layer-wise diagnostic. The training and evaluation sets (9,000 and 1,200 recordings respectively) are self-constructed via GPT-4.1 prompting and commercial TTS voices rather than drawn from naturalistic speech, and are not stated to be publicly released. Paralinguistic-awareness judgments (PA-score, PA-rate) rely on an LLM judge (GPT-4.1) rather than human raters, so reported gains reflect an automated proxy for appropriateness rather than validated human judgments of empathetic or safe responding.
Wiki Connections
- Spoken Language Model — probes and selectively fine-tunes existing large audio language models (Qwen2.5-Omni, Kimi-Audio) to localize and restore paralinguistic sensitivity that is suppressed by content-centred modality tuning.
- Speech-to-Speech — targets the dialogue sub-paradigm, aiming to make audio-in/audio-out conversational agents condition their responses on a speaker’s implicit age, gender, and emotional context rather than query content alone.
- Evaluation Metrics — introduces PA-score and PA-rate as LLM-judged metrics to discriminate paralinguistic-aware responses, addressing a gap where existing audio-centred scores could not distinguish between fine-tuning strategies with substantially different attribute-conditioning behavior.
- Self-Reflective Alternating Inference for Empathetic Spoken Dialogue — cited as related recent work on empathetic spoken dialogue; contrasted as improving empathy through inference-time self-reflective reasoning rather than layer-localized fine-tuning.
- OSUM-EChat — cited as related work on understanding-driven empathetic spoken chatbots; contrasted as an end-to-end architectural approach to empathetic response generation rather than a diagnostic-plus-fine-tuning protocol applied to existing LALMs.