arXiv · 2025 · Preprint
Yadavalli, Pimentel, Regev, Wilcox & Warstadt (UC San Diego / ETH Zürich / MIT / Georgetown University) · → Paper · Demo: ? · Code: ?
Proposes an information-theoretic framework that quantifies not just how much information prosody conveys beyond text, but specifically what that information is about, applying it to sarcasm, emotion, and questionhood using natural speech from television and podcasts.
Problem
Prosody, the melody of speech, conveys meaning that written text alone does not capture, but prior information-theoretic work on the text-prosody relationship has been limited to estimating the overall redundancy between the two channels (how much they overlap), which says nothing about what unique information prosody actually conveys, whether it signals emotion, syntax, speaker intent, or something else. This gap matters for practical spoken-language technology: as LLM-based systems increasingly interact through voice, understanding exactly which dimensions of meaning depend on audio versus text, and by how much, is directly relevant to designing systems that use or generate prosody appropriately, particularly when context beyond the current sentence is unavailable, as in a single voice-assistant turn.
Method
The paper reframes the question around a specific meaningful feature Y (e.g., whether an utterance is sarcastic) rather than the raw redundancy between text T and prosody P. Since mutual information between two continuous channels (audio and prosody) is difficult to estimate reliably and uninterpretable in isolation, the authors instead estimate mutual information between each channel and a discrete labeled feature Y, using the identity MI(X;Y) = H(Y) - H(Y|X), where the conditional entropy H(Y|X) is approximated as the cross-entropy of a classifier fine-tuned to predict Y from channel X. Because no existing architecture accepts isolated prosodic features as input, the authors approximate prosody’s unique contribution, MI(Y;P|T), using the full audio signal’s unique contribution instead, MI(Y;A|T) = MI(Y;A) - MI(Y;T), reasoning that the audio information not attributable to either text or prosody (e.g., speaker identity, background noise) is likely small relative to the quantity of interest. Text-only classifiers are built by fine-tuning GPT-2 (small through xl) on transcripts; audio-only classifiers are built by fine-tuning Whisper (tiny through large, encoder only) and wav2vec 2.0 on raw audio. The method is applied to three case studies: sarcasm detection (MUStARD, 690 TV sitcom utterances), affect classification (MSP-Podcast, 10 emotion categories), and questionhood classification (a new binary dataset curated from MSP-Podcast via sentence segmentation and forced alignment).
Key Results
For both sarcasm and affect, mutual information between the audio channel and the target feature exceeds mutual information between text and the feature by over an order of magnitude (uncertainty coefficients of 0.22 and 0.20 respectively), and audio-based classifiers consistently outperform text-only classifiers in raw accuracy. The estimated unique contribution of prosody (approximated via the full audio signal, conditioned on text) is 0.20 bits for sarcasm and 0.46 bits for affect, a large fraction of each feature’s total audio-conveyed information. For questionhood, the audio-to-text mutual information ratio is only about 2.4x, far smaller than for sarcasm or affect, consistent with the fact that English questions are also strongly marked syntactically (subject-auxiliary inversion, wh-words), which is redundant with the prosodic cues. Larger audio-only models generally perform better (though Whisper large does not consistently beat Whisper medium), while text-only model size has comparatively little effect on classifier performance.
Novelty Assessment
The genuine methodological contribution is reframing prosodic information estimation around a discrete, labeled meaning dimension rather than the raw, continuous prosody signal: this sidesteps two specific problems in prior work, the uninterpretability of differential entropy for continuous variables, and the instability of density estimation for conditional prosody distributions, while requiring only standard classifier fine-tuning with cross-entropy loss. The tradeoff is that the approach requires channel-specific models (a speech encoder for audio, a text LM for text) and labeled data for each meaning dimension studied, rather than the automatically-extracted prosodic features and text-only models used by the prior redundancy-estimation approach it responds to.
Field Significance
moderate — This paper does not build, train, or evaluate any speech generation system; its relevance to this wiki is as methodology and empirical grounding for prosody-aware speech technology rather than as a synthesis contribution. Its central empirical finding, that prosody’s marginal informational contribution varies substantially by meaning dimension (large for sarcasm and affect, comparatively small for questionhood, at least without cross-sentence context) offers a concrete, quantified basis for prioritizing which aspects of prosody expressive TTS and dialogue systems most need to model faithfully, and the general information-theoretic framework is explicitly designed to extend to further meaning dimensions, channels, and languages.
Claims
- supports: The audio channel of spoken communication can convey substantially more information about affective and pragmatic meaning, such as sarcasm or emotion, than the text transcript alone conveys, when only sentence-local context is available.
Evidence: Mutual information between audio and sarcasm/affect labels exceeds mutual information between text and the same labels by over an order of magnitude, with audio-based classifiers also achieving higher accuracy than text-only classifiers on both tasks. (§6, Figures 2-3)
- complicates: Prosody’s marginal informational contribution beyond text varies substantially by the type of meaning being conveyed, rather than being uniformly large across all pragmatic or grammatical functions prosody is known to signal.
Evidence: For questionhood, mutual information between audio and the label is only about 2.4 times greater than between text and the label, far smaller than the order-of-magnitude gap observed for sarcasm and affect, reflecting that English questions are also strongly signaled by syntax redundant with prosodic cues. (§6)
- supports: A discrete-feature-based mutual information estimation framework, which trains classifiers to predict a meaningful label from each communication channel, can quantify what specific kind of meaning a channel like prosody conveys, not merely how much raw information it shares with another channel like text.
Evidence: The framework’s entropies (H(Y) and H(Y|X)) are directly interpretable as expected information gain in bits about a specific labeled feature and require only standard cross-entropy classifier training, avoiding the uninterpretable, unstable differential-entropy estimation that prior text-prosody redundancy approaches required. (§3.2, §4)
- complicates: Estimating how much unique information prosody conveys about a meaning dimension currently requires an indirect approximation rather than a direct measurement, since no available model architecture accepts isolated prosodic features as input.
Evidence: The paper approximates prosody-specific mutual information using the full audio signal’s conditional mutual information instead, an approximation whose accuracy depends on an untested assumption that audio information beyond text and prosody (e.g., speaker identity, background noise) is minimal. (§3.2, §8)
Limitations and Open Questions
The authors restrict both audio and text to a single sentence, explicitly noting that including longer discourse context would likely increase text’s informational contribution for sarcasm and affect substantially, while prosody’s unique contribution might remain comparatively stable, since prior work suggests prosody’s unique information relative to prior text is mostly local. The sarcasm dataset is artificially class-balanced, unlike real conversational sarcasm rates, and the affect dataset’s ten discrete emotion categories are a coarse approximation of the full range of human emotional expression, both of which affect the magnitude (though not necessarily the ratio) of the reported estimates. The authors also note they did not fine-tune text models larger than GPT-2 XL due to computational cost, and acknowledge that stronger text or audio models could shift the specific quantitative estimates reported.
Wiki Connections
No qualifying concept connections: this paper’s contribution is a linguistic information-theory framework for characterizing communication channels, not a speech generation, control, or evaluation mechanism meeting any of this wiki’s tracked concept definitions.