NAACL · 2025 · Conference
Chi et al. · → Paper · Demo: ? · Code: ?
Using controlled audio manipulations on a natural-speech SQA dataset, this paper demonstrates that models can utilize prosodic cues alone for question answering, but overwhelmingly default to lexical information when it is available, even when only a small fraction of training data carries lexical content.
Problem
Spoken language understanding research has relied heavily on text-derived data and ASR transcriptions, discarding prosodic information in the process. For tasks such as Spoken Question Answering (SQA), this creates a gap: prosody (pitch, stress, rhythm, pauses) carries structural and emphatic cues that may guide comprehension, yet most models never access it. Earlier SQA datasets also predominantly used TTS-generated speech, whose prosodic characteristics do not reflect natural speech. The paper asks two targeted questions: whether prosodic information alone is sufficient for SQA, and whether models exploit prosodic cues when lexical information is simultaneously available.
Method
The study builds on the DUAL (Discrete Spoken Unit Adaptive Learning) framework (Lin et al., 2022), which bypasses ASR transcription by encoding raw audio through WavLM Large, then quantising the resulting representations into discrete units via k-means clustering (1000 clusters). The discrete token sequence is fed to a Longformer-base language model that predicts answer span start and end positions. WavLM representations are kept frozen throughout training.
To isolate prosodic versus lexical information, the authors modify the SLUE-SQA-5 dataset into three conditions. In the lexical condition, fundamental frequency and intensity are flattened to utterance-level averages, removing intonation and stress while preserving phonetic content. In the prosodic condition, a 300 Hz low-pass filter removes most consonant and vowel energy while retaining F0 contour, rhythm, and low-frequency amplitude patterns. The natural condition uses unmodified audio. All three conditions are used for training and evaluation in a cross-condition matrix, with an additional random question-context pairing experiment to assess whether prosodic cues are question-sensitive or merely highlight salient passage regions.
Evaluation uses Frame-level F1 (FF1) and Audio Overlapping Score (AOS), both measuring temporal overlap between predicted and ground-truth answer spans, on the standard test and a hand-verified test subset.
Key Results
When trained and tested within the same condition, the natural condition achieves the highest FF1 (33.27 on test), followed by the lexical condition (32.37), and then the prosodic condition (18.49). A white-noise chance baseline scores 6.03 FF1, confirming that the prosodic condition’s 18.49 reflects genuine prosodic cue utilization and not random performance (Table 2, §4.1).
Cross-condition results expose the lexical dominance clearly. A model trained on lexical data performs nearly as well as the natural-condition model on natural audio (26.72 vs. 33.27 FF1), but both collapse to around 6-9 FF1 when tested on prosodic audio. Conversely, a prosodic-trained model reaches only 8-9 FF1 on natural or lexical audio.
The lexical dominance is further confirmed in the mixing experiment (§4.2): even when 90% of training data is prosodic and only 10% is lexical, the model rapidly shifts toward lexical strategies, with evaluation loss on lexical and natural sets quickly falling to match the level of the prosodic set. Random question-context pairing in the prosodic condition drops performance from 18.49 to 9.77 FF1, indicating that prosodic cues do partially track question-relevant content rather than being entirely question-agnostic (Table 4, §4.1).
Novelty Assessment
The contribution is primarily empirical. The DUAL framework used for experiments is an existing system, WavLM is a pre-trained model applied without modification, and the SLUE-SQA-5 dataset was published separately. The novelty lies in the experimental design: a carefully controlled three-way audio manipulation (natural, lexical, prosodic) applied to a real-speech SQA corpus, combined with a cross-condition training matrix and a mixing experiment that tests whether lexical exposure overrides prosodic learning. The methodology for isolating prosodic and lexical information using low-pass filtering and F0/intensity flattening follows established psycholinguistic practice; the application to SQA model analysis is new. The paper is honest about the imperfect nature of the separation.
Field Significance
Moderate — This paper provides controlled empirical evidence that current SSL-based spoken language understanding models do not effectively integrate prosodic information even when it is behaviorally sufficient for the task. The finding that even 10% lexical training data overrides predominantly prosodic training data quantifies a known intuition and motivates architectural work on explicit prosody-lexical fusion. The relevance to speech synthesis is indirect but real: if downstream comprehension systems discard prosody, TTS evaluations that assess prosodic naturalness through comprehension tasks may underestimate the contribution of prosodic quality.
Claims
-
supports: Prosodic cues in natural speech carry information sufficient to partially guide spoken comprehension models above chance level.
Evidence: A WavLM-based SQA model trained and tested on 300 Hz low-pass filtered (prosodic condition) audio achieves FF1 18.49 on SLUE-SQA-5 test, substantially above the white-noise chance baseline of 6.03. (§4.1, Table 2)
-
complicates: SSL-based spoken language models do not effectively leverage prosodic information when lexical cues are simultaneously available, even when lexical data is a small minority of training.
Evidence: When only 10% of training data carries lexical information and 90% is prosodic, models rapidly shift toward lexical strategies, with lexical and natural evaluation losses converging near prosodic-only levels within training. (§4.2, Figure 5)
-
complicates: The use of TTS-synthesised speech in SQA training data may not capture the prosodic characteristics of natural speech, limiting the study of prosody in comprehension models.
Evidence: The paper selects SLUE-SQA-5 specifically because prior SQA datasets relied on TTS synthesis, whose prosodic properties differ from natural speech; the prosodic condition experiments are only valid under the natural-speech assumption. (§1, §3.1)
-
refines: Prosodic cues in SQA are at least partially question-sensitive rather than globally highlighting salient passage regions.
Evidence: Random pairing of questions and contexts in the prosodic condition reduces FF1 from 18.49 to 9.77, showing the model’s prosodic utilization depends on question-context alignment, not only on passage-level salience signals. (§4.1, Table 4)
Limitations and Open Questions
Warning
The prosodic and lexical conditions cannot achieve perfect information separation: the 300 Hz low-pass filter retains some residual lexical information (WER ~57% at 300 Hz cut-off), and flattening F0/intensity preserves rhythm. Results for the prosodic condition therefore reflect a lower bound on prosodic-only performance, and the lexical condition’s prosodic residue may slightly inflate cross-condition scores.
The study uses extractive SQA where answers are timestamped spans in spoken passages. This frames prosody as a localization cue; its role in open-ended or inferential comprehension tasks is not addressed. The choice of deeper WavLM layers (stronger semantic encoding) likely underrepresents prosodic information relative to earlier layers, which may encode prosody more directly. The corpus (Spoken Wikipedia) is read-aloud speech, so findings may not generalize to conversational or spontaneous speaking styles where prosody is more variable and communicatively richer.
Wiki Connections
- Prosody Control — this paper provides behavioral evidence that current models fail to integrate prosodic cues, motivating research into explicit prosody conditioning and control mechanisms.
- Self-Supervised Speech — the paper uses WavLM Large as the core speech encoder, and the layer-selection choice (deeper semantic layers vs. earlier prosodic layers) is identified as a key limitation affecting how much prosodic information the model can access.
- Evaluation Metrics — the paper uses Frame-level F1 and Audio Overlapping Score as temporal span metrics specific to spoken QA, illustrating how audio-domain evaluation requires task-adapted metrics distinct from text-based F1.