ACL 2026 (Findings) · 2026 · Conference
Yosha et al. (The Hebrew University of Jerusalem) · → Paper · → Preprint · Demo: ✓ · Code: ✓
Introduces StressTest, a studio-recorded benchmark in which the same sentence carries different meanings depending on which word is stressed, shows that open speech LMs answer it near chance, and closes much of the gap by LoRA-finetuning Qwen2-Audio on Stress-17k, a synthetic stressed-speech training set built with a commercial TTS and an automatic stress verifier.
Problem
Sentence stress (emphasis on particular words to mark contrast, focus, new information or intensity) can change what an utterance means without changing its words, as in “I didn’t take your book” versus “I didn’t take your book”. A cascade that transcribes speech and passes text to an LLM discards this cue, and speech-aware LMs that consume audio directly are, in principle, able to use it. Existing speech LM evaluation suites cover prosody only loosely, and prior stress benchmarks target word-level stress detection rather than whether a model can infer the speaker’s intended meaning from stress. The paper asks whether current speech LMs can do that inference and, if not, whether it can be taught.
Method
Benchmark. StressTest contains 101 texts, each recorded in two or three stress patterns by a single professional actor in a treated studio (218 audio samples; 85 texts with 2 interpretations, 16 with 3; 170 interpretations stress one word, 43 two words, 5 three words). Texts were written by annotators to admit stress-dependent readings, cross-checked by a second annotator, and refined by a third where needed. A complementary set, StressPresso, re-annotates 202 neutral-emotion utterances (96 texts, 4 speakers) from Expresso post hoc. The paper treats StressPresso as secondary evidence because its intended meanings were inferred after recording rather than recorded to a script, but it adds speaker and recording-condition diversity (§3.1, Appendix A).
Two tasks are defined (§3, §3.2):
- Sentence Stress Reasoning (SSR), new in this paper: given only the audio, pick which of two interpretations the speaker intended. Because models follow answer formats inconsistently, a GPT-4o judge (GPT-4o System Card) maps each free-form output to an answer before accuracy is computed. An open-ended variant drops the answer options and has the LLM judge score free-form explanations on a 1-5 scale.
- Sentence Stress Detection (SSD), an established task: given audio and the ground-truth transcript, name the stressed word(s); scored by precision, recall and F1.
Stress-17k training data (§4). GPT-4o, run as an agent in the CrewAI framework and seeded with a domain, topic and sentence type, writes a sentence with two stress patterns, a description of each implied meaning, and a short answer pair. Each stress pattern is synthesized twice with the OpenAI TTS API using random male or female voices; stressed words are marked by surrounding them with asterisks, which the authors report the TTS realizes as emphasis and which sounded more natural in preliminary tests than editing prosodic features directly. Because the TTS often misplaces or drops the emphasis, WhiStress (a Whisper-based stress detector from the same group) re-transcribes each clip and keeps only those whose detected stress matches the target. This leaves 1,311 of 4,400 clips (Table 7). Each clip is used in four prompt templates (stress detection, direct answer, elaborate-then-answer using the meaning description, and transcribe-with-stress-then-answer), giving roughly 17K training examples, about 4.5K of them from verified clips.
StresSLM (§5, Appendix B.3). Qwen2-Audio-7B-Instruct (a Whisper encoder feeding a Qwen-7B LLM) is finetuned with LoRA (rank 16, alpha 32, rsLoRA) on the query and value projections, with the speech encoder also trained. LibriLight ASR and MELD emotion-recognition samples, roughly matched in audio duration to the verified subset, are mixed in to limit forgetting. Training is staged: one epoch over all of Stress-17k, then continued training on the verified subset (1,595 steps in total, 1,261 in stage 1) on a single L40S GPU. The loss covers the full answer, including the elaboration text.
Key Results
Existing speech LMs fail at stress reasoning (Table 2). With audio only, the open speech LMs score close to the 50% chance level on SSR: Qwen2-Audio-7B-Instruct 53.2 / 51.4 (StressTest / StressPresso), SALMONN 55.9 / 52.4, LLaMA-Omni 49.5 / 52.9, Phi-4-multimodal 52.2 / 50.4, Audio Flamingo 3 56.8 / 52.9 and Qwen3-Omni-30B 64.6 / 64.8. Among proprietary systems, gpt-4o-audio reaches 68.8 / 64.8 and Gemini 2.5 Pro 77.5 / 72.7. Human listeners score 92.6 / 89.6, rising to 96.0 on both sets by three-annotator majority vote (100 sampled items per set).
Cascades beat off-the-shelf speech LMs. WhiStress feeding predicted transcripts and stress labels to text LLMs reaches 83.4 / 79.7 with gpt-4o and 76.1 / 73.2 with gpt-4o-mini, above every off-the-shelf speech LM. Compared with oracle transcripts and stress labels, the cascade loses accuracy (Llama-3.1-8B-Instruct 73.3 oracle vs. 65.5 cascade on StressTest), which the paper reads as error propagation. Text-only results track general LLM strength (gpt-4o 86.2 / 83.6 with oracle inputs; Qwen-7B-Chat 61.4 / 64.8).
StresSLM. SSR accuracy is 86.2 on StressTest and 87.6 on StressPresso. On StressTest this ties text-only gpt-4o given ground-truth transcripts and stress labels (86.2) and is above every audio-input system, including the WhiStress → gpt-4o cascade (83.4). On StressPresso it is the highest non-human score in Table 2 (next: oracle gpt-4o 83.6). Humans remain ahead on both sets (92.6 / 89.6).
Stress detection (Table 3). StresSLM has the highest SSD F1 of any speech LM on all three sets (StressTest 86.9, StressPresso 80.6, Expresso 59.1; the next best speech LM is Gemini 2.5 Pro at 48.5 / 40.7 / 39.6). It still trails the dedicated WhiStress detector on F1 everywhere (88.3 / 83.5 / 68.9), most clearly on Expresso, and its Expresso recall (68.6) is lower than Gemini 2.5 Pro’s (82.1). The baseline speech LMs show a consistent pattern of high recall and low precision (for example Gemini 2.5 Pro on StressTest: P 35.9, R 74.9), which suggests they mark many words as stressed.
Open-ended reasoning (Table 11, Figure 3). StresSLM scores 3.70 / 3.83 on the 1-5 judge scale. Gemini 2.5 Pro scores higher on StressTest (3.82) but drops to 3.29 on the multi-speaker StressPresso; the open speech LMs fall between 1.89 and 2.84. Binary SSR accuracy and open-ended scores correlate strongly across models (Figure 3), and the paper notes the correlation weakens among near-chance models.
Oracle inputs (Table 10). Giving speech LMs ground-truth stress labels alongside audio generally helps them; Gemini 2.5 Pro reaches 98.1 / 96.5 with ground-truth transcript and stress. StresSLM benefits on StressPresso (92.5 with transcript and stress, 92.0 with stress only) but not on StressTest, where it drops from 86.2 (audio only) to 83.4 and 78.8.
Retention of base skills (Table 4). Against the Qwen2-Audio-7B-Instruct starting point, StresSLM keeps or slightly improves WER (LibriSpeech clean/other 2.29 / 4.59 vs. 2.31 / 4.92; CommonVoice 10.49 vs. 11.49) and improves MELD emotion accuracy (57.6 vs. 26.4). It remains behind the non-instruct Qwen2-Audio-7B base model on all three ASR sets (1.73 / 4.01 and 8.72), though ahead of it on MELD (54.6); the paper’s “no degradation” statement is made relative to the Instruct model.
Ablations (§5.6, Tables 5, 6, 12, 13; trained without rehearsal data under a fixed compute budget). On StressTest, training only on the verified 4K subset gives the best SSR (79.3) but the lowest SSD F1 (81.4); the full noisy 17K gives the reverse (84.5 F1, 76.6 SSR); staged training sits between (85.9 F1, 78.4 SSR). On StressPresso the staged and verified-only runs are nearly identical (82.4 / 85.6 vs. 82.3 / 85.5). Training the speech encoder raises StressTest SSD F1 from 85.9 to 89.3 and SSR from 78.4 to 83.0. No task mixture wins on every metric: dropping the elaborate-reasoning task gives the best SSD F1 (90.5, above WhiStress’s 88.3) but lowers SSR to 78.4, and dropping the stress-detection task gives the best SSR (84.4) but collapses SSD precision to 45.5. A small human check of Stress-17k (20 clips per subset, 3 annotators each; Table 8) finds verified clips match their intended stress far better than unverified ones (SSD F1 78.81 vs. 50.75; SSR majority vote 85.0 vs. 70.0).
Novelty Assessment
The new element is the SSR task: testing whether a model infers intended meaning from stress, not just where the stress falls, with recordings scripted to a known meaning. StressTest is small (218 clips, one speaker) but carefully controlled, and the human ceiling and LLM-judge protocol make it usable as a diagnostic. The training recipe is an engineering integration of known pieces: LLM-generated text, prompted emphasis in a commercial TTS, filtering with the group’s earlier WhiStress detector, multi-template instruction data, and LoRA finetuning of an existing speech LM with rehearsal data. The useful finding is that this cheap synthetic pipeline transfers to real recorded speech, including the four-speaker StressPresso set. StresSLM is a specialist model; its detection component does not beat a dedicated detector in the final configuration.
Field Significance
Moderate — The paper gives clear evidence that current open speech LMs barely use sentence stress when interpreting speech, even though they receive the audio, and that a text cascade with an explicit stress detector does better. It also shows that a modest amount of verified synthetic stressed speech can teach this skill and transfer it to human recordings. The contribution is a focused diagnostic benchmark plus a reproducible data recipe, rather than a new architecture.
Claims
- supports: End-to-end speech LMs that consume raw audio often fail to use prosodic emphasis when inferring speaker intent, performing near chance on stress-disambiguated meaning.
Evidence: Six open speech LMs score 49.5-64.6 SSR accuracy on StressTest and 50.4-64.8 on StressPresso on a two-way choice where human listeners reach 92.6 / 89.6; gpt-4o-audio reaches 68.8 / 64.8. (§3.3, Table 2)
- complicates: Passing audio directly to an LLM does not by itself preserve paralinguistic information better than a cascade; a cascade with an explicit prosody transcriber can outperform end-to-end speech LMs on prosody-dependent understanding.
Evidence: WhiStress → gpt-4o reaches 83.4 / 79.7 SSR, above every off-the-shelf end-to-end model (best: Gemini 2.5 Pro 77.5 / 72.7), although the cascade loses accuracy relative to oracle stress labels (Llama-3.1-8B 73.3 vs. 65.5). (§5.2, Table 2)
- supports: Finetuning a speech LM on synthetic TTS speech with controlled emphasis can teach prosody-dependent understanding that transfers to real human recordings.
Evidence: StresSLM, trained only on OpenAI TTS clips plus ASR/SER rehearsal data, reaches 86.2 / 87.6 SSR and 86.9 / 80.6 SSD F1 on the actor-recorded StressTest and the four-speaker StressPresso, up from 53.2 / 51.4 SSR and 33.1 / 25.5 F1 for its Qwen2-Audio-Instruct starting point. (§5.1, §5.2, Tables 2-3)
- complicates: Text markup requesting emphasis in an expressive TTS system does not reliably produce the intended stress, so synthetic prosody data needs automatic verification.
Evidence: After asterisk-marked stress prompting of the OpenAI TTS, the WhiStress verifier keeps 1,311 of 4,400 clips; in a small human check, verified clips match their target stress at SSD F1 78.81 versus 50.75 for unverified clips. (§4, Appendix B.1-B.2, Tables 7-8)
- refines: Quality filtering of synthetic training data can trade one capability against another, rather than improving all of them.
Evidence: On StressTest, training on the verified subset raises SSR (79.3 vs. 76.6) but lowers SSD F1 (81.4 vs. 84.5) compared with the full noisy set; staged noisy-then-verified training recovers SSD (85.9) at an SSR of 78.4. (§5.6, Table 5)
Limitations and Open Questions
Warning
The primary benchmark is small and narrow: 218 clips of 101 sentences recorded by a single professional actor, in English only. The second set, StressPresso, has four speakers but post-hoc meaning annotation, which the authors treat as complementary rather than primary evidence. Both SSR metrics also depend on a GPT-4o judge to read model outputs.
- The paper limits itself to English and few speakers, and conversational settings are untested (§Limitations).
- Binary SSR offers two options per item, so a model can sometimes guess the stressed word from the answer wording; the paper suggests this may explain part of Gemini 2.5 Pro’s score (§5.2).
- StresSLM does not beat the dedicated WhiStress detector on SSD F1 in its final configuration (Table 3), and on StressTest its SSR falls when given ground-truth stress labels (78.8 vs. 86.2 audio only; Table 10), so how it uses explicit stress information is unclear.
- ASR is retained relative to Qwen2-Audio-7B-Instruct but remains worse than the Qwen2-Audio-7B base model (Table 4); only ASR and emotion recognition are checked, so effects on broader speech LM benchmarks are not reported.
- The §5.4 text states that StresSLM leads on SSD with 88.3 and 83.5, but those values are WhiStress’s F1 scores in Table 3; StresSLM’s are 86.9 and 80.6 and lead only among speech LMs.
- The Stress-17k audio comes from a single commercial TTS voice set, and the verifier is the same group’s WhiStress model, which is also the cascade baseline; the paper does not test whether these choices bias the training data toward what WhiStress can detect.
- Training-data validation uses only 20 clips per subset (Table 8).
Wiki Connections
- Spoken Language Model — Benchmarks how existing speech LMs use sentence stress in the input audio and adapts one (Qwen2-Audio) to reason about stress-implied meaning.
- Qwen2-Audio — The backbone StresSLM is LoRA-finetuned from, and a near-chance baseline on StressTest before finetuning.
- Expresso — Source of the real-speech StressPresso subset and of the Expresso stress-detection test set.
- Gemini 2.5 — The strongest off-the-shelf speech LM baseline on SSR and open-ended reasoning, and the best model when given ground-truth stress labels.
- SALMONN — Evaluated speech LM baseline that performs near chance on stress reasoning.
- LLaMA-Omni — Evaluated speech LM baseline with the lowest open-ended stress reasoning scores.
- Qwen3-Omni — Evaluated omni-modal baseline; the strongest open model on SSR but well below StresSLM.
- Audio Flamingo 3 — Evaluated audio-language model baseline that performs near chance on stress reasoning.