Interspeech · 2026 · Conference

Turetzky et al. (The Hebrew University of Jerusalem / IBM Research) · → Paper · Demo: ? · Code: ✓

Introduces CAST, a benchmark of contrastive context pairs (one sentence, two discourse contexts, two different words to stress) showing that current TTS systems almost never shift word-level stress in response to discourse context, even though text-only LLMs recover the intended stress word reliably.

Problem

Contrastive focus, such as stressing manager versus flight in “The manager booked the flight”, is decided by the preceding discourse, not by the sentence alone. Existing stress-related evaluations cover stress detection from speech, stress understanding in speech LMs (StressTest), or emphasis realization and transfer when the stress target is given explicitly (EmphAssess, Prosaudit). Prosody-controllable TTS work likewise assumes the stress target is supplied at inference time. None of these tests whether a TTS system can infer which word to stress from context and then realize it acoustically, so it is unclear whether the expressive prosody of modern LLM-based TTS reflects discourse meaning.

Method

Task (§3.1). Each item is a tuple (context c, sentence s, intended stressed word w*), with w* defined semantically by the discourse. The system receives c and s with no stress markup and must synthesize s with emphasis on w*. Items come in contrastive pairs: the same sentence appears with two contexts that require stress on two different words. Because lexical content is identical across the pair, any difference in stress can be attributed to the context rather than to sentence-internal bias.

Construction (§3.2). The benchmark is built entirely in text. gpt-5-mini generates the contexts, sentence, and stress targets jointly through structured prompting. Planning codes fix the stress position and the discourse function of the first context (correction, role disambiguation, exclusivity, and so on), and each sentence has exactly two plausible stress candidates. Items survive only if two independent judges (gpt-5-nano and gemini-2.5-flash, the latter from a different model family than the generator) agree on the stressed word. No audio is used in construction. The final set has 113 contrastive pairs (226 items), with stress positions spread across initial (23.9%), early (23.9%), medial (23.0%), and final (29.2%) positions and eight context types ranging from alternative (18.6%) to modality (5.3%) (§3.3, Table 2). An extended pipeline that adds synthesis and automatic acoustic stress filtering produced a separate synthetic corpus of about 10k context-sentence-audio triples, released for training but not used in the benchmark (§3.4).

Conditioning modes (§4.1). Systems are tested in each mode they natively support: baseline (no context), Concat (context prepended to the sentence, with the target segment cut out by Whisper-based forced alignment), Instruct (context wrapped in a natural-language instruction such as “The following context determines which word should be emphasized in the sentence: [context]”), and Explicit (target word marked with model-supported markup, treated as an oracle upper bound). The evaluated systems are Kokoro (a conventional non-LLM system), Chatterbox (LLM-based, no instruction support), CosyVoice3, HiggsAudio V2 and Qwen3TTS (open instruction-capable LLM-based systems), and the proprietary GPT-4o-mini-tts.

Metrics (§4.2). Stress is detected from the generated audio by WhiStress, a Whisper-based stress detector that outputs the set of stressed words. Three metrics are defined over a pair with targets A and B. Hit asks whether the target word is among the detected stressed words. Pair-Contrast additionally requires the other context’s target to be absent, which penalizes stressing many words. Pair-Correct requires Pair-Contrast on both sides of the pair at once, which a system with a fixed sentence-internal stress pattern cannot satisfy. Confidence intervals are 95% bootstrap intervals over 10K resamples.

Human validation (§4.3). Three fluent English speakers labelled the stressed word in 50 items (25 pairs). Individual accuracy against the benchmark labels ranged from 80% to 94%, a majority consensus formed on 94% of items, and the majority matched the benchmark label on 93.6% of those. A second panel of three listeners marked perceived stressed words in 50 synthesized utterances (25 CosyVoice Explicit, 25 GPT-4o-mini-tts Instruct). Inter-listener agreement was Fleiss’ κ = 0.36. Agreement between the majority vote and WhiStress was κ = 0.32 (F1 = 0.48), within the pairwise listener range of κ = 0.29 to 0.40.

Key Results

TTS systems barely respond to context (§5.1, Table 3). Pair-Correct stays between 0.0 and 3.5% for every system in every non-oracle mode. Adding context does not clearly help any system: CosyVoice3 reaches 0.9 ± 1.4 Pair-Correct without context, 0.0 with Concat, and 0.9 ± 1.4 with Instruct, and Qwen3TTS reaches 2.7 ± 3.1, 2.7 ± 3.1, and 3.5 ± 3.1. The best Pair-Contrast in a context mode is GPT-4o-mini-tts Instruct at 26.1 ± 5.3. The highest non-oracle Hit belongs to Qwen3TTS without context (42.5 ± 6.0), and its Hit falls with Concat (35.0) and Instruct (38.9). The confidence intervals for the context modes overlap heavily, so the paper’s conclusion is that no conditioning mode reliably gets context into stress.

Explicit markup helps but remains far from solved (§5.1, Table 3). With the target word marked, CosyVoice3 reaches Hit 52.2 ± 6.0, Pair-Contrast 40.3 ± 5.8, and Pair-Correct 10.6 ± 5.8, and GPT-4o-mini-tts reaches 51.8 ± 6.2, 43.8 ± 5.3, and 11.5 ± 5.8. The intervals do not overlap with the context conditions. Even when told which word to stress, the systems realize the full contrastive pair correctly about one time in ten.

Text-only LLMs mostly know the right word (§5.2, Table 4). Predicting the stressed word zero-shot from context and sentence text, Qwen3 models reach Pair-Correct 5.3 (0.6B), 24.8 (1.7B), and 37.2 (4B), and Claude-haiku-4.5 reaches Contrast 88.1 and Pair-Correct 76.1. Hit equals Contrast for these models because each predicts exactly one word. The large gap between text-level inference and speech-level realization is the paper’s central finding.

The comparison is fair in that all systems face the same items and detector, but systems are tested only in the modes they natively support, so not every system appears in every mode (for example, only CosyVoice3 and GPT-4o-mini-tts have Explicit results).

Novelty Assessment

The contribution is an evaluation design rather than a model. Its central idea is the contrastive-pair setup with the Pair-Correct metric, which separates real sensitivity to context from a fixed default stress pattern. Hit-style or F1-style stress scores cannot make that distinction, so this is a genuine methodological improvement over evaluations with explicit stress targets. The construction pipeline (LLM generation plus cross-family LLM-judge filtering) and the stress detector (WhiStress, from the same group) are existing tools put together. The benchmark is small (113 pairs) and English-only. The labels are validated by humans on only 50 items, and the stress detector on another 50 utterances. The text-versus-speech gap is clear and convincing at the headline level. Finer comparisons between TTS systems mostly fall within overlapping confidence intervals and should not be over-read.

Field Significance

Moderate. CAST supplies a targeted test and metric for whether TTS systems turn discourse context into word-level stress, a capability that natural-sounding expressive TTS evaluations do not currently probe. Its main lesson is a cautionary one: both instruction-capable and LLM-based TTS systems in this evaluation leave the context-to-stress mapping largely unexploited, even though the mapping is mostly recoverable from text.

Claims

  • complicates: Expressive, LLM-based TTS systems do not reliably infer contextually appropriate word-level stress from discourse context, whether the context is concatenated to the input or supplied as a natural-language instruction.

    Evidence: Pair-Correct stays at or below 3.5% for Kokoro, Chatterbox, CosyVoice3, HiggsAudio V2, Qwen3TTS and GPT-4o-mini-tts across baseline, Concat and Instruct modes, with heavily overlapping confidence intervals between the no-context and context conditions. (§5.1, Table 3)

  • supports: The information needed to place contrastive stress is largely recoverable from text alone, so the bottleneck in context-aware stress lies in TTS conditioning and acoustic realization rather than in semantic inference.

    Evidence: Text-only LLMs predicting the stressed word zero-shot reach up to 88.1% Contrast and 76.1% Pair-Correct (Claude-haiku-4.5), with steady gains across Qwen3 sizes from 0.6B to 4B, far above every TTS system. (§5.2, Table 4)

  • complicates: Even with the stress target given explicitly, current TTS systems realize word-level emphasis inconsistently, so explicit prosody markup is not a solved upper bound.

    Evidence: With explicit stress markup, CosyVoice3 and GPT-4o-mini-tts reach about 52% Hit but only 10.6% and 11.5% Pair-Correct, clearly above the context conditions but far below reliable realization. (§5.1, Table 3)

  • refines: Evaluating context-dependent prosody requires contrastive designs that hold lexical content fixed, because per-utterance hit rates can reward fixed sentence-internal stress patterns that ignore context.

    Evidence: Pair-Correct requires the correct stress and the absence of the competing stress on both sides of a pair sharing one sentence, and it falls to near zero for systems whose Hit scores reach 30 to 42%. (§4.2, §5.1, Table 3)

  • complicates: Automatic stress detectors are only a moderate proxy for human perception of prominence, which limits the precision of automatic prosodic-stress evaluation.

    Evidence: WhiStress agrees with the listener majority vote at κ = 0.32 (F1 = 0.48) on 50 synthesized utterances; this is within the inter-listener range (κ = 0.29 to 0.40), but human agreement is itself low (Fleiss’ κ = 0.36). (§4.3, §6)

Limitations and Open Questions

Warning

All system scores come from one automatic stress detector (WhiStress) whose agreement with the human majority is modest (κ = 0.32, F1 = 0.48) and which the authors note may favor loudness over subtler pitch-based emphasis. Absolute scores and fine-grained system rankings depend on this proxy.

  • The benchmark is small (113 contrastive pairs, 226 items) and English-only, so confidence intervals are wide and most comparisons between systems in the context modes are not statistically separable.
  • Contexts, sentences and labels are all generated and filtered by LLMs. Human label validation covers 50 items, and three of those had majority labels that disagreed with the benchmark because the sentences had more than one plausible focal word.
  • Concat mode depends on Whisper-based forced alignment to cut out the target sentence, which may add segmentation errors not quantified in the paper.
  • Systems are evaluated only in their natively supported modes, so the mode-by-system grid is incomplete (for example, Explicit results exist only for CosyVoice3 and GPT-4o-mini-tts).
  • Only lexical stress is evaluated. The authors name intonation contour and phrasing as untested dimensions of contextual intent (§6).
  • The released ~10k synthetic training corpus is not used to train any system, so whether it improves context-aware stress remains open.

Wiki Connections

  • Evaluation Metrics — Introduces the Hit, Pair-Contrast and Pair-Correct metrics for context-conditioned stress, built on an automatic stress detector and designed to expose context-insensitive default stress patterns.
  • Prosody Control — Measures how well TTS systems control word-level stress, both from discourse context and through explicit stress markup as an oracle, and finds both well short of reliable.
  • Instruction-Conditioned TTS — Tests instruction-capable TTS systems with the discourse context wrapped in a natural-language instruction to emphasize the right word, and finds no clear gain over no context.
  • Subjective Evaluation — Uses human listeners to check the benchmark labels and the automatic stress detector, and reports low inter-listener agreement on perceived prominence (Fleiss’ κ = 0.36).
  • CosyVoice 3 — Evaluated system in all four modes. With explicit stress markup it gives the paper’s oracle upper bound for open systems (Pair-Correct 10.6%).
  • Qwen3-TTS — Evaluated instruction-capable system. Its Hit is highest with no context and does not improve when context is supplied by concatenation or instruction.