ACL · 2025 · Conference
Tamar I Regev et al. (MIT) · → Paper · Demo: ✗ · Code: ✓
Using mutual information estimation with fine-tuned language models over systematically varied context lengths, this paper establishes that prosodic features (pitch, loudness, prominence) are redundant with 3–8 past words of linguistic context but only 1–2 future words, supporting a long-past / short-future asymmetry theory of prosody’s communicative role.
Problem
Prior work (Wolf et al. 2023) established that prosody is substantially redundant with the surrounding linguistic context, but used the full available context without examining how that redundancy depends on context length. The time scale of this relationship — how many words in the past and future carry prosodic information — was unknown, despite its importance for understanding both how listeners use prosody and how speech synthesis should model prosodic generation from text.
Method
The study quantifies the mutual information MI(P_t; W_{n,m}) between a prosodic feature P_t at word t and a linguistic context window W_{n,m} of n past words and m future words (including the word itself), systematically varying n and m from 0 to 9.
Dataset: LibriTTS audiobooks with prosodic features extracted via Montreal Forced Aligner (duration, pause) and the Suni et al. (2017) wavelet-based algorithm (pitch f0, energy/loudness, absolute and relative prominence).
Mutual information estimation: H(P_t) estimated by Gaussian KDE over training prosodic values. H(P_t | W_{n,m}) estimated by fine-tuning BERT/BERT-large/RoBERTa-large with a linear head predicting parametric distributions (Gaussian/Gamma/Laplace depending on feature) over prosody given a context window. A single model handles all (n,m) combinations by training on randomly sampled spans of 1–10 words. MI = H(P_t) − H(P_t | W).
The time scale is defined as the context length beyond which MI no longer increases significantly.
Key Results
For pitch, loudness, and prominence (both absolute and relative):
- Past context: MI increases until ~3–8 words, then plateaus — consistent with the long-scale past redundancy hypothesis.
- Future context: MI saturates after 1–2 words — consistent with the short-scale future redundancy hypothesis.
- Past MI is higher than future MI at all context lengths above 1 word.
For duration and pause:
- Past context adds little beyond the word identity itself (MI does not increase with n > 0).
- Future context saturates after ~4 words for pause, ~1 word for duration.
- Duration and pause have higher MI with future context than with past context, opposite to other features.
The authors attribute duration/pause’s different behavior to sentence-boundary effects (pauses predominantly occur after sentence-final words, which are predictable from future context structure).
Optimal prediction: combining ~5–8 past words with ~1 future word maximizes MI across features.
Novelty Assessment
Methodologically rigorous extension of Wolf et al. (2023), adding the time-scale dimension via systematic context-length manipulation. The long-past / short-future asymmetry is a clean, well-supported finding with direct implications for prosody modeling in TTS. The duration/pause exception is an honest anomaly that the paper explains plausibly. The computational analysis is thorough and the code is public. Primary contribution is analytical rather than applied to synthesis, but the findings are directly relevant to how TTS systems should model prosodic features from text.
Field Significance
Moderate — This paper provides empirical grounding for a specific and useful question in prosody modeling: how many words of past and future context are necessary to reliably predict prosodic features. The long-past / short-future asymmetry it establishes can inform the design of prosody predictors in TTS systems, suggesting that duration predictors and pitch/loudness predictors may warrant different context window configurations. Its scope is limited to English audiobook speech, which constrains how broadly the quantitative findings transfer.
Claims
- Prosodic features such as pitch, loudness, and prominence are redundant with an extended past linguistic context of 3–8 words, but are redundant with future context only over a short span of 1–2 words. (§5, §7)
- Duration and pause exhibit the opposite asymmetry to other prosodic features: they carry little redundancy with past context beyond word identity, but show stronger redundancy with future context, likely due to sentence-boundary predictability. (§5, §6.4)
- Fine-tuned masked language models provide a viable framework for quantifying how much linguistic context is needed to predict prosodic features, with mutual information serving as an interpretable upper-bounded estimator. (§3.1, §4.2)
- The asymmetric time scale of prosody-context redundancy implies that prosody serves at least two distinct communicative functions: reinforcing information from longer-range past context and signaling short-term upcoming structure. (§6.2, §6.3, §7)
Limitations and Open Questions
- Dataset limited to English audiobook narration (LibriTTS), which may exhibit higher prosody-context redundancy than spontaneous speech.
- Mutual information values are cross-entropy upper bounds; estimates may be loose for short-span BERT inputs.
- Single model for all (n,m) combinations may be suboptimal for any specific combination.
- Duration/pause results are noisy due to the highly skewed pause distribution (89.4% are zero seconds); a zero-inflated model is suggested for future work.
- Open question: how do these time scales generalize across languages? (Companion work Wilcox et al. 2025 begins to address this.)
Wiki Connections
This paper is most directly relevant to prosody-control — it provides empirical grounding for how many words of past/future context a TTS system needs to accurately model prosodic features. The finding that pitch, loudness, and prominence require 3–8 past words suggests that prosody predictors in FastSpeech-style and autoregressive TTS systems should attend to an extended past linguistic context. The mutual information framework is relevant to evaluation-metrics as an information-theoretic approach to characterizing TTS prosody quality.