arXiv · 2026 · Preprint
O’Connor Russell et al. (Trinity College Dublin) · → Paper · Demo: ✓ · Code: ✓
Uses a vocoder-based resynthesis method to cleanly isolate prosodic and lexical cues in speech, then probes a self-supervised-representation-based turn-taking model to determine which cue drives its predictions and whether the two are encoded interdependently.
Problem
Predictive turn-taking models for spoken dialogue systems increasingly rely on self-supervised speech representations (S3Rs) rather than hand-engineered prosodic features, because S3Rs jointly encode phonetic, lexical, and prosodic information relevant to anticipating speaker changes. This richness comes at a cost: because S3Rs collapse many cues into a single representation, it is difficult to establish which cue a downstream turn-taking model, such as the voice-activity projection (VAP) model, actually relies on. Prior interpretability attempts corrupted only a single acoustic dimension at a time (e.g., flattening pitch or intensity alone), which leaves the untouched cue available to the model and cannot cleanly separate the contribution of prosody from lexical content. Common corruption methods such as low-pass filtering also disrupt both intelligibility and prosody simultaneously, further confounding any interpretability claims. It is therefore unresolved whether S3R-based turn-taking prediction depends primarily on prosodic cues, lexical cues, or both, and whether these cues are encoded independently or with interdependence in the representation.
Method
The paper introduces a vocoder-based stimulus manipulation method built on the WORLD vocoder to control prosodic and lexical information in speech more cleanly than prior corruption-based probing. Two manipulation families are generated from the CANDOR corpus (1,657 US English dyads, 850 hours, timing information from Amazon Transcribe): (1) prosody-matched noise, which replaces the spectral envelope with pink noise while preserving the original pitch and amplitude contours, removing intelligibility while retaining prosody; and (2) prosodic manipulation, which preserves the spectral envelope (intelligibility) while flattening pitch, intensity, or both to the utterance mean. Background noise (babble, music, competing speech) is also generated at controlled signal-to-noise ratios (-10 to 10 dB) as an additional comparison condition. Word error rate from Whisper transcription is used as a proxy to verify that each manipulation removes or preserves lexical content as intended.
These manipulated stimuli are used to probe the VAP model, a transformer-based turn-taking model with a frozen, causal pre-trained S3R front-end (three self-attention layers, one per channel with tied weights, and one cross-attention layer, 256-dimensional) that predicts the next two seconds of speaking activity at 20 Hz. Ten VAP model variants are trained under different data conditions (clean speech; each individual manipulation; and mixed conditions combining 75% clean with 25% manipulated speech), evaluated with five-fold cross-validation while withholding 70 test sessions. Models are scored on two binary discrimination tasks: S/H-Pred (speech before a speaker-change shift versus speech before a held turn) and S-Pred (pre-shift speech versus mid-turn speech), using F1 and balanced accuracy with significance tested via t-tests. To assess whether findings are specific to one self-supervised encoder, a VAP variant with a wav2vec2.0 front-end (projected to 256 dimensions) is trained and evaluated under the same protocol as the CPC-based model.
Key Results
On clean speech, the VAP model reaches 85% balanced accuracy (S-Pred) and 80% (S/H-Pred), consistent with prior reported performance. Flattening pitch or intensity individually reduces S-Pred accuracy to 68-71%, remaining above chance, while removing lexical content entirely via prosody-matched noise causes a larger drop to 58-60% balanced accuracy, indicating the model is more sensitive to losing lexical cues than to losing prosody in this direct-transfer setting. When models are instead trained on a mix of clean and manipulated speech, accuracy on prosody-matched noise recovers substantially: training and testing exclusively on noise matched to the original prosody achieves 87-91% of clean-speech accuracy, comparable to training on prosody-flattened but lexically intact speech (89-96% of clean-speech accuracy). At -10 dB SNR, a clean-speech-trained model retains 66% balanced accuracy on fully unintelligible prosody-matched noise (WER above 100%), notably higher than the 52% achieved on generic background noise at similarly high WER, indicating that clean prosodic structure, not merely low WER, supports the residual performance. The wav2vec2.0 variant shows minimal accuracy change (93% versus 92%) when either cue is removed, which the authors attribute to non-causal information leakage from future frames rather than genuine robustness, distinguishing it from the causal CPC-based results.
Novelty Assessment
The architectural components in this paper (the VAP transformer model, the CPC and wav2vec2.0 S3R front-ends, and the WORLD vocoder) are all drawn from prior work; the paper proposes no new model architecture. The contribution is a methodological one: applying vocoder-based analysis-resynthesis, a technique established in neuroscience for isolating prosody from intelligibility, to interpretability analysis of S3R-based turn-taking models for the first time. This is a meaningfully cleaner probing method than prior single-dimension corruption (flattening only pitch or only intensity), because it fully removes one cue class while cleanly preserving the other rather than partially degrading both. The empirical findings, that prosodic and lexical cues are each independently sufficient and are exploited flexibly by the model without retraining, and that these findings replicate across two different S3R encoders, are the paper’s primary evidentiary contribution rather than a new system or training recipe.
Field Significance
moderate — This paper contributes a cleaner interpretability methodology for a specific but consequential capability of spoken dialogue systems: turn-taking timing prediction. It clarifies that self-supervised representations used for turn-taking encode prosodic and lexical cues with limited interdependence, and demonstrates that prosody alone can support competitive turn-taking accuracy. This motivates a specific, narrow direction (prosody-only turn-taking models for privacy or efficiency) rather than reshaping how the field approaches speech generation broadly. The scope is limited to one turn-taking architecture family, one corpus, and English only.
Claims
- supports: Self-supervised speech representations encode prosodic and lexical cues supporting a downstream prediction task largely independently, such that either cue alone can sustain prediction accuracy close to using both.
Evidence: A VAP model trained and tested exclusively on prosody-matched unintelligible noise retains 87-91% of clean-speech balanced accuracy, and a model trained on prosody-flattened but lexically intact speech retains 89-96% (§4.3, Table 3, Figure 4).
- supports: A model trained only on clean speech can flexibly fall back on whichever cue (prosodic or lexical) remains available at test time, without any additional training on the degraded condition.
Evidence: The clean-speech-trained VAP model still achieves 66% balanced accuracy on prosody-matched noise with WER above 100% at -10 dB SNR, versus 85% on clean speech and only 52% on generic background noise at comparably high WER (§4.2, Figure 2A, Figure 3).
- complicates: Interpretability probes for self-supervised speech models that manipulate only a single acoustic dimension (e.g., flattening pitch or intensity alone) can conflate prosodic and lexical contributions and understate cue independence.
Evidence: Individually flattening pitch or intensity only partially degrades accuracy (68-71% S-Pred) because the untouched cue remains available, whereas the vocoder-based method that fully removes lexical content while preserving prosody produces a larger, cleaner drop (58-60%) (§4.1, Table 2).
- supports: Prosodic structure alone, stripped of all lexical or intelligible content, can support a spoken-dialogue timing task at performance approaching that of full speech, suggesting privacy-preserving deployments are feasible.
Evidence: A VAP model trained specifically on prosody-matched unintelligible noise reaches 91% of the balanced accuracy of the clean-speech model, despite the stimulus containing no recoverable lexical content (WER at or above 100%) (§4.3, §5).
- complicates: Findings about cue independence derived from one self-supervised encoder may not transfer cleanly to encoders with different temporal properties, since some encoders exhibit information leakage that inflates apparent robustness.
Evidence: The wav2vec2.0-based VAP variant shows near-ceiling accuracy (93% vs. 92%) regardless of which cue is removed, which the authors attribute to wav2vec2.0 embeddings containing future-frame information rather than genuine causal robustness, unlike the causal CPC-based model (§4.4, Table 4).
Limitations and Open Questions
The authors acknowledge a residual risk that prosody-matched noise could retain trace phonetic information despite full spectral-envelope replacement, though they consider this unlikely given that WER on this condition confirms no recoverable lexical content. The study is limited to English speech from a single corpus (CANDOR), and the authors explicitly flag cross-linguistic generalization as future work rather than something this paper establishes. The turn-taking model itself (VAP) is adopted from prior work rather than proposed here, so the findings characterize how existing S3R-based turn-taking systems behave rather than proposing an improved system. The wav2vec2.0 results are also flagged by the authors as likely inflated by non-causal information leakage, limiting how directly those numbers can be compared to the causal CPC-based results.
Wiki Connections
- Self-Supervised Speech — probes what a CPC-based and a wav2vec2.0-based self-supervised representation each encode about prosodic and lexical turn-taking cues, and whether these are represented independently.
- Prosody Control — introduces a WORLD-vocoder-based resynthesis method that isolates and controls prosodic information independently of lexical content, used here for diagnostic stimulus generation rather than synthesis quality.
- Evaluation Metrics — contributes a cleaner probing methodology for testing which acoustic cues a self-supervised speech model relies on, extending prior single-dimension corruption approaches.