IWSDS · 2025 · Workshop
Castillo-López et al. (Université Paris-Saclay, CEA, List) · → Paper · Demo: ? · Code: ?
A structured review of turn-taking modeling for spoken dialogue systems, covering end-of-turn prediction, backchannel prediction, and multi-party scenarios, with the first systematic corpus inventory of the field and a concrete diagnosis that 72% of surveyed works do not compare against prior methods.
Problem
Human turn transitions average 200ms gap; current spoken dialogue agents exhibit 700-1000ms response delays, producing unnatural interactions. Prior work mostly addressed end-of-turn detection using voice activity detection with fixed silence thresholds, which either interrupts users mid-utterance or imposes excessive waits. Beyond the technical gap, the field lacks shared evaluation infrastructure: without standardized benchmarks and public test corpora, it is difficult to measure whether newer neural approaches actually improve over earlier baselines. The 2021 survey by Skantze provided a foundation, but Voice Activity Projection (VAP) models, large language model integration, and several new datasets have emerged since then.
Method
This work is a structured literature survey covering turn-taking modeling research, with particular attention to publications from 2021 onwards. The authors systematically searched major conference proceedings (SIGDIAL, ACL-family venues, Interspeech, IWSDS, ICASSP) and supplemented coverage by reviewing scholar profiles and forward citations. The result covers more than 35 papers published after 2021.
The survey organizes the field around two core subtasks. End-of-turn (EOU) prediction is divided into continuous methods, which predict turn transitions at regular intervals (e.g., every 50ms) regardless of silence, and IPU-based methods, which trigger predictions only when a silence is detected. Backchannel prediction, a binary classification task distinguishing listener feedback from actual turn-grabs, is treated separately. Multi-party conversation (MPC) modeling receives dedicated coverage given its additional complexity (addressee selection, out-of-scope utterances, gaze-based cues).
The survey also provides the first systematic corpus inventory for the field. Fourteen datasets are catalogued across English, Japanese, Mandarin, and multilingual settings, annotated for modality (speech, transcript, video), duration, and multi-party status. Key technical trends identified: multimodal feature fusion consistently outperforms single-modality approaches; VAP models are emerging as a self-supervised framework that unifies EOU and backchannel prediction; syntactic completeness derived from language models (TurnGPT and similar) improves EOU accuracy; and LLMs alone are ineffective at detecting mid-utterance turn opportunities.
Key Results
The survey’s central empirical finding is a reproducibility deficit: only 28% of reviewed papers compare their method against prior work, and one third of EOU detection papers use no public corpus. This is not a minor caveat but the paper’s main thesis: the field cannot reliably measure progress.
Among technical findings reviewed across surveyed works: combining prosodic and word features outperforms either alone in multiple independent studies; fine-tuning wav2vec 2.0 for ASR to extract linguistic features improves turn-completion estimation over acoustic-only systems; VAP models trained on English and fine-tuned on Japanese outperform models trained on Japanese directly; and aligning silence threshold conventions across datasets is identified as a prerequisite for valid cross-lingual evaluation. Less than 30% of multi-party conversation papers were published after 2021, confirming the relative neglect of MPC scenarios. The Switchboard Corpus appears in 69% of backchannel and 41% of EOU prediction papers, making it the dominant evaluation resource despite covering a narrow domain (telephone conversations).
Novelty Assessment
The contribution is entirely a literature synthesis and gap analysis. No new model, dataset, or metric is proposed. The novelty over Skantze (2021) comes from three additions: the first comprehensive corpus inventory for the field, the reproducibility audit (the 72% no-comparison finding), and coverage of VAP-based and LLM-based approaches that postdate the prior survey. The corpus inventory and the benchmark critique are the most practically useful outputs; the technical summaries are accessible but not deeper than the surveyed papers themselves. For researchers entering the field, the paper provides a reliable starting point; for those already active in it, the reproducibility data is the salient new contribution.
Field Significance
Moderate — This survey provides an actionable diagnosis of the turn-taking modeling field’s evaluation problem. The 72% no-comparison finding is concrete and reproducible, and the corpus inventory addresses a genuine gap not covered by the prior survey. The impact is proportional to whether the community acts on the benchmark recommendation; the paper itself does not build the benchmark.
Claims
-
complicates: Systematic progress measurement in turn-taking modeling is impeded by the absence of shared evaluation benchmarks.
Evidence: 72% of surveyed papers do not compare their methods with prior work; one third of EOU detection papers use no public corpus, making cross-study comparison unreliable. (§7.1)
-
supports: Combining multimodal cues (prosodic, lexical, and visual) consistently improves turn-taking prediction over single-modality approaches.
Evidence: Multiple independent surveyed works demonstrate that joint use of prosodic and word features outperforms either in isolation, consistent with prior findings on the additive effect of turn-taking cues across human communication. (§4.1)
-
supports: Voice Activity Projection models provide a self-supervised framework that unifies end-of-turn and backchannel prediction in continuous spoken dialogue.
Evidence: VAP models evaluated zero-shot across shift prediction, hold prediction, upcoming backchannel, and backchannel-vs-turn-shift classification outperform earlier silence-threshold and IPU-based baselines in multiple reviewed works. (§4.1, §5)
-
complicates: Large language models are insufficient as standalone turn-taking detectors in spoken dialogue.
Evidence: A surveyed study demonstrates that LLMs are ineffective at detecting mid-utterance turn initiation points without integration of acoustic signals. (§4.1)
-
complicates: Cross-lingual transfer for turn-taking models requires alignment of speech segmentation criteria, not merely multilingual training data.
Evidence: VAP models show poor zero-shot cross-lingual transfer; alignment of IPU silence threshold definitions across datasets is identified as a prerequisite for valid multilingual evaluation. (§4.1, §7.3)
Limitations and Open Questions
The literature search focuses on post-2021 work from major conference proceedings and may underrepresent preprints and workshop papers outside the primary venues searched. Coverage of non-English languages reflects the state of available annotated corpora rather than a selection decision. The multi-party section is acknowledged as sparse.
Key open questions identified by the survey: how can VAP models incorporate visual signals for richer multimodal turn prediction; how should benchmarks handle varying silence threshold conventions across datasets; and how can intent recognition and response obligation detection be integrated for agents in multi-party settings where out-of-scope utterances are common.
Wiki Connections
- Evaluation Metrics — The survey’s central finding is the absence of standardized benchmarks for turn-taking modeling, making it directly relevant to evaluation methodology challenges across spoken dialogue research.
- Speech-to-Speech — Turn-taking timing is a core capability of end-to-end spoken conversational agents; this survey maps the current state of that component.
- Subjective Evaluation — The paper critiques inconsistent evaluation protocols across surveyed works, a challenge parallel to the subjective evaluation standardization problem in speech synthesis.