ACL · 2026 · Conference

Hyunjong Ok et al. (POSTECH / HJ AILAB / KAIST) · → Paper · → Preprint · Demo: ✗ · Code: ?

Releases OpenETD, an end-turn detection corpus of TTS-voiced MultiWOZ dialogues with injected pauses and filler words plus diarized real conversations, and proposes SpeculativeETD, which runs a tiny GRU on-device to detect silence and calls a server-side fine-tuned wav2vec 2.0 only once per silence to decide whether it is an end of turn or a pause.

Problem

Spoken dialogue systems built around LLMs must decide when the user has finished a turn. When a user pauses to think, a system that treats silence as the end of the turn responds prematurely; one that waits too long feels sluggish. Pause duration alone does not separate hesitation from turn completion (§1). The authors identify two bottlenecks: the lack of a public dataset for end-turn detection (ETD), since Fisher is paid and prior turn-taking work trains on private data, and the cost of running an accurate transformer classifier at every frame of a real-time stream, while models small enough for a phone are much less accurate (Table 1: 79.7% vs. 99.3% accuracy on the synthetic set) (§1, §2).

Method

Task definition. At each time step the speaker is in one of three states: speaking unit (SU), Pause (silent but intending to continue), or Gap (turn finished). OpenETD labels each audio file as a sequence of (state, start time) segments (§3).

OpenETD synthetic data. Text dialogues from MultiWOZ are voiced with Google Cloud Text-to-Speech. Because plain TTS of a text dialogue contains only speech and gaps, three variants are generated: V1 base (no pauses), V2 with pauses (micro-hesitations already produced by the TTS model are stretched to durations drawn from a truncated Erlang distribution with shape 3 and rate 4.29, range 0.1 to 3.0 s, mean 0.70 s), and V3 with filler words (“um”, “uh” and similar) injected into the text and followed by a pause. For some samples, the audio after a pause is cut so that the clip does not end in a gap (§3.1, §B). The synthetic set has 122,481 samples and 148.26 h, split following the MultiWOZ partition (Table 2).

OpenETD real data. Two-speaker conversations are collected from YouTube podcast queries and the Buckeye corpus, filtered with pyannote diarization (more than two speakers removed) and with Whisper language identification (English only; 8,022 of 11,127 files kept), then labelled automatically: any silence longer than 200 ms is a Pause if the same speaker resumes and a Gap if the speaker changes (§3.2, §G). The real set has 8,987 samples and 165.75 h, split at file level (Table 3). YouTube audio is distributed as URLs plus download scripts rather than files (§3.3).

SpeculativeETD. A two-stage cascade (§4, Figure 3):

  • On-device: a GRU classifier processes 100 ms chunks of a 40-dim log-mel spectrogram through a two-layer Conv2D frontend and a single GRU layer (hidden size 64), making a binary SU vs. non-SU decision. §5 and §B give about 202K parameters; Table 1 and the Figure 3 caption describe it as a 1M-parameter GRU.
  • Server-side: once the GRU predicts non-SU for 200 ms (two consecutive chunks), the audio accumulated from the start of that silence is sent to a fully fine-tuned wav2vec 2.0 base model (94M parameters), which classifies it as Gap or Pause. A Gap triggers the LLM response.

The name borrows from speculative decoding, but the authors note the two models predict different label sets: the large model makes the harder fine-grained decision conditioned on the small model’s trigger rather than verifying the same output (§4). All models are trained with AdamW for 10 epochs on the mixture of synthetic and real training splits (§5, Table 15). The baselines are VAP (pretrained encoder frozen, “comparative” predictor head trained on OpenETD), the GRU alone, and wav2vec 2.0 alone.

Key Results

Binary Gap vs. Pause classification (Table 4, whole segment given). Fine-tuned wav2vec 2.0 scores highest on every column: 99.3% accuracy and 99.2 F1 on synthetic data, 81.2% accuracy and 75.2 F1 on real data. VAP reaches 92.3% / 69.6% accuracy and the GRU 79.7% / 69.0%. On real data the GRU’s F1 is 49.8. SpeculativeETD is not evaluated on this task because it assumes a streaming setting (§5).

Real-time 3-class segmentation (Table 5, macro F1 / IoU over SU, Pause and Gap at 100 ms steps):

SystemSynthetic F1Synthetic IoUReal F1Real IoU
VAP90.684.833.225.9
GRU58.052.234.231.7
wav2vec 2.094.790.258.446.2
SpeculativeETD94.088.945.637.8

SpeculativeETD outperforms VAP and the standalone GRU on all four columns and comes within 0.7 F1 / 1.3 IoU of wav2vec 2.0 on synthetic data. On real data it trails wav2vec 2.0 by 12.8 F1 and 8.4 IoU, a large accuracy cost that the paper’s “comparable to wav2vec 2.0” framing in §6.2 applies only to the synthetic set. The Table 5 caption also reports 26.7x fewer wav2vec 2.0 calls on real audio.

Compute and latency. Over 100 synthetic samples, SpeculativeETD uses 919.64 MFLOPs (45.34 GRU + 874.30 server) against 34,971.68 for frame-by-frame wav2vec 2.0 (38x fewer) and 10,354.98 for VAP (Table 6). FLOPs are not reported for the real set. On an iPhone 12 mini, the GRU executes in 0.26 ms per 100 ms step (load 1.16 ms, init 3.85 ms), while wav2vec 2.0 on-device takes 1,500.32 ms to execute and 874.06 ms to load (Table 7); wav2vec 2.0 latency grows with input length while the GRU’s stays flat (Figure 5). The server’s own GPU inference time is not reported. Network round-trip time from the phone to a server (no inference) is 106.0 to 116.2 ms mean on 5G and 98.8 to 139.7 ms mean on Wi-Fi across 100 ms to 10 s payloads, though the Wi-Fi 95th percentile for a 10 s payload is 225.0 ms, above the 200 ms turn-taking threshold the authors use as the budget (§E, Table 12).

Data analyses. Synthetic pause and gap durations match real ones closely (KS 0.092 and 0.083), but positions within the utterance do not (KS 0.238 for both) (§A, Table 8). Training SpeculativeETD on the synthetic + real mixture gives 45.6 real F1, versus 43.1 with real only and 44.0 with synthetic only (§A, Table 9). A human check of 96 real clips by three annotators found 94.0% agreement with automatic Pause labels but only 76.1% for Gap labels (85.4% overall) (§C, Table 10).

Novelty Assessment

The dataset is the more distinctive contribution: the authors describe OpenETD as the first public dataset for end-turn detection, and its design choices (stretching TTS micro-hesitations with an Erlang duration model, filler-word injection, truncating clips after a pause, diarization-derived Pause/Gap labels with a human audit) are documented and partly validated. SpeculativeETD itself combines two standard classifiers (a small GRU and a fine-tuned wav2vec 2.0) in an event-triggered cascade; the idea of gating an expensive model behind a cheap silence detector is an engineering design rather than a new architecture or training objective. The evaluation is confined to the authors’ own dataset, with VAP as the only external turn-taking baseline, and no end-to-end test inside a running dialogue system.

Field Significance

Moderate — The paper provides an openly constructed end-turn detection benchmark with both synthetic and real conversational audio, which addresses the reproducibility gap the authors identify for turn-taking research on private data. Its results also make concrete how far a synthetic test set can overstate real-world performance: every model loses between 24 and 57 F1 points moving from synthetic to real data across Tables 4 and 5, and the cascade’s accuracy cost relative to a full wav2vec 2.0 detector is small on synthetic audio but large on real audio.

Claims

  • supports: Gating an expensive turn-end classifier behind a lightweight on-device silence detector, so that the large model runs once per silence instead of every frame, cuts end-turn detection compute by more than an order of magnitude.

    Evidence: A ~202K-parameter GRU triggers a fine-tuned wav2vec 2.0 only after 200 ms of predicted silence; the cascade uses 919.64 MFLOPs vs. 34,971.68 for frame-level wav2vec 2.0 on 100 synthetic samples (38x fewer), with GRU execution at 0.26 ms per 100 ms step on an iPhone 12 mini. (§4, §6.3, Table 6, Table 7)

  • complicates: The accuracy cost of cascaded end-turn detection depends strongly on the test domain, being negligible on clean synthetic dialogue but large on spontaneous real conversation.

    Evidence: SpeculativeETD is 0.7 F1 below wav2vec 2.0 on the synthetic test set (94.0 vs. 94.7) but 12.8 F1 and 8.4 IoU below it on real YouTube/Buckeye data (45.6 vs. 58.4 F1; 37.8 vs. 46.2 IoU). (§6.2, Table 5)

  • complicates: End-turn detectors evaluated only on TTS-voiced dialogue substantially overestimate performance on real conversational speech.

    Evidence: Binary Gap-vs-Pause accuracy drops from 99.3% to 81.2% for wav2vec 2.0 and from 92.3% to 69.6% for VAP between synthetic and real test sets; the synthetic data has much higher SNR (39.7 vs. 10.5 dB), less speaking-rate variability, and pause positions that diverge from real ones (KS 0.238). (§6.1, Table 4, §A, Table 8, §D, Table 11)

  • supports: Synthetic dialogue speech with artificially inserted pauses and filler words is useful as augmentation for end-turn detection even when it does not replicate real pause placement.

    Evidence: Training SpeculativeETD on synthetic + real data yields 45.6 real-set F1 vs. 43.1 (real only) and 44.0 (synthetic only); synthetic pause durations match real ones (KS 0.092) while positions do not. (§A, Table 8, Table 9)

  • complicates: Automatically derived turn-boundary labels from speaker diarization are reliable for within-speaker pauses but noticeably noisier for speaker-change gaps.

    Evidence: Three annotators agreed with automatic labels on 94.0% of Pause clips but only 76.1% of Gap clips (96 clips total), attributed to ambiguous speaker-change boundaries. (§C, Table 10)

Limitations and Open Questions

Real-data performance remains low

On the real test split, SpeculativeETD’s 3-class segmentation reaches 45.6 macro F1 and 37.8 IoU (Table 5), far below its synthetic-set scores, and the real-set Gap labels it is scored against agree with human judgment only 76.1% of the time (Table 10). The headline “comparable to wav2vec 2.0” result holds only on synthetic data.

  • Model size reported inconsistently. The on-device GRU is described as about 202K parameters in §5 and §B but as 1M parameters in Table 1 and the Figure 3 caption; the same GRU accuracy (79.7%) appears in Table 1 and Table 4.
  • Synthetic voice count reported inconsistently. §D text and Table 11(b) say the synthetic set uses two TTS voices (EN-US, EN-BR), while Table 11(a) lists “3 TTS voices”.
  • Latency accounting is partial. On-device GRU latency and network round-trip time are measured, but server-side wav2vec 2.0 inference time is not, and the paper’s Limitations section states that network latency considerations are not addressed in the main evaluation. The Wi-Fi 95th-percentile RTT exceeds 200 ms for a 10 s payload (Table 12).
  • FLOPs only on synthetic data. The 38x compute saving is measured on 100 synthetic samples; on real audio only a call-count ratio (26.7x fewer wav2vec 2.0 calls) is given, without a FLOPs figure.
  • English only. Both the synthetic and real data are English (non-English YouTube files were discarded), so turn-taking patterns in other languages are untested (§Limitations).
  • No downstream dialogue evaluation. The detector is not tested inside a running spoken dialogue system, so the effect on interruption rate or response timing seen by users is not measured.
  • Release details. The proceedings state that the dataset and processing code are released (§1, §3.3, Ethics Statement), but the text gives no repository URL, so code availability is recorded as unknown.

Wiki Connections

  • Self-Supervised Speech — The server-side end-turn classifier is a fully fine-tuned wav2vec 2.0 base model, and it gives the highest Gap-vs-Pause accuracy of the models compared.
  • Moshi — Cited as a full-duplex LLM-based dialogue agent; the paper argues such systems are computationally expensive and positions a lightweight explicit end-turn detector as an alternative.
  • Language Model Can Listen While Speaking — Cited as a real-time duplex dialogue model trained on private data, part of the motivation for releasing an open end-turn dataset.
  • Whisper — Used as a tool for English language identification when filtering the real YouTube data and for speaking-rate transcripts in the domain analysis.