arXiv · 2026 · Preprint
Kangxiang Xia et al. (Qwen Team, Alibaba) · → Paper · Demo: ✗ · Code: ✓
Introduces a real-world benchmark, a composite responsiveness-robustness metric, and an LLM-based detector for semantic interruption handling in full-duplex spoken dialogue systems.
Problem
Full-duplex spoken dialogue systems (SDS) must decide, in real time, whether an overlapping user utterance is a genuine interruption that should stop the system’s speech or a non-substantive backchannel (“uh-huh”, “right”) that should be ignored. Mainstream VAD-based systems are “trigger-happy,” treating any detected vocal energy as a stop signal and frequently misfiring on backchannels, while robust end-to-end models tend to respond too slowly to genuine interruptions. Prior automated benchmarks for this problem, including the Full-Duplex-Bench series and FD-Bench, rely on synthetic dialogue: text generated by LLMs and rendered by TTS. This sidesteps the timing, prosody, and emotional texture of spontaneous human speech, and existing metric suites either lack interpretability or fail to combine responsiveness and robustness into a single actionable score. The paper argues that progress on interruption handling is bottlenecked by the absence of a real-world benchmark, a holistic metric, and a validated modeling recipe, rather than only by architecture.
Method
The paper contributes three linked pieces: a benchmark, a metric, and a model.
SID-Bench. The benchmark is built entirely from real, recorded human-human conversation rather than synthetic TTS dialogue: professionally recorded bilingual (Chinese/English) vendor data plus a curated subset of the Switchboard corpus for English. It comprises 3,700 evaluation instances from about 10 hours of audio, split into interruption-at-the-beginning, interruption-in-the-middle, uninterrupted (backchannel-only), and a dedicated noise/silence category to test whether a system reacts to arbitrary acoustic energy rather than semantic speech content (Table 1). Ground-truth interruption points are produced by a semi-automated pipeline that combines LLM-based semantic tagging of the transcript (identifying the word at which an utterance shifts from backchannel to substantive new intent) with Kaldi forced alignment to anchor that semantic boundary to a precise word-level timestamp.
APT metric. Building on two primary indicators adopted from prior full-duplex benchmarks, False Interruption Rate (FIR) and Interruption Response Latency (IRL), the paper defines Average Penalty Time (APT), a composite score that assigns an asymmetric time cost to each of the four possible outcomes of a user barge-in: true positives are penalized by their latency (IRL); false positives (stopping for a backchannel) are penalized by the entire remaining duration of the turn, since this is treated as a catastrophic conversational failure; false negatives (failing to stop for a genuine interruption) are penalized by the duration of the superfluous audio the user is forced to hear; true negatives incur zero penalty. APT is the mean penalty across all test instances, giving a single number that trades off responsiveness against robustness.
SID-model. The detection model reframes interruption detection as sequential binary classification over a streaming audio context. An Audio Transformer (AuT) encoder extracts acoustic embeddings from raw audio, which are fed into Qwen3-0.6B, a lightweight LLM that serves as the semantic reasoning engine, in an architecture inspired by multimodal designs such as Qwen3-Omni. Training follows a two-stage “pretrain for alignment, fine-tune for detection” paradigm: the full pipeline is first pretrained on large-scale ASR data to align acoustic and linguistic representations, then fine-tuned on a bilingual dialogue corpus with ground-truth interruption timestamps using randomly cropped audio segments labeled positive or negative relative to the true interruption point. Two boundary-aware training strategies address the ambiguity near the decision boundary: a one-word temporal shift applied to the ground-truth label to require the model to accumulate more evidence before predicting positive, and oversampling of segment endpoints that fall close to the true interruption boundary so the model must discriminate fine-grained cases. At inference, the model consumes cumulatively concatenated audio chunks and only registers a stop decision after K consecutive “interrupt” predictions (K=3 in the reported experiments), a decision-smoothing policy intended to suppress momentary prediction jitter near ambiguous boundaries.
Key Results
On SID-Bench, the proposed model achieves the lowest average APT (0.711s) among all systems tested, versus 2.129s for the next-best baseline (Freeze-Omni with Silero-VAD), a reduction of roughly threefold (Table II). It also attains the lowest average FIR (0.124) and a competitive average IRL (0.389s), indicating it does not trade one property for the other the way the baselines do. A pure energy-based VAD baseline (FSMN-VAD) achieves zero latency but the highest FIR (0.840) and highest APT (3.627s), confirming that acoustic-energy-only triggering cannot distinguish backchannels from interruptions. The end-to-end model Moshi achieves the lowest FIR on English conversational data (0.010) but the highest average IRL (2.517s) and, on the dedicated Noise/Silence test set, the worst APT of any system (7.810s) because it misfires on non-speech acoustic energy such as environmental noise (Table III). Two VAD-augmented cascaded systems, Freeze-Omni and FireRedChat, land between these extremes but each retains a residual FIR above 0.5 or a fragility to specific acoustic conditions (Table II, Table III). Because the compared systems differ structurally (cascaded VAD-gated pipelines vs. Moshi’s end-to-end architecture), the paper uses different proxies for “user barge-in time” for each system class: detected VAD onset for cascaded systems, and a fixed-seed, fixed-timing test-audio injection protocol for Moshi.
Novelty Assessment
The contribution is best understood as a benchmark-and-metric paper with a validating model, rather than an architectural paper. SID-Bench’s novelty lies in being built from authentic recorded human dialogue (plus Switchboard) rather than synthetic LLM-generated, TTS-rendered dialogue used by prior full-duplex benchmarks, and in its semi-automated LLM-plus-forced-alignment annotation pipeline for locating semantically grounded interruption points. APT’s novelty is in unifying two previously separate axes (accuracy via FIR, latency via IRL) into a single asymmetric-cost score that specifically penalizes catastrophic false positives more heavily than transient ones. The SID-model itself combines existing components, an audio transformer encoder and a small pretrained LLM decoder, in a fusion pattern the paper itself describes as inspired by prior multimodal designs; its novelty is concentrated in the training recipe (boundary-aware label shifting and oversampling, two-stage ASR-to-detection transfer) and the streaming decision-smoothing inference policy, not in a new architectural class.
Field Significance
moderate — the paper adds a real-world-data benchmark and a composite metric to a still-narrow but growing sub-area (turn-taking and interruption handling for full-duplex spoken dialogue systems), and demonstrates with its own baseline comparisons that the benchmark can differentiate systems that appear similar under simpler metrics. Its contribution is scoped to this specific sub-problem rather than to speech generation broadly, and the model’s architecture is an application of existing components rather than a new design.
Claims
- supports: Automated turn-taking benchmarks built from synthetic, TTS-rendered dialogue may fail to capture the acoustic and semantic texture of authentic human interruption behavior, motivating benchmarks built from real recorded conversation.
Evidence: SID-Bench is constructed entirely from recorded human-human bilingual dialogue plus a curated Switchboard subset, yielding 3,700 evaluation instances spanning genuine interruptions, backchannels, and a dedicated noise/silence category absent from prior synthetic benchmarks. (§II)
- supports: A composite metric that assigns asymmetric time-cost penalties to false alarms versus late responses can surface trade-offs between systems that similar or lower aggregate error rates on a single-axis metric would otherwise obscure.
Evidence: Under APT, FireRedChat’s pVAD incurs a higher penalty than Freeze-Omni on the Noise/Silence set (2.458s vs. 1.658s) despite a lower overall FIR (0.098 vs. 0.154), because its rare false positives occur during long noise segments that each carry a full-turn-duration penalty. (§V.D, Table III)
- complicates: Voice-activity-detection triggered purely by acoustic energy cannot reliably distinguish genuine interruption intent from non-substantive backchannels.
Evidence: FSMN-VAD achieves zero measured latency (IRL) but a False Interruption Rate of 0.840 on average, exceeding 0.90 in conversational contexts, because it treats any detected vocal energy as a stop trigger. (§V.C, Table II)
- complicates: End-to-end speech-to-speech models that learn turn-taking behavior implicitly from training data can be robust to conversational backchannels yet fail to generalize to non-speech acoustic interference.
Evidence: Moshi attains the lowest FIR on English conversational speech (0.010) but the worst APT of any evaluated system under the Noise/Silence condition (7.810s), a failure the authors attribute to pattern-matching between speech and non-speech acoustic energy rather than semantic understanding of interruption intent. (§V.D, Table III)
- supports: Coupling a semantic LLM-based classifier with boundary-aware supervised training and multi-step inference-time decision smoothing can substantially narrow the responsiveness-robustness trade-off in real-time interruption detection relative to VAD-based or purely end-to-end approaches.
Evidence: The proposed audio-encoder-plus-Qwen3-0.6B model, trained with ASR pretraining followed by boundary-aware interruption fine-tuning and a K=3 decision-smoothing inference policy, achieves an average APT of 0.711s versus 2.129s for the best baseline. (§V.C, Table II)
Limitations and Open Questions
The cross-system evaluation protocol is not uniform: VAD-gated cascaded baselines are scored from their detected VAD onset time, while the end-to-end model Moshi is scored using a fixed-seed, fixed-timing test-audio injection heuristic devised specifically because Moshi exposes no explicit barge-in signal. This structural difference in how "user barge-in time" is operationalized per system class is a necessary accommodation for architecturally dissimilar systems, but it means the head-to-head APT comparison between cascaded and end-to-end systems rests on two different measurement procedures rather than one shared one.
Beyond this, the benchmark is limited to two languages (Chinese and English) and a modest scale (3,700 instances from about 10 hours of audio). The paper does not report a correlation between APT and human perceptual judgments of interruption handling; the authors explicitly list this as future work, alongside expanding SID-Bench to more languages and integrating the interruption module into a fully generative SDS rather than evaluating it as a standalone classifier.
Wiki Connections
- Spoken Language Model — the proposed detector feeds an external streaming audio signal, via an audio transformer encoder, into a pretrained Qwen3-0.6B LLM adapted to make sequential interrupt/continue decisions.
- Speech-to-Speech — targets the interruption-handling and turn-taking behavior of full-duplex spoken dialogue systems, evaluating several existing cascaded and end-to-end speech-to-speech dialogue models as baselines.
- Evaluation Metrics — introduces the Average Penalty Time (APT) composite metric and the SID-Bench benchmark as new evaluation infrastructure for interruption detection.
- Moshi — used as the end-to-end speech-to-speech baseline, evaluated via a custom fixed-seed test-audio injection protocol since it exposes no explicit barge-in signal.
- Full-Duplex-Bench — prior full-duplex turn-taking benchmark built from synthetic TTS-rendered dialogue; SID-Bench is positioned as a real-world-data alternative addressing this synthetic-data limitation.
- Full-Duplex-Bench v1.5 — cited as part of the synthetic-data benchmark lineage this paper contrasts SID-Bench against.
- Full-Duplex-Bench-v2 — cited as part of the same synthetic-data benchmark lineage predating SID-Bench.
- FireRedChat — evaluated as a cascaded full-duplex baseline using a proprietary VAD (pVAD) for barge-in detection.
- OSUM-EChat — cited as related work on end-to-end empathetic spoken dialogue systems relevant to the full-duplex SDS landscape this paper targets.
- MinMo — cited as related work on multimodal LLMs for seamless voice interaction, part of the broader full-duplex SDS context motivating this benchmark.