arXiv · 2025 · Preprint
Lin et al. · → Paper · Demo: ? · Code: ✓
Full-Duplex-Bench is the first scenario-driven benchmark for evaluating turn-taking behaviors — pause handling, backchanneling, smooth turn-taking, and user interruption management — in full-duplex spoken dialogue models using automatic, reproducible metrics.
Problem
Most existing evaluation frameworks for spoken dialogue models assume half-duplex, turn-based interaction, leaving the real-time interactive dynamics of full-duplex systems unmeasured. Prior work either relies on coarse corpus-level statistics (voice activity patterns, gap distributions) that are hard to interpret and anchored to a specific dataset, or on user studies that are expensive and difficult to reproduce. As full-duplex models capable of simultaneous listening and speaking become more prevalent — GPT-4o Voice Mode and Moshi being prominent examples — the field lacks a standardised, automated way to diagnose and compare their conversational behaviours.
Method
Full-Duplex-Bench feeds a unified user audio stream to a model and collects time-synchronous output. A word-level, time-aligned ASR transcript (using Nvidia parakeet-tdt-0.6b-v2) is then produced from the output, enabling automatic evaluation across four dimensions without human annotators in the evaluation loop.
The four dimensions and their metrics are:
Pause Handling tests whether the model interrupts a speaker who pauses mid-utterance but has not yielded the turn. The metric is Takeover Rate (TOR) — the fraction of turns where the model produces non-silent, non-backchannel speech during the user’s pause. Lower TOR is better.
Backchanneling assesses whether the model produces timely short acknowledgements (less than 1 second, fewer than two words) during a dominant speaker’s turn. Three metrics are used: TOR (should remain low), Backchannel Frequency (events per second, conditioned on no takeover), and Jensen-Shannon Divergence (JSD) between the model’s backchannel timing distribution and a human ground-truth distribution derived from the ICC corpus.
Smooth Turn-Taking measures how quickly the model begins speaking after the user genuinely finishes. The metric is average response latency (seconds from end of user speech to onset of model speech), reported only when a takeover occurs.
User Interruption Management evaluates coherence and timing when a user interjects mid-response. Metrics are TOR (should be high — the model should yield), response latency after interruption, and a GPT-4o coherence score (0–5) on the model’s reply.
Data comes from three sources: the 850-hour Candor corpus of spontaneous two-channel English conversation (used for pause handling and turn-taking), the ICC corpus with human-annotated backchannel timing from 118 native speakers (55 stimuli), and synthetic dialogues generated using GPT-4o text with ChatTTS synthesis across 10 speaker voices (200 interruption samples, 137 pause samples).
Key Results
Four models are evaluated: dGSLM, Moshi, Freeze-Omni, and Gemini Live.
On pause handling, end-to-end models (dGSLM TOR 0.934–0.935, Moshi TOR 0.980–0.985) interrupt user pauses at high rates across both real and synthetic data. Freeze-Omni, which uses an explicit turn-state classifier, reduces TOR markedly (0.481–0.642). Gemini Live achieves the lowest TOR (0.255–0.310), suggesting its VAD-based architecture is well-calibrated to intra-turn pauses.
On backchanneling, only dGSLM produces backchannel responses at a measurable rate (Freq = 0.015) with the lowest JSD among open-source models (0.934). Moshi and Freeze-Omni produce near-zero backchannel frequency. Gemini Live achieves the best JSD (0.896), indicating better alignment with human backchannel timing distributions.
On smooth turn-taking, dGSLM and Moshi respond with low latency (around 0.3 s) but high TOR. Freeze-Omni’s cascaded pipeline increases latency (0.953 s) and yields a low TOR (0.336), suggesting it misses turn-end cues rather than intentionally withholding the turn.
On user interruption, Freeze-Omni produces the most coherent responses (GPT-4o score 3.615) at moderate latency (1.409 s). Gemini Live scores comparably (3.376) with faster response (1.183 s). End-to-end models fare poorly: Moshi responds fast (0.257 s) but scores only 0.765 on coherence, while dGSLM produces low-coherence responses (0.201) with high latency (2.531 s).
The benchmark reveals a consistent tension: models that are aggressive at taking the turn (high TOR on turn-taking) tend to over-interrupt on pauses and miss backchanneling, while cautious models (low pause-handling TOR) miss turn-end cues and produce low smooth-turn-taking TOR.
Novelty Assessment
The contribution is primarily an evaluation framework. The individual metrics — TOR, response latency, JSD against a human reference distribution — draw on established concepts from conversation analysis and speech technology. The novelty lies in operationalising these behaviours into a unified, fully automated benchmark with scenario-controlled stimuli rather than relying on corpus-level statistics or user studies. The dataset construction is careful: pause thresholds are grounded in psycholinguistic literature, backchannel ground truth is derived from a human annotation study (ICC), and synthetic interruption data fills a gap in available public corpora. The benchmark’s deliberate framing of metrics as descriptive rather than prescriptive — allowing developers to weight different behaviours according to application needs — is a useful design choice. Coverage is limited to four models and English only at the time of publication.
Field Significance
Moderate — Full-Duplex-Bench fills a concrete gap in the SCA evaluation landscape by providing reproducible, behaviour-specific metrics for real-time interactive dynamics, which prior benchmarks largely ignore. Its diagnostic decomposition of turn-taking into four distinct dimensions exposes a clearer picture of where current full-duplex systems fail than any previous automated method. The result that explicit turn-state control modules outperform end-to-end models on pause sensitivity, while end-to-end models still struggle with semantic coherence during interruptions, is a practically useful finding for system designers.
Claims
- Explicit turn-state prediction modules reduce unwanted interruptions during intra-turn speaker pauses more effectively than purely end-to-end architectures trained on dialogue data. (§V, Table III)
- End-to-end full-duplex speech models respond with lower latency but produce less semantically coherent replies when confronted with user interruptions than cascaded systems with a frozen LLM backbone. (§V, Table III)
- Backchannel generation remains a largely unsolved capability in current full-duplex spoken dialogue models, with most open-source systems producing near-zero backchannel frequency. (§V, Table III)
- Commercial full-duplex systems exhibit better-calibrated turn-taking behaviour on naturalistic data than open-source research models, suggesting that VAD design and deployment-scale tuning contribute substantially to perceived conversational naturalness. (§V, Table III)
- Automatic metrics for full-duplex dialogue can be made reproducible and model-agnostic without human raters, enabling rapid cross-system comparisons that corpus-level statistics and user studies cannot efficiently provide. (§I, §III-A)
Limitations and Open Questions
Warning
The benchmark evaluates only four models — three open-source (dGSLM, Moshi, Freeze-Omni) and one commercial API (Gemini Live) — and is restricted to English. Conclusions about which architectural choices drive observed behaviours are correspondingly narrow.
The metrics do not currently capture human preference: a model with low pause-handling TOR is assumed to be better, but users in some contexts may prefer a more assertive interlocutor. The benchmark explicitly notes that it is descriptive rather than prescriptive. Backchannel detection relies on a heuristic duration-and-word-count rule (under 1 second, fewer than two words), which may not generalise across speaking styles or languages. Synthetic interruption data generated with GPT-4o and ChatTTS may not reflect the prosodic and timing characteristics of natural human interruptions, potentially creating an evaluation distribution mismatch. Future work is needed to link benchmark metrics to subjective preference scores and to extend coverage to multilingual settings.