arXiv · 2025 · Preprint
Lin et al. (NTU) · → Paper · Demo: ? · Code: ✓
Full-Duplex-Bench v1.5 is the first fully automated benchmark for evaluating how spoken dialogue models handle overlapping speech, covering four controlled scenarios (interruption, backchannel, talking to others, background speech) with metrics spanning categorical dialogue behavior, timing latency, and prosodic adaptation.
Problem
Mainstream speech benchmarks focus on single-turn, half-duplex settings and provide no mechanism for evaluating how a system manages overlapping speech during generation. Overlap accounts for over 40% of conversational turns yet existing evaluations rely on pause statistics, floor-transfer offsets, or human ratings, all of which are either too coarse, too expensive, or tied to specific training corpora. As full-duplex dialogue systems proliferate, the absence of a reproducible, automated, and scenario-controlled evaluation framework makes it impossible to compare systems on the behaviors that matter most in real interaction.
Method
The benchmark streams pre-recorded user audio to a model via its native streaming interface while simultaneously capturing its audio output. This model-agnostic design supports both open-weight systems (Moshi, Freeze-Omni) and closed commercial APIs (Gemini, Nova Sonic, GPT-4o Realtime) without requiring access to model internals.
Four controlled overlap scenarios are defined. In the User Interruption scenario, 200 contextually relevant interruptions are synthesized with the same speaker voice as the initial query and injected mid-utterance, testing whether the system yields and addresses the new intent. The User Backchannel scenario uses 99 short affirmations (e.g., “uh-huh”, “yeah”) to test whether the model correctly ignores non-floor-taking cues. Talking to Others injects 100 utterances acoustically processed to simulate off-axis speech (volume reduced by 8 dB, high-shelf attenuation, added reflections), probing addressee detection. Background Speech uses 100 utterances from a different speaker at -15 dB with low-pass filtering and echo to simulate ambient noise.
Three evaluation dimensions are measured. Dialogue behavior categorizes the model’s post-overlap response (Respond, Resume, Uncertain, or Unknown) using a GPT-4o evaluator applied to ASR transcripts from Parakeet-TDT, strictly separating the evaluation modality from the generation modality. Interaction timing captures stop latency (onset of user speech to model stop) and response latency (end of user speech to model restart). Prosodic adaptation compares speaking rate (WPM), pitch mean and standard deviation, intensity mean and standard deviation, and UTMOSv2-predicted MOS between pre-overlap and post-overlap segments using paired t-tests.
Key Results
Across five systems, two divergent behavioral strategies emerge. GPT-4o is the most responsive: it achieves the highest interruption handling rate (RESPOND=0.78) and fastest stop latency (0.23 s) for genuine interruptions, and the shortest response latency (1.35 s) among evaluated systems. However, this responsiveness misfires on non-addressee and background speech, where GPT-4o incorrectly yields in 91% and 93% of cases respectively. Nova Sonic and Gemini exhibit strong floor-holding: Sonic achieves RESUME=0.98 on both backchannel and background speech scenarios, but leaves long post-overlap gaps (2.75 s response latency on interruptions) and struggles with true interruptions (RESPOND=0.24). Freeze-Omni offers the most balanced interruption handling (RESPOND=0.72, t_resp=1.35 s) but over-reacts to background speech. Moshi is unstable across non-floor-taking scenarios, with UNKNOWN rates of 0.92, 0.59, and 0.71 for backchannel, talking-to-others, and background speech respectively.
Prosodic adaptation analysis reveals two post-interruption regimes. GPT-4o and Gemini adopt a tempo-and-pitch-lift strategy: accelerated speech with elevated mean pitch, consistent with emphatic floor re-acquisition. Freeze-Omni and Sonic instead reduce mean intensity while increasing intensity variability, producing quieter but more contoured re-entries. Moshi’s re-entry is erratic: extreme speed-up with reduced pitch and inconsistent intensity, suggesting brittle delivery under pressure. UTMOSv2 scores are largely stable across the pre/post transition for most systems, with only Sonic showing a significant decline.
Novelty Assessment
The primary contribution is a benchmark and evaluation framework, not a new model or architecture. The four scenario taxonomy (interruption, backchannel, talking-to-others, background speech) is a useful operationalization of overlap types from conversation analysis literature, and the combination of behavioral, timing, and prosodic metrics is more comprehensive than prior corpus-level analyses. The use of GPT-4o as an objective text-level evaluator, operating independently of the audio generation modality, is a reasonable design choice that avoids common biases in speech evaluation. The benchmark extends the authors’ prior Full-Duplex-Bench (ASRU 2025) by adding prosodic analysis and more controlled acoustic simulation of the non-addressee scenarios. The benchmark is accepted at ICASSP 2026. The principal limitation is that all audio stimuli are synthesized rather than drawn from real conversational corpora, and the evaluation covers only English.
Field Significance
Moderate — Full-Duplex-Bench v1.5 provides a reproducible, automated test bed for a genuinely under-evaluated capability in spoken conversational agents: overlap handling. By quantifying the responsiveness-versus-floor-holding trade-off across five state-of-the-art systems, it establishes a concrete diagnostic framework that can serve as a shared evaluation standard as full-duplex architectures mature. The benchmark does not advance model capabilities but fills a methodological gap that has constrained progress measurement in this area.
Claims
- Current full-duplex spoken dialogue systems face a fundamental responsiveness-versus-floor-holding trade-off, with no single system achieving strong performance across both genuine interruptions and non-addressee overlap scenarios. (§4, Table 2)
- Stop latency below roughly 0.5 s under genuine user interruptions is achievable for commercial API-based systems but comes at the cost of significantly elevated false-yield rates on backchannel and background speech. (§4.1, Table 2)
- Full-duplex systems exhibit measurable prosodic adaptation post-interruption, but the adaptation strategies diverge across architectures, with some systems showing unstable or brittle re-entry delivery. (§5, Table 3)
- Automated evaluation of full-duplex overlap handling is feasible without human annotation by combining ASR transcription, LLM-based semantic classification, VAD-based timing measurement, and acoustic feature analysis. (§2.3)
Limitations and Open Questions
Warning
All evaluation stimuli are synthesized rather than drawn from real conversational corpora. The acoustic simulations for off-axis and background speech are rule-based (fixed dB offsets, fixed filters), which may not capture the full distribution of real-world overlap conditions. Generalisability of the behavioral profiles to naturalistic speech is unverified.
The benchmark covers English only and uses a single synthesized speaker voice for most scenarios; performance under multilingual or multi-accent conditions is unknown. The behavioral classification relies on GPT-4o applied to ASR output, introducing two error sources (ASR errors, LLM judgment errors) that are not quantified. The prosodic adaptation analysis is limited to the User Interruption scenario and does not examine how systems adapt prosody during backchannel or background speech handling. The benchmark captures reactive behavior after overlap events but does not evaluate proactive overlap prevention or turn-yield signaling. The five evaluated systems include two open-source models and three commercial APIs, with the API systems accessed at specific versioned endpoints that may change over time.
Wiki Connections
- spoken-language-model — this paper provides the primary evaluation framework for assessing overlap handling in full-duplex spoken language models, revealing systematic behavioral differences across architectures.
- speech-to-speech — the benchmark directly targets real-time speech-to-speech systems and characterizes their overlap management strategies through timing and behavioral metrics.
- evaluation-metrics — introduces a multi-dimensional evaluation suite combining categorical behavior classification, latency measurement, and prosodic adaptation analysis for full-duplex dialogue assessment.
- 2501.06282 (MinMo) — benchmarked as one of the evaluated full-duplex systems, providing a reference point for MinMo’s overlap handling profile.
- 2411.00774 (Freeze-Omni) — evaluated directly in the benchmark; documents Freeze-Omni’s behavioral profile as offering the most balanced interruption handling among open-source systems.
- 2507.06261 (Gemini 2.5) — the Gemini Live commercial API endpoint is one of the five systems benchmarked, with results showing strong floor-holding but slow yielding under true interruptions.