arXiv · 2025 · Preprint
Yuan Ge et al. (Northeastern University) · → Paper · Demo: ? · Code: ✓
Introduces FLEXI, a benchmark covering six full-duplex human-LLM speech interaction scenarios, including model-initiated interruption in emergencies, and uses it to expose systematic gaps between open-source and commercial full-duplex speech language models.
Problem
Full-duplex speech-to-speech language models are meant to support natural, human-like conversational behavior, including barge-in, turn-taking, and backchanneling, but existing evaluation protocols cover only fragments of this behavior. Prior full-duplex benchmarks either restrict evaluation to a narrow set of open-source systems and a small number of interaction patterns (backchannels and user pauses) or evaluate turn-taking capability without considering the full diversity of duplex interaction, and none systematically test a model’s ability to proactively interrupt a user in a time-sensitive situation.
Method
FLEXI is a benchmark, not a trained model. It formalizes full-duplex conversation using standard turn-taking primitives (inter-pausal units, silences, pauses, gaps, overlaps, backchannels, and interrupts, following the terminology of dGSLM and prior turn-taking benchmarking work) and organizes evaluation around six scenarios: standard turn-taking, pause handling, user interrupt, model interrupt (emergency-triggered), user backchannel, and model backchannel.
Benchmark data is constructed synthetically: user queries for each scenario are generated with Qwen-plus, synthesized into speech with CosyVoice2, concatenated into multi-turn conversational flows with inserted silences to approximate natural timing, and then manually validated. Each scenario is paired with scenario-specific metrics: Takeover Rate (TOR) and Jump-in Rate (JIR) for turn-taking and pause handling; Turn-Taking Rate (TTR), interruption latency, and a GPT-4o-rated Topic Shift Score (TSS) for user interrupts; TOR and a GPT-4o-rated Emergency Detection Score (EDS) for model-initiated interrupts; TTR and a coherence measure for user backchannels; and TOR, Backchannel Rate (BCR), and Jensen-Shannon Divergence (JSD) against GPT-4o-annotated reference backchannel timing for model backchannels. Models under test interact with the benchmark in real time over WebSocket, simulating a live spoken conversation. The paper evaluates three open-source end-to-end full-duplex systems (Moshi, Freeze-Omni, Vita-1.5) and one commercial system (Gemini-2.5-flash-live), the only commercial model tested due to budget constraints.
Key Results
Standard turn-taking and pause handling reveal a trade-off tied to how aggressively a system claims the turn. Moshi has the lowest response latency (0.696s) but the highest jump-in rate (JIR = 0.785), frequently responding before the user’s turn is complete; the other three systems have near-zero JIR (0.12, 0, 0) but instead take over prematurely during user pauses at around 90% TOR (0.937, 0.916, 0.835), versus 0.55 for Moshi (Table 2).
On user-initiated interruption, the commercial system substantially outperforms the open-source systems: Gemini reaches 0.935 TTR and a Topic Shift Score of 4.51 (out of 5, GPT-4o-rated), compared to 0.226–0.539 TTR and 0.75–2.9 TSS for the open-source models (Table 3). On model-initiated interruption in emergencies, no model achieves both proactive interruption and accurate emergency recognition: Gemini and Vita-1.5 never interrupt proactively (TOR = 0) despite Gemini scoring highest on emergency detection (EDS = 2.19), while the open-source models that do interrupt more often (TOR up to 0.535) score lowest on emergency detection, indicating their interruptions stem from generally aggressive turn-taking rather than recognizing the emergency (Table 3). Backchannel behavior is weak across all four systems: backchannel rates are near zero (0.002–0.031) and JSD against GPT-4o-annotated reference timing is close to the maximum divergence of 1 (0.947–0.986) for every model (Table 4).
Novelty Assessment
The contribution is an evaluation-methodology and dataset one, not an architectural one. FLEXI extends the scenario coverage of prior full-duplex benchmarks (which the authors identify as focused mainly on user interruption and backchannels) by adding pause handling and, most distinctively, model-initiated interruption in emergency scenarios. The formalization of turn-taking primitives is adopted directly from earlier work (dGSLM, Arora et al.), and the synthetic data construction pipeline (LLM-generated queries plus TTS synthesis) is a standard recipe already used by the benchmark it explicitly builds on. The paper’s discussion section argues for next-token-pair prediction architectures based on the benchmark’s findings, but this is an interpretive recommendation citing prior work, not a new architecture proposed or tested by the paper itself.
Field Significance
Moderate — FLEXI broadens full-duplex speech interaction evaluation to a scenario the field had not previously tested (model-initiated interruption in emergencies) and provides a head-to-head comparison of an aggressively-tuned open-source system against a commercial one across turn-taking, interruption, and backchanneling behavior. Its contribution is evaluation methodology and a synthetic benchmark rather than a new model or training technique.
Claims
- complicates: Full-duplex speech systems face a trade-off between avoiding mid-turn jump-ins and avoiding premature takeover during a user’s pause; tuning to reduce one failure mode appears to increase the other.
Evidence: Moshi shows the lowest turn-taking latency but the highest jump-in rate (JIR = 0.785) and the lowest (best) pause-handling takeover rate (0.55), while Freeze-Omni, Vita-1.5, and Gemini show near-zero JIR but ~85-94% pause-handling takeover rate. (§3.2, Table 2)
- supports: Commercial full-duplex speech systems currently produce more topically coherent responses to user interruptions than open-source full-duplex systems.
Evidence: Gemini reaches Turn-Taking Rate 0.935 and GPT-4o-rated Topic Shift Score 4.51/5 on the user-interrupt scenario, versus 0.226–0.539 TTR and 0.75–2.9 TSS for Moshi, Freeze-Omni, and Vita-1.5. (§3.2, Table 3)
- complicates: A full-duplex system’s ability to recognize an emergency in its generated response content does not imply it will proactively interrupt the user to act on it.
Evidence: Gemini attains the highest Emergency Detection Score (2.19) but a model-interrupt takeover rate of 0, while open-source models with higher takeover rates (up to 0.535) score lower on emergency detection, indicating their interruptions are driven by generally aggressive turn-taking rather than emergency recognition. (§3.2, Table 3)
- complicates: Current full-duplex speech language models generate listener backchannels rarely and with poor timing alignment to human-judged reference positions.
Evidence: All four evaluated systems show near-zero Backchannel Rate (0.002-0.031) and Jensen-Shannon Divergence near the 1.0 maximum (0.947-0.986) against GPT-4o-annotated reference backchannel timing, though isolated strong backchannel cases were observed in the model-interrupt scenario. (§3.2, Table 4; §4)
Limitations and Open Questions
Warning
Benchmark data is fully synthetic (LLM-generated queries synthesized with CosyVoice2, not real human speech), and only one commercial system (Gemini-2.5-flash-live) is evaluated due to budget constraints, excluding GPT-4o-realtime, Nova Sonic, and DouBao, which limits how far the open-source-versus-commercial comparison generalizes.
The GPT-4o-based scoring used for Topic Shift Score, Emergency Detection Score, and backchannel-timing reference annotation is an automated judge rather than a human rater, so the benchmark’s quality measurements are not independently validated by human listeners. The paper also does not report inter-scenario correlations or statistical significance for the observed gaps between models.
Wiki Connections
- Spoken Language Model — FLEXI is designed specifically to benchmark full-duplex speech language models’ turn-taking, interruption, and backchanneling behavior.
- Speech-to-Speech — the benchmark targets end-to-end full-duplex speech-to-speech systems and evaluates their real-time conversational behavior over a live WebSocket interaction.
- Evaluation Metrics — introduces scenario-specific metrics (Takeover Rate, Jump-in Rate, Turn-Taking Rate, Topic Shift Score, Emergency Detection Score, Jensen-Shannon Divergence for backchannel timing) for full-duplex conversational behavior.
- Full-Duplex-Bench — FLEXI explicitly builds on this benchmark, arguing it evaluates full-duplex models comprehensively but overlooks the diversity of interaction scenarios and the role of controllable modules that FLEXI adds.
- CosyVoice 2 — used as the text-to-speech engine to synthesize the benchmark’s user query audio from LLM-generated text.
- SALM-Duplex — listed among the open-source end-to-end full-duplex systems surveyed in the paper’s landscape overview of full-duplex speech language models.