arXiv · 2025 · Preprint
Lin et al. (National Taiwan University / Carnegie Mellon University / MIT) · → Paper · Demo: ? · Code: ?
Full-Duplex-Bench-v2 (FDB-v2) introduces the first automated, multi-turn evaluation framework for full-duplex spoken dialogue systems, using a speech language model as an automated examiner that drives staged goal-oriented scenarios while scoring turn-taking fluency, instruction following, and task-specific competence across four task families.
Problem
Existing benchmarks for full-duplex dialogue systems assess isolated interaction phenomena such as turn-taking timing, backchannel handling, or interruption response, but none evaluate whether models can sustain coherent performance over extended multi-turn exchanges. Single-turn and scripted scenarios fail to capture the compounding challenges of real conversation: state drift, cumulative reference tracking, mid-turn corrections that must overwrite prior context, and safety consistency under pressure. Human multi-turn evaluation is too expensive and hard to reproduce at scale, while corpus-level statistics such as pause distributions and VAD metrics capture timing but ignore semantic appropriateness.
Method
FDB-v2 structures evaluation around two roles: an automated Examiner and the system under test (the Evaluatee). The Examiner is itself a spoken language model (GPT-Realtime, chosen for instruction-following stability), which communicates via synthesized speech to ensure identical acoustic conditions across all sessions. The Examiner organizes dialogue into stepwise semantic subgoals; it advances only when a goal is satisfied, rephrases or repeats if not, and may interrupt to generate natural overlaps.
The framework runs both parties through a streaming orchestrator using bidirectional WebRTC connections at 48 kHz 16-bit mono PCM in 10 ms frames. Each model connects via a lightweight adapter, making the framework model-agnostic. Two pacing regimes vary the Examiner’s initiative: in Fast mode the Examiner may interrupt or backchannel before the Evaluatee finishes; in Slow mode it waits for an end-of-turn or long pause.
Four task families cover representative assistant and safety scenarios. Daily tasks (ordering, scheduling, reservations) test multi-turn goal tracking under natural conditions. Correction scenarios evaluate whether models detect and propagate mid- or cross-turn self-repairs (e.g., revising a coffee order from cold to hot). Entity Tracking scenarios test reference resolution and attribute carry-over using ordinals, attributes, and landmark references across turns. Safety scenarios cover 11 policy-aligned hazard classes and evaluate refusal consistency under conversational pressure and barge-in.
Dual-channel audio is transcribed with Parakeet-TDT (fast conformer ASR), and scoring is performed by a Gemini LLM judge conditioned on the Examiner’s role and stage-level goals. Three metrics are reported: Turn-Taking Fluency (TT, per turn-taking event, 1-5), Multi-Turn Instruction Following (IF, per event, 1-5), and a global Task-Specific score (1-5) for Correction, Entity Tracking, and Safety.
Key Results
Three systems are evaluated: GPT-Realtime (closed commercial), Moshi (open-weights, multi-stream full-duplex), and Freeze-Omni (open-weights, streaming adapter-based). GPT-Realtime leads across all tasks and pacing setups. Under Fast pacing, GPT-Realtime achieves task-specific scores above 4.0 on all three specialized tasks (Correction: 4.02, Entity: 4.51, Safety: 4.44), while Moshi and Freeze-Omni remain below 3.0 on Correction and Entity (Moshi: 2.88/2.76, Freeze-Omni: 2.74/2.62). Slow pacing yields improvements for open-source models, particularly for Moshi on Entity Tracking (+1.1) and Freeze-Omni on Correction (+0.76).
Temporal trajectory analysis shows that all systems degrade over the course of a session. Turn-taking fluency declines gradually, while instruction-following is more volatile and rarely recovers to its initial level. Entity Tracking is comparatively stable (explicit references serve as grounding anchors), while Daily and Correction tasks degrade faster because errors in state tracking accumulate. Safety remains the most robust task for all systems.
LLM judge agreement with human annotators reaches Pearson r = 0.59-0.69 across all metrics (Table 2), with Instruction Following and Safety showing the strongest correlations (r = 0.68, 0.69). Turn-Taking Fluency has the weakest correlation (r = 0.61), reflecting inherent subjectivity in timing judgments.
Novelty Assessment
The primary contribution is methodological: an automated, streaming-native, multi-turn evaluation framework that combines a spoken LM as examiner with staged semantic goals and a standardized WebRTC orchestration layer. Compared to earlier Full-Duplex-Bench versions (v1.0 single-turn, v1.5 overlap phenomena), FDB-v2 extends evaluation to the session level and introduces task-specific metrics for correction handling and entity tracking. The use of a spoken LM as examiner rather than pre-recorded human speech is a practical design choice that trades prosodic naturalness for reproducibility. The framework itself is the contribution; the evaluation results on existing systems (GPT-Realtime, Moshi, Freeze-Omni) serve as a demonstration rather than a novel empirical advance.
The gap between GPT-Realtime and open-source models is substantial on the hardest tasks (entity, correction), though this reflects the capability difference between the systems evaluated, not a claim enabled by the framework itself. The LLM-as-judge approach relies on Gemini and carries known risks of prompt sensitivity and model bias.
Field Significance
Moderate — FDB-v2 fills a concrete gap in evaluation infrastructure for full-duplex spoken dialogue systems: prior frameworks covered single-turn or isolated overlap events, while real conversational scenarios require multi-turn consistency. The framework’s streaming-native, model-agnostic design makes it practical to adopt across system types, and the open-source standardized protocol provides a reproducible testbed that the community can extend to new task families.
Claims
- Multi-turn performance in full-duplex spoken dialogue systems degrades over session time, with instruction-following declining more sharply and with greater volatility than turn-taking fluency. (§3.2.1, Table 3)
- Examiner pacing substantially affects model performance in multi-turn settings, with slower pacing improving instruction-following and task accuracy particularly for open-source systems on memory-intensive tasks. (§3.2.2)
- Entity tracking is more robust to multi-turn degradation than correction handling or daily task instruction following, because explicit reference anchors help models stay grounded. (§3.2.1, §3.2.3)
- LLM-based automated judges can serve as reliable proxies for human assessment in multi-turn spoken dialogue evaluation, with moderate to strong agreement (Pearson r = 0.59-0.69) across turn-taking, instruction-following, and task-specific dimensions. (§3.3, Table 2)
- Correction handling and entity tracking represent the hardest capabilities for current full-duplex spoken dialogue systems under fast-paced interaction, with open-source models scoring below 3.0 out of 5 on both dimensions. (§3.2.3, Table 1)
Limitations and Open Questions
Warning
The automated examiner uses synthesized speech from GPT-Realtime, which reduces prosodic naturalness and may not fully represent the variability of real human interlocutors. Evaluation depends on ASR transcription quality, and LLM judge scoring is sensitive to prompt design and model biases.
Scenario coverage is restricted to four task families in English; open-domain tasks, richer social intents (negotiation, tutoring), and multilingual settings are excluded. The current metric suite does not reward audio-expressive behaviors such as emotional prosody, entrainment, or active-listening cues, meaning systems may underexpress paralinguistic adaptation. Reliance on a closed commercial model (GPT-Realtime) as both the examiner and the top-performing evaluatee creates a potential circularity that may favor response styles aligned with that model family.
Wiki Connections
- spoken-language-model — FDB-v2 directly targets the evaluation of full-duplex spoken LMs, measuring whether they can sustain multi-turn coherence in realistic streaming conditions.
- speech-to-speech — the framework evaluates end-to-end speech-in, speech-out systems including Moshi and Freeze-Omni in bidirectional real-time interaction.
- evaluation-metrics — introduces three new evaluation dimensions (Turn-Taking Fluency, Multi-Turn Instruction Following, Task-Specific Competence) for the multi-turn full-duplex setting.
- 2504.08528 (Survey on spoken language models) — provides the landscape context for the full-duplex systems and evaluation approaches that FDB-v2 targets.
- 2510.02066 (Chain-of-thought reasoning in full-duplex dialogue) — evaluates a system (SCoT) that is a direct target system type for FDB-v2’s multi-turn framework.
- 2410.00037 (Moshi) — Moshi is one of the three systems evaluated in the FDB-v2 experiments, and its multi-turn performance under Fast vs. Slow pacing is directly reported.
- 2411.13577 (WavChat survey) — referenced as background on spoken dialogue models and the landscape FDB-v2 addresses.
- 2503.04721 (Full-Duplex-Bench v1.0) — FDB-v2 directly extends and supersedes the earlier single-turn benchmark, adding multi-turn evaluation and the automated examiner design.
- 2411.00774 (Freeze-Omni) — Freeze-Omni is one of the three systems evaluated in the FDB-v2 experiments and shows the largest gap relative to GPT-Realtime on correction and entity tasks.