arXiv · 2026 · Preprint

Smit Nautambhai Modi et al. (Labelbox) · → Paper · Demo: ? · Code: ?

EchoChain benchmarks whether full-duplex voice assistants correctly revise their task state when a user barges in mid-response with new information, using pre-synthesized user audio and onset-anchored interruption timing so the same stimulus reaches every model, and a three-way failure taxonomy (contextual inertia, interruption amnesia, objective displacement) to diagnose what goes wrong.

Problem

Spoken-dialogue evaluation has mostly assumed half-duplex turn-taking: each user turn is complete before the assistant responds. Single-turn benchmarks (VoiceBench, AudioBench, Big Bench Audio) and half-duplex multi-turn benchmarks (Audio MultiChallenge, SpokenWOZ, MTalk-Bench) test memory and coherence across exchanges but never deliver new information while the assistant is still speaking. Existing full-duplex benchmarks mostly score timing and turn-taking behavior (Full-Duplex-Bench v1 and v1.5, FD-Bench) or topic redirection (FLEXI’s Topic Shift Score), not whether the continuation after an interruption actually reflects the updated task. Full-Duplex-Bench v2 moves toward task-level scoring, but its adaptive examiner varies interruption timing and content per model, which the authors argue undermines controlled cross-model comparison. The paper also notes that these failures are often invisible on the surface: a model can acknowledge an interruption fluently while continuing under stale assumptions.

Method

EchoChain is a generation-and-evaluation harness rather than a trained model. Each conversation starts from a scenario seed (topic, subtopic, persona; e.g. voice support calls, interview coaching, multi-constraint planning; §Appendix B). A Dialogue Planner Module (GPT-5.2) produces and updates a conversation blueprint after each exchange, steering toward states where an interruption that changes the correct answer is natural. A User Simulator Module (Gemini 3 Pro) generates the next user message and, from the second exchange on, a separate short barge-in utterance labeled with one of seven interruption types (§4.1, §Appendix A.2). Because both modules condition on the live conversation history, interruptions are grounded in what the responder actually said.

All user speech is pre-synthesized before the interaction, so runtime TTS latency cannot shift barge-in timing across models. A Persona Voice Engine (Minimax Voice Cloning) clones voices from a pool of recorded voice actors and a User TTS Engine (Minimax Speech-02-HD) renders the user utterance and barge-in stimulus, one cloned voice per conversation (§4.2). To validate the cloned voices, the authors run a paired listening study: 80 real-recording/cloned pairs, three ratings each (240 judgments), on a five-point preference scale centered on “Neutral”.

At runtime a Full-Duplex Orchestrator streams the user audio in incremental chunks to a pluggable realtime responder model, leaves the first exchange uninterrupted to establish context, then for later exchanges detects assistant speech onset and injects the barge-in after a fixed delay (§4.3). A GPT-5.2 Failure Classifier applies the taxonomy to segmented transcripts (not audio) with majority voting. When it flags a failure, the history up to the preceding exchange is given as text and the flagged exchange is replayed as audio to the other responder models, so all four systems face identical conditions. Flagged conversations then go to blinded human review: annotators screen for coherence, naturalness, and audio quality; author an instance-specific discrete rubric; apply it pass/fail to all four model responses; and label failures with the taxonomy, followed by an expert audit (§5).

The taxonomy (§2) distinguishes contextual inertia (the update is acknowledged but the content does not actually change, or changes incorrectly), interruption amnesia (earlier constraints are correctly applied before the barge-in but dropped after it), and objective displacement (the model answers only the interruption and abandons the original request, even when the interruption was additive). Two metrics are reported: mean pass rate (MPR), the fraction of conversations where every rubric criterion is met, and mean criterion passed percentage (MCP), the fraction of individual criteria met (§6).

Key Results

Four closed-source realtime voice models are evaluated on 200 interrupted conversations (§6, Table 1):

ModelContextual InertiaInterruption AmnesiaObjective DisplacementOtherFailedMPR (%)MCP (%)
Grok Voice Agent423916610348.585.5
GPT-realtime-2025-08-28294535111045.081.1
Amazon Nova Sonic 2735316514726.578.1
Gemini Live-2.5-flash-native-audio386564016716.564.8

No model exceeds 50% MPR. The gap between conversation-level and criterion-level scores is informative: GPT-realtime’s MPR is about 70% higher than Nova Sonic 2’s in relative terms, but the MCP gap shrinks to about 4%, because 49.09% of GPT-realtime’s failed conversations violate two or more criteria versus 42.86% for Nova. Failure profiles differ by model: contextual inertia accounts for about half of Nova Sonic 2’s failures, Gemini Live’s concentrate in amnesia and displacement, and Grok shows little objective displacement.

A paired half-duplex control on 48 sampled conversations across all four models (the same content delivered after a complete response, scored with the same rubric) yields 55 failures versus 92 for the interrupted versions, a 40.22% reduction, which the authors take as evidence that a large share of errors comes from the interruption rather than task difficulty (§1, §6). The voice-cloning listening study finds 97 ratings favoring the cloned sample, 92 favoring the human recording, and 51 neutral; a Pearson goodness-of-fit test against a symmetry null gives X² = 0.92 (p = 0.63), so there is no evidence of directional preference (§4.2, Figure 3).

Comparisons are fair in the sense that all four systems receive the same audio and timing rule, but the benchmark items are not a random sample of conversations (see Limitations), so absolute pass rates should be read as performance on hard, classifier-flagged cases.

Novelty Assessment

The contribution is evaluation methodology rather than modeling. Interruption handling is already covered by several full-duplex benchmarks; what EchoChain adds is (1) scoring the semantic correctness of the post-interruption continuation with instance-specific human rubrics, rather than timing or topic-shift proxies, (2) a controlled protocol that pre-synthesizes user audio and anchors barge-in timing to detected speech onset while still conditioning interruption content on the observed conversation, and (3) a three-way failure taxonomy that separates distinct ways state revision breaks down. The half-duplex control is a useful piece of experimental design that most interruption benchmarks lack, though at 48 conversations it is small. The harness components (LLM planner, LLM user simulator, commercial voice cloning, LLM-judge pre-filter) are all off the shelf. The evaluated systems are all closed commercial APIs, so the benchmark says nothing yet about open full-duplex speech LMs.

Field Significance

Moderate — EchoChain provides a controlled test of whether full-duplex speech LMs revise their task state under mid-speech interruption, and shows that four commercial realtime systems all pass fewer than half of these cases, with distinct per-model failure profiles. Its main value is diagnostic: it separates reasoning failures under concurrent input from turn-taking timing, which earlier full-duplex benchmarks mostly measured, and its half-duplex control isolates the effect of the interruption itself.

Claims

  • supports: Full-duplex spoken conversational agents frequently fail to revise their task state correctly when new task-relevant information arrives mid-response, even when they acknowledge the interruption.

    Evidence: Across 200 interrupted conversations no commercial realtime model exceeds 48.5% mean pass rate (Gemini Live 16.5%, Nova Sonic 2 26.5%, GPT-realtime 45.0%, Grok 48.5%), and contextual inertia (acknowledging but not applying the update) is a major failure category for every model. (§6, Table 1)

  • supports: Interruptions are themselves a significant source of reasoning error in spoken dialogue models, beyond the difficulty of the underlying task.

    Evidence: In a paired control on 48 conversations across four models, delivering the same content after a complete assistant response instead of as a barge-in reduces total failures from 92 to 55 (40.22%) under the same rubric. (§1, §6)

  • refines: Post-interruption failures in full-duplex dialogue decompose into distinct mechanisms (ignoring the update, forgetting earlier constraints, abandoning the original request) whose relative frequency differs across systems, so a single aggregate score hides model-specific weaknesses.

    Evidence: Nova Sonic 2’s failures concentrate in contextual inertia (73 of 147), Gemini Live’s in interruption amnesia (65) and objective displacement (64), while Grok shows only 16 objective-displacement failures. (§2, §6, Table 1)

  • complicates: Timing- and turn-taking-based full-duplex metrics do not capture whether a model’s continuation after an interruption is semantically correct.

    Evidence: The authors argue that Full-Duplex-Bench v1/v1.5 and FD-Bench measure yielding, resumption, and detection timing, and FLEXI’s Topic Shift Score rewards any pivot to the interrupting topic, whereas EchoChain’s rubric failures (e.g. recommending a caffeinated drink after acknowledging a no-caffeine constraint) are invisible to such measures. (§3.3, §Appendix C.1)

  • complicates: Conversation-level pass rates and criterion-level scores can rank spoken dialogue systems very differently in magnitude, depending on whether failures are concentrated or spread across criteria.

    Evidence: GPT-realtime’s MPR is roughly 70% higher than Nova Sonic 2’s in relative terms, but the MCP gap narrows to about 4% because 49.09% of GPT-realtime’s failed conversations violate two or more criteria versus 42.86% for Nova. (§6, Table 1)

Limitations and Open Questions

Warning

Benchmark items are selected through the failure pipeline: a conversation is generated against one randomly assigned responder, and only exchanges flagged as failures by the GPT-5.2 classifier are replayed to the other three models and enter human review. Absolute pass rates therefore describe performance on hard, failure-enriched cases rather than a representative sample, and the model that generated a given conversation may be disadvantaged on it; the paper does not report per-originating-model breakdowns.

  • Only four closed-source commercial systems are evaluated; no open-weight full-duplex speech LMs are included (§7).
  • The benchmark covers only information-bearing barge-ins; backchannels, side-speech, and ambient noise are left to future work, and acoustic dimensions such as response timing and prosodic adaptation are not scored (§7).
  • Barge-ins occur at a fixed offset from detected speech onset rather than at conversationally chosen points, and model-specific buffering can introduce small timing deviations (§4.3, §7).
  • Each conversation is screened, rubric-authored, and scored by a single rater (followed by an expert audit); inter-annotator agreement is not reported (§5).
  • The half-duplex control uses 48 conversations, much smaller than the 200-conversation main set (§1).
  • No release URL for code or data is given in the paper, so reproducibility claims cannot yet be checked.

Wiki Connections

  • Spoken Language Model — benchmarks commercial realtime speech LMs that consume streamed user speech, testing their ability to update reasoning while generating.
  • Speech-to-Speech — targets full-duplex spoken dialogue, the dialogue sub-paradigm of speech-to-speech interaction, with audio in and audio out under overlap.
  • Evaluation Metrics — introduces conversation-level (MPR) and criterion-level (MCP) rubric metrics plus a three-way failure taxonomy for post-interruption continuations.
  • Subjective Evaluation — validates cloned user voices with a paired human listening study (240 preference judgments) and relies on blinded human rubric adjudication for final labels.
  • Full-Duplex-Bench — contrasted as a timing and turn-taking benchmark that does not score the reasoning in the post-interruption response.
  • Full-Duplex-Bench v1.5 — contrasted as an overlap-handling benchmark focused on yielding, resumption, and speech quality rather than task-state revision.
  • Full-Duplex-Bench v2 — closest prior work; EchoChain argues its adaptive examiner varies interruption timing and content per model and does not separate failure mechanisms.
  • FD-Bench — contrasted because its user script is fixed in advance and it scores interruption detection and timing rather than continuation correctness.
  • FLEXI — challenged on its Topic Shift Score, which rewards any pivot to the interrupting query regardless of reasoning quality.
  • Audio MultiChallenge — half-duplex multi-turn benchmark whose context-tracking focus EchoChain extends to mid-speech updates.
  • MTalk-Bench — half-duplex multi-turn speech-to-speech benchmark cited as unable to test updates arriving during generation.
  • VoiceBench — single-turn voice-assistant benchmark cited as treating user speech as atomic input.