arXiv · 2026 · Preprint

Lu et al. (Zhejiang University / Alibaba Qwen Business Unit / Tencent HunYuan Team / ByteDance) · → Paper · Demo: ✓ · Code: ✗

Replaces the binary “is it full-duplex?” label for spoken dialogue systems with three linked frameworks (an L0-L3 hierarchy for where the duplex decision is made, a T × I × R interaction ontology, and a five-state decision state machine), then uses them to audit published systems, public training corpora, and benchmarks.

Scope and Coverage

The survey covers spoken dialogue systems that claim full-duplex behaviour (concurrent listening and speaking, with overlap, backchannels, and barge-in), concentrating on the post-GPT-4o wave of 2024-2026. It sorts systems into three groups (§1): primary targets (systems that explicitly claim full-duplex behaviour, such as Moshi, MinMo, OmniFlatten, SyncLLM, LSLM, FireRedChat, Freeze-Omni, Mini-Omni2, Fun-Audio-Chat, Covo-Audio, DuplexMamba, and SALMONN-omni); historical baselines (pre-LLM industrial modular pipelines such as Google Duplex and the Ant Group and Alibaba DAMO call-centre systems); and taxonomy references (adjacent streaming speech LMs such as GLM-4-Voice, Qwen2.5-Omni and Qwen3.5-Omni, and Step-Audio R1.1, which share an architectural shape but make no full-duplex claim). These are used for context and left out of the per-cell audits. TTS-only, ASR-only, voice-cloning, and non-dialogue audio generation are out of scope.

The paper has nine chapters: background on audio tokenizers and encoders (§2, which the authors say is not a tokenizer survey), a history of full-duplex systems (§3), three conceptual frameworks (§4-§6), a training-data audit (§7), an evaluation audit (§8), and a frontier chapter on hypothetical L3 architectures (§9). The authors position it against two earlier surveys: WavChat, which organises spoken dialogue models along a cascaded/end-to-end axis and treats full-duplex as a binary attribute, and a full-duplex survey by Chen and Yu that uses an engineered-vs-learned axis. The authors argue that both put the design space on a single axis (§3). Several authors of this survey also wrote WavChat. The survey runs no experiments of its own and reports no metrics of its own.

Method

The survey’s method is three frameworks that are meant to be used together.

L0-L3 Architectural Hierarchy (§4). Systems are placed by where in the stack the listen/speak/wait decision is made. L0 (module level) puts it in external modules outside the LLM, typically VAD, an end-of-turn classifier, and a dialogue manager (FireRedChat, FlexDuo, X-Talk, SoulX-Duplug, plus turn detectors such as Easy Turn and FastTurn). L1 (hidden-state level) uses a sidecar predictor that reads the LLM hidden state (MinMo’s single-layer full-duplex predictor; Freeze-Omni, described as sitting on the L0/L1 boundary because the LLM is frozen). L2 (token level) encodes the decision in the generated token sequence itself, through several sub-modes: parallel streams (Moshi, Fun-Audio-Chat), a flattened sequence (OmniFlatten), chunk alternation (SyncLLM), layer fusion (LSLM), control tokens (Covo-Audio’s THINK/SHIFT/BREAK), a state-space backbone (DuplexMamba), and continuous embeddings without a codec (SALMONN-omni). L3 (representation level) would share a latent across the user and assistant streams, and no published system does this yet. The survey links the L2 sub-designs to the tokenizer’s K-tokens-per-frame constraint (§2.3) and argues that the hierarchy describes design choices, not a ladder of progress (§4.1).

T × I × R Interaction Ontology (§5). Each interaction moment is a triple: a temporal relation (T1 sequential, T2 latched, T3 overlap, T4 concurrent, T5 silence), a user intent (I1 information, I2 backchannel, I3 repair, I4 floor-claim, I5 floor-yield, I6 disfluency, I7 third-party), and a required system response (R1 continue, R2 stop, R3 wait, R4 backchannel, R5 ignore, R6 initiate). This gives 210 nominal cells, of which “a few dozen” are judged physically and pragmatically meaningful (§5, Table 3). Six “acid-test” cells form the diagnostic core: standard turn, latched zero-gap, cooperative barge-in, backchannel during system speech, third-party speech, and hesitation with long silence. Sustained concurrent speech (T4·I1·R1) is added as a seventh cell in the coverage tables.

Decision State Machine (§6). Five states (IDLE, LISTEN, SPEAK, WAIT, DUAL) and eleven transitions, each labelled with its (T, I) trigger and R action (Table 4). This extends earlier two-state and three-state (FlexDuo) formulations with a WAIT state for semantic hesitation and a DUAL state for overlap. Each ontology cell maps to a small sequence of transitions; for example, barge-in is SPEAK → DUAL → LISTEN and a backchannel is SPEAK → DUAL → SPEAK (§6.4). The authors state explicitly that the machine is a description of observable behaviour, not a controller (§6.6).

The frameworks are then applied in three audits: a system-by-cell coverage matrix (Table 1) and a layer-by-reachable-state matrix (Table 2); a corpus-by-cell matrix for public two-channel training data (Table 6); and a benchmark-by-cell matrix (Table 7).

Key Results

The survey’s findings are interpretive syntheses, not measurements.

  • Layer placement (§4.1-§4.2, Table 2). L2 is the most populated layer and also internally varied. The survey treats the L1 shape as a recurring pattern because MinMo (full-duplex) and the Qwen2.5-Omni Thinker-Talker (streaming text plus speech, no full-duplex claim) have structurally similar designs. L0 is still active, with new pluggable modules appearing in 2025-2026.
  • Cell coverage (Table 1). Coverage is taken from each system paper’s own claims combined with the authors’ reading of the evidence. No surveyed paper explicitly claims sustained concurrent speech. Only OmniFlatten and SyncLLM claim overlapping speech as an architecturally supported behaviour. Only FireRedChat (via personalised VAD) claims third-party rejection. Mini-Omni2’s interruption is treated as keyword-triggered command mode. The authors stress that an empty cell means the capability is undocumented, not that the system fails it.
  • Data-bottleneck thesis (§7). The authors argue that, across L0-L2, training-data cell coverage rather than architecture sets the ceiling on demonstrated full-duplex behaviour. Their evidence: coverage differs among L2 systems in ways that do not follow sub-design; Moshi’s own report places turn-taking acquisition in its Fisher fine-tuning stage (the survey notes that Moshi reports no ablation that removes Fisher); and MinMo’s one-layer predictor depends on a 4,000-hour dialogue mix (figures as quoted from the respective papers in §4.8 and §7). The survey estimates the public two-channel conversational data in Table 5 at about 5k hours, dominated by English telephone audio. Its corpus-by-cell matrix finds no public corpus with sustained concurrent speech, and third-party speech only partially covered by HumDial Track 2 (§7.5, Table 6).
  • Realization gap and evaluation (§8). The survey defines the realization gap as the difference between the cells an architecture admits and the behaviour a trained system demonstrates. It recommends that each system report three things: its L-layer, the cell coverage of its training data, and its per-cell behaviour on the Full-Duplex-Bench v1.5/v2 protocols (§8.3). Its benchmark audit (Table 7) finds that standard turns and barge-in are tested by every benchmark it examines, while sustained concurrent speech is only partially probed (by the FDB v2 examiner). It also finds that no public benchmark includes a causal-influence probe that would separate substantive full-duplex systems from apparent ones, such as chunk-aligned designs (§8.4). It also clarifies that the Full-Duplex-Bench series (Lin et al.) and FD-Bench (Peng et al.) are separate benchmark lines whose numbers are not comparable (§8.1).

Novelty Assessment

The contribution is a taxonomy and an audit, not empirical work. The main new elements are the layer-based placement of the duplex decision, the factored T × I × R ontology, and the coupling of every state-machine transition to an ontology cell, which turns system descriptions into checklists that can be compared. The state machine extends existing FSM formulations (a two-state SPEAK/LISTEN scheme and FlexDuo’s three-state version) rather than starting from scratch. The training-data and benchmark coverage matrices are the most practically useful additions, and the FDB/FD-Bench disambiguation is a small but concrete service.

The empirical backing is thinner than the framing suggests. The coverage matrices are built from self-reported capabilities and the authors’ reading of them. The data-bottleneck thesis relies on correlational evidence from a few systems, with no controlled comparison. Its central contrast pairs Mini-Omni2 with OmniFlatten as “flattened-family” systems with the same architectural reach (§7, §8.3), but the survey’s own description of Mini-Omni relies on parallel text and audio heads and keyword-triggered interruption tokens (§4.5), so that pair does not isolate training data as the only difference. The L3 chapter (§9) is explicitly speculative: it proposes continuous-latent autoregressive streaming, JEPA-style dual-stream prediction, and world-model-conditioned dialogue based on text and vision results. The authors’ affiliations overlap with some of the systems and prior surveys they discuss: Qwen Business Unit authors, while Qwen-Omni models serve as taxonomy references, and shared authorship with WavChat.

Field Significance

Moderate. The survey provides a shared vocabulary for full-duplex spoken dialogue: decision layer, interaction cell, and state transition. With it, a claim such as “system X fails the backchannel cell” can be checked, and capability, training-data coverage, and benchmark coverage can be reported separately. Its value is organisational and diagnostic. Its specific coverage judgements and the data-bottleneck thesis are stated as interpretations of self-reported evidence, not as independently verified findings.

Claims

  • refines: Full-duplex capability in spoken dialogue systems is better described by where the turn-taking decision is made in the model stack and which interaction types are handled than by a binary cascaded versus end-to-end label.

    Evidence: The L0-L3 audit places systems claiming full-duplex at the module level (FireRedChat, FlexDuo, SoulX-Duplug), the hidden-state level (MinMo, Freeze-Omni), and the token level (Moshi, OmniFlatten, SyncLLM, and others). It assigns each layer a different set of reachable states; for example, DUAL is effectively unreachable at L0 and only limited at L1 (§4.2, Table 2).

  • complicates: Architectural capacity for overlapping speech does not guarantee demonstrated full-duplex behaviour, because training-data coverage of interaction types limits what a trained system realises.

    Evidence: Token-level systems that are all architecturally able to reach overlap states differ in the backchannel and concurrent-speech capabilities they claim. The survey attributes this to post-training data (Moshi’s Fisher fine-tuning, MinMo’s 4,000-hour dialogue mix) but acknowledges that the coverage matrix comes from self-reported claims and that no data-removal ablation exists (§7, §8.3, Table 1).

  • complicates: Public two-channel conversational corpora under-represent sustained concurrent speech and third-party speech, which limits open training for these full-duplex behaviours.

    Evidence: The corpus-by-cell matrix finds no representation of sustained concurrent speech in Fisher, CANDOR, MagicData-RAMC, Easy Turn, or HumDial Track 2, and third-party speech only partially in HumDial Track 2. The survey estimates the public pool at about 5k hours, mostly English telephone audio (§7.2, §7.5, Tables 5-6).

  • complicates: Full-duplex benchmarks over-test standard turns and barge-in and lack causal-influence probes, so stop-latency scores can reward systems that do not integrate user audio mid-turn.

    Evidence: Every benchmark in the audit covers standard turns and barge-in, only FDB v2 partially probes sustained concurrent speech, and none tests whether mid-turn behaviour actually responds to user audio. The survey argues that chunk-alternating designs such as SyncLLM can do well on stop latency without passing such a test (§8.2, §8.4, Table 7).

  • refines: Telling a user backchannel apart from a floor-claiming interruption during system speech requires classifying the user’s intent, not just detecting voice activity.

    Evidence: The state machine sends both cases through the same SPEAK → DUAL transition and separates them only by intent (continue versus yield). The survey describes a half-duplex system as treating every overlap as a handoff, and describes energy-based VAD as breaking off system speech mid-clause (§5.4, §6.3, Table 4).

Limitations and Open Questions

Warning

The per-system and cell-coverage audits rely on each paper’s self-reported capabilities rather than independent reproductions, and several industrial systems are closed-source. Coverage marks in Tables 1, 6, and 7 are therefore the authors’ interpretations, not measured behaviour (§Limitations, Table 1).

  • The frameworks are calibrated on existing L0-L2 systems. The L3 hypotheses are provisional, and the authors expect the ontology and state machine to need extending if L3 systems bring new interaction patterns (§Limitations, §6.6).
  • The data-bottleneck thesis is stated as conditional on the current, mostly static evaluation regime. The authors themselves note that adaptive multi-turn evaluation could expose reasoning-related gaps that more data would not close (§7.7).
  • Industrial training corpora are largely undisclosed, so the survey cannot quantitatively compare public and proprietary data, and the open-source versus industrial gap it describes stays unmeasured (§7.3).
  • The survey proposes no metric or benchmark of its own. The causal-influence probe it recommends is only sketched, not built or validated (§8.3, §8.4).

Wiki Connections

  • Spoken Language Model — The survey organises full-duplex speech LMs by where the turn-taking decision sits relative to the LLM (external module, hidden-state predictor, or token stream).
  • Speech-to-Speech — It addresses spoken dialogue systems specifically, covering both modular cascades and end-to-end speech-to-speech dialogue models within one hierarchy.
  • Evaluation Metrics — Its evaluation audit maps full-duplex benchmarks onto interaction-ontology cells, identifies untested cells, and proposes a three-part reporting frame (layer, data coverage, per-cell behaviour).
  • WavChat — An earlier survey with shared authors; this work critiques its cascaded/end-to-end organisation as too coarse for full-duplex behaviour.
  • Moshi — The main parallel-stream L2 example in the hierarchy and a key piece of evidence for the data-bottleneck thesis through its Fisher fine-tuning stage.
  • MinMo — The main L1 hidden-state example; its full-duplex predictor and 4,000-hour dialogue mix are central to the survey’s layer and data arguments.
  • OmniFlatten — The flattened-sequence L2 example, credited with claiming both backchannel and overlapping-speech support.
  • FireRedChat — The main L0 modular example and the only surveyed system credited with third-party speech rejection (via personalised VAD).
  • Mini-Omni2 — Treated as an apparent full-duplex system with keyword-triggered interruption, and used as the low-coverage half of the realization-gap contrast.
  • Full-Duplex-Bench v1.5 — Singled out as the benchmark whose four overlap scenarios best match the ontology’s overlap row, and recommended as the empirical anchor.
  • Full-Duplex-Bench-v2 — Its multi-turn examiner is the only audited benchmark that partially probes sustained concurrent speech.