arXiv · 2026 · Preprint
Jianing Yang et al. (SB Intuitions Corp. / The University of Tokyo) · → Paper · Demo: ? · Code: ?
A VAD-free cascaded ASR-LLM-TTS pipeline that converts long dialogue turns into chunk-wise micro-turns with dedicated control tokens, achieving full-duplex turn-taking without sacrificing the text LLM’s conversational intelligence.
Problem
Cascaded spoken dialogue systems (ASR transcribes speech, an LLM reasons over text, TTS synthesizes the reply) inherit the strong instruction-following and reasoning abilities of modern text LLMs, but they typically depend on an external voice activity detector (VAD) to decide turn boundaries. VAD-based endpointing struggles to use conversational semantics, so cascaded systems interrupt users during meaningful pauses or stay silent when a backchannel or immediate response is expected, producing brittle, half-duplex “listen-then-speak” behavior. End-to-end (E2E) speech-text models remove VAD segmentation and support full-duplex listening and speaking, but jointly learning crossmodal representations and dialogue policy tends to degrade conversational intelligence relative to a dedicated text LLM. The paper targets this trade-off: how to obtain VAD-free, full-duplex turn-taking control while preserving the reasoning quality of a cascaded, LLM-centered design.
Method
DuplexCascade restructures a conventional cascaded pipeline (streaming ASR, an LLM, and streaming TTS) around fixed-length “micro-turns” instead of full utterances. User audio is fed continuously into a streaming ASR module, and partial transcriptions are periodically flushed (every Δt = 0.6s) into a text micro-turn. The LLM consumes the interleaved history of user and system micro-turns, separated by a dedicated end-of-turn token <EOS>, and predicts the next system micro-turn, which is sent incrementally to a streaming TTS module.
To make turn-taking decisions controllable under these streaming constraints, the LLM’s vocabulary is extended with conversational special tokens. On the user side, <no voice> marks silence for the current buffer interval. On the system side, tokens such as <user is speaking>, <user finish speaking>, <user is interrupting>, <user backchannel>, <user is thinking>, and <system backchannel> steer whether the system should stay silent, take the turn, abandon a response after an interruption, ignore a backchannel, wait while the user thinks, or emit a pre-synthesised backchannel clip. A micro-turn therefore does not necessarily carry speech content; it can consist solely of a control token.
Because annotated full-duplex speech corpora are scarce, the authors adapt a Qwen2-7B-Instruct backbone (following Freeze-Omni’s choice of backbone) using only 50k text-only multi-turn dialogues sampled from UltraChat. A dynamic construction procedure converts each text dialogue into a micro-turn sequence: user and system long turns are split into chunks, silent user micro-turns (<no voice>) are inserted between system micro-turns, and <user finish speaking> prepends the first system micro-turn of each original turn. The procedure further simulates six interaction phenomena with sampled probabilities: randomized micro-turn length (to emulate streaming ASR variability), natural pauses (inserted <no voice> runs), user interruptions (abandoning the current response mid-generation), user backchannels (short acknowledgements the system should ignore), system backchannels (inserted via an LLM-based post-processing pass with Qwen2-72B-Instruct to mark natural backchannel points), and user “thinking” pauses after a system response. Training applies next-token-prediction LoRA fine-tuning (rank 16, alpha 32, on query/value projections) only on system micro-turns, while the token embedding matrix, the new special-token embeddings, and the prediction head are fully fine-tuned; the remaining backbone parameters stay frozen except through LoRA. A weighted loss addresses class imbalance across the six system control tokens. Two variants are trained: DuplexCascade (no system-backchannel supervision) and DuplexCascade-β (with the <system backchannel> token and its simulation). The streaming ASR and streaming TTS components are instantiated from DSM-ASR and DSM-TTS (Delayed Streams Modeling). Training uses AdamW, a linear warmup over 500 steps, batch size 32, 5k steps, and a maximum sequence length of 4096 tokens, on 8 NVIDIA H100 GPUs for 5 hours.
Key Results
On Full-Duplex-Bench, the authors introduce an Averaged Turn-Taking Accuracy metric that unifies the benchmark’s direction-inconsistent Take-Over Rate (TOR) subsets into a single score. DuplexCascade attains the best Averaged Turn-Taking Accuracy (0.858) among the evaluated systems, ahead of Gemini Live (0.778), the VAD-based Freeze-Omni (0.489), the E2E Moshi (0.395), and PersonaPlex (0.759), and clearly ahead of dGSLM (0.466) and the inferred MiniCPM-Duplex checkpoint (0.598). It is also the best system on Synthetic and Candor Pause-Handling TOR (0.058 and 0.222, both lower-is-better).
On VoiceBench, which measures the preserved conversational intelligence of the underlying LLM, DuplexCascade reaches an Overall score of 65.41 and DuplexCascade-β 65.81, both well above the other duplex baselines (Freeze-Omni 55.2, PersonaPlex 30.59, Moshi 29.51) and close to a naive, non-duplex DSM-ASR + Qwen2-7B-Instruct cascade (69.66). DuplexCascade-β additionally places second-best on the backchannel-specific ICC Freq and JSD metrics on Full-Duplex-Bench, though at a lower Averaged Turn-Taking Accuracy (0.748) than the non-backchannel DuplexCascade variant. A micro-turn duration analysis (Δt from 0.3s to 1.8s) shows Averaged Turn-Taking Accuracy improving up to Δt = 1.2s before degrading, while latency increases monotonically with Δt; the authors select Δt = 0.6s in the main experiments as a practical accuracy/latency trade-off rather than the accuracy-optimal setting.
Novelty Assessment
The architecture itself, streaming ASR, an LLM, and streaming TTS, is not new; the contribution is the micro-turn reformulation of the cascaded pipeline and the accompanying set of conversational control tokens that let the LLM make VAD-free turn-taking decisions directly from dialogue context rather than from an external endpointing head. The dynamic, probabilistic construction of duplex training sequences from ordinary text-only dialogues (rather than requiring scarce annotated full-duplex speech corpora) is a second concrete contribution, since it lets the method sidestep the largest practical obstacle to training duplex control policies. The result is best characterized as an architectural-novelty contribution at the level of interaction protocol and training-data design, layered on top of otherwise standard streaming ASR/TTS/LLM components, evaluated with a moderate-scale LoRA adaptation (5k steps, single 7B backbone) rather than at large model or data scale.
Field Significance
moderate — DuplexCascade offers a lightweight, backbone-agnostic recipe (small-scale LoRA fine-tuning on synthetically constructed text-only dialogues) for retrofitting full-duplex, VAD-free turn-taking control onto an existing cascaded ASR-LLM-TTS pipeline, while empirically demonstrating on two independent benchmarks (Full-Duplex-Bench, VoiceBench) that this can be done without the conversational-intelligence degradation typically seen in end-to-end duplex systems. Its contribution is scoped to a single backbone and a single training-data recipe rather than a new architecture family.
Claims
- supports: Converting long dialogue turns into short, chunk-wise micro-turns paired with an explicit set of control tokens lets a cascaded (ASR-LLM-TTS) pipeline make VAD-free, full-duplex turn-taking decisions directly from LLM context.
Evidence: DuplexCascade achieves the best Averaged Turn-Taking Accuracy (0.858) on Full-Duplex-Bench among evaluated open-source duplex systems, outperforming the VAD-driven Freeze-Omni (0.489) and the end-to-end Moshi (0.395). (§4.2, Table 1)
- supports: Adapting a cascaded dialogue system for full-duplex behavior via text-only fine-tuning of the LLM backbone, rather than joint speech-text training, can retain most of the backbone’s conversational intelligence.
Evidence: DuplexCascade’s VoiceBench Overall score (65.41) sits close to a naive, non-duplex DSM-ASR + Qwen2-7B-Instruct cascade (69.66) and far above end-to-end duplex baselines Moshi (29.51) and PersonaPlex (30.59). (§4.3, Table 2)
- supports: Full-duplex conversational phenomena, pauses, interruptions, and backchannels, can be synthetically simulated from ordinary text-only multi-turn dialogue corpora to train duplex turn-taking policies, avoiding dependence on scarce annotated full-duplex speech corpora.
Evidence: The training pipeline dynamically converts 50k UltraChat text dialogues into micro-turn sequences by probabilistically inserting silence, interruption, and backchannel patterns (probabilities 0.10, 0.30, and 0.01 respectively), with no real full-duplex speech corpus required. (§3.3)
- complicates: The turn-taking accuracy gains from finer-grained micro-turn segmentation trade off against response latency, and the accuracy-optimal segmentation granularity is not the latency-optimal one.
Evidence: Averaged Turn-Taking Accuracy on Full-Duplex-Bench improves as micro-turn duration Δt increases up to 1.2s and then degrades, while Smooth Turn-Taking Latency increases monotonically with Δt; the authors adopt Δt = 0.6s as a practical trade-off rather than the accuracy-maximizing setting. (§4.4, Figure 3)
Limitations and Open Questions
Evaluated with a single 7B-parameter LLM backbone (Qwen2-7B-Instruct) and a single training-data recipe (50k UltraChat-derived dialogues, 5k LoRA steps); the paper does not report whether the micro-turn and control-token approach generalizes to other backbone sizes, languages, or dialogue domains.
The synthetic training data construction relies on fixed, hand-set probabilities for pauses, interruptions, and backchannels (e.g., 0.10, 0.30, 0.01) and on an auxiliary LLM (Qwen2-72B-Instruct) to mark backchannel points in text-only dialogues; how well these heuristics match real conversational statistics is not directly validated against a real full-duplex corpus. Enabling the <system backchannel> token (DuplexCascade-β) improves backchannel-specific metrics but reduces Averaged Turn-Taking Accuracy relative to the non-backchannel variant, indicating a tension between the two objectives that is not fully resolved. The chosen micro-turn interval (Δt = 0.6s) is explicitly a trade-off rather than an optimum, and Smooth Turn-Taking latency under DuplexCascade (1.724s) is higher than several baselines, including PersonaPlex (0.24s) and Moshi (0.257s), despite DuplexCascade’s stronger aggregate accuracy.
Wiki Connections
- Speech-to-Speech — proposes a cascaded (ASR-LLM-TTS) sub-paradigm for full-duplex spoken dialogue that removes VAD dependence via micro-turn segmentation and control tokens.
- Streaming TTS — relies on streaming TTS synthesis (DSM-TTS) to incrementally produce system audio from LLM-generated micro-turns, a prerequisite for full-duplex interaction.
- Evaluation Metrics — introduces an Averaged Turn-Taking Accuracy metric that aggregates Full-Duplex-Bench’s direction-inconsistent Take-Over Rate subsets into a single comparable score.
- Full-Duplex-Bench — uses this benchmark as the primary evaluation suite for turn-taking quality and builds its aggregate accuracy metric on top of its reported TOR subsets.
- VoiceBench — uses this benchmark to evaluate whether the backbone LLM’s conversational intelligence is preserved after duplex adaptation.
- Freeze-Omni — adopts the same Qwen2-7B-Instruct backbone choice and is evaluated as a VAD-based cascaded baseline, exhibiting weaker turn-taking robustness on Full-Duplex-Bench.
- Moshi — compared as an end-to-end VAD-free duplex baseline, showing strong turn-taking metrics but substantially degraded VoiceBench conversational intelligence.
- PersonaPlex — compared as a full-duplex conversational speech baseline on both Full-Duplex-Bench and VoiceBench.
- OmniFlatten — cited as related work on restructuring interaction into short alternating segments (time-division multiplexing) without external VAD.
- WavChat — cited as a survey reference situating this cascaded pipeline within the broader landscape of spoken dialogue model designs.