arXiv · 2026 · Preprint
Ruiqi Yan et al. (Shanghai Jiao Tong University; Soul AI Lab; Northwestern Polytechnical University) · → Paper · Demo: ? · Code: ✓
Introduces a plug-and-play streaming module that unifies VAD, ASR, and turn detection into a single interleaved-token architecture so any half-duplex spoken dialogue model can gain full-duplex turn management without retraining its backbone.
Problem
Full-duplex spoken dialogue systems (FD-SDSs) either train end-to-end models that jointly learn speech understanding, turn-taking, and response generation, or bolt a modular “VAD-ASR-turn detection” pipeline onto an existing half-duplex system. End-to-end approaches entangle turn-taking policy with language modeling, which limits controllability and interpretability, and they require large-scale full-duplex conversational data that is costly to collect. Modular pipelines are easier to deploy but their VAD stage relies on acoustic features alone, without access to semantic content, while non-streaming ASR and turn-detection components add latency that grows with input length. Prior semantic-aware VAD variants (e.g., Phoenix-VAD) improve turn-boundary awareness but do not explicitly consume text as an input signal, leaving the integration of speech recognition and semantic turn detection unresolved. Separately, existing FD-SDS benchmarks are mostly self-constructed or English-only, making bilingual comparison difficult.
Method
SoulX-Duplug reformulates duplex interaction control as a streaming state-prediction problem over incrementally available observations, rather than a cascade of independent VAD, ASR, and turn-detection modules. User-side audio is tokenized with the GLM-4-Voice speech tokenizer (block-causal, 12.5 Hz), producing discrete audio tokens that are linearly projected into the LLM’s embedding space. The model, built on a Qwen3-0.6B backbone, processes 160 ms audio chunks and generates an interleaved sequence of audio, text (streaming ASR), and dialogue-state tokens: for each chunk it first predicts the ASR output conditioned on the chunk history, then predicts a dialogue-state token conditioned on the ASR output for that same chunk. This ordering ensures that textual (semantic) information is available before the state decision is made, so the module functions as a “semantic VAD.”
Five state tokens capture interaction dynamics: <|user_idle|> (no semantic content), <|user_nonidle|> (meaningful speech present), <|user_backchannel|>, <|user_complete|> (assistant may take the turn), and <|user_incomplete|> (assistant should keep waiting). Training uses a token-type-weighted cross-entropy loss to balance the highly imbalanced frequencies of text tokens, end-of-ASR tokens, and state tokens. Training proceeds in three stages: non-streaming ASR pretraining, streaming ASR adaptation, and duplex state-prediction fine-tuning (LoRA, rank 32, applied only in the final stage; the GLM-4-Voice tokenizer stays frozen throughout). At inference, rather than relying on the model’s own streaming ASR output, a separate lightweight state-of-the-art ASR model (SenseVoice Small for English, Paraformer for Mandarin) supplies teacher-forced text tokens each chunk, which the authors find more accurate and efficient than self-generated ASR under the model’s short chunk size. Because the module treats VAD, ASR, and state prediction as one streaming decoding process, it can be attached to any half-duplex spoken dialogue model as an external controller without modifying that model’s backbone.
Under a uniform-distribution assumption over speech endpoints within an 160 ms chunk, the module has a theoretical average algorithmic latency of 240 ms (80 ms expected endpoint offset + 160 ms chunk duration).
The paper also releases SoulX-Duplug-Eval, a bilingual evaluation suite: an English extension of the Easy Turn testset (Complete/Incomplete utterances synthesized with ChatTTS from ChatGPT-generated text), and a Chinese counterpart to the Full-Duplex-Bench series covering turn-taking, pause handling, user backchannel, and user interruption scenarios (synthesized with ChatTTS and SoulX-Podcast).
Key Results
On Bilingual Full-Duplex-Bench, a cascaded FD-SDS built from SoulX-Duplug (state control) + Qwen2.5-7B-Instruct (response generation) + IndexTTS-1.5 (synthesis) attains the highest overall turn-management score among all compared systems in both English (0.812) and Chinese (0.916), and the best overall latency in Chinese (0.847 s) while ranking second-best in English (0.691 s) (Table 2). It reaches a Turn-Taking take-over rate of 0.933, close to end-to-end continuous-output baselines (dGSLM, PersonaPlex, Moshi), while showing substantially lower false-takeover rates on Pause Handling than those same baselines, and it outperforms Freeze-Omni across all three shared metrics in both languages (§6.1.1, Table 2).
On the Bilingual Easy Turn testset, SoulX-Duplug’s streaming state-prediction accuracy (83.32% EN, 84.33% ZH average) trails a non-streaming pipeline of a state-of-the-art ASR model plus a 7B-parameter TEN Turn Detection model by only about 3 percentage points, despite operating in a fully streaming mode with no external VAD dependency (§6.1.2, Table 3). In standalone latency measurement on the Turn Taking task, SoulX-Duplug achieves a deployed latency of 205 ms (EN) / 295 ms (ZH), lower than FlexDuo (343 ms, numbers taken from the FlexDuo paper) and a standard VAD-based pipeline (500 ms) (§6.1.3, Table 4). Ablations show that removing the ASR-pretraining stages drops accuracy from 84.33% to 80.50%, and removing teacher-forced external ASR at inference drops it further to 73.50%, confirming that both the semantic pretraining and the teacher-forced textual guidance are load-bearing for state prediction (§6.2, Table 5).
Novelty Assessment
The core novelty is architectural: unifying VAD, ASR, and turn detection into a single streaming decoder that interleaves audio, text, and state tokens so that state prediction is always conditioned on freshly generated text for that same chunk, rather than composing independently trained VAD/ASR/turn-detection modules in a cascade. The paper’s own comparison table (Table 1) positions this as the first plug-and-play state-prediction module that is simultaneously streaming, ASR-capable, and end-to-end optimized. The hybrid train/inference strategy (end-to-end training over VAD+ASR+state, but teacher-forced external ASR at inference) is a pragmatic engineering choice rather than a new architectural idea, but the ablations show it materially improves the resulting accuracy. The SoulX-Duplug-Eval benchmark is a genuine, if incremental, evaluation contribution: it extends two existing benchmarks (Easy Turn, Full-Duplex-Bench) with the missing language variant rather than introducing new evaluation dimensions.
Field Significance
moderate — the paper contributes a plug-and-play alternative to prior modular full-duplex controllers (Easy Turn, FireRedChat, Phoenix-VAD) that closes much of the gap to non-streaming, larger baselines while remaining streaming and low-latency, and it demonstrates that a downstream FD-SDS built on it can outperform several published systems on turn management and latency. It also releases a bilingual evaluation suite that fills a concrete gap (Easy Turn was Chinese-only; Full-Duplex-Bench was English-only). The contribution is scoped to a single module rather than a full dialogue system, and its downstream FD-SDS evaluation depends on off-the-shelf LLM and TTS components rather than novel generation techniques.
Claims
- supports: A streaming state-prediction module can be attached to an existing half-duplex spoken dialogue model to grant full-duplex turn-taking behavior without modifying or retraining that model’s backbone.
Evidence: SoulX-Duplug is trained independently and then composed with an off-the-shelf Qwen2.5-7B-Instruct LLM and IndexTTS-1.5 TTS system to form a complete FD-SDS evaluated on Full-Duplex-Bench, with no modification to either downstream component. (§5.2, §6.1.1)
- supports: Explicitly conditioning dialogue-state prediction on jointly generated streaming ASR text, rather than acoustic features alone, improves turn-taking and interruption-handling accuracy over acoustic-only VAD.
Evidence: Ablating ASR pretraining from the training pipeline drops Easy Turn testset (Zh) average accuracy from 84.33% to 80.50%, and removing the teacher-forced external ASR guidance at inference drops it further to 73.50%. (§6.2, Table 5)
- complicates: Streaming state prediction under very short audio chunks trades some accuracy for latency relative to non-streaming pipelines that use larger, non-streaming ASR and turn-detection models.
Evidence: SoulX-Duplug’s average accuracy (83.32% EN, 84.33% ZH) is about 3 percentage points lower than a non-streaming pipeline combining a state-of-the-art ASR model with a 7B-parameter TEN Turn Detection model, though the non-streaming pipeline incurs additional, input-length-dependent VAD latency not reflected in its reported accuracy. (§6.1.2, Table 3)
- complicates: Streaming ASR at very small chunk sizes remains inherently unreliable for languages where word boundaries can span adjacent chunks, limiting how aggressively latency can be reduced without accuracy loss.
Evidence: The authors report that short chunk durations frequently cut across phoneme, syllable, or word boundaries, an effect “particularly pronounced in English,” causing recognition instability, and note that LLM-based streaming ASR does not necessarily outperform conventional architectures like RNN-T under this constraint. (§6.3)
Limitations and Open Questions
The full downstream FD-SDS results (Table 2) evaluate SoulX-Duplug only as part of a cascaded system with a fixed external LLM (Qwen2.5-7B-Instruct) and TTS (IndexTTS-1.5); the paper does not report how much of the overall turn-management or latency improvement is attributable to the state-prediction module in isolation versus the choice of these downstream components, beyond the standalone latency measurement in Table 4.
The Chinese state-prediction training data relies on a proprietary ten-thousand-hour in-house corpus rather than an open-source dataset, since the authors report no suitable open-source Mandarin equivalent to the Fisher corpus exists; this limits reproducibility of the Mandarin training pipeline outside the authors’ organization. The authors also note that streaming ASR under short chunk constraints remains an open problem, particularly for English, and that current open-source streaming speech encoders and ASR models remain immature for fully real-time deployment, an issue the paper’s own approach only partially mitigates via teacher-forced external ASR.
Wiki Connections
- Spoken Language Model — reformulates full-duplex turn control as a single autoregressive decoder over interleaved audio, text, and state tokens rather than a cascade of separately trained models.
- Streaming TTS — applies streaming, chunk-based decoding principles to the dialogue-state-control side of a full-duplex system, complementing streaming synthesis work with streaming turn management.
- Speech-to-Speech — supplies the turn-taking and interruption-handling controller for a full-duplex spoken dialogue pipeline, one of the paper’s own defined sub-paradigms.
- Evaluation Metrics — introduces SoulX-Duplug-Eval, a bilingual extension of the Easy Turn testset and Full-Duplex-Bench series, to standardize cross-lingual FD-SDS comparison.
- Easy Turn — extends Easy Turn’s Chinese-only testset with an English counterpart and uses Easy Turn’s non-streaming ASR+turn-detection pipeline as its main non-streaming accuracy baseline.
- Phoenix-VAD — contrasts with Phoenix-VAD’s LLM-based semantic endpoint detector, which enhances semantic awareness but does not incorporate ASR or explicit text input the way SoulX-Duplug does.
- PersonaPlex — compared as an end-to-end continuous-output baseline on Bilingual Full-Duplex-Bench, where PersonaPlex achieves very high turn-taking rates but also a high pause-handling take-over rate.
- FireRedChat — cited as a representative cascaded VAD-ASR-turn-detection modular system, the architectural pattern SoulX-Duplug’s unified streaming design is proposed as an alternative to.
- SoulX-Podcast — used as the TTS system to synthesize the User Backchannel subset of the paper’s own SoulX-Duplug-Eval benchmark.
- Open-Source Full-Duplex Conversational Datasets — cited as the closest existing open bilingual dual-stream conversational corpus, noted as limited to about 15 hours of total duration.
- X-Talk — cited in support of the paper’s argument that modular full-duplex designs offer deployment flexibility advantages over end-to-end architectures.