arXiv · 2026 · Preprint

Hao Zhang et al. (Tencent AI Lab) · → Paper · Demo: ? · Code: ?

Offloads full-duplex turn-taking decisions in a cascaded spoken dialogue system to a separate 0.5B LLM (“semantic VAD”) that reads streaming ASR text and emits one of four control tokens, so the larger response-generating LLM is invoked only when the system should speak.

Problem

Full-duplex spoken dialogue systems must decide, in real time, whether a user has finished a query, whether a pause is only hesitation, and whether a barge-in during system speech is a genuine interruption or a backchannel, aside, or acknowledgment. Earlier dialogue managers built from small neural classifiers over shallow acoustic and lexical features (Duplex Conversation, speech collision detection) have limited semantic understanding. More recent approaches fold interaction control into the main LLM (NeuralFSM, time-division multiplexing duplex models, VITA, Moshi), which raises inference cost because the large model must run at every decision step and couples turn-taking behaviour to the response model’s training.

Method

The system is a cascade of six modules: acoustic echo cancellation, a speaker- and distance-aware acoustic VAD, ASR, the proposed semantic VAD acting as dialogue manager (DM), a larger LLM serving as the core dialogue engine (CDE), and TTS (§2.2, Figure 1a). The first four modules run on short intervals of incoming speech; the CDE is activated only on demand. The TTS and ASR components are not specified and are not the paper’s subject.

The DM is a Hunyuan 0.5B-dense-8k LLM fine-tuned with four added control tokens (§3.1, §3.4):

  • <|C-L|> (Continue-Listening): the query is incomplete; keep listening and cache the partial query.
  • <|S-S|> (Start-Speaking): the query is complete; invoke the CDE and speak.
  • <|S-L|> (Start-Listening): a real interruption (denial, follow-up inquiry, topic change) during system speech; stop speaking. Also emitted after a response finishes.
  • <|C-S|> (Continue-Speaking): a fake interruption (acknowledgment, backchannel, speech directed at someone else, unrelated comment); keep speaking.

The DM reads user query tokens from ASR plus the CDE’s historical response tokens and predicts the next action (§3.2, Figure 1c). Only user-initiated interruptions are allowed by design. The DM operates entirely on text; it does not consume audio.

Training data is synthetic and text-only (§3.3, §3.4). An algorithm samples a topic (200-topic pool), a speaking style (10 personas), a number of QA rounds (2 to 12), and per-round scenario types (real interruption, fake interruption, normal QA, with random truncation for incomplete queries), then builds a prompt asking the Tencent Yuanbao LLM to generate an annotated conversation. Of 20,000 prompted generations, about 60% survived cleaning, so the authors also generate ordinary QA dialogues and inject interaction patterns by rule-based post-processing. Query-final punctuation is altered to resemble ASR output, and matching uninterrupted dialogues are included to limit degradation of the base model. The final set has 11,990 Chinese conversations with 80,338 rounds (47% normal, 21% real interruption, 19% fake interruption, 13% incomplete query; Figure 1e). Fine-tuning runs 1,500 steps at batch size 128 with learning rate decaying linearly from 0.001 to 0.0001. The authors report that training sets with over 50% (or exclusively) interruption or incomplete-query cases overfit and degrade performance.

Key Results

On a synthetic test set of 1,000 samples per control token, generated with the same procedure from reconstructed topic and style pools, the DM reaches 0.9785 overall accuracy (Table 1). Barge-in handling is near perfect: <|C-S|> F1 1.000 and <|S-L|> F1 0.999. Query-completion detection is weaker: <|C-L|> recall 0.926 (74 of 1,000 incomplete queries misread as complete) and <|S-S|> precision 0.930, both F1 around 0.96.

Table 2 places these numbers next to figures reported by Duplex Conversation and RTTL-DG in their own papers. The authors state they had no access to those models or test sets, so the comparison is across different data and task definitions. As reported, the proposed DM has higher F1 on every row: user state detection 0.96 versus 0.91 (Duplex Conversation, single pooled score) and 0.85/0.52 (RTTL-DG, remain silent/initiate speaking); user intention detection 0.96/1.00 versus 0.89 (Duplex Conversation, pooled) and 0.62/0.95 (RTTL-DG, stop/keep speaking). The proposed system’s Start-Listening row in Table 2 (precision 0.99, recall 0.93, F1 0.96) differs from the same token in Table 1 (1.000, 0.999, 0.999); the Table 2 values match Table 1’s Continue-Listening row.

On internal real recordings of users interacting with a half-duplex system and told to hesitate naturally, a silence-threshold acoustic VAD alone can only emit <|S-S|>, so every incomplete query is answered prematurely (532 of 1,540 utterances at 300 ms, 132 of 1,240 at 1800 ms; Table 3). Adding the semantic VAD raises accuracy to 0.935, 0.962, 0.966, and 0.971 at 300, 500, 800, and 1800 ms thresholds. The <|C-L|> F1 stays between 0.880 and 0.920 across thresholds, lower than on synthetic data; precision on <|C-L|> drops to 0.800 at the 1800 ms threshold (from 0.887 to 0.892 at 300 to 500 ms), because fewer incomplete queries remain while about 32 to 63 complete queries are still wrongly held. The authors attribute some errors to ASR mistakes rather than the DM. No latency, compute savings, end-to-end conversation quality, or human evaluation is reported.

Novelty Assessment

The contribution is engineering integration. Instruction-tuning an LLM to emit turn-taking state tokens is close to prior state-token duplex schemes (NeuralFSM, VITA); what differs is placing that decision in a separate small text LLM in front of an untouched large CDE, which decouples DM optimisation from the response model. The synthetic data recipe (scenario-sampled prompts plus rule-based post-processing to recover from poor LLM instruction following) is a practical detail of moderate interest. The efficiency argument (fewer CDE calls) is asserted but not measured, the evaluation is mostly on self-generated synthetic text from the same pipeline used for training, the real-recording test covers only query completion, and the baseline comparison reuses numbers from other papers on other data. All data is Chinese.

Field Significance

Low — The paper shows that a 0.5B text LLM fine-tuned on synthetic annotated dialogues can separate real from fake interruptions and complete from incomplete queries well on in-distribution synthetic text, and can reduce premature responses relative to a silence-threshold VAD on real recordings. It provides a concrete design point for modular cascaded full-duplex systems, but without latency, cost, or end-to-end interaction measurements it does not establish the claimed efficiency trade-off.

Claims

  • supports: In cascaded full-duplex dialogue, semantic analysis of transcribed partial queries reduces premature responses compared with silence-duration endpointing alone.

    Evidence: On real recordings with instructed hesitations, a silence-threshold acoustic VAD answers every incomplete query (132 to 532 cases depending on threshold); adding the fine-tuned 0.5B semantic VAD raises accuracy to 0.935 to 0.971 across 300 to 1800 ms thresholds. (§4.3, Table 3)

  • refines: Distinguishing genuine interruptions from backchannels during system speech is easier for a text-based turn-taking model than detecting whether a user’s query is complete.

    Evidence: On the synthetic test set, Continue-Speaking and Start-Listening reach F1 1.000 and 0.999, while Continue-Listening and Start-Speaking reach F1 0.956 and 0.959, with 74 of 1,000 incomplete queries classified as complete; the authors attribute this to query completeness depending on speaking style and linguistic nuance. (§4.2, Table 1)

  • complicates: Turn-taking models trained and tested on LLM-generated synthetic dialogue overstate performance relative to real spoken interaction.

    Evidence: Continue-Listening F1 is 0.956 on the synthetic test set but 0.880 to 0.920 on internal real recordings, where ASR errors also introduce mispredictions. (§4.2, §4.3, Tables 1 and 3)

  • complicates: Training turn-taking models on interaction-heavy synthetic data requires balancing scenario proportions to avoid overfitting.

    Evidence: Training sets with over 50% or exclusively real/fake interruptions or incomplete queries overfit and degrade performance, and uninterrupted dialogues are added to preserve base-model behaviour. (§3.4)

Limitations and Open Questions

Warning

The main evaluation uses synthetic text dialogues produced by the same generation pipeline as the training data, and the baseline comparison (Table 2) quotes numbers from other papers on different test sets; neither establishes performance on real full-duplex speech. The claimed computational savings are never measured.

  • The real-recording test (Table 3) covers only query completion; barge-in handling (real versus fake interruptions) is evaluated only on synthetic text.
  • No latency figures are given for the DM, the ASR-to-DM loop, or time to response; the authors list system delay as future work (§5).
  • Chinese only; data, model, and test sets are not released.
  • The DM sees only ASR text, so prosodic and acoustic turn-taking cues are ignored beyond the acoustic VAD’s silence threshold, and ASR errors propagate directly into turn decisions.
  • The TTS and CDE components are unspecified, and no end-to-end conversational quality or user study is reported.

Wiki Connections

  • Speech-to-Speech — Contributes a modular turn-taking dialogue manager for a cascaded (AEC, VAD, ASR, LLM, TTS) full-duplex spoken dialogue system, as an alternative to folding interaction control into an end-to-end model.
  • Moshi — Cited as an end-to-end full-duplex model that integrates turn-taking into the main model; the paper argues its separate small DM avoids that inference overhead, but does not compare against it.
  • VITA — Cited as a state-token duplex scheme inside a large multimodal LLM; the proposed DM applies a similar control-token idea in a separate lightweight model.