arXiv · 2026 · Preprint

Qingkai Fang et al. (Institute of Computing Technology, Chinese Academy of Sciences / Key Laboratory of AI Safety, Chinese Academy of Sciences / University of Chinese Academy of Sciences) · → Paper · Demo: ? · Code: ✓

Converts a turn-based speech LM (GLM-4-Voice) into a full-duplex one without adding any module, by laying user speech, assistant text and assistant speech out as a block-interleaved token sequence in which four dialogue-state tokens turn every listen, speak and stop decision into ordinary next-token prediction.

Problem

Most speech LMs, whether they extend the LLM vocabulary with speech tokens or wrap the LLM with an encoder and decoder, assume turn-based interaction: an external voice activity detector (VAD) segments the user’s utterance and the model replies once it is complete. That front end caps interactive behaviour, since acoustic VAD cannot tell a mid-utterance pause from an end of turn, cannot handle barge-in, and has no access to dialogue semantics. Native full-duplex models such as Moshi avoid the VAD but rely on millions of hours of pretraining and large amounts of paired two-channel dialogue. Partially duplex systems (LSLM, Mini-Omni2, Freeze-Omni) let the user interrupt but do not let the model decide on its own when to start or stop speaking. The paper asks whether a strong turn-based speech LM can be converted into a competitive full-duplex one with a small, structured fine-tuning recipe.

Method

The backbone is GLM-4-Voice used unchanged: a speech tokenizer built from a Whisper-large-v3 encoder with a vector quantizer (16 kHz input, 12.5 Hz tokens, one token per 80 ms), a 9B decoder-only LLM initialized from GLM-4-9B with speech tokens in its vocabulary, and a speech decoder made of a flow-matching model and a HiFi-GAN vocoder adapted from CosyVoice. The tokenizer and speech decoder are frozen; only the LLM is fully fine-tuned, and no parameters or heads are added.

The contribution is the multi-channel interleaved sequence. A dialogue is rendered as two equal-length audio tracks (user and assistant, with silence where a party is not speaking), both tokenized by the same tokenizer; silence is tokenized as audio rather than replaced by a special token. The sequence is organized in blocks, each holding N user-speech tokens, then M text tokens, then N assistant-speech tokens; the paper uses N = 10 and M = 5, so each block spans 0.8 s and carries 6.25 text tokens per second on average. The assistant channels are shifted one block ahead of the user channel (a causal shift), so the text and speech emitted at block b correspond to the next 0.8 s of wall-clock time, and inference plays output back with the same offset.

The text channel works as an inner monologue that is never shown to the user but conditions the assistant speech in the same block. It defaults to a [SILENCE] token and is overwritten for each assistant utterance with [ASSISTANT] (reply onset), the text content, [PAD] (text finished, speech still being emitted) and [EPAD] (text and speech both complete). Turn-taking, being interrupted and restarting therefore reduce to next-token prediction over the standard vocabulary, without a classification head, attention-mask scheme or state machine.

Figure 1: Multi-channel interleaved sequence in BayLing-Duplex. The user speech, assistant text and assistant speech channels are interleaved block-by-block at a fixed N : M : N ratio; here N = M = 2 for clarity (we use N = 10, M = 5 in practice). The text channel embeds the dialogue-state tokens [SILENCE] ([S]), [ASSISTANT] ([A]), [PAD] ([P]) and [EPAD] ([E]). The illustrated dialogue starts with the user asking 'Hi, what time is it?'; the assistant takes the turn at 2.0 s with 'The time is...', is interrupted at 3.0 s by 'Wait, Beijing Time!', and re-starts with '9PM' at 4.5 s. Turn-taking, being interrupted, and re-starting are all expressed as ordinary next-token prediction over the standard LLM vocabulary.

Training has two stages on top of the public GLM-4-Voice checkpoint. Stage I is supervised fine-tuning with cross-entropy only at text-channel and assistant-speech positions (the user channel is conditioning only). Because [SILENCE] dominates the sequence and [ASSISTANT] appears once per turn, per-token weights rebalance the loss: the [SILENCE] weight is set to 0.1 and the [ASSISTANT]/[EPAD] weight to 10. Stage II is Direct Preference Optimization (DPO) with an auxiliary SFT term on the positive sequence (β = 0.5, auxiliary weight 0.5, 200 steps, reference policy = Stage I checkpoint). Preference pairs share user audio and response text and differ only in timing, so the update concentrates on the dialogue-state tokens.

Training data is derived from the 200K-sample multi-turn speech dialogue corpus of LLaMA-Omni 2 (Alpaca and UltraChat rewrites synthesized with CosyVoice voice cloning, diverse user voices and a single assistant voice). Each dialogue is converted into a turn-taking sample (0.8 s response gap) and an interruption sample (the user re-enters mid-response and the assistant stops after a 0.8 to 2.0 s reaction delay), giving 400K full-duplex samples mixed 1:1. DPO negatives stretch the response gap to 2 to 5 s or the interruption reaction delay to 3 to 5 s.

At inference, decoding proceeds block by block: the model receives 10 user-speech tokens from the live stream, samples 5 text-channel tokens (masked to text and state tokens), then 10 assistant-speech tokens (masked to speech tokens), and decodes and plays that audio. Without the masks the LLM occasionally emits cross-channel tokens that corrupt the speech decoder input.

Key Results

All evaluation is fully automatic: assistant audio is transcribed with Whisper-large-v3, segmented with Silero VAD (used only for evaluation, not inference), and response quality is judged by GPT-4o on a 1 to 5 scale. The only external baseline is Moshi (public Moshika checkpoint).

  • Full-duplex spoken QA (Table 1): with the spoken question fed directly and no external VAD, BayLing-Duplex (+DPO) reaches 46.0% on Llama Questions and 18.1% on Web Questions, against 21.0% and 9.2% for Moshi. The SFT-only model (44.3% / 18.0%) already exceeds Moshi on both sets.
  • Turn-taking (Table 2, InstructS2S-Eval, 199 items): the share of items answered within 3 s of the user’s end rises from 71.9% (Moshi) to 88.9% (SFT) and 92.0% (+DPO); the GPT-4o speech-response score goes from 2.17 to 3.23 and 3.39.
  • Interruption (Table 2): the overlap from user barge-in to assistant stop (lower is better) drops from 2.07 s (Moshi) to 1.51 s (SFT) and 1.10 s (+DPO); the share of items stopping within 2 s rises from 81.9% to 91.4% and 100%; the score for the reply to the interrupting question rises from 2.45 to 2.95 and 3.27. Both BayLing-Duplex variants beat Moshi on every column of Tables 1 and 2.
  • Token-weight ablation (Table 3): uniform weights collapse the model toward near-permanent silence (60.3% turn-taking success; its 100% ISR@2s is a degenerate effect of rarely speaking, as the paper notes). Down-weighting [SILENCE] alone gives 82.4%, up-weighting role tokens alone gives 73.9%, and both together give 88.9%.
  • DPO sweep (Table 4): ISR@2s is 100% for all nine (β, auxiliary weight) settings. The default (0.5, 0.5) has the highest turn-taking success (92.0%), speech-response score (3.39) and Q2 score (3.27), but not the lowest overlap (1.10 s versus 1.01 s for three settings with auxiliary weight 0.3).
  • Duplex versus turn-based (Table 5): against a turn-based SFT model trained on the same data and backbone, the duplex SFT model scores 1.0 point lower on Llama Questions (44.3 vs 45.3), 2.1 points higher on Web Questions (18.0 vs 15.9) and 0.07 higher on Alpaca-Eval S2S (3.23 vs 3.16). The authors read this as “on par or stronger”; the differences are small in both directions and no significance testing is reported.

Comparison fairness: Moshi was trained under a different recipe and backbone than GLM-4-Voice, so the gap mixes the layout’s effect with backbone differences. Other full-duplex systems discussed in related work (OmniFlatten, SALMONN-omni, FLM-Audio, SyncLLM) are not evaluated. Absolute response latency in milliseconds is not reported; timing is reported only through threshold success rates and overlap.

Novelty Assessment

The novel element is the sequence layout and the in-vocabulary dialogue-state tokens: block-level interleaving of three channels with a causal one-block shift, so that turn-taking and interruption become next-token prediction with no new module. Related ideas exist (Moshi’s inner monologue, OmniFlatten’s flattened token stream, FLM-Audio’s natural monologues); the differentiator claimed here is the coarse block granularity that keeps text contiguous, which the paper conjectures matches the backbone LLM’s text distribution better, but this conjecture is not tested (no block-size sweep and no comparison with per-token interleaving). The training recipe (loss reweighting for rare state tokens, timing-only DPO negatives) is simple and its components are ablated. The practical point is data efficiency: converting a public turn-based checkpoint with 400K synthesized samples rather than pretraining a duplex model from scratch.

Field Significance

Moderate. The paper shows that a turn-based speech LM can acquire full-duplex turn-taking and interruption behaviour through a sequence-layout change, a reweighted SFT loss and a timing-focused DPO stage, without architectural additions, and that timing-only preference pairs move interruption behaviour more than turn-taking. The evidence is limited by fully synthetic audio, a single external baseline and automatic-only scoring.

Claims

  • supports: Full-duplex turn-taking and interruption can be learned by a single autoregressive speech LM as ordinary next-token prediction over in-vocabulary dialogue-state tokens, without an external VAD or a dedicated turn-taking module.

    Evidence: Four state tokens in a block-interleaved text channel on top of an unmodified GLM-4-Voice backbone reach 92.0% turn-taking success within 3 s and 100% interruption success within 2 s on InstructS2S-Eval, versus 71.9% and 81.9% for Moshi. (§2.1, §4.3, Table 2)

  • supports: Converting a turn-based speech LM to full-duplex operation through fine-tuning on interleaved dialogue data need not degrade its response content quality.

    Evidence: Against a turn-based SFT model on the same data and backbone, the duplex model scores 44.3 vs 45.3 on Llama Questions, 18.0 vs 15.9 on Web Questions and 3.23 vs 3.16 on Alpaca-Eval S2S. (§4.4, Table 5)

  • refines: When dialogue-state tokens are heavily imbalanced in a duplex training sequence, uniform cross-entropy drives the model toward permanent silence, so rare turn-taking tokens need explicit loss reweighting.

    Evidence: Uniform weights give 60.3% turn-taking success; reducing the [SILENCE] weight to 0.1 gives 82.4%, raising the role-token weight to 10 alone gives 73.9%, and both together give 88.9%. (§4.4, Table 3)

  • supports: Preference optimization over pairs that differ only in response timing improves the timing behaviour of a spoken dialogue model while preserving or improving response content.

    Evidence: DPO with timing-perturbed negatives lowers interruption overlap from 1.51 s to 1.10 s and raises ISR@2s from 91.4% to 100%, while the speech-response score rises from 3.23 to 3.39 and spoken-QA accuracy from 44.3/18.0 to 46.0/18.1. (§3, §4.3, Tables 1 and 2)

  • complicates: Block-based duplex layouts trade response latency against the text budget available per block, setting a floor on how quickly the model can respond.

    Evidence: With N = 10 the block spans 0.8 s, which caps minimum response latency at 0.8 s; smaller blocks leave too few text slots and produce jittery turn-taking, and no systematic sweep over N is reported. (§2.1, §Limitations)

Limitations and Open Questions

Warning

All training and evaluation audio is synthesized (described by the authors as single-speaker, near-field and noise-free), the only external baseline is Moshi, and every quality score is automatic (Whisper transcription plus GPT-4o judging) with no human listening test. Robustness to real conversational audio with noise, reverberation and competing speakers is untested.

  • The block size is fixed at N = 10 (0.8 s granularity) and not swept, so the latency/quality trade-off and the conjectured benefit of contiguous text over per-token interleaving remain unverified.
  • Backchannels, multi-party conversation and emotion-aware turn-taking are not addressed; the training data contains only clean turn-taking and interruption scenarios with fixed or uniformly sampled gaps.
  • Absolute end-to-end latency (for example, time from user end to first audio) is not reported beyond threshold success rates.
  • The system inherits GLM-4-Voice’s limitations on rare languages, code-switching and out-of-distribution acoustics, and evaluation is English-only.
  • Gains over Moshi cannot be attributed solely to the layout, since backbones and training data differ.

Wiki Connections

  • Spoken Language Model — Fine-tunes the GLM-4-Voice speech LM to consume a live external user-speech stream and decide when to listen, speak and stop.
  • Speech-to-Speech — An end-to-end full-duplex spoken dialogue system evaluated on spoken QA, turn-taking and interruption.
  • RLHF Speech — Adds a DPO stage with timing-only preference pairs and ablates its KL and auxiliary-SFT coefficients.
  • GLM-4-Voice — Provides the backbone checkpoint (tokenizer, LLM and frozen speech decoder) that BayLing-Duplex converts to full-duplex operation.
  • Moshi — The sole external baseline and the main native full-duplex reference; BayLing-Duplex outperforms it on all reported spoken-QA, turn-taking and interruption metrics.
  • LLaMA-Omni — Earlier turn-based speech LM from the same group; supplies the InstructS2S-Eval protocol used for turn-taking evaluation.
  • LLaMA-Omni 2 — Source of the 200K multi-turn speech dialogue corpus converted into full-duplex training data.
  • OmniFlatten — Related full-duplex design that flattens channels at per-token granularity, contrasted with BayLing-Duplex’s block-level interleaving (not evaluated).
  • SALMONN-omni — Continuous-embedding full-duplex speech LM discussed as an alternative design (cited in its arXiv version, not evaluated).
  • Freeze-Omni — Cited as a partially duplex system that supports interruption but cannot decide on its own when to speak.
  • LSLM — Cited as an early listen-while-speaking model with input-side barge-in only.