arXiv · 2026 · Preprint

Hui Lu et al. (The Chinese University of Hong Kong / SenseTime Research / Tsinghua University) · → Paper · Demo: ✓ · Code: ?

A controlled comparison of two ways to feed the user’s speech stream into a frozen LLM backbone during full-duplex generation, showing that channel fusion gives much stronger spoken QA and interruption handling, while cross-attention routing keeps the model’s own generation context intact when an interruption is missed.

Problem

Full-duplex spoken dialogue requires a model to keep processing user speech while it is still generating its own response, but text LLMs are pre-trained to extend a single sequence. Existing full-duplex systems answer the question of how user speech enters the LLM only implicitly: some interleave fixed-size chunks of both streams into one sequence (doubling sequence length and filling the context with silent user chunks), others sum the streams at each time step (channel fusion), which removes any architectural separation between user input and the model’s own generation context. The paper argues that this routing choice is a central design axis and that its failure modes under overlapping speech, in particular corrupted continuations after a missed interruption, have not been characterised or compared under matched conditions.

Method

Both variants share one system built on a frozen Qwen3-1.7B backbone with LoRA adapters (rank 16, α = 32). User audio passes through a streaming speech encoder initialised from Whisper and made causal (left-padded convolutions, causal self-attention masks, sinusoidal positions replaced by RoPE), processed in fixed-size chunks with KV caching, then a three-layer adapter that halves the frame rate. Output speech uses the supervised semantic tokenizer of CosyVoice 2 (25 Hz) and its frozen token2wav decoder. An autoregressive audio head, a small decoder LM initialised from Qwen3-0.6B, reads the backbone’s last-layer hidden state and emits groups of G = 4 audio tokens per text position, starting after a delay of D = 2 text tokens; each generated group is embedded and fed back as the model-audio stream. The user stream, model text stream and model audio stream share one duplex timeline.

The two routing variants differ only in how the user stream reaches the backbone. CF-Duplex (channel fusion) sums the user, model-text and model-audio embeddings at each aligned step, plus a sigmoid-gated MLP over their concatenation, so the backbone sees one autoregressive stream. XA-Duplex (cross-attention routing) keeps the user stream as external memory: Flamingo-style dense cross-attention adapters, inserted at every second backbone layer (14 layers), use backbone hidden states as queries and RoPE-indexed user embeddings as keys and values. XA-Duplex is larger (3.40B vs. 2.90B parameters including the backbone; Table 8).

Waiting and interruption behaviour is learned through ordinary next-token supervision of special tokens (<USER_WAIT>, <TEXT_WAIT>, <AUDIO_WAIT>, <TEXT_INT>, <AUDIO_INT>), with wait tokens down-weighted to 0.001 and interruption tokens up-weighted to 50 in the loss. Training follows a three-stage curriculum identical for both variants: stage 1 trains everything except the backbone on ASR and streaming TTS; stage 2 also freezes the encoder and adds speech-to-text and speech-to-text-and-speech turn-based dialogue; stage 3 adds full-duplex dialogue. Training data comprise about 217k hours of English ASR data, 104k hours of VoxBox TTS data, and about 1.9M turn-based spoken dialogues built by rewriting text QA and dialogue corpora into spoken style with Qwen3-30B-A3B and synthesising them with IndexTTS2. Full-duplex examples (context-dependent and context-independent interruptions, backchannels) are composed on the fly with randomised insertion timing and a randomised model reaction delay of 2 to 6 steps (320 to 960 ms).

Key Results

Spoken QA (Table 1). XA-Duplex is below CF-Duplex on all four OpenAudioBench sets in both speech and text responses: LLaMA Questions 38.3 | 40.3 vs. 50.7 | 57.3, TriviaQA 8.0 | 8.2 vs. 18.1 | 19.6, WebQuestions 18.4 | 18.7 vs. 28.0 | 30.3, AlpacaEval 3.87 | 4.04 vs. 3.94 | 4.16 (the AlpacaEval gap is small). Against external systems, CF-Duplex has the highest AlpacaEval scores and the highest WebQuestions speech score, but Freeze-Omni (7B backbone) is clearly ahead on LLaMA Questions (56.2 | 74.2) and TriviaQA (28.5 | 45.1), and GLM-4-Voice is ahead on TriviaQA (26.5 | 39.1) and ties CF-Duplex on LLaMA Questions speech responses while leading on text responses (50.7 | 64.7). Answers are judged by GPT-5.4-mini with OpenAudioBench prompts; the paper does not state whether baseline numbers were re-scored with the same judge.

Full-duplex behaviour (Tables 2 and 3). On Full-Duplex-Bench v1.0 User Interruption, CF-Duplex reaches TOR 1.000 with the highest GPT-4o response score (3.96) at 0.374 s latency; XA-Duplex reaches TOR 0.971 but a much lower response score (2.23). On Smooth Turn Taking, XA-Duplex is the best system in the table (TOR 0.983, latency 0.161 s vs. CF-Duplex 0.924 and 0.336 s). On v1.5 Interruption, CF-Duplex ties Freeze-Omni on Respond (0.72) with the lowest stop latency (0.74 s) and respond latency (0.72 s); XA-Duplex responds in only 0.32 of cases and resumes its old response in 0.41. On v1.5 Backchannel, CF-Duplex has the highest Resume rate (0.96 vs. 0.86 for XA-Duplex), while XA-Duplex has the lowest respond latency (1.67 s). The authors note that XA-Duplex sometimes talks over the user.

Missed interruptions (§6.3.3, Appendix G). On cases where each model fails to yield the floor, CF-Duplex’s continuation often becomes semantically incoherent, mixing content from the user’s interruption into its ongoing answer, while XA-Duplex continues its original answer coherently. This evidence is qualitative: three example pairs are shown, and no count, rate or automatic coherence score is reported.

Intermediate stages (Table 4). ASR WER on LibriSpeech test-clean | test-other favours CF-Duplex at every stage (stage 3: 3.90 | 10.04 vs. 4.56 | 12.49). Streaming TTS WER on seed-tts-eval is close and does not consistently favour either variant: CF-Duplex is marginally lower at stage 1 (2.37 | 2.13 vs. 2.40 | 2.21 EN | ZH), but XA-Duplex is lower at stage 2 (2.29 | 2.88 vs. 2.41 | 3.13) and stage 3 (2.83 | 3.20 vs. 2.93 | 3.37). Both variants degrade on ASR and TTS after full-duplex training. The S2TD and S2TSD QA gap in favour of CF-Duplex appears at stage 2 as soon as dialogue data are introduced. (Table 10 lists the stage-1 CF-Duplex TTS pair in the opposite EN | ZH order, 2.13 | 2.37.)

Ablations (Table 5, CF-Duplex only). Removing interruption-token supervision lowers v1.5 Respond from 0.72 to 0.56 and raises stop latency from 0.74 s to 1.90 s; a fixed overlap of 2 or 3 steps instead of the randomised 2 to 6 range gives Respond 0.58 or 0.67.

Novelty Assessment

Neither routing mechanism is new: channel fusion follows Moshi-style and SALM-Duplex-style designs, and cross-attention adapters come from Flamingo and Audio Flamingo. The contribution is the matched comparison: same backbone, encoder, audio head, data and curriculum, differing only in routing, plus the identification of context corruption after missed interruptions as a failure mode tied to channel fusion. The full-duplex framework itself (streaming Whisper encoder, grouped audio-token head, special-token supervision, on-the-fly interruption synthesis) is solid engineering integration. The comparison has caveats: XA-Duplex has about 0.5B more parameters, the cross-attention placement was selected on stage-1 ASR/TTS only, the headline robustness finding has no quantitative measure, and everything runs at one model scale.

Field Significance

Moderate: the paper isolates user-stream routing as a variable in full-duplex speech LMs and provides matched-condition evidence of a trade-off: fusing user speech into the generation context buys semantic grounding and interruption responsiveness, while keeping it in a separate cross-attention memory protects the ongoing response from contamination but costs QA accuracy and interruption handling. The robustness side of the trade-off is shown only qualitatively, so the result serves mainly as a well-controlled data point and a design consideration rather than a settled conclusion.

Claims

  • supports: Injecting the user speech stream directly into the LLM’s input context gives stronger semantic grounding for spoken question answering than exposing it through cross-attention memory.

    Evidence: Under the same Qwen3-1.7B backbone, curriculum and data, channel fusion beats cross-attention routing on all four spoken QA sets in speech and text responses (e.g., LLaMA Questions 50.7 | 57.3 vs. 38.3 | 40.3), and the gap is already present on S2TD/S2TSD at stage 2. (§6.3.1, Table 1; §6.3.4, Table 4)

  • complicates: Merging user speech into the same context that drives the model’s own generation leaves full-duplex speech LMs vulnerable to semantically corrupted continuations when an interruption is not acted on.

    Evidence: In missed-interruption cases, the channel-fusion model blends the user’s interrupting content into its ongoing answer, while the cross-attention model continues coherently but still fails to yield; shown on three qualitative examples with no quantified rate. (§6.3.3; §Appendix G, Figures 5-7)

  • complicates: Structurally separating the user stream from the generation context does not by itself improve full-duplex turn management, and can weaken interruption handling.

    Evidence: The cross-attention variant has the best Smooth Turn Taking TOR (0.983) and latency (0.161 s) but responds to only 0.32 of v1.5 interruptions (vs. 0.72), resumes its old response in 0.41, scores 2.23 vs. 3.96 on v1.0 interruption response quality, and sometimes talks over the user. (§6.3.2, Tables 2-3)

  • refines: Interruption handling in end-to-end full-duplex models improves with explicit interruption-token supervision and with randomised overlap timing during training.

    Evidence: Removing interruption tokens drops v1.5 Respond from 0.72 to 0.56 and raises stop latency from 0.74 s to 1.90 s; fixed overlaps of 2 or 3 steps give Respond 0.58 or 0.67 vs. 0.72 for the randomised 2 to 6 range. (§6.4, Table 5)

  • complicates: Adding full-duplex dialogue training to a unified speech LLM degrades its base ASR and streaming TTS accuracy.

    Evidence: From stage 1 to stage 3, channel-fusion WER rises from 2.90 | 8.17 to 3.90 | 10.04 on LibriSpeech and TTS WER from 2.37 | 2.13 to 2.93 | 3.37 on seed-tts-eval EN | ZH; the cross-attention variant degrades similarly. (§6.3.4, Table 4)

Limitations and Open Questions

The robustness advantage of cross-attention routing, which the paper presents as one side of its central trade-off, rests on three qualitative examples selected from missed-interruption cases. No rate of incoherent continuation, no coherence metric and no count of missed interruptions per variant is reported, so the size of the effect is unknown.

  • Only two routing strategies are compared, at a single model scale with one frozen backbone and one training recipe; the authors state that how the trade-off changes with scale is open (§Limitations).
  • The variants are not parameter-matched: XA-Duplex has 469M of cross-attention adapters on top of a 27.3M fusion module (3.40B vs. 2.90B total), and its adapter placement was tuned on stage-1 ASR/TTS rather than dialogue.
  • Interleaved routing, the third strategy discussed in the introduction, is not included in the experiments.
  • All dialogue training data are synthetic (LLM-rewritten text QA voiced by IndexTTS2), with simulated interruptions and backchannels; behaviour on natural conversational overlap is untested.
  • External baseline numbers in Tables 1-3 come from systems with different backbones and data, and the paper does not say whether they were re-evaluated with the same GPT-5.4-mini judge.
  • No human listening evaluation of response speech quality is reported.
  • No public model release at submission time (§Appendix A).

Wiki Connections

  • Spoken Language Model — builds a full-duplex speech LM from a frozen text LLM with a streaming Whisper encoder and compares two ways of routing the external user speech stream into it.
  • Speech-to-Speech — the evaluated system is an end-to-end spoken dialogue model producing text and speech responses, tested on interruption, backchannel and turn-taking behaviour.
  • Streaming TTS — the audio head generates speech tokens incrementally from the model’s own text with a fixed delay and grouping, and streaming TTS WER is reported at every training stage.
  • Moshi — representative channel-fusion full-duplex model and a baseline on QA and Full-Duplex-Bench v1.0/v1.5.
  • SALM-Duplex — prior channel-fusion duplex model with a small backbone, used as a QA baseline.
  • Freeze-Omni — frozen-LLM full-duplex baseline that outperforms CF-Duplex on LLaMA Questions and TriviaQA and ties it on v1.5 interruption Respond.
  • Full-Duplex-Bench — benchmark used for interruption and smooth turn-taking evaluation (TOR, GPT-4o score, latency).
  • Full-Duplex-Bench v1.5 — benchmark used for interruption and backchannel behaviour tags and stop/respond latency.
  • CosyVoice 2 — supplies the 25 Hz supervised semantic tokenizer and frozen token2wav decoder for speech output, and serves as a TTS WER reference.
  • Fun-Audio-Chat — channel-fusion system whose grouped audio-token generation the audio head follows (with G = 4 instead of 5).
  • OmniFlatten — representative interleaved-stream full-duplex model, cited as the alternative routing strategy and an ASR reference.