arXiv · 2026 · Preprint
Tao Zhong et al. (The Chinese University of Hong Kong / AudioLab Hong Kong, Huawei Leibniz Research Center / Nanyang Technological University) · → Paper · Demo: ? · Code: ?
Adds a frame-level, target-speaker-aware reliability gate to the user audio stream of an end-to-end dual-channel full-duplex dialogue model, so that interfering speakers leaking into the microphone are down-weighted before the LLM sees them, improving response quality and turn-taking under simulated interference.
Problem
End-to-end dual-channel full-duplex speech LMs fuse the user audio stream and the agent stream into a single LLM input, treating every user audio frame as equally trustworthy. When other talkers leak into the user microphone, their speech is encoded as if it were part of the user query. The paper argues this corrupts the LLM’s conditioning, which harms both reasoning (wrong or incoherent answers) and interaction control (missed responses, mistimed or false barge-in). Prior end-to-end duplex work has concentrated on relatively clean conditions or on agent echo, and conventional fixes (speech separation front-ends, post-hoc filtering) conflict with the strict causality and latency budget of duplex interaction.
Method
The base system is a NeMo-implemented dual-stream duplex model. User audio passes through a 100M-parameter streaming FastConformer encoder (80 ms right context) and a modality adapter at 12.5 Hz, producing frame embeddings that are summed with the agent’s text-token embeddings and fed to a 1.1B TinyLlama backbone, which predicts agent text tokens. A separate 12-layer causal T5-style Transformer speech decoder, conditioned on the LLM’s last hidden states, autoregressively predicts agent speech tokens from NanoCodec (FSQ, 12.5 Hz, 0.6 kbps, four codebooks of 4,037 entries), decoded to audio by the frozen causal codec decoder. Everything except the codec decoder is jointly fine-tuned with a weighted multi-channel next-token loss (text weight 1.0, audio weight 5.0).
IRAF (Interference-Resilient Adaptive Fusion) replaces the unconditional sum with a gated one. At each frame the target-speaker embedding (from a pretrained ECAPA-TDNN) is concatenated with the user audio embedding and passed through an input projection, a single causal Transformer layer, and a linear output; a scaled sigmoid gives a scalar gate in [0, 2], and the LLM input becomes the gated user embedding plus the agent text embedding. Because the gate can exceed 1, it can amplify as well as suppress user frames. The gate is trained jointly with the full model, plus an auxiliary binary loss (weight 0.1) against frame-level target-speaker activity labels derived from the clean training utterances. The module is causal and adds no lookahead.
Training data are simulated. MS MARCO QA pairs are voiced with CosyVoice 2 for single-turn spoken QA, and InstructS2S-200K supplies multi-turn dialogue. Sessions are laid out as two synchronized channels with a fixed 0.64 s inter-turn pause; on InstructS2S-200K, barge-in is simulated on the fly (probability 0.5 per turn) by overlapping the next user turn with agent speech and truncating the agent 0.64 s after interruption onset. Interference uses MUSAN speech (librivox and us-gov subsets) as competing talkers and MUSAN noise as background, with disjoint MUSAN partitions per split, at SNRs of 0 to 10 dB (MS MARCO) or 0 to 20 dB (InstructS2S-200K).
Key Results
Three configurations share the same architecture: CleanBase (trained on clean speech, no IRAF), NoisyAug (trained with interference augmentation, no IRAF), and IRAF (augmentation plus the gate). All are tested on interference-augmented test sets. Response quality is ASR-transcript BLEU and Sentence-BERT similarity against the reference text (no LLM judge, no human raters). Turn-taking uses Silero-VAD-based response latency (RL) and response success rate (RSR, response within 1.5 s); barge-in uses stop latency (SL) and stop success rate (SSR).
- CleanBase collapses under interference. On MS MARCO it scores BLEU 0.66 and RSR 6.2% with interfering speakers, and BLEU 0.00 and RSR 2.8% with added background noise (Table 1). Augmentation alone recovers most of the gap.
- MS MARCO, single-turn (Table 1). IRAF exceeds NoisyAug on BLEU, sBERT and RSR, and has lower RL, in all six matched rows (LIBRI, US-GOV, ALL interference subsets; with and without background noise). With interfering speakers only (ALL), BLEU goes from 12.74 to 14.20 and RSR from 93.1% to 95.7%; with added background noise (ALL), BLEU goes from 11.33 to 12.01 and RSR from 88.2% to 92.5%. RL differences are 0.01 to 0.04 s.
- InstructS2S-200K, multi-turn (Table 2). Deltas are IRAF minus NoisyAug, so negative is better for latency. With interfering speakers only: BLEU +4.12 (9.64 to 13.76), sBERT +0.11 (0.47 to 0.58), RL -0.15 s (0.97 to 0.82), RSR +21.8 points (69.2% to 91.0%). With added background noise the gains shrink: BLEU +1.51 (8.32 to 9.83), sBERT +0.03, RL -0.07 s, RSR +13.2 points (56.0% to 69.2%). Barge-in handling is already near ceiling with augmentation, so IRAF’s SL and SSR gains are marginal (SL -0.01 s; SSR 99.0% to 99.8% and 99.6% to 100.0%).
- SNR sweep. Figure 3 shows IRAF above NoisyAug on BLEU and RSR across all SNR settings on InstructS2S-200K with interfering speakers (values are plotted, not tabulated).
The comparisons are internally fair (same architecture, data and training recipe), but the only baselines are ablated versions of the authors’ own model. No external full-duplex system, no separation or target-speaker-extraction front-end, and no results on clean test audio are reported.
Novelty Assessment
The contribution is a small, targeted fusion change: a learned scalar gate on the user stream, conditioned on a target-speaker embedding and supervised by a speaker-activity auxiliary loss. Speaker-conditioned gating and target-speaker voice-activity estimation are established ideas in speech enhancement and target-speaker ASR; the novelty is applying them inside the input fusion of an end-to-end duplex speech LM, where the paper argues interference harms turn-taking as well as content. The base duplex architecture (streaming encoder, small LLM, separate AR codec-token decoder, NanoCodec) follows prior duplex work and is not new. Evidence is limited to a with/without-gate comparison: there is no ablation of the speaker embedding, the auxiliary loss, the [0, 2] gate range, or the choice of a scalar rather than a vector gate, so it is unclear which component carries the gain.
Field Significance
Moderate — The paper isolates acoustic interference as a failure mode specific to end-to-end full-duplex speech LMs and shows that, in simulation, it degrades turn-taking reliability (missed responses) as much as answer content, especially in multi-turn dialogue. It provides a cheap, causal mitigation whose gains go beyond data augmentation, a useful reference point for robustness work on duplex agents. The evidence base (simulated interference, internal baselines only, small LLM) keeps it an incremental robustness contribution rather than a demonstrated general solution.
Claims
- supports: Acoustic interference in the user channel degrades not only response content but also turn-taking reliability in end-to-end full-duplex speech LMs.
Evidence: A model trained on clean speech reaches RSR of 6.2% and BLEU 0.66 on MS MARCO with interfering speakers, and RSR 13.9% on InstructS2S-200K; augmentation-trained models still miss 30.8% of responses in multi-turn dialogue with interfering speakers. (§4.3, Tables 1 and 2)
- supports: Frame-level, speaker-aware gating of user audio before LLM fusion improves the robustness of full-duplex dialogue models to interfering speakers beyond what interference augmentation alone achieves.
Evidence: Against the same architecture trained with the same augmentation, the gate raises multi-turn BLEU from 9.64 to 13.76 and RSR from 69.2% to 91.0%, and lowers response latency from 0.97 s to 0.82 s; it also improves BLEU, sBERT and RSR in all six single-turn test conditions. (§2.2, §4.3, Tables 1 and 2)
- refines: Interference robustness gains in full-duplex dialogue are larger for multi-turn interaction than for single-turn QA, and shrink when competing speech is combined with background noise.
Evidence: The RSR gain over augmentation is +2.6 points on single-turn MS MARCO (ALL, interfering speakers) but +21.8 points on multi-turn InstructS2S-200K; on InstructS2S-200K, adding background noise reduces the BLEU gain from +4.12 to +1.51 and the RSR gain from +21.8 to +13.2 points. (§4.3, Tables 1 and 2)
- complicates: Barge-in stop behaviour in simulated full-duplex training can saturate under augmentation, leaving little headroom for input-robustness methods to show gains on interruption handling.
Evidence: The augmentation-only baseline already reaches SSR of 99.0% and 99.6% with SL of 0.74 s, and the gate changes these by at most 0.8 points and 0.01 s. (§4.3, Table 2)
Limitations and Open Questions
Warning
All results come from simulated interference (MUSAN speech and noise mixed at controlled SNRs, synthetic CosyVoice 2 user speech for MS MARCO), compared only against ablations of the authors’ own model, and no clean-condition test results are reported, so it is unknown whether the gate costs anything on clean input or holds up on real recorded multi-talker audio.
- Target-speaker enrollment is under-specified. The gate depends on an ECAPA-TDNN target-speaker embedding, but the paper does not say where the enrollment audio comes from at test time, and speakers from different conversations are simply assumed to be distinct identities. Robustness to enrollment mismatch or a target speaker resembling an interferer is not tested.
- No component ablations. The auxiliary activity loss, the speaker embedding and the [0, 2] gate range are never removed individually.
- No external baselines. The paper does not compare against other full-duplex systems or against a cascaded target-speaker extraction front-end, which is the most obvious alternative.
- Evaluation is fully automatic. Response quality is ASR-transcript BLEU and sBERT against references, and turn-taking timing comes from VAD; there is no human evaluation, no assessment of output speech quality, and no significance testing.
- Small backbone. Results use a 1.1B TinyLlama backbone; behaviour with larger duplex speech LMs is untested.
- Code and demo availability are not stated.
Wiki Connections
- Spoken Language Model — The system is an adapted LLM that consumes the user’s streaming speech, and the paper targets how corrupted speech input distorts the LLM’s conditioning.
- Speech-to-Speech — An end-to-end full-duplex speech-to-speech dialogue model evaluated on response quality, turn-taking and barge-in.
- Streaming TTS — The agent’s speech is generated incrementally while listening, and evaluation measures response latency and stop latency in interaction.
- Autoregressive Codec TTS — A jointly fine-tuned causal Transformer decoder autoregressively predicts NanoCodec tokens from the LLM hidden states; the paper’s contribution sits on the input side, not in this decoder.
- SALM-Duplex — Cited as duplex speech-to-speech modeling that reduces reliance on speech pretraining; IRAF’s base system is likewise a NeMo-implemented dual-channel model, and IRAF targets the interference robustness that such clean-condition duplex models leave open.
- Moshi — Cited as a representative native dual-channel full-duplex speech LM whose joint user/agent stream modeling assumes a trustworthy user channel.
- SALMONN-omni — Cited as prior end-to-end duplex work that addresses agent echo rather than third-party interference.
- Chronological Thinking — Cited among end-to-end duplex models evaluated in relatively clean conditions.
- Freeze-Omni — Cited as a controller-style duplex system that coordinates listening and speaking with explicit control, the alternative to IRAF’s end-to-end approach.
- LLM-Enhanced Dialogue Management — Cited as a semantic-VAD-based dialogue manager for full-duplex systems, contrasted with native dual-channel modeling.
- CosyVoice 2 — Used to synthesize user speech for the MS MARCO spoken QA data.
- MUSAN — Supplies the interfering speech and background noise used for training augmentation and test conditions.