arXiv · 2026 · Preprint
Donghang Wu et al. · → Paper · Demo: ? · Code: ?
Introduces FLAIR, a full-duplex spoken dialogue model that performs continuous latent reasoning during the user’s speaking turn instead of emitting silence tokens, trained with an ELBO-based objective that transfers a non-causal expert’s global reasoning into the causal model without explicit reasoning annotations.
Problem
Full-duplex spoken dialogue language models (SDLMs) continuously ingest streaming user speech while generating a response, which lets them support backchanneling and barge-in the way turn-based systems cannot. But existing full-duplex systems waste the listening window: while the user is speaking, the model is typically forced to predict repeated, non-informative <SIL> padding tokens, contributing nothing to the eventual response. A natural fix borrowed from text LLMs, generating an explicit chain-of-thought in parallel with the incoming audio, breaks down in the streaming setting: discrete text reasoning cannot causally precede speech that has not finished unfolding, and if the model commits to a rigid textual thought chain it must interrupt and restart when the user finishes speaking at an unpredictable moment, introducing latency and state-management complexity. The paper asks whether a full-duplex model can “think while listening” using implicit, continuous latent computation instead of discrete tokens, and whether such a mechanism can be trained without constructing costly explicit reasoning-annotated datasets.
Method
FLAIR (Full-duplex LAtent and Internal Reasoning) modifies what a full-duplex SDLM feeds itself as input during the user’s speaking phase. Instead of the argmax token embedding from the previous step, the model’s last hidden state is passed through the LLM head to produce text logits, which are softmax-weighted and used to compute a weighted average over the vocabulary embedding matrix; this continuous vector, rather than a discrete <SIL> token, becomes the next-step input. Because this latent recursion has no ground-truth target, the paper frames training as a latent-variable inference problem and derives a conditional Evidence Lower Bound (ELBO) objective for maximizing the log-likelihood of the response text Y_txt given the streaming user input X through a latent variable Z.
To supervise this latent state, the method introduces a non-causal “Global-aware Expert” model Q_φ: a BERT-based encoder that sees the full dialogue context (user speech embeddings plus the assistant’s response text embeddings) and produces an approximate posterior over Z, expressed as a softmax weighting over the vocabulary embedding matrix. During teacher-forced supervised fine-tuning, the LLM’s input during listening-phase timesteps is replaced with the Expert’s derived latent-reasoning embedding label, and the loss combines (1) a masked next-token reconstruction loss on the response segment, (2) a KL-divergence term aligning the causal model’s predicted vocabulary distribution with the (stop-gradient) Expert posterior during listening, and (3) a binary loss on a predicted timing indicator that decides, at inference, whether to output a text token or continue latent reasoning. At inference the Expert model is discarded entirely: the causal LLM alone decides per-step whether to emit a token or recurse a latent embedding, so no additional inference-time latency or compute is introduced relative to a standard full-duplex model.
The LLM backbone is Qwen2.5-7B-Instruct; a frozen 600M-parameter Parakeet-based streaming conformer encoder with a Transformer adapter maps 12.5 Hz audio frames into the LLM’s embedding space. A frozen audio codec and pretrained streaming flow-matching vocoder (from CosyVoice 2) generate speech tokens and waveform from the LLM’s text tokens and hidden states, trained in a final Speech Synthesizing SFT stage while the rest of the model is frozen. Training data is entirely synthetic and constructed from three sources: 530K hours of speech-continuation data built by converting text corpora into alternating pseudo-dialogue turns and synthesizing them with a diverse pool of TTS voices, 70K hours of instruction-following QA data generated by an LLM ensemble and voice-cloned across a 20K-speaker prompt bank, and 20K hours of ASR-QA data built from real noisy ASR corpora to improve robustness. Synthetic user barge-ins are injected during training by forcing early termination of agent responses.
Key Results
On QA and reasoning benchmarks (Table 1), enabling latent reasoning (“FLAIR w/ thk”) improves over the same model without it (“FLAIR w/o thk”) on nearly every benchmark, with the largest gains on reasoning-heavy tasks (MMSU accuracy rises from 50.2% to 56.2%, Llama Questions from 73% to 78%) and smaller gains on tasks dominated by factual retrieval (WebQuestions moves only from 41.7% to 43%). FLAIR also outperforms other published full-duplex SDLMs (Moshi, Freeze-Omni, SALMONN-omni, SALM-Duplex) on most benchmarks, though some half-duplex models (e.g., Kimi-Audio, Baichuan-Audio) still score higher on open-ended GPT-scored tasks, which the paper attributes to FLAIR’s conversational training data favoring concise responses.
Conversational-dynamics metrics (Tables 2 and 3) show that adding latent reasoning does not degrade full-duplex behavior: turn-taking latency, barge-in latency, and barge-in success rate for FLAIR with and without thinking are close (e.g., 100% barge-in success rate either way on the Impatient dataset), and results on Full-Duplex-Bench’s noisy CANDOR-derived turn-taking evaluation and barge-in evaluation are comparable to or better than Moshi, Freeze-Omni, dGSLM, and Gemini Live. Synthesized-speech quality, measured by UTMOS-computed MOS, is 4.3 for both FLAIR variants. A t-SNE visualization (Figure 3) shows the latent-reasoning embeddings trace a trajectory in representation space between the input audio embeddings and the target text embeddings, consistent with the claim that the latent state functions as a bridge between perception and generation.
Novelty Assessment
The core contribution is architectural and training-recipe: recursively feeding softmax-weighted vocabulary embeddings back into a causal LLM during the listening phase is not itself new (the paper cites prior latent chain-of-thought work such as Coconut and CODI as direct inspiration), but applying this mechanism to the full-duplex spoken dialogue setting, and deriving a variational (ELBO) objective with a non-causal expert model to supervise it via teacher forcing without any explicit reasoning annotations, is a genuine adaptation with a real engineering payoff: it sidesteps the causality and latency problems that make explicit streaming chain-of-thought infeasible for real-time speech. The paper’s own related-work section is candid that other “think-while-listening” approaches exist (Wu et al., Chiang et al., Arora et al.) but either require constructing specialized CoT datasets or cannot guarantee causality; FLAIR’s comparison excludes these baselines because they lack open code or common benchmark results, so the empirical claim of superiority over other listen-and-think methods is not directly demonstrated, only argued. The data-construction pipeline (620K hours of purely synthetic dialogue audio) is itself a substantial engineering effort but is presented as a means to the architectural end rather than a standalone contribution.
Field Significance
Tip
High — this paper is, by the authors’ own framing, the first application of latent (non-token) reasoning to speech language models, and it resolves a specific structural tension (causal, low-latency reasoning during streaming listening) that explicit chain-of-thought approaches cannot handle. It demonstrates that a variational, teacher-forced training recipe can install this capability without new annotated data or extra inference cost, which is a reusable training pattern beyond this specific system.
The paper is accepted at ICML 2026, a top-tier machine learning venue, and provides a full training recipe (ELBO derivation, teacher-forcing schedule, timing-indicator prediction) that other full-duplex SDLM work can adopt directly. It stops short of foundational status because it builds on an active, already-populated line of full-duplex and latent-reasoning research rather than establishing a wholly new subfield.
Claims
- supports: Recursing a continuous latent embedding through a causal model during a listening phase can substitute for explicit token-level reasoning without introducing additional inference latency.
Evidence: At inference, the model discards the non-causal expert entirely and performs step-by-step decoding, choosing per-timestep between emitting a text token and feeding back a softmax-weighted vocabulary embedding, so its architecture is identical in cost to a standard full-duplex model that outputs
<SIL>tokens (§4.2). - supports: A variational (ELBO) objective with a non-causal expert model providing posterior supervision can train implicit latent reasoning via teacher forcing, avoiding the need for explicit reasoning-annotated data.
Evidence: The Global-aware Expert model Q_φ, trained on full dialogue context, supplies a stop-gradient vocabulary-weighting target that the causal LLM’s listening-phase output distribution is trained to match via KL divergence, with no CoT annotations required at any stage (§4.2, Eq. 10-11).
- complicates: The benefit of latent reasoning in spoken dialogue models is not uniform across task types, and is smaller for tasks requiring mostly factual retrieval rather than multi-step reasoning.
Evidence: Enabling latent reasoning raises MMSU accuracy from 50.2% to 56.2% and Llama Questions from 73% to 78%, but WebQuestions improves only from 41.7% to 43%, which the paper attributes to factual extraction requiring less reasoning (§7.1, Table 1).
- complicates: Open-ended GPT-judged QA metrics conflate response quality with response length, which can disadvantage conversational full-duplex systems trained to produce concise turns relative to longer-form half-duplex LLM responses.
Evidence: A case study shows FLAIR’s response to a query is markedly shorter than the same underlying LLM’s unconstrained response, and the paper notes that longer responses tend to yield higher GPT scores on AlpacaEval and CommonEval, explaining why some half-duplex baselines outscore FLAIR on those benchmarks despite FLAIR’s stronger reasoning-benchmark results (§7.1, Table 5).
- refines: The dimensional and semantic alignment between a streaming speech encoder and the LLM backbone it feeds is a critical, not incidental, design factor for full-duplex turn-taking robustness.
Evidence: Ablations show a 120M-parameter streaming encoder degrades response success rate on the Llama Questions test set to 70.7%, while scaling to a 600M-parameter encoder with the same simple linear adapter restores a 100% success rate (§Appendix E).
Limitations and Open Questions
Warning
The comparison against other explicit “think-while-listening” full-duplex baselines (Wu et al. 2025’s Chronological Thinking, Chiang et al.’s SHANKS, Arora et al.’s streaming CoT) is omitted entirely because those systems lack open-source code, released CoT datasets, or common benchmark results, so FLAIR’s central claim of being a superior listen-and-think mechanism relative to the most directly comparable prior work is argued qualitatively rather than demonstrated empirically.
All training and evaluation data is either fully synthetic (speech-continuation and instruction QA data, built with LLM-generated text and cloned TTS voices) or drawn from existing benchmarks; only CommonEval within VoiceBench uses real human speech for the response-quality evaluation, leaving open how much the synthetic-data reliance affects generalization to natural spontaneous speech. The paper reports MOS via the automatic UTMOS predictor rather than a human listening test, so speech-quality claims rest on a model-based proxy. The ablation study (Appendix E) is reported only in aggregate outcome terms without full architectural variant tables, and the choice of α=3, β=5 loss weights is described as selected empirically without a systematic sensitivity analysis being shown.
Wiki Connections
- Spoken Language Model — proposes a training and inference mechanism specifically for the LLM backbone of a full-duplex spoken dialogue language model, targeting how that backbone should use idle computation during listening.
- Speech-to-Speech — is a full-duplex spoken dialogue agent that continuously ingests streaming user speech and produces spoken responses, fitting the dialogue sub-paradigm of speech-to-speech interaction.
- Streaming TTS — relies on a streaming flow-matching vocoder to convert LLM-predicted speech tokens into audio incrementally as part of its full-duplex response pipeline.
- Neural Audio Codec — depends on the CosyVoice 2 audio codec to tokenize target speech for training the speech-generation module and to reconstruct waveform at inference.
- Chronological Thinking in Full-Duplex SDLMs — addresses the same think-while-listening problem in full-duplex dialogue models but is excluded from FLAIR’s own experimental comparison due to lack of released code and CoT data.
- SHANKS — is cited as another explicit streaming chain-of-thought approach to simultaneous hearing and thinking, contrasted with FLAIR’s implicit latent alternative.
- Chain-of-Thought Reasoning in Streaming Full-Duplex Systems — represents the explicit-CoT approach to think-while-listening that FLAIR argues cannot guarantee causality in the streaming setting.
- CosyVoice 2 — supplies the audio codec and pretrained streaming flow-matching model that FLAIR’s speech-generation module builds on directly.
- Full-Duplex-Bench — is used as an evaluation benchmark to measure FLAIR’s turn-taking and barge-in performance against other full-duplex baselines.