arXiv · 2026 · Preprint

Chung-Ming Chien et al. (Toyota Technological Institute at Chicago / Kyutai / Gradium) · → Paper · Demo: ✓ · Code: ✓

Adds asynchronous retrieval-augmented generation to the full-duplex speech LM Moshi: the model emits a retrieval trigger token, keeps talking while a text back end (LLM or web search) fetches a reference, and absorbs the result mid-response before the key content is spoken (accepted to ICML 2026).

Problem

Full-duplex speech LMs listen and speak at the same time, which gives them fast turn-taking, backchannels and interruption handling, but their factual accuracy lags behind text LLMs and turn-based speech LMs. Vanilla Moshi reaches only 10.5% on spoken HaluEval questions and 22.8% on TriviaQA (§5.1, Table 1). Scaling the model would raise inference cost under strict real-time constraints, and conventional retrieval-augmented generation assumes the model can pause before answering, which a full-duplex system cannot do without breaking conversational flow. Concurrent approaches either target non-duplex settings with pre-indexed corpora (Stream RAG) or call an LLM at fixed intervals regardless of context (KAME) (§2).

Method

The design rests on a timing observation. The authors define time-to-first-audio-token (TTFAT), keyword delay (time from response onset to the first word that actually answers the query) and end-to-end keyword delay (E2EKD = TTFAT + keyword delay). Across existing speech LMs, E2EKD often exceeds 3 seconds, so a retrieval back end that returns within about 2 seconds can deliver information before the answer-bearing word is needed (§3.1, Figure 2, Table 1).

The system splits into a front end that interacts with the user in real time and an asynchronous back end (§3.2). The front end is a 7B Moshi model (an RQ-Transformer with a 12.5 Hz temporal Transformer and a depth Transformer predicting 8 Mimi codec tokens per step, with a parallel text channel) plus a 1B streaming ASR model with 0.5 s latency (Kyutai delayed-streams ASR) that transcribes the user for retrieval purposes. The only changes to Moshi are a special retrieval trigger token in the text stream and a reference text encoder (§3.2, §3.3).

Figure 3. Illustration of the front-end and back-end components in MoshiRAG. When the model needs external information, it outputs a ⟨ret⟩ token. The conversation transcript is sent to the back end which operates asynchronously. Once ready, the result is injected into Moshi which then adapts its response with no interruption.

When the trigger is predicted, the system waits 0.5 s for the ASR transcript, then sends the conversation transcript to a text-in, text-out back end: either a prompted LLM (Gemma 3 27B by default, also GPT-4.1, Llama 4 Maverick, Mistral Medium 3.1 and Gemini 2.5 Flash) or the Tavily web search API (§3.3.3, Appendix B.3). Meanwhile Moshi keeps generating “pre-RAG” content (a coarse answer or filler such as “Let me check that for you”). The returned reference is encoded by a frozen ARC-Encoder that compresses sequence length by a factor of four, projected by a trainable linear layer, and added to the temporal Transformer input over successive frames (additive injection), so the input sequence length does not grow (§3.3.1, Equation 2).

Figure 4. Text and audio token streams of the inputs and outputs of MoshiRAG. Front-end Moshi receives at all time its previous step token predictions and the user speech tokens. When the retrieval result is ready, its representation is summed with the embeddings from other token streams and ingested over a number of time steps.

Training data is fully synthetic. About 474k topics come from Natural Questions, HotpotQA and TriviaQA, plus 5.5k LLM-generated topics across 111 expert domains. Three Gemma 3 27B instances (user, Moshi, reference) write multi-turn scripts in which knowledge-intensive Moshi turns are split into a knowledge-free lead, a reference-grounded body and an optional tail, under three interaction styles (basic, user challenges, small talk and distractions) plus a single-turn QA subset. A multi-channel TTS model voices the scripts, giving about 1.9M conversations and 47,770 hours (§4.1, Table 4). The trigger token is placed using TTS forced alignment just before the lead portion, and the retrieval delay is simulated by sampling mostly from a window that leaves at least 1 s of buffer before the body, with a 20% fallback to a wider range. All parameters except the reference encoder are fine-tuned from Moshi for 100k updates, with 20% reference dropout replaced by a learned vector (§4.2).

Key Results

Factuality (Table 1, OpenAudioBench LLM-judge protocol). With the default Gemma 3 27B back end, response accuracy rises from vanilla Moshi’s 26.6 / 22.8 / 10.5 on Web Questions / TriviaQA / HaluEval to 67.2 / 69.6 / 36.3, and from 62.3 to 80.3 on Llama Questions. A vanilla Moshi fine-tuned on the same synthetic data without retrieval reaches only 37.0 / 29.7 / 18.7, so most of the gain comes from retrieval rather than the data (§5.1, Table 1). The comparison with other speech LMs is mixed for the default configuration: GPT-4o Audio (88.4 / 81.0 / 90.6 / 68.7) leads on every set, and Qwen3-Omni (84.7 / 68.8 / 73.6 / 38.9) is ahead of MoshiRAG-Gemma on all four; Step-Audio-Chat and Kimi-Audio are also ahead on Web Questions (75.1, 70.2), and Kimi-Audio on HaluEval (43.2). Swapping in GPT-4.1 lifts responses to 80.6 / 68.9 / 78.2 / 51.3, above every open speech LM on TriviaQA and HaluEval but still behind Qwen3-Omni on Llama Questions and Step-Audio-Chat and Kimi-Audio on Web Questions; Tavily search gives 78.2 / 66.1 / 77.5 / 47.0. These GPT-4.1 and Tavily runs assume a uniform 1.5 s retrieval delay rather than measured API latency (§5.1, footnote 11). Several baseline numbers are copied from the original papers rather than re-run (underlined in Table 1).

Retrieval versus response. The retrieved reference is on average about 5 points more accurate than the final spoken answer (Gemma: 83.0 vs 80.3 on Llama Questions, 42.0 vs 36.3 on HaluEval), measuring information loss during injection (§5.1, Table 1). With ground-truth user transcripts instead of ASR, HaluEval response accuracy rises from 36.3 to 50.8; with ground-truth HaluEval references (97.2% correct), responses reach only 65.1, a 32-point integration loss (Appendix B.2, Table 8).

Delay and compute. MoshiRAG-Gemma has a TTFAT of 0.0 s, keyword delay of 3.1 s (one second longer than vanilla Moshi’s 2.1 s, due to the lead-in template) and E2EKD of 3.1 s, lower than every other measured speech LM except LLaMA-Omni2 (2.9 s) and vanilla Moshi (2.1 s), and equal to the non-retrieval fine-tuned Moshi. The reported compute (FLOPs per second of generated audio) is 0.37, versus 0.22 for vanilla Moshi and 0.18 for Freeze-Omni (§5.2, Table 1). Accuracy drops sharply when retrieval latency exceeds 1.5 s (Appendix C, Figure 6b).

Interactivity (Full-Duplex-Bench, Table 2). MoshiRAG’s takeover rate during user pauses falls to 0.32 / 0.56 (synthetic / Candor) from vanilla Moshi’s 0.99 / 0.98, though Gemini is lower still on both (0.26 / 0.31). Turn-taking latency is 0.18 s versus 0.95 s for Freeze-Omni and 1.3 s for Gemini. User-interruption GPT score rises from 0.77 to 3.75, but the non-retrieval fine-tuned Moshi scores higher (4.19) with lower interruption latency (0.52 s vs 1.02 s), and vanilla Moshi’s interruption latency is 0.26 s. The authors attribute the turn-taking changes mostly to the training data distribution rather than retrieval (§5.3, Table 2). Backchannel frequency stays low (0.010/s). On Full-Duplex-Bench v1.5, MoshiRAG resumes speaking after user backchannels far more often than vanilla Moshi (0.61 vs 0.06), while its response rate to real interruptions is similar (0.51 vs 0.50) (Appendix B.4, Table 11).

Out-of-domain math (Table 3). On spoken AddSub / MultiArith / SinglEq / SVAMP / GSM8K, MoshiRAG-Gemma scores 61.7 / 69.0 / 68.2 / 55.0 / 33.9 versus vanilla Moshi’s 8.3 / 9.8 / 18.4 / 9.7 / 2.1. It beats GLM-4-Voice on four of five sets but trails it on SinglEq (68.2 vs 71.0), and the math-specialized STITCH-S is ahead on all five (81.7 / 87.9 / 91.7 / 72.2 / 56.7). Summarizing the reference before injection helps (62.0 / 73.1 / 66.4 / 57.5 / 51.2), as does GPT-4.1 (64.8 / 76.0 / 72.9 / 61.1 / 43.2) (§5.4, Table 3).

Ablations. Inserting reference embeddings into the sequence outperforms additive injection (ARC-Encoder HaluEval 49.0 vs 41.8; T5 47.1 vs 13.9), but the additive form is kept to bound sequence length for long conversations. ARC-Encoder with 4x compression beats 8x compression and T5 under the default setup (Appendix B.1, Tables 6 and 7).

Novelty Assessment

The genuinely new piece is the inference procedure: a learned trigger token that launches retrieval asynchronously, paired with streaming additive injection of a compressed reference into a dual-stream full-duplex model, and a training recipe that teaches the model to fill the retrieval window with knowledge-free lead content. The keyword-delay framing turns this into an explicit latency budget and doubles as a reusable measurement. The components themselves (Moshi, ARC-Encoder, a streaming ASR, off-the-shelf LLMs and Tavily) are existing systems, and concurrent work (Stream RAG, KAME) exploits the same temporal-gap idea, so the advance is the full-duplex, trigger-on-demand combination rather than the retrieval idea itself. The factuality gains are large relative to Moshi, but they are measured only on single-turn QA, with LLM judges, and the strongest results depend on external frontier back ends with simulated latency.

Field Significance

Moderate. This paper provides a working demonstration that a compact full-duplex speech LM can delegate knowledge to an external text system mid-utterance without pausing, and it quantifies the latency budget that makes this possible. It also exposes the main bottlenecks for this design (ASR errors upstream of retrieval and information loss at the injection step), which gives a concrete agenda for tool use in full-duplex speech LMs.

Claims

  • supports: Delegating knowledge to an external text retrieval system substantially improves the factual accuracy of a compact speech LM, with gains largest on harder knowledge-intensive questions.

    Evidence: Retrieval raises Moshi’s spoken HaluEval accuracy from 10.5% to 36.3% (Gemma back end) and 51.3% (GPT-4.1), while fine-tuning on the same synthetic conversations without retrieval reaches only 18.7%. (§5.1, Table 1)

  • supports: The interval between the onset of a spoken response and its answer-bearing content provides a usable latency budget for asynchronous back-end computation in full-duplex dialogue.

    Evidence: Measured end-to-end keyword delays of speech LMs mostly exceed 3 s, inference-time retrieval delays are almost always shorter than the keyword delay, and the retrieval-augmented model keeps a 3.1 s E2EKD with 0.0 s time-to-first-audio-token. (§3.1, §5.2, Figure 5, Table 1)

  • complicates: Injecting retrieved text into a streaming speech LM loses a substantial share of the retrieved information, so reference correctness is an upper bound that the spoken answer does not reach.

    Evidence: Responses trail the retrieved reference by about 5 points on average, and with ground-truth HaluEval references (97.2% correct) the spoken response reaches only 65.1%. (§5.1, Table 1; Appendix B.2, Table 8)

  • complicates: Retrieval-augmented speech dialogue is sensitive to upstream ASR errors, since the retrieval query is built from transcribed user speech.

    Evidence: Replacing streaming ASR transcripts (11.5% WER on HaluEval) with ground-truth text raises HaluEval response accuracy from 36.3% to 50.8%, and retrieval trigger rate falls as user-speech WER increases. (Appendix B.2, Table 8; Appendix C, Figure 6a)

  • refines: In streaming speech LMs, the timing of when conditioning information arrives matters as much as its content, favouring compact reference encodings over longer ones.

    Evidence: With additive streaming injection, an uncompressed T5 reference encoding collapses to 13.9% HaluEval accuracy versus 41.8% for 4x-compressed ARC-Encoder, while sequence insertion narrows the gap (47.1% vs 49.0%); accuracy also drops sharply once retrieval delay exceeds 1.5 s. (Appendix B.1, Table 6; Appendix C, Figure 6b)

Limitations and Open Questions

Warning

All factuality and math evaluations are single-turn, scored by LLM judges, and the headline GPT-4.1 and Tavily results assume a uniform 1.5 s retrieval delay rather than measured API latency, so the multi-turn conversational ability the training data targets and the real-time behaviour of the strongest configurations are not directly validated.

  • The retrieval trigger is learned entirely from synthetic training data; the authors list linking it to query difficulty or learning it with reinforcement learning as future work (§6).
  • Training conversations are LLM-scripted and TTS-voiced with a single fixed Moshi voice, and HaluEval and math questions are also synthesized with the authors’ TTS, so evaluation audio partly shares the training-data generation pipeline.
  • Retrieval reduces some interactivity relative to the non-retrieval fine-tuned Moshi (lower interruption GPT score, higher interruption latency), and backchannel frequency remains low (§5.3, Table 2).
  • The more effective insertive injection was rejected for sequence-length reasons, leaving an open question of how to integrate references both efficiently and losslessly (Appendix B.1.1).
  • No human listening or conversational user study is reported; naturalness of the filler-based lead-ins is not assessed.

Wiki Connections

  • Spoken Language Model — Extends a full-duplex speech LM that consumes live user speech with an asynchronous retrieval trigger and streaming reference injection.
  • Speech-to-Speech — Targets end-to-end full-duplex spoken dialogue, comparing factuality, delay and turn-taking against turn-based and duplex speech-to-speech systems.
  • Evaluation Metrics — Introduces keyword delay and end-to-end keyword delay as latency measures of when answer-bearing content is spoken, measured across a range of public speech LMs.
  • Moshi — The base full-duplex model that MoshiRAG fine-tunes, keeping its Mimi tokens and RQ-Transformer and adding only a trigger token and reference encoder.
  • Stream RAG — Concurrent work exploiting conversational temporal gaps for retrieval, contrasted as limited to non-duplex settings with pre-indexed corpora.
  • Delayed Streams Modeling — Supplies the 1B streaming ASR that transcribes user speech for building retrieval queries.
  • Full-Duplex-Bench — Benchmark used to measure pause handling, backchannels, turn-taking and interruption behaviour.
  • Full-Duplex-Bench v1.5 — Benchmark used to evaluate reactions to overlapping user and background speech.
  • Qwen3-Omni — Strongest open baseline in the factuality table, ahead of the default Gemma-backed configuration on all four QA sets.
  • End-to-End RAG for Speech-to-Speech Dialogue — Prior retrieval-augmented speech dialogue work cited as non-full-duplex related work.
  • GLM-4-Voice — Turn-based baseline on QA and math, beaten on most sets but ahead on SinglEq.