arXiv · 2025 · Preprint
Siddhant Arora et al. (Meta, Carnegie Mellon University) · → Paper · Demo: ? · Code: ?
Introduces the first framework for integrating external tool use (web search, knowledge graphs) directly into end-to-end speech-in speech-out dialogue systems, and shows that issuing tool queries in parallel with incoming user speech recovers most of the resulting latency cost.
Problem
End-to-end speech-in speech-out dialogue systems generate expressive, low-latency responses directly from audio, avoiding the error propagation of cascaded ASR-LLM-TTS pipelines. However, because they answer purely from internalized training-data knowledge, they hallucinate on knowledge-seeking and time-sensitive questions. Text-based conversational assistants already address this with retrieval-augmented generation (RAG), issuing calls to web search or knowledge-graph APIs, but tool calls take seconds to resolve. In a spoken dialogue setting this delay produces awkward silences that break conversational flow, and prior speech-in speech-out systems have not been trained or evaluated on tool use at all. The paper frames the core tension as a trade-off between factual accuracy (which tool use improves) and responsiveness (which tool use degrades), and asks how a spoken system can have both.
Method
The paper first formalizes tool integration for speech-in speech-out LLMs as a two-stage inference process: a Query Generation stage that maps an audio question to one or more tool queries, and a Response Generation stage that conditions the final spoken answer on the audio question plus the retrieved tool results. This baseline (“Open Book”) formulation is applied to three existing speech-in speech-out models (Qwen-Omni’s Thinker-Talker architecture, OpusLM’s chain-of-thought speech LM, and Kimi-Audio’s unified audio foundation model) without any streaming behavior, and already improves factual accuracy substantially over closed-book responses, at the cost of roughly doubling first-token latency.
To recover that latency, the paper introduces Streaming RAG, built around three design choices: a trigger (when to issue a new tool query), the number of parallel query threads, and a reflector (whether an intermediate tool result is already sufficient). Two concrete instantiations are proposed. Fixed-Interval Streaming RAG fires a tool query after every fixed-size audio block, running many query threads in parallel and using a heuristic reflector (matching top-5 web documents or exact-match KG results) to pick the earliest intermediate query whose retrieval would match the final query; it requires no additional training and can be applied to any existing speech-in speech-out model at inference time. Model-Triggered Streaming RAG instead post-trains the model itself to decide, after each block, whether to emit a new tool query or a NO_QUERY token, conditioning on the accumulated partial speech and the most recent non-empty query; this collapses the design to a single tool-call thread and removes the need for an explicit reflector. Post-training uses a multi-task objective over streaming query generation and response generation, with labels derived by comparing an LLM-generated pseudo-ground-truth query at each block to the previous query (assigning NO_QUERY when retrieval results would not change). A key auxiliary technique is negative-sample injection during training: the model is sometimes shown a deliberately incorrect “previous query” so it learns to recover from its own intermediate query errors at inference rather than getting stuck emitting NO_QUERY after an early mistake.
To evaluate these approaches, the paper builds AudioCRAG by converting the text-based CRAG factual-QA benchmark into spoken form: AudioCRAG-Synthetic (1,862 queries synthesized with an in-house TTS system and filtered for zero-WER Whisper transcription and UTMOS ≥ 3.5) and AudioCRAG-Human (618 human-recorded queries). Post-training uses a 16,000-question subset of TriviaQA, converted to speech and paired with pseudo-ground-truth queries and retrieved web/KG documents. Web retrieval uses a BGE-based reranker over the CRAG document corpus; KG retrieval uses CRAG’s simulated API. Evaluation scores answers with an LLM judge (LLaMA-4-Maverick) on accuracy, hallucination, and miss rates, following CRAG’s own protocol.
Key Results
Tool integration alone (Open Book vs. Closed Book) raises accuracy by up to 140% relative (11.1% to 26.3% absolute for Qwen2.5-7B on AudioCRAG-Synthetic) but increases first-token latency by 2.3x. Model-Triggered Streaming RAG improves on this further: Qwen2.5-7B reaches 34.2% accuracy on AudioCRAG-Synthetic (37.4% on AudioCRAG-Human) — over 200% relative improvement from the closed-book baseline — while cutting first-token tool-use latency by 20.7% (Synthetic) and 53.4% (Human) relative to the non-streaming Open Book setting. OpusLM shows a similar pattern (23.6% accuracy, 14.8-41.5% latency savings). Notably, the streaming-trained models generalize from the synthetic training and evaluation set to the human-recorded benchmark, with equal or larger gains observed on AudioCRAG-Human. An ablation confirms Model-Triggered Streaming RAG needs only a single parallel tool-call thread and yields larger latency savings than Fixed-Interval Streaming RAG, which needs multiple threads and a reflector and only achieves a 10.7% latency reduction for Qwen-Omni without post-training. A separate ablation shows that negative-sample injection during post-training is necessary: removing it drops text-output accuracy from 39.8% to 36.5%, because the model otherwise cannot recover from incorrect intermediate query predictions.
A secondary analysis (Table 2) shows a persistent modality gap: every evaluated model scores higher when producing a text answer than a spoken answer from identical retrieved references (e.g., Qwen2.5-7B: 39.6% text vs. 26.3% speech in Open Book), and an intermediate “self-cascade” (generate text, then synthesize speech from it) only partially closes this gap. Kimi-Audio could not be evaluated under Streaming RAG because it errors when handling long tool-result references, an architectural constraint of its rotary-embedding-based audio detokenization. OpusLM scored 0% accuracy in the vanilla Open Book speech-out setting, since it does not support tool-result references as speech output in a zero-shot manner, only after Streaming RAG post-training.
Novelty Assessment
The contribution is genuinely architectural at the level of inference-time control flow: the trigger/threads/reflector formulation for streaming tool calls, and specifically the model-triggered variant that learns a NO_QUERY decision from partial speech, are new mechanisms not previously proposed for any dialogue system, spoken or text. The underlying speech-in speech-out backbones (Qwen-Omni, OpusLM, Kimi-Audio) are not modified architecturally; the paper’s method is a post-training and inference-time addition layered on top of them, which places part of the contribution in the engineering-integration category as well. The AudioCRAG benchmark is a straightforward but useful adaptation of an existing text benchmark (CRAG) to spoken form, rather than a novel evaluation methodology. The negative-sampling strategy for query-generation robustness is a targeted, well-motivated training trick rather than a broad architectural innovation, but the ablation evidence for its necessity is convincing.
Field Significance
Tip
High — this paper is the first to demonstrate that tool use (web search, knowledge graphs) can be integrated into end-to-end speech-in speech-out dialogue systems, and the first to show that streaming, incremental tool-query prediction can substantially offset the latency cost that tool use otherwise introduces into spoken interaction.
The paper opens a research direction (tool-augmented, low-latency spoken dialogue) that had not previously been explored for genuinely end-to-end speech-in speech-out systems, evaluates it consistently across three independently developed SOTA models, and releases a benchmark (AudioCRAG) specifically designed for this evaluation. Its significance is bounded by the fact that the underlying speech backbones are used off-the-shelf and the framework is validated only on single-turn factual QA with mock retrieval APIs.
Claims
- supports: Augmenting speech-in speech-out dialogue systems with external tool retrieval substantially improves factual question-answering accuracy over relying on internalized model knowledge alone.
Evidence: Across Qwen2.5-7B, OpusLM, and Kimi-Audio, moving from Closed Book to Open Book tool integration raises AudioCRAG-Synthetic accuracy by up to 140% relative (11.1% to 26.3% absolute for Qwen2.5-7B), though at the cost of roughly 2.3x higher first-token latency. (§1, Table 1)
- supports: Predicting tool queries incrementally from partial speech input, before the user finishes speaking, can recover most of the latency overhead that external tool calls introduce into spoken dialogue, without sacrificing accuracy.
Evidence: Model-Triggered Streaming RAG post-training on Qwen2.5-7B reaches 34.2% accuracy on AudioCRAG-Synthetic (higher than the non-streaming Open Book’s 26.3%) while cutting first-token tool-use latency by 20.7% on AudioCRAG-Synthetic and 53.4% on AudioCRAG-Human. (§5.1, Table 1)
- complicates: Generating a spoken response degrades factual QA accuracy relative to generating a text response from the same tool-augmented model and identical retrieved evidence.
Evidence: In the Open Book setting, all three evaluated models score lower for speech output than text output from the same references (Qwen2.5-7B: 39.6% text vs. 26.3% speech; Kimi-Audio: 45.8% text vs. 21.8% speech), and an intermediate text-then-speech self-cascade only partially closes the gap. (§5.2, Table 2)
- complicates: Streaming tool-query generation without a learned trigger requires substantially more parallel computation to achieve latency savings comparable to a model that learns when to query.
Evidence: Fixed-Interval Streaming RAG requires multiple parallel tool-call threads per query and reduces Open-Book tool-use latency by only 10.7% for Qwen-Omni without any post-training, whereas Model-Triggered Streaming RAG achieves larger average latency reductions using a single thread. (§5.4, Figure 4)
Limitations and Open Questions
Warning
All reported accuracy scores rely on automated LLM-judge evaluation (LLaMA-4-Maverick transcribing and scoring Whisper transcripts of the spoken output), and no human listening study validates either the naturalness of the spoken responses or the paper’s central claim about reduced user-perceived latency; the framework’s core motivation (avoiding “awkward silences”) is argued from measured latency numbers rather than from subjective user studies.
The evaluation is restricted to single-turn factual QA with simulated web and knowledge-graph APIs drawn from the CRAG corpus; multi-turn dialogue, ambiguous follow-up queries, and real (non-mocked) retrieval latency are not tested. Kimi-Audio could not be evaluated under Streaming RAG at all due to an architectural limit on tool-result reference length, and OpusLM only supports tool-result conditioning in speech-out mode after post-training, not zero-shot, indicating the framework’s applicability still depends on properties of the underlying backbone. The proposed Streaming RAG methods are demonstrated only in this speech-in speech-out setting on English factual QA; the paper notes the approach is modality-agnostic in principle but does not evaluate typed-input or multilingual scenarios.
Wiki Connections
- Spoken Language Model — evaluates and post-trains three existing speech-in speech-out language models (Qwen-Omni, OpusLM, Kimi-Audio) to add tool-use capability rather than proposing a new backbone.
- Speech-to-Speech — targets genuinely end-to-end speech-in speech-out dialogue, contrasting its tool-augmented approach with cascaded ASR-LLM-TTS pipelines throughout.
- Streaming TTS — introduces a streaming inference strategy (query prediction from partial audio blocks) specifically designed to reduce user-perceived latency in spoken response generation.
- On the Landscape of Spoken Language Models — cited as background survey characterizing the E2E spoken dialogue system landscape this paper extends with tool use.
- Kimi-Audio — used as one of three evaluated speech-in speech-out backbones, exposing an architectural constraint (limited tool-reference length) that prevents Streaming RAG from being applied to it.
- Qwen2.5-Omni — used as the primary evaluated backbone (as “Qwen-Omni”/Qwen2.5-7B), showing the largest accuracy and latency gains from Streaming RAG.
- AURA — prior work integrating tool use into voice-driven tasks via a cascaded pipeline, contrasted with this paper’s genuinely end-to-end speech-in speech-out approach.
- Mini-Omni — cited as one of the recent E2E spoken dialogue systems that, prior to this work, had not been trained or evaluated for external tool use.
- Retrieval-Augmented Generation for Large Language Models: A Survey — cited as the text-domain RAG survey whose techniques this paper extends into speech-in speech-out systems for the first time.