arXiv · 2026 · Preprint
Wenhao Zou et al. (Meituan / University of Chinese Academy of Sciences) · → Paper · Demo: ? · Code: ?
Introduces LTS-VoiceAgent, a cascaded (ASR-LLM-TTS) streaming voice-agent framework that separates when to begin reasoning (a Dynamic Semantic Trigger detecting semantically complete prefixes) from how to reason incrementally (a Dual-Role Stream Orchestrator coordinating a background state-tracking “Thinker” with a foreground speculative “Speaker”), achieving near-real-time latency without the wasted computation or semantic fragmentation of chunk- or VAD-based streaming baselines.
Problem
Real-time voice agents face a latency-versus-intelligence trade-off: end-to-end speech models respond quickly but reason shallowly, so cascaded ASR-LLM-TTS pipelines remain the dominant choice for complex agentic tasks, at the cost of high latency from executing the three stages strictly in sequence, unlike human listeners who begin thinking before a speaker finishes. Existing streaming mitigations fall into two failure modes: aggressive speculative generation “guesses” the user’s intent from incomplete input and must frequently roll back predictions when that guess is wrong (wasting computation and disrupting responses when intent shifts mid-utterance), while mechanical segmentation (fixed chunks or VAD-based silence detection) triggers reasoning on acoustically but not semantically complete fragments, since VAD relies on silence duration rather than whether enough information has actually accumulated, producing “invalid thinking” during meaningless hesitations like “um…”.
Method
LTS-VoiceAgent has two coordinated components operating on the incrementally growing ASR transcript. The Dynamic Semantic Trigger is a lightweight streaming binary classifier (DistilBERT) that estimates the “semantic saturation probability” of the current transcript prefix and activates reasoning only once this probability exceeds a threshold τ=0.65, rather than reacting to every new token or every silence gap. Because large-scale annotated “semantic breakpoint” data does not exist, the trigger’s training data is synthesized: GPT-4o is prompted as a “linguistic analysis expert” to insert boundary markers into reasoning-intensive seed texts (GSM8K, MMLU-Pro) at points of semantic completeness while explicitly skipping disfluent hesitation markers, the annotations are manually verified, and the marked texts are sliced into streaming prefixes labeled positive or negative depending on whether they cross a marker (yielding ~100k balanced training examples). At inference, threshold gating combines with a deduplication-suppression mechanism that blocks re-triggering on transcripts unchanged since the last trigger, preventing jitter from noisy ASR output.
The Dual-Role Stream Orchestrator addresses what happens once triggered. Rather than running understanding and generation as one serial process (which still causes interruptions and “thought discontinuity” even with a good trigger), it concurrently schedules two roles via the inference engine’s batch processing: a background Thinker that performs input sanitization (correcting likely ASR phonetic errors using context, e.g., “sign”→“sine”), extracts key variables, and maintains an evolving plan, packaged as a compact JSON state snapshot; and a foreground Speaker that performs speculative “Answer-First” response generation following a Restate-Consult-Solve pattern (restate the problem for implicit self-verification, consult the Thinker’s shared plan if available, then solve). The Thinker’s prior-turn plan and key variables are injected into the Speaker’s context as prior knowledge, letting the Speaker benefit from deep background planning without waiting on the Thinker’s real-time inference. When the orchestrator detects a substantial semantic shift in the incoming stream (e.g., a self-correction), it terminates the current Speaker process but preserves and merges the Thinker’s valid accumulated state into the next context, an efficient “thought rollback” rather than discarding all prior computation. The paper also introduces a Pause-and-Repair benchmark: GSM8K/MMLU-Pro questions rewritten into first-person spoken help-seeking scenarios, injected with density-controlled filler words and four self-correction patterns (entity correction, intent reset, etc.) via GPT-4o, and converted to waveform via CosyVoice2, specifically to stress-test streaming robustness against natural spoken disfluencies rather than only clean or heuristically-noised text.
Key Results
Across VERA, Spoken-MQA, BigBenchAudio, and the authors’ own Pause-and-Repair benchmark, LTS-VoiceAgent achieves streaming-level latency (sub-second time-to-first-sentence, 209-415ms depending on benchmark) while substantially closing the accuracy gap to a full-context serial “Think” upper bound that requires tens-to-hundreds of seconds of TTFS (e.g., BigBenchAudio: 71.9% vs. 73.77% upper bound, at a fraction of the latency). Against chunk-based (PredGen) and VAD-augmented (LTS-VAD) speculative baselines at comparable latency, LTS-VoiceAgent achieves markedly higher accuracy on most benchmarks. Efficiency analysis reveals a stark contrast in wasted computation: PredGen triggers 59-177 forward-pass evaluations per query depending on dataset, with 90-99% of them interrupted and rolled back; LTS-VAD reduces evaluation count substantially but still shows 41-90% interruption rates; LTS-VoiceAgent uses only ~2-2.4 forward passes per query with single-digit interruption rates (5.5-9.8%) across all four benchmarks, and trigger inference itself averages roughly 5ms. An end-to-end speech baseline (Qwen2.5-Omni) trails the cascaded LTS-VoiceAgent substantially on accuracy across all benchmarks despite comparable latency, illustrating the paper’s stated end-to-end-vs-cascaded intelligence gap directly. Ablations isolate each component’s distinct contribution: removing the semantic trigger (falling back to mechanical triggering) increases forward-pass count from 2.10 to 7.23 and interruption rate from 8.20% to 60.12%, showing the trigger is primarily responsible for computational efficiency and interruption robustness; removing the orchestrator degrades accuracy more than removing the trigger does (49.37% vs. 55.37% full-system average) while leaving forward-pass count nearly unchanged, showing orchestration is primarily responsible for reasoning quality and stability rather than raw efficiency. Scaling the backbone from Qwen3-8B to Qwen3-32B (Appendix C) consistently improves accuracy while preserving sub-second responsiveness, indicating the framework’s latency benefits are not backbone-size-dependent.
Novelty Assessment
The contribution is a genuinely new streaming-reasoning architecture rather than a new speech-generation mechanism: the Dynamic Semantic Trigger replaces acoustic silence-based (VAD) or fixed-length triggering with a learned semantic-completeness signal, and the Dual-Role Stream Orchestrator’s asynchronous Thinker/Speaker coordination with explicit state injection and selective rollback is a distinct design from both fully serial reasoning and from prior “listen-and-guess” speculative approaches that roll back entire predictions rather than preserving accumulated background state. Both components are individually ablated with clear, distinguishable effects (trigger → efficiency/interruption-robustness; orchestrator → accuracy/stability), which is stronger evidence of genuine mechanism-level novelty than a single combined ablation would provide. The Pause-and-Repair benchmark, evaluated via a genuine audio-only pipeline with real ASR output rather than simulated text-chunk or heuristic-noise approximations, is a reusable methodological contribution independent of the core framework. The underlying components (ASR, LLM backbone, TTS for benchmark audio) are all off-the-shelf; no new speech synthesis or recognition architecture is proposed.
Field Significance
high — this paper demonstrates that gating streaming-dialogue reasoning on a learned semantic-completeness signal, combined with an explicitly decoupled background-state/foreground-response architecture with selective state preservation on interruption, can substantially narrow the accuracy gap to full-context serial reasoning while achieving near-real-time latency and near-zero wasted speculative computation, directly addressing the cascaded-voice-agent latency-intelligence trade-off with individually ablated evidence for why each of its two components matters.
Claims
- supports: Gating a streaming spoken-dialogue reasoning module on learned semantic completeness, rather than acoustic silence or fixed-length chunking, can substantially close the accuracy gap to full-context reasoning while achieving near-real-time voice-agent latency.
Evidence: LTS-VoiceAgent reaches 71.9% accuracy on BigBenchAudio (vs. 73.77% for a full-context serial upper bound) at 209-415ms time-to-first-sentence across benchmarks, versus tens-to-hundreds of seconds TTFS for the serial upper bound. (§5.2, Table 1)
- complicates: Acoustic voice-activity-detection-based triggering for streaming spoken-dialogue reasoning is fundamentally limited because silence duration is not a reliable proxy for semantic completeness, producing substantial wasted computation and unstable reasoning even when paired with speculative generation.
Evidence: VAD-augmented speculative baselines (LTS-VAD) incur 41-90% interruption rates depending on dataset, since silence-based triggering fires during meaningless hesitations rather than genuine semantic boundaries, whereas the semantic-trigger-based system achieves 5.5-9.8% interruption rates across all four benchmarks. (§5.2, Table 2)
- supports: Separating a streaming spoken-dialogue system into an asynchronous background state-tracking role and a foreground speculative-response role, coordinated via explicit state injection, preserves reasoning quality under evolving partial transcripts more effectively than a semantic trigger alone.
Evidence: Removing the dual-role orchestrator (while keeping the semantic trigger) degrades average accuracy across four benchmarks more than removing the trigger alone (49.37% vs. 55.37% for the full system vs. 52.30% without the trigger), while the forward-pass count stays nearly identical, isolating the orchestrator’s contribution to quality rather than latency. (§5.2, Table 3)
- complicates: Standard spoken-agent evaluation pipelines that approximate streaming ASR via fixed text chunks or heuristic noise injection understate the difficulty real disfluent speech poses to streaming voice agents, since natural hesitation and self-correction patterns are not adequately represented by mechanical chunking or synthetic noise alone.
Evidence: The authors construct a dedicated audio-only Pause-and-Repair benchmark specifically because prior evaluation setups approximate streaming ASR behavior rather than exposing systems to genuine ASR output under natural hesitations, self-corrections, and non-uniform speaking rates. (§5.1.1, §4)
Limitations and Open Questions
The authors’ own limitations note that evaluation covers a limited range of backbones, languages, and acoustic conditions, and that even with the audio-only protocol and the Pause-and-Repair benchmark’s disfluency stress-testing, the benchmarks do not capture the full diversity of real conversations, including multi-turn dialogue dynamics, background noise, accents, domain-specific terminology, or mid-conversation intent shifts beyond those explicitly synthesized. The paper does not include human-subject studies measuring perceived responsiveness or usefulness, nor a comprehensive analysis of safety and privacy risks in deployment, both of which the authors flag as requiring broader data and application-specific evaluation.
Wiki Connections
- Speech-to-Speech — proposes a cascaded (ASR-LLM-TTS) streaming architecture specifically targeting the latency-intelligence trade-off in spoken dialogue reasoning.
- Evaluation Metrics — introduces the Pause-and-Repair benchmark, an audio-only evaluation pipeline with synthesized natural disfluencies (fillers, self-corrections) for stress-testing streaming voice-agent robustness.
- PredGen — used as a direct chunk-based speculative-generation baseline, compared head-to-head on accuracy, latency, and interruption-rate metrics across all four benchmarks.
- Qwen2.5-Omni — used as a direct end-to-end speech-model baseline to quantify the accuracy gap between end-to-end and cascaded voice-agent architectures.
- CosyVoice 2 — used as the off-the-shelf TTS component to synthesize waveform audio for the Pause-and-Repair benchmark.
- VERA — one of three public benchmarks (alongside Spoken-MQA and BigBenchAudio) used to evaluate LTS-VoiceAgent’s reasoning accuracy and latency.
- VoiceAgentBench — cited as related work on evaluating whether voice assistants are ready for complex agentic tasks, part of the motivating context for cascaded architectures remaining dominant for such tasks.
- Moshi — cited as a representative end-to-end speech-text foundation model, part of the end-to-end-vs-cascaded trade-off this paper’s cascaded design is positioned against.
- LLaMA-Omni — cited as a representative end-to-end speech-interaction model achieving low latency at the cost of reasoning depth, motivating the paper’s cascaded alternative.
- Freeze-Omni — cited as a related low-latency speech-to-speech dialogue system built on a frozen LLM, part of the broader landscape of latency-focused voice-agent architectures.