arXiv · 2025 · Preprint
Maben et al. (Carnegie Mellon University) · → Paper · Demo: ? · Code: ?
AURA is the first open-source cascaded speech-to-speech agent that integrates ASR, an LLM reasoning layer (ReAct), and TTS with real-world tool use (calendar, email, web search, contact lookup) for multi-turn spoken task completion.
Problem
Existing spoken dialogue systems fail to bridge two gaps simultaneously. End-to-end speech models (SpeechGPT, Moshi, Mini-Omni) aim to generate speech directly from speech, but VoiceBench benchmarks show they substantially underperform cascaded pipelines on practical tasks. Meanwhile, cascaded systems with strong text LLM backbones have not been extended to support real-world tool use in an open-source form. Evaluation benchmarks such as SpokenWOZ focus narrowly on dialogue state tracking from ASR transcripts without supporting execution of real-world APIs (calendar, email, web search), limiting their relevance to practical deployment.
Method
AURA adopts a modular cascaded architecture with four components: a Gradio-based User Interface, a Dialog Processing Unit (DPU), an LLM server, and external API integrations. Speech input is transcribed by either Whisper-v3-large or ESPnet’s OWSM (Open Whisper-style Speech Models); TTS output uses the ESPnet VITS model (kan-bayashi/ljspeech).
The DPU implements a ReAct-style agent loop. At each turn the LLM (LLaMA-3.3-Instruct-70B, hosted via vLLM with PagedAttention) produces a structured response with three fields: Thought (reasoning trace), Action type, and Payload. Supported action types include Chat (user-facing message), Calendar (event booking and editing), WebSearch (Wikipedia and web queries returning top-3 results), Email (composition with configurable recipient whitelist), and Contact retrieval. Observations from executed actions are appended to the action-observation history, enabling the agent to reason over prior results in compound multi-turn tasks.
Dialog State Tracking (DST) is optional and performed via prompt-based LLM inference on the conversation history without dedicated fine-tuning. New tools can be added by describing them in natural language and implementing an action class, without modifying the agent’s core loop.
Key Results
On VoiceBench OpenBookQA, AURA with Whisper-v3-large and LLaMA-3.3-70B achieves 92.75% accuracy, outperforming all open-weight spoken systems including Kimi-Audio (83.52%) and approaching the closed GPT-4o pipeline at 92.97% (Table I). With ESPnet-OWSM as the ASR component, enabling mandatory web search raises accuracy from 87.69% to 89.23%, demonstrating the practical value of retrieval augmentation for knowledge-intensive QA. On AlpacaEval (open-ended, GPT-4o-mini scored), AURA reaches 4.39 out of 5, competitive with open-weight systems and below closed models (GPT-4o-Audio: 4.78).
Human evaluation of 30 multi-turn compound tasks yields 90% overall task success (100% on easy, 91.67% on medium, 83.33% on hard) with satisfaction scores above 4.0 across all difficulty levels (Table II). On SpokenWOZ dialog state tracking, AURA achieves 28.76% JGA without fine-tuning, surpassing the best prior baseline (SPACE + WavLMalign: 25.65%) by 3.1 points (Table III). The human evaluation uses a single evaluator who is a co-author, and comparisons are made against leaderboard baselines without controlled re-evaluation.
Novelty Assessment
The contribution is engineering integration. ASR (Whisper, OWSM), LLM reasoning (LLaMA-3.3, ReAct), and TTS (VITS via ESPnet) are all independently established components; the ReAct reasoning paradigm is adopted directly from prior text-based work. The system’s originality lies in being the first open-source implementation that combines these into a speech-native agent capable of real-world tool invocation across multiple API types. The modular action-class design, which allows tool addition via natural language description, is a practical software engineering contribution. The benchmarking result showing open cascaded systems can approach GPT-4o on VoiceBench is empirically informative for the field, even without novel architecture.
Field Significance
Moderate — AURA provides concrete evidence that open-weight cascaded spoken conversational agents can match or approach closed proprietary systems on spoken QA benchmarks, while also demonstrating a practical path to multi-turn agentic task execution via standard API integration. The paper introduces no new architecture or training technique; its value is as a reproducible open-source baseline and as evidence that the performance gap between open and closed speech assistants on task-oriented benchmarks is closable without end-to-end speech modeling.
Claims
-
supports: Cascaded ASR+LLM+TTS pipelines outperform end-to-end speech models on practical spoken task benchmarks when a high-quality instruction-tuned text LLM serves as the reasoning backbone.
Evidence: AURA with Whisper+LLaMA-3.3-70B achieves 92.75% on VoiceBench OpenBookQA versus 25.93% (Moshi) and 26.59% (Mini-Omni2) for end-to-end speech models, and 83.52% for Kimi-Audio. (§III.A, Table I)
-
supports: Retrieval augmentation via web search at inference time improves spoken question answering on knowledge-intensive multiple-choice tasks without retraining.
Evidence: Prompting AURA (OWSM ASR) to perform at least one web search before answering raises OpenBookQA accuracy from 87.69% to 89.23%. (§III.A)
-
supports: ReAct-style interleaved reasoning and action enables reliable tool selection and goal completion in multi-turn spoken conversational agents.
Evidence: AURA achieves 90% overall task success and satisfaction scores above 4.0 across 30 human-evaluated tasks spanning compound goals requiring calendar, email, contact, and web search APIs. (§III.B, Table II)
-
complicates: Dialog state tracking in spoken task-oriented systems remains far from reliable even with large instruction-tuned LLMs applied without fine-tuning.
Evidence: AURA achieves 28.76% JGA on SpokenWOZ, improving over the best prior baseline by 3.1 points, but remaining well below full accuracy, indicating that structured state extraction from spoken conversation is a persistent challenge. (§III.C, Table III)
Limitations and Open Questions
Warning
The human evaluation of multi-turn task success uses a single evaluator (a co-author), which introduces potential bias and limits the reliability of reported success rates. No inter-annotator agreement or independent replication is provided.
The TTS component is a single English, single-speaker VITS model (LJSpeech); naturalness, speaker diversity, and multilingual coverage are not assessed. Response latency is not reported, which is critical for real-time spoken dialogue with tool calls that may have variable completion times. The 28.76% JGA on SpokenWOZ, while an improvement, remains too low for robust structured task completion in production settings. AlpacaEval scoring by GPT-4o-mini introduces evaluator-model variance that is not characterized.
Wiki Connections
- Speech-to-Speech — AURA instantiates a full speech-in, speech-out agent loop with tool use, extending speech-to-speech systems to real-world task execution via external APIs.
- Evaluation Metrics — the paper benchmarks against VoiceBench (OpenBookQA accuracy, AlpacaEval) and SpokenWOZ JGA, contributing evidence on how open cascaded systems score relative to end-to-end models.
- Subjective Evaluation — human evaluation of 30 multi-turn compound tasks provides task success rates and satisfaction scores as a complement to automated benchmark results.
- OpenOmni — cited as an end-to-end speech model that AURA’s cascaded approach contextualizes against in the broader landscape of open spoken conversational agents.