arXiv · 2025 · Preprint
Dhruv Jain et al. (Ola Electric / Krutrim) · → Paper · Demo: ✗ · Code: ✓
Introduces a 6,000+ query benchmark that evaluates SpeechLMs and ASR-LLM pipelines on agentic tool-use tasks (single/parallel/sequential tool calls, multi-turn dialogue, adversarial safety) across English and six Indic languages.
Problem
Prior speech benchmarks (SUPERB, SLUE, AIR-Bench, AudioBench, VoiceBench) evaluate isolated speech capabilities such as transcription, translation, or single-turn instruction following, while LLM agent benchmarks (ToolBench, BFCL, API-Bank, AgentHarm) evaluate tool-use and safety only in the text modality. No existing benchmark systematically tests whether voice assistants built on SpeechLMs or ASR-LLM pipelines can perform the agentic behaviors expected of a real voice assistant: selecting and invoking the right tool, filling parameters correctly, chaining dependent tool calls, tracking multi-turn dialogue state, and refusing unsafe requests. Multilingual and culturally grounded coverage of these agentic settings is also absent from prior work.
Method
VoiceAgentBench (VAB) is a benchmark, not a proposed model. It comprises over 6,000 synthetic spoken queries spanning six task categories: single tool invocation, single tool invocation with retrieval, parallel (independent) tool calls, sequentially dependent tool calls (a novel category built around 21 custom tools across cab-booking, food-ordering, and payment agents), multi-turn dialogue-based tool invocation (adapted from API-Bank), and safety evaluation (adapted from AgentHarmBench, covering 11 harm categories). Query text and ground-truth tool calls are sourced from BFCL, API-Bank, and AgentHarmBench and extended with LLM-generated (Gemma3 27B, GPT-4o-mini) Indian-context variants, then translated into six Indic languages (Hindi, Bengali, Marathi, Tamil, Telugu, Malayalam) to form a Multilingual subset alongside the English subset.
To convert queries to speech, the benchmark uses ElevenLabs and Coqui-TTS for English and Krutrim-TTS for Indic languages, with a novel diversity-driven sampling strategy for selecting source audios for TTS voice conversion: ECAPA-TDNN speaker embeddings are used to select maximally diverse reference audios (comparing Determinantal Point Processes, Farthest Point Sampling, a density-based method, and random sampling), with Farthest Point Sampling found to maximize acoustic diversity as measured by mean distance to the nearest selected point. This diversity-sampling component is the paper’s own methodological contribution; the tool-use tasks themselves reuse or adapt existing text-domain benchmarks.
Evaluation defines four metrics: Tool Selection (TS, exact tool-name match), Tool Call Structure (TCS, schema validity via Pydantic), Parameter Filling (PF, GPT-4o-mini-judged argument correctness against ground truth), and Refusal Rate (RR, GPT-4o-mini-judged refusal of unsafe requests). The paper benchmarks two classes of systems as external subjects under test: three end-to-end SpeechLMs (KimiAudio 7B, Qwen2.5-Omni 7B, AudioFlamingo3 7B) and three ASR-LLM cascade pipelines (Whisper v3 Large transcription followed by Qwen-3 8B, Gemma3 27B, or Llama 3.3 70B), all prompted in a one-shot setting to produce tool calls in Python syntax.
Key Results
On the English subset, ASR-LLM pipelines outperform end-to-end SpeechLMs overall: Whisperv3-Llama3 70B reaches the highest average Parameter Filling (60.64%), with Whisperv3-Gemma3 27B close behind (59.28%), compared to KimiAudio 7B’s 57.57% as the strongest SpeechLM. KimiAudio approaches ASR-LLM pipelines on simple Single Tool Calling but falls further behind on more complex categories (SinTC with Retrieval, Parallel, Sequential Dependent). AudioFlamingo3 7B and Qwen2.5-Omni 7B trail well behind all other systems.
On the Multilingual subset (six Indic languages), every system degrades sharply relative to English: the strongest ASR-LLM pipelines drop from ~59-61% average PF on English to 35-39% on Indic languages, and SpeechLMs degrade even more steeply (KimiAudio 7B falls from 57.57% to 28.21% average PF). Sequential Dependent tool calling is the hardest category throughout: even the best English result is only 14.8% PF (Whisperv3-Qwen3 8B), falling to 4.3% on the multilingual subset.
On safety, refusal rates are generally low and inconsistent across languages: KimiAudio 7B’s English refusal rate of 51.78% collapses to 2.67% on Indic queries; ASR-LLM pipelines are more consistent (Whisperv3-Gemma3 27B: 59.56% English, 38.20% Indic) but still degrade. An ablation replacing Whisper transcripts with ground-truth text improves average PF on non-Hindi Indic languages by at least 24 percentage points, and substituting an Indic-specialized ASR model (IndicConformer) recovers roughly 40-55% of the ASR-induced loss, isolating transcription quality as a major bottleneck for ASR-LLM performance in low-resource languages. A further ablation shows removing the one-shot example or refusal prompt from the system message causes large drops in Parameter Filling and refusal rate respectively, and adding adversarial phrasing (harmful hints) lowers refusal rates to 35-40% across all tested models.
The paper excludes closed-source systems (GPT-4o-audio, Gemini 2.5 Pro) due to cost, so comparisons are limited to open-weight SpeechLMs and open ASR/LLM combinations.
Novelty Assessment
The contribution is primarily a new evaluation resource: an engineering integration of existing tool-calling benchmarks (BFCL, API-Bank, AgentHarmBench) into a speech modality, extended with a genuinely novel category (sequentially dependent multi-tool workflows across custom cab/food/payment agents) and a genuinely novel diversity-sampling method for selecting voice-conversion source audio. It proposes no new model architecture, training objective, or inference procedure for speech generation or understanding; TTS and voice conversion are used purely as a data-construction tool to render text queries as audio, not as a research contribution being evaluated in its own right. The value of the paper lies in the systematic multilingual, agentic evaluation design and the resulting empirical findings (ASR-LLM pipelines beating SpeechLMs, severe multilingual and sequential-task degradation), not in architectural innovation.
Field Significance
Moderate — this paper provides the first systematic evaluation of agentic tool-use capability for SpeechLM-based voice assistants, extending function-calling evaluation methodology (previously confined to text-only LLM agents) into the speech modality with multilingual coverage. Its findings that ASR-LLM cascades currently outperform end-to-end SpeechLMs on complex tool orchestration, and that both classes of system degrade sharply on low-resource languages and sequential workflows, are useful empirical data points for practitioners choosing a voice-assistant architecture, though the benchmark itself introduces no new modeling technique.
Claims
- supports: Cascaded ASR-LLM pipelines currently retain an advantage over end-to-end SpeechLMs on complex agentic tool-use tasks, even though end-to-end systems avoid the latency of a separate transcription step.
Evidence: On the English subset, Whisperv3-Llama3 70B and Whisperv3-Gemma3 27B achieve average Parameter Filling of 60.64% and 59.28% respectively, versus 57.57% for the strongest SpeechLM (KimiAudio 7B), with the gap widening on harder categories such as Sequential Dependent and Parallel Tool Calling. (§4.2, Table 2)
- complicates: Agentic tool-use performance of both SpeechLM and ASR-LLM voice assistants generalizes poorly from high-resource to low-resource languages.
Evidence: Average Parameter Filling for the strongest ASR-LLM pipelines drops from 59.3-60.6% on English to 35.3-39.2% on the six-language Indic subset, with SpeechLMs showing even larger declines (e.g. KimiAudio 7B falls from 57.57% to 28.21%). (§4.2, Table 3)
- complicates: Sequential, dependency-aware multi-tool orchestration remains a largely unsolved capability for current speech-based agents, independent of whether the system is a SpeechLM or an ASR-LLM cascade.
Evidence: Sequential Dependent Tool Calling yields the lowest Parameter Filling of any category for every model tested, with the best English result reaching only 14.8% PF (Whisperv3-Qwen3 8B) and the best multilingual result only 4.3% PF. (§4.2, Table 2, Table 3)
- refines: The multilingual performance gap in ASR-LLM voice-assistant pipelines is substantially attributable to ASR transcription error rather than to the downstream LLM’s reasoning or tool-use ability.
Evidence: Replacing Whisper v3 transcripts with ground-truth text improves average Parameter Filling by at least 24 percentage points across non-Hindi Indic languages, and substituting an Indic-specialized ASR model recovers roughly 40-55% of the ASR-induced performance loss. (§4.3)
- complicates: Safety refusal behavior learned by speech-based voice assistants is fragile and does not transfer robustly across languages or under adversarial phrasing.
Evidence: KimiAudio 7B’s refusal rate on adversarial safety queries falls from 51.78% in English to 2.67% in Indic languages, and across all tested models refusal rates drop further (to 35-40%) when adversarial hints are added to unsafe requests, or collapse when explicit refusal prompts are removed from the system message. (§4.2, Table 4; §4.3, Figure 3)
Limitations and Open Questions
Warning
The benchmark excludes closed-source voice assistants (GPT-4o-audio, Gemini 2.5 Pro) due to cost constraints, so the reported ASR-LLM-vs-SpeechLM ranking may not hold against the strongest proprietary systems.
The evaluation does not include speech with background noise, so robustness to acoustic degradation is untested. Multi-turn dialogue evaluation is not conducted for all Indic languages (results reported are Hindi-only for that category). The benchmark evaluates static, single-shot tool invocation rather than dynamic, real-time interactive tool use with a live environment. Parameter Filling correctness relies on an LLM judge (GPT-4o-mini) rather than exact-match scoring, which the authors support with a human-agreement study but which nonetheless introduces judge-model bias as a potential confound.
Wiki Connections
- Spoken Language Model — evaluates three end-to-end SpeechLMs (KimiAudio, Qwen2.5-Omni, AudioFlamingo3) specifically on agentic tool-use tasks that go beyond their originally reported transcription and QA capabilities.
- Evaluation Metrics — introduces four task-specific automatic metrics (Tool Selection, Tool Call Structure, Parameter Filling, Refusal Rate) for scoring agentic speech interactions, extending function-calling evaluation methodology into the speech modality.
- Voice Conversion — proposes a novel speaker-embedding-based diversity sampling strategy (comparing DPP, Farthest Point Sampling, and density-based methods) for selecting source audio used in TTS voice conversion during benchmark construction.
- Stream RAG — both papers target agentic tool-use capability in spoken dialogue systems; VoiceAgentBench provides the benchmark that could be used to evaluate systems like Stream RAG’s streaming tool-use approach.
- VoiceBench — VoiceAgentBench extends VoiceBench’s approach of converting text instructions to spoken form for LLM-based voice assistant evaluation, adding agentic tool-use, multi-turn, and safety dimensions that VoiceBench does not cover.
- Qwen2.5-Omni — evaluated directly as one of the three end-to-end SpeechLMs under test, where it underperforms both KimiAudio and all ASR-LLM pipelines on agentic tool-use tasks.
- Kimi-Audio — evaluated directly as the strongest end-to-end SpeechLM under test, approaching but not matching ASR-LLM pipeline performance on tool-use tasks.
- Step-Audio 2 — a contemporaneous SpeechLM technical report cited as related work on instruction-following, chat-style SpeechLMs, though not itself benchmarked in this paper.
- AudioPaLM — cited as foundational related work establishing LLM-based speech understanding and generation, which this paper extends into agentic, tool-using evaluation settings.