Interspeech · 2026 · Conference
Ramit Pahwa et al. (Rivian and Volkswagen Group Technologies) · → Paper · Demo: ? · Code: ✓
Audio2Tool is a roughly 30,000-query spoken tool-calling benchmark over 152 smart-car, smart-home and wearable functions, organised into eight complexity tiers and voiced with zero-shot voice-cloning TTS plus noise. It shows that speech LMs pick the right tool far more reliably than they fill its arguments, and that neither end-to-end speech LMs nor ASR-LLM cascades handle compositional, corrective or distractor-speech queries well.
Problem
Voice assistants are moving from intent classification to audio-native function calling, where a speech LM maps a spoken request directly onto an executable API call with valid arguments. Classic spoken language understanding corpora (ATIS, SNIPS, SLURP, STOP, MAC-SLU) score intents and slots, not executable calls, and the newer function-calling benchmarks (BFCL, BFCL Audio, MFCL, VoiceAgentBench) offer limited domain breadth, little acoustic diversity and no graded query hierarchy for isolating specific audio-to-tool failure modes. The authors want a large, acoustically varied benchmark that separates tool selection, argument grounding and multi-step composition.
Method
Tool taxonomy. The benchmark covers 152 verified functions grounded in real APIs (including standard Android Automotive functionality), grouped into 23 categories across three domains: Smart Car (14 categories, such as climate, driving dynamics, charging and maintenance), Smart Home (lighting, appliances, sensors, security) and Wearables (health tracking, connectivity). Smart Car makes up the majority of queries in every tier. Compared with prior SLU sets (Table 1), Audio2Tool has fewer domains (3) but more intents (152), with 22 slot types, and models multi-intent requests explicitly.
Eight complexity tiers. Queries are written by closed-source LLMs (GPT-5.2, Gemini 2.5 Pro, Claude Opus) to fill eight tiers: (1) direct 2-6 word commands; (2) parametric slot-filling commands; (3) multi-intent utterances (2 to 5 calls, mostly 2 or 3); (4) implicit requests needing pragmatic inference (“I’m freezing”); (5) needle-in-a-haystack extraction from long, mostly irrelevant audio; (6) mid-utterance corrections; (7) multi-turn dialogues of 4-8 turns needing state tracking; and (8) intent blending, where a background speaker (radio, podcast, passenger) utters a second valid command that must be ignored. Tier 1 has 1,080 queries, Tier 2 has 5,800, and Tiers 3-7 have 4,560 each. Queries are checked by an LLM-as-judge pair (GPT-5.1, Gemini-2.5-pro), and any query where at least one judge failed is checked by hand.
Speech generation. Each query is voiced by zero-shot voice-cloning TTS, using Qwen3-TTS and CosyVoice 3, with reference speakers drawn from SPGISpeech 2.0, Emilia-YODAS, 3D-Speaker and VoxPopuli (100, 100, 30 and 100 selected speakers respectively, Table 2) to cover US, Asian, Latin American, Chinese and European accents. Speakers are subsampled from embedding space by stratified farthest-point sampling, several speakers are sampled per query, and each rendering is mixed with automotive and indoor noise (engine, road, wind, HVAC, rain, turn signal, room noise) (Algorithm 1).
Evaluation protocol. Systems are scored zero-shot on Tool Accuracy (ordered tool sequence matches), Exact Match (all tools and normalised arguments match) and Slot F1 (micro-averaged per-argument F1). Two families are compared: open end-to-end speech LMs (Step-Audio 2, Audio Flamingo 3, Kimi-Audio, Qwen-2.5-Omni, Qwen3-Omni; 7B to 30B), and cascades of Whisper large-v3 with Qwen 3 or Gemma 3 back-ends (1.7B to 27B). An oracle setting feeds ground-truth text to the same LLMs to isolate ASR error; it is not run on Tier 8.
Key Results
All numbers are from Table 3.
- Simple commands are largely solved. Qwen-3-Omni-30B reaches 92.4% Tool Accuracy on Tier 1, and the Whisper + Gemma 27B cascade 87.9%.
- Tool selection outpaces argument grounding. On Tier 2, Tool Accuracy reaches 84.6% (Qwen-3-Omni-30B) but no system exceeds 15.6% Exact Match, including the oracle-text LLMs (at most 12.3% EM). On Tier 4 (implicit requests) every system stays at or below 33.4% Tool Accuracy.
- End-to-end vs. cascade. Qwen-3-Omni-30B has the highest Tool Accuracy on Tiers 1-6, but the Whisper + Gemma 27B cascade beats it on Exact Match and Slot F1 on Tiers 3, 5, 6 and 7 (for example Tier 6 EM 44.1 vs. 24.8; Tier 3 EM 36.9 vs. 28.6), and on Tool Accuracy on Tiers 7 and 8. The paper’s §5.3 statement that Qwen-3-Omni-30B outperforms others on Tiers 5-7 holds only for Tool Accuracy on Tiers 5 and 6. The other speech LMs (Kimi-7B, Step-Audio2-7B, Audio-Flamingo-8B, Qwen-2.5-Omni-7B) generally trail the larger cascades.
- ASR cost in cascades. Replacing Whisper transcripts with ground-truth text consistently helps; for Gemma 12B, Tier 3 Tool Accuracy rises from 50.4 to 59.2 and Tier 4 from 18.9 to 27.4.
- Scale matters in cascades. Whisper + Qwen Tier 2 Tool Accuracy goes from 33.6 (1.7B) to 67.4 (8B).
- Intent blending is hardest. On Tier 8 the best Tool Accuracy is 50.5 (Whisper + Gemma 27B) and the best Exact Match is 16.3 (Qwen-3-Omni-30B); no system was designed to attribute commands to the primary speaker.
- Noise. In an ablation with MS-SNSD babble, mechanical hum and impulsive noise at +15, +5 and -5 dB SNR, intent F1 falls as noise increases for Qwen-3-Omni-30B, Kimi and Whisper + Gemma 12B, and the cascade falls most steeply (Figure 3). Exact values are only shown graphically.
Comparisons are fair within the benchmark (same queries, same zero-shot prompting), but only open models are tested, and each family’s prompting setup is described only briefly.
Novelty Assessment
The contribution is a dataset and an evaluation design, not a model. What is new relative to VoiceAgentBench and BFCL Audio is the domain-grounded IoT/automotive taxonomy, the eight-tier curriculum that separates failure modes (implicit intent, correction, long-context extraction, distractor speech), and the accent-diverse speaker selection with mixed-in noise. The construction pipeline itself (LLM-written queries, LLM-judge filtering, zero-shot TTS rendering) is standard for recent synthetic speech benchmarks. The analysis stays at the table level: there is no error breakdown by accent, TTS system or noise type beyond the small Figure 3 ablation, and no real recordings are used to check whether synthetic-speech difficulty transfers.
Field Significance
Moderate — Audio2Tool provides a graded, IoT-grounded test set for speech-driven function calling, and its main finding, that speech LMs often choose the right tool while getting its arguments wrong, gives a concrete argument-grounding target that intent-level SLU benchmarks do not expose. Its evidence comes entirely from synthetic speech, so it describes robustness to TTS-rendered accents and added noise rather than to real in-car or in-home recordings.
Claims
- supports: In speech-driven tool calling, end-to-end speech LMs can match or exceed ASR-LLM cascades at selecting the right tool while still trailing them at producing exactly correct arguments.
Evidence: Qwen-3-Omni-30B has the highest Tool Accuracy on Tiers 1-6, but the Whisper-v3 + Gemma 27B cascade has higher Exact Match on Tiers 3, 5, 6 and 7 (Tier 6: 44.1 vs. 24.8 EM). (§5.3, Table 3)
- supports: Argument grounding, not tool selection, is the main bottleneck for spoken function calling, and it persists even when transcription errors are removed.
Evidence: Tier 2 parametric commands reach up to 84.6% Tool Accuracy but at most 15.6% Exact Match across all systems; the oracle-text LLMs also stay at or below 12.3% EM. (§5.3, Table 3)
- supports: ASR errors measurably degrade downstream tool calling in cascaded voice-assistant pipelines relative to ground-truth transcripts.
Evidence: The same Gemma 12B back-end drops from 59.2 to 50.4 Tool Accuracy on Tier 3 and from 27.4 to 18.9 on Tier 4 when given Whisper-v3 transcripts instead of oracle text. (§5.1, Table 3)
- complicates: Spoken assistants remain poor at ignoring valid commands spoken by background or secondary speakers, a failure that intent-level benchmarks with a single clean speaker do not measure.
Evidence: On Tier 8 (intent blending with a distractor speaker) the best Tool Accuracy is 50.5 and no system exceeds 16.3 Exact Match. (§3.1, §5.3, Table 3)
- supports: Additive background noise substantially degrades spoken tool-call understanding, with cascades more sensitive than the strongest end-to-end speech LM.
Evidence: Under babble, mechanical hum and impulsive noise at +15 to -5 dB SNR, intent F1 declines for all three tested systems, most steeply for Whisper + Gemma 12B and least for Qwen-3-Omni-30B. (§5.4, Figure 3)
Limitations and Open Questions
Warning
All benchmark audio is synthetic: queries are LLM-written and voiced by two zero-shot voice-cloning TTS systems, which the authors acknowledge. Whether the measured tier and accent difficulty carries over to real speech in cars or homes is untested.
- Queries and their ground truth come from LLMs and are filtered by LLM judges, with human checks only where a judge failed; residual label noise is not quantified.
- Exact Match depends on deterministic argument normalisation, which is not specified in detail; the very low Tier 2 EM even for oracle-text LLMs may partly reflect normalisation strictness rather than model failure.
- Only open models are evaluated; no closed real-time voice assistants are included.
- The noise ablation (Figure 3) covers three systems and reports only an intent F1 score, and there is no breakdown by accent, TTS source or speaker region despite the accent-diversity design.
- The domain mix favours Smart Car, so aggregate scores mostly reflect automotive tools.
Wiki Connections
- Spoken Language Model — Benchmarks five open speech LMs on audio-native tool calling and shows that they select tools well but ground arguments poorly compared with strong ASR-LLM cascades, including an oracle-transcript setting and multi-turn dialogue tiers.
- Evaluation Metrics — Contributes an eight-tier spoken tool-calling test set scored with Tool Accuracy, Exact Match and Slot F1 to separate tool selection from argument grounding.
- VoiceAgentBench — Closest prior agentic voice benchmark; Audio2Tool positions itself as broader in taxonomy, acoustic diversity and query hierarchy.
- Qwen3-TTS — One of the two zero-shot voice-cloning TTS systems used to render benchmark queries.
- CosyVoice 3 — The second zero-shot voice-cloning TTS system used to render benchmark queries.
- Qwen3-Omni — Strongest end-to-end system on Tool Accuracy, but behind the Gemma 27B cascade on Exact Match in most complex tiers.
- Kimi-Audio — Evaluated end-to-end speech LM baseline and one of three systems in the noise ablation.
- Step-Audio 2 — Evaluated end-to-end speech LM baseline.
- Audio Flamingo 3 — Evaluated end-to-end speech LM baseline, weakest on implicit-intent queries.
- Whisper — ASR front-end (large-v3) of every cascaded baseline.