arXiv · 2024 · Preprint
Chen et al. (National University of Singapore) · → Paper · Demo: ? · Code: ✓
VoiceBench introduces the first multi-faceted benchmark for LLM-based voice assistants, revealing that pipeline-based systems outperform end-to-end models by more than 20 points and that mispronunciation and disfluency cause far greater performance degradation than environmental noise.
Problem
Evaluation of LLM-based voice assistants has been dominated by ASR metrics and clean-speech question-answering datasets. This narrow scope misrepresents real deployment conditions: it ignores the variability introduced by speaker characteristics (accent, age, pitch, speaking rate), acoustic environment (noise, reverberation, packet loss), and natural speech content imperfections (disfluencies, mispronunciations, grammatical errors). Systems that perform well under clean synthetic speech conditions may fail substantially in real-world interactions, yet no benchmark existed to expose these gaps.
Method
VoiceBench constructs spoken instruction data covering three evaluation dimensions: general capability, instruction following, and safety. For general capability, text questions from AlpacaEval, CommonEval, SD-QA, OpenBookQA, and MMSU are converted to speech via Google TTS (for synthetic data) and supplemented with real recorded speech from CommonEval and SD-QA. Instruction-following capability uses the IFEval dataset filtered to speech-appropriate prompts under 50 words. Safety is assessed with AdvBench harmful prompts, measuring the refusal rate.
To probe real-world robustness, the benchmark then systematically perturbs the base speech along three axes. Speaker variations are simulated by controlling speaking speed, pitch, volume, and age via Google TTS and CosyVoice, while accent robustness is probed using the SD-QA multi-accent corpus (11 accents) and synthetic accents via MeloTTS. Environmental variations include far-field low-pass filtering, amplitude clipping (distortion), reverberation at multiple RT60 values, packet loss simulation, and white or babble noise injection. Content variations are generated by using GPT-4o to rewrite clean instructions into versions containing repairs, repetitions, filled pauses, interjections, false starts, mispronunciations, or grammatical errors, which are then re-synthesised via TTS.
Nine voice assistants are evaluated: Qwen2-Audio, LLaMA-Omni, Mini-Omni, Mini-Omni2, VITA, Moshi, DiVA, GPT-4o-Audio, and two naive cascade pipelines (Whisper + LLaMA-3.1-8B and Whisper + GPT-4o). Text-form and speech-form instructions are tested separately to isolate the degradation introduced by the speech modality. Response scoring uses GPT-4o-mini for open-ended tasks, rule-based extraction for multiple-choice, and refusal-phrase matching for safety.
Key Results
The performance gap between cascade pipelines and end-to-end models is the headline finding. The naive Whisper + LLaMA-3.1-8B pipeline achieves 79.1 on the composite VoiceBench score (speech form), while the best open-source end-to-end model, DiVA, scores 55.7 (Table 3). The Naive-4o pipeline (GPT-4o backend) reaches 87.2, and GPT-4o-Audio achieves 86.4 as a proprietary end-to-end system, closing most but not all of the gap. The smallest open-source models, Mini-Omni and Mini-Omni2, score 27.9 and 31.3 respectively on speech-form instructions.
The text-to-speech performance gap is pronounced for end-to-end models. VITA drops more than 35 points from text-form to speech-form instructions, whereas the naive pipeline drops only 4.4 points. For multiple-choice tasks, most end-to-end models except Qwen2-Audio perform near random guessing on speech-form inputs.
Mispronunciation causes the largest content degradation, with an average score reduction of 20.3% across models; repair disfluencies cause a 12.6% reduction (Table 5). Grammatical errors are largely tolerated. Real accent data (SD-QA) is substantially harder than synthetic accent data, particularly for low-resource accents such as Philippine and Indian English. VITA shows the greatest accent sensitivity, hypothesised to stem from its non-Whisper CNN+Transformer speech encoder.
Safety robustness degrades considerably when instructions are delivered via speech: LLaMA-Omni’s refusal rate drops from 98.5% (text) to 11.4% (speech), and Mini-Omni’s from 86.4% to 37.1%, raising practical safety concerns for deployed voice assistants.
The model ranking is consistent across synthetic TTS voices from different systems (Table 7), supporting the validity of using Google TTS synthetic data as an evaluation proxy.
Novelty Assessment
VoiceBench’s contribution is methodological and dataset-level. The individual perturbation techniques (adding noise, reverberation, disfluencies via GPT rewriting) are not novel, and the base QA datasets it draws from are all pre-existing. The novelty lies in systematically assembling these axes into a unified evaluation framework targeted at end-to-end voice assistants specifically, rather than ASR systems or audio analysis models.
The benchmark’s design principle, separating text-form from speech-form instruction evaluation, is a particularly useful diagnostic that reveals how much of a model’s capability loss is attributable to speech processing versus LLM reasoning degradation from additional training. The finding that LLaMA-Omni’s additional training reduces its text-instruction performance by over 11 points compared to the same base LLM is a valuable diagnostic result that prior ASR-focused evaluations would not surface. The safety dimension in voice interaction is underexplored elsewhere and the finding of substantial safety regression in speech mode is a notable empirical contribution.
Field Significance
Tip
High: VoiceBench fills a genuine gap by providing a benchmark specifically targeting voice assistant quality across real-world speech variability, and its finding that end-to-end models lag pipeline systems by more than 20 points on spoken instructions gives the field a concrete, reproducible diagnostic for tracking progress on this gap. The result that end-to-end training can degrade safety alignment in the speech modality is a cautionary finding that had not been clearly quantified before.
Claims
- Pipeline-based voice assistants (ASR plus LLM) substantially outperform end-to-end audio LLMs on spoken instruction understanding, with the gap persisting even for state-of-the-art proprietary end-to-end systems. (§3.3, Table 3)
- End-to-end audio LLM training can degrade the underlying LLM’s text-instruction capabilities, including safety alignment, compared to using the same base model in a pipeline configuration. (§3.3)
- Mispronunciation causes significantly greater performance degradation in voice assistants than disfluencies or grammatical errors, because it increases effective word error rate rather than merely adding irrelevant context. (§6.2, Table 5)
- Evaluations using clean synthetic speech substantially overestimate voice assistant robustness relative to real recorded speech, particularly for models with non-standard speech encoders. (§3.3, Table 4)
- The choice of speech encoder determines a voice assistant’s robustness to accent and speaker variation more than the choice of base LLM. (§4.2)
Limitations and Open Questions
Warning
VoiceBench evaluates only the text content of responses, not speech output quality. Moshi, the only tested model that natively outputs speech, is therefore evaluated under the same protocol as text-output models, which means the benchmark does not measure naturalness, prosody, or latency in the response speech stream.
The benchmark covers evaluation in English only, and generalisation claims about accent robustness are limited by the diversity of the accent TTS systems used. Content perturbations are all generated synthetically via GPT-4o rewriting, which may not fully capture the distribution of natural human disfluencies.
The paper notes that future work should develop evaluation protocols for speech-based responses. Real-time interaction quality, turn-taking behaviour, and multi-turn coherence are not covered.
Wiki Connections
Concept pages this paper most directly informs: spoken-language-model, speech-to-speech, evaluation-metrics, subjective-evaluation.
In-corpus papers evaluated: 2410.00037 (Moshi), 2409.06666 (LLaMA-Omni), 2407.10759 (Qwen2-Audio), 2408.16725 (Mini-Omni), 2410.11190 (Mini-Omni2), 2408.05211 (VITA), 2310.13289 (SALMONN), 2407.05407 (CosyVoice).