VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Models

arXiv · 2026 · Preprint

Heyang Liu et al. (Shanghai Jiao Tong University) · → Paper · Demo: ? · Code: ?

Introduces VocalBench, a ~24k-instance bilingual (English/Mandarin) benchmark spanning 14 capability dimensions across semantic, acoustic, conversational, and robustness quality, and evaluates 27 mainstream speech interaction systems (cascade pipelines, SpeechLLMs, Omni-LLMs, and real-time APIs) to surface shared bottlenecks in current architectures.

Problem

Speech large language models (SpeechLLMs) and Omni-LLMs have rapidly improved vocal human-machine interaction, but systematic evaluation has lagged behind. The paper identifies three specific limitations in existing benchmarks: (1) they primarily evaluate the quality of a system’s text responses, largely neglecting speech generation quality (fluency, paralinguistic fidelity); (2) their capability categorizations are narrow or ambiguous, failing to capture the multifaceted nature of vocal communication; and (3) their instances and methodologies are mostly derived from text-based evaluation, resulting in a scarcity of speech scenarios that actually mimic authentic conversations. Existing benchmarks are also each narrow in scope: some target only paralanguage (SD-Eval) or instruction following (SpeechInstructBench), most are English-only, and none simultaneously covers knowledge, reasoning, creativity, acoustic quality, multi-round dialogue, empathy, instruction following, safety, latency, and robustness in a single, comparable evaluation.

Method

VocalBench comprises approximately 24,395 curated instances across 14 capability dimensions (knowledge, reasoning, creativity, single-round dialogue, multi-round dialogue, instruction following, empathy, safety alignment, code-switching, dialect, and robustness, among others), split into VocalBench-en and VocalBench-zh subsets, and drawing on 29 public resources. Construction combines multiple sources: public text QA pairs adapted into spoken form with verified reference answers; additional LLM-generated cases (for diverse task formats and query types) with mandatory human revision and validation; and manually collected cases for further diversity. Non-English-sourced content is translated and culturally adapted (including named-entity replacement) rather than directly ported.

Speech queries are synthesized with CosyVoice under strict manual inspection, using speaker prompts sourced from Seed-TTS and Common Voice. For capability sets emphasizing clarity (knowledge, reasoning), prompts are filtered for higher UTMOS scores and better human ratings; for conversational sets, non-fluent and lower-quality speaker clips are deliberately included to mimic real interaction diversity. Every synthesized query is transcribed with Whisper-large-v3, and any transcription mismatch triggers native-speaker review (with an online dictionary reference for uncommon words) and resynthesis until the query passes manual examination. For capability-specific construction, additional acoustic post-processing is applied: noise and echo simulation for robustness, and model-based speech emotion recognition (emotion2vec_plus_large, requiring >0.9 classification confidence on source prompts and >0.8 on final synthesized queries) for the empathy subset, which draws speaker prompts from CREMA-D and ESD.

27 systems are evaluated across four paradigms: cascade pipelines (including an upper-bound cascade using GPT-4o and Qwen3-Max), offline SpeechLLMs and Omni-LLMs (grouped into tiny <2B, base 7-9B, and large ~30B parameter tiers), and real-time APIs (Qwen-Omni-Turbo, GPT-Realtime). Evaluation indicators are tailored per task type: objective accuracy plus following/refusal rates for instruction following and safety; LLM-judged 1-5 scores for open-ended creativity and dialogue; an Emotional Empathy Rate (EER) combining semantic and acoustic appropriateness for empathy; a Preserve Rate (PR, performance retained under distraction/accent/multilingual conditions relative to clean conditions) for code-switching, dialect, and robustness; and real-time factor (RTF) plus first-chunk latency (FCL) for responsiveness. An overall weighted score aggregates across dimensions.

Key Results

Across both languages, LLM backbone strength (parameter count, architecture) is the dominant driver of semantic-dimension performance: Qwen3-Omni’s 30B mixture-of-experts backbone substantially outperforms 7-8B dense models on knowledge and reasoning, but this advantage does not extend to acoustic fidelity or emotional empathy. A scaling effect is also visible within a size tier: MiMo-Audio-Instruct, pretrained on over 100 million hours of audio (far more than comparable models), leads other base-sized (7-9B) models in Mandarin conversational tasks. The gap between cascaded and end-to-end (SpeechLLM) approaches is narrowing on semantic tasks, where cascades retain a slight edge, but cascades consistently underperform end-to-end models on emotional empathy, a deficit traced specifically to weaker acoustic emotional expression rather than semantic misunderstanding. On the acoustic dimension, models with stronger semantic/linguistic sophistication tend to show lower UTMOS fluency, and speech-text alignment (WER/PER) degrades more for longer-form responses than short ones. On the conversational dimension, instruction-following gains are driven almost entirely by semantic-content adherence, not paralinguistic control: models struggle broadly to manipulate speech speed or emotional tone even as content-following improves. On robustness, end-to-end models (VocalNet2, Qwen3-Omni) maintain roughly 90% preserve-rate scores under acoustic perturbation in both languages, while cascaded systems degrade more, attributed to transcription-error propagation. A validation study finds an LLM judge (Qwen2.5-max) achieves 85.9% consistency with 5 bilingual human evaluators across 400 paired open-ended-response comparisons, with the LLM’s preference matching the human majority in 94% of cases.

Novelty Assessment

The paper’s central contribution is scope and construction fidelity rather than a new evaluation technique: per its own comparison table, VocalBench is the only benchmark among 11 compared that covers all listed capability columns (knowledge, reasoning, creativity, acoustic quality, single- and multi-round dialogue, empathy, instruction following, safety, latency, and robustness) simultaneously and bilingually, in a fully speech-instruction format rather than multiple-choice or text-derived formats. The construction pipeline’s emphasis on speech-native validation (Whisper-transcription mismatch checks, native-speaker review, resynthesis until passing, emotion-classifier-gated prompt selection for empathy) is a genuine methodological improvement over benchmarks whose instances and validation are inherited from text-only evaluation design. The human-consistency validation of the LLM judge (85.9% agreement) is a reasonable, if modest-scale (400 cases, 5 evaluators), check on the automated-scoring methodology’s reliability. As the authors themselves note in Limitations, the benchmark currently relies exclusively on synthetic (TTS-generated) speech rather than human-recorded speech, which constrains how far its acoustic-dimension findings generalize to genuinely natural human speech variation.

Field Significance

Tip

High, VocalBench’s breadth (14 capabilities, both languages, 27 systems spanning all major current architectural paradigms) and its explicit diagnosis of shared bottlenecks, particularly the disconnect between semantic scaling and acoustic/paralinguistic quality, and the empathy generation gap, make it directly useful as a reference point for both model development and model selection. Its scope and rigor position it to become a standard comparison benchmark for speech interaction systems going forward, similar in role to how VoiceBench and AIR-Bench functioned for earlier, narrower evaluation needs.

Claims

• supports: The strength of a speech interaction model’s underlying LLM backbone chiefly determines semantic-quality performance, but provides little to no corresponding improvement in acoustic fidelity or emotional empathy.

  Evidence: Qwen3-Omni’s 30B mixture-of-experts backbone substantially outperforms 7-8B dense models on knowledge and reasoning, but this advantage does not carry over to acoustic fluency or empathy scores, where its performance is comparable to or below several smaller models. (§4.3.1, Table 2, Table 3)

• complicates: Speech interaction models with stronger semantic and linguistic capability often produce lower acoustic naturalness in their speech responses, indicating a trade-off rather than joint improvement between response sophistication and speech fluency.

  Evidence: Models with strong semantic performance, such as LLaMA-Omni2-Bilingual and MiMo-Audio-Instruct, achieve comparatively low UTMOS fluency scores, attributed to complex structured textual response patterns that are harder to convert into natural, well-paced speech. (§4.3.3, Observation 6)

• supports: End-to-end speech interaction models are inherently more robust to acoustic perturbations than cascaded ASR-LLM-TTS pipelines, because cascaded systems propagate transcription errors from degraded audio while end-to-end models process speech directly.

  Evidence: VocalNet2 and Qwen3-Omni maintain approximately 90% preserve-rate scores under acoustic perturbation across both languages, while cascaded systems show significant degradation under the same conditions, attributed to error propagation from inaccurate transcription. (§4.3.5, Observation 10)

• complicates: Improvements in speech interaction models’ instruction-following ability are driven almost entirely by better adherence to semantic content, not by improved control over paralinguistic attributes such as speaking rate or emotional tone.

  Evidence: On VocalBench-zh’s instruction-following leaderboard, the leading and second-place models differ substantially in accuracy (73.3% vs. 52.4%), but this gap is driven entirely by semantic-content adherence; both models show limited ability to manipulate paralinguistic features like speech speed and emotional tone. (§4.3.4, Observation 8)

• supports: An LLM-based automated judge can substitute for costly human evaluation on open-ended speech-interaction assessment tasks with a high degree of agreement with human raters.

  Evidence: Qwen2.5-max’s automated scoring achieves an 85.9% consistency rate with five bilingual human evaluators across 400 paired-response comparisons, with the LLM’s preferred response matching the human majority preference in 94% of cases and at least one human evaluator’s preference in over 99% of cases. (§4.4)

Limitations and Open Questions

Warning

The current version relies exclusively on synthetic (TTS-generated) speech queries rather than human-recorded speech; the authors state that real human recordings are needed to evaluate nuanced aspects such as subtle pronunciation variation, prosody, and natural dialogue dynamics, which synthetic speech only adequately approximates for fundamental capability dimensions.

• The scope of evaluated capabilities and metrics is stated by the authors as needing further expansion in future versions, including broader multilingual coverage beyond English and Mandarin.
• The human-consistency validation of the LLM judge covers only 400 paired cases from three task types (creativity, single-round dialogue, empathy) rated by 5 evaluators, a modest scale relative to the benchmark’s overall size.
• Nearly all evaluated models exhibit latency exceeding 500ms of model-side streaming latency and show limited acoustic empathy, code-switching, and dialect handling, which the authors frame as open problems for the field rather than limitations specific to this benchmark’s design.

Wiki Connections

• Speech-to-Speech — benchmarks the full space of current spoken dialogue system paradigms (cascade, SpeechLLM, Omni-LLM, real-time API) across conversational, acoustic, and robustness dimensions.
• Evaluation Metrics — introduces a large-scale, 14-capability, bilingual benchmark with dedicated indicators (Emotional Empathy Rate, Preserve Rate under perturbation, real-time factor) spanning semantic, acoustic, conversational, and robustness evaluation.
• Subjective Evaluation — validates its automated LLM-judge scoring methodology against ratings from human evaluators fluent in both English and Mandarin.
• Emotion Synthesis — includes a dedicated empathy evaluation dimension assessing both semantic and acoustic appropriateness of emotionally responsive speech generation.
• Spoken Language Model — evaluates SpeechLLMs and Omni-LLMs that consume external speech input and generate speech output across a comprehensive capability suite.
• VoiceBench — a prior English-only benchmark covering general knowledge, instruction following, safety, and robustness, which VocalBench extends with bilingual coverage, acoustic-quality metrics, and additional capability dimensions.
• WildSpeech-Bench — a prior single-language benchmark covering simple single-round dialogue and paralinguistic perception, compared against in VocalBench’s benchmark landscape table.
• CS3-Bench — the authors’ own prior code-switching evaluation benchmark, whose speech instances VocalBench directly incorporates for its code-switching and dialect subsets.
• CosyVoice — used, together with CosyVoice 2, to synthesize all of VocalBench’s speech queries from source text and reference speaker prompts.
• CosyVoice 2 — used alongside CosyVoice for VocalBench’s speech-query synthesis pipeline.
• Qwen3-Omni — the top-performing evaluated model overall, central to the paper’s finding that strong LLM backbones drive semantic gains without corresponding acoustic or empathy improvements.
• Kimi-Audio — one of the 27 evaluated speech interaction systems, notable for requiring different evaluation hardware (NVIDIA A100) due to memory usage.
• GLM-4-Voice — one of the 27 evaluated end-to-end spoken chatbot systems, discussed in the robustness and acoustic-dimension analyses.