arXiv · 2026 · Preprint
Li et al. (Xiamen University, Zhejiang University, CUHK-Shenzhen) · → Paper · Demo: ? · Code: ✓
WavBench introduces a 17,577-item, 76.5-hour benchmark that jointly stress-tests end-to-end spoken dialogue models on complex reasoning delivered conversationally, everyday colloquial “listenability,” and comprehensive paralinguistic understanding and generation.
Problem
Prior spoken dialogue benchmarks each cover only a fragment of what a genuinely conversational, audio-native agent needs to demonstrate. SUPERB and AIR-Bench assess general audio comprehension largely through text-generation-style content accuracy, without checking whether responses actually sound spoken. SD-Eval and VStyle probe paralinguistic perception (emotion, accent, age) but restrict themselves to input comprehension using utterances not drawn from real dialogue scenarios. VoxDialogue broadens paralinguistic attributes and aligns data to dialogue scenarios but stops short of evaluating end-to-end spoken dialogue models. MMSU integrates linguistic theory into 47 tasks yet remains confined to the perception phase, leaving generative fidelity unassessed. On the reasoning side, BigBenchAudio and MultiChallenge adapt text-based reasoning suites to audio but treat speech merely as a transmission medium, never checking whether a model can articulate complex logic in a colloquial, listener-friendly way. No existing benchmark simultaneously stresses high-difficulty reasoning under spoken delivery constraints, defines an explicit standard for spoken (as opposed to written) colloquialism, and evaluates paralinguistic understanding and generation across both explicit-instruction and implicit-inference settings for genuinely end-to-end models.
Method
WavBench organizes 17,577 items (76.5 hours of audio) into two evaluation dimensions built through two separate construction pipelines.
The Colloquial Expression Set targets “spoken friendliness” across seven cognitive domains (Code, Creative Writing, Instruction Following, Logic, Math, Common QA, Safety) and is split into a Pro subset (3,176 items, high-difficulty reasoning drawn mainly from BBEH, MMLU, and Arena-Hard) and a Basic subset (4,486 items, routine interactions drawn mainly from OpenBookQA, WildSpeech, AlpacaEval, AlignBench, and MMLU). Construction proceeds in five stages: (1) aggregating 15 open-source text datasets and stratifying them into Basic/Pro tiers, using GPT-4.1 for fine-grained complexity stratification on Math and Safety and Qwen3-Max to filter content unsuitable for verbalization; (2) colloquial adaptation and rewriting, where Qwen3-Max converts static text queries into scenario-grounded spoken inquiries and rewrites answers into full conversational replies, with domain-specific strategies for mathematics (LaTeX to spoken description), code (syntax to explained logic), and structured multiple-choice content (linearized into natural sentences); (3) human verification by five expert annotators screening 11,000 samples for semantic and acoustic-suitability errors; (4) high-fidelity audio synthesis using IndexTTS2 in zero-shot cloning mode, with 1,088 Seed-TTS-Eval English samples as speaker prompts; and (5) automated audio-quality filtering using Whisper-Large-V3 transcription, discarding any sample with WER above 5%.
The Acoustic Interaction Set (9,915 items) covers 10 paralinguistic dimensions spanning speaker information (age, gender, accent, language), acoustic characteristics (pitch, speed, volume, emotion), and background sounds (audio, music), split into explicit-instruction (understanding and generation under directive cues, e.g. “please adopt a childlike voice”) and implicit-dialogue settings (no lexical cues, including multi-turn conditions with roughly a 1:3 single-turn-to-multi-turn ratio). Its pipeline uses Qwen3-Max and Qwen-Plus to synthesize dialogue scripts paired with target paralinguistic labels, then produces audio through attribute-specific conditioning: IndexTTS2 handles pitch, speed, volume, gender, and language control via zero-shot synthesis conditions and speaker prompts; GPT-4o-mini-TTS handles accent and emotion via stylistic instruction prompting; and AudioCaps/MusicCaps segments are concatenated for background audio and music conditions. Quality is filtered automatically (Whisper-Large-V3 for WER, Emotion2Vec for emotion-label confidence below 0.5) and then verified by ten expert human annotators.
For evaluation, the Colloquial Expression Set is scored by Gemini 3 Pro Preview using a hierarchical rubric: a score of 1 for task failure (incorrect answer, instruction violation, or unsafe response), 3 for factually correct but stiff/mechanical delivery, and 5 for correct delivery that satisfies lexical appropriateness, linguistic naturalness, interactive rapport, and emotional-contextual matching. The Acoustic Interaction Set is scored differently per subtask: explicit understanding is scored by direct accuracy against ground-truth labels; explicit generation is scored by having Gemini 3 Pro Preview annotate the paralinguistic attributes of synthesized responses and comparing to ground truth; implicit dialogue is scored on a 0-10 scale by Gemini 3 Pro Preview separately for paralinguistic style and semantic transcription accuracy.
Key Results
Five end-to-end spoken dialogue models were evaluated: Qwen3-Omni, Kimi-Audio, MiMo-Audio, Step-Audio-2-mini, and GPT-4o Audio. On the Pro subset, GPT-4o Audio leads with an average score of 58.23, but this still falls well short of ceiling performance, and open-source models trail substantially (Qwen3-Omni 39.53, MiMo-Audio 32.02, Kimi-Audio 30.79, Step-Audio-2-mini 30.40); the gap is sharpest in Logic and Math, where Step-Audio-2-mini drops to 22.40 on Math and Kimi-Audio to 26.03 on Logic, and even GPT-4o Audio falls to 42.60 on Logic. On the Basic subset, GPT-4o Audio again leads (68.80 average, 81.00 on Safety, 75.60 on QA), with Qwen3-Omni second (55.80) and the remaining open-source models showing sharp category-specific dips (Step-Audio-2-mini 30.20 on Math, MiMo-Audio 33.56 on Instruction).
In explicit acoustic understanding, all five models achieve relatively high accuracy on Language (83.5-97.0%) and Emotion (75.4-92.9%) but perform far worse on Pitch (23.1-34.2%) and Volume (31.1-50.7%); Step-Audio-2-mini leads overall (57.36% average). In explicit generation, GPT-4o Audio dominates across nearly every attribute (79.23% average), with all models scoring under 50% on Background Audio generation. In implicit multi-turn dialogue, text-content scores rise across all models (e.g., Qwen3-Omni reaches 4.88/10) while paralinguistic-style audio scores fall sharply to the 1.04-1.25 range, versus 2.47-3.50 in single-turn settings.
Novelty Assessment
The primary contribution is a large-scale (17,577-item), purpose-built dataset plus a specific evaluation methodology rather than a new model or architecture. The novelty lies in the framing: the Pro subset operationalizes a distinct “reasoning under spoken-delivery constraints” stress test that prior reasoning-audio benchmarks (BigBenchAudio, MultiChallenge) did not target, and the Basic subset’s colloquialism rubric (lexical appropriateness, linguistic naturalness, interactive rapport, emotional-contextual matching) gives a concrete, reproducible operationalization of “spoken listenability” that earlier benchmarks left largely implicit. The Acoustic Interaction Set’s explicit/implicit split and multi-turn extension are incremental relative to prior paralinguistic benchmarks (SD-Eval, VStyle, VoxDialogue) but combine a broader attribute set (10 dimensions) with genuine generation evaluation, which several predecessors (SD-Eval, MMSU, VoxDialogue) do not attempt. The data-construction pipeline itself is an engineering assembly of existing tools (Qwen3-Max, GPT-4.1, IndexTTS2, GPT-4o-mini-TTS, Whisper-Large-V3, Emotion2Vec) rather than a novel technique.
Field Significance
Tip
High — WavBench provides a substantially more comprehensive and larger-scale test of end-to-end spoken dialogue models than prior benchmarks, uniquely combining a reasoning-under-cognitive-load stress test with an explicit colloquialism standard and a 10-dimension paralinguistic evaluation spanning both explicit and implicit settings. Its cross-model results surface a specific, reproducible failure pattern (multi-turn paralinguistic-style degradation despite improving semantic coherence) that gives the field a concrete target for future work on maintaining acoustic consistency across dialogue turns.
Claims
- supports: Benchmarks that combine high-cognitive-load reasoning tasks with spoken-delivery constraints reveal a distinct performance gap beyond what text-only reasoning evaluation captures, even for the strongest end-to-end spoken dialogue models.
Evidence: On the Pro subset, GPT-4o Audio leads with only a 58.23 average and drops to 42.60 on Logic; open-source models score 30.40-39.53 on average, with Step-Audio-2 falling to 22.40 on Math and Kimi-Audio to 26.03 on Logic. (§4.3.1, Table 2, Panel A)
- complicates: Evaluating spoken dialogue naturalness and paralinguistic fidelity through an automated LLM-judge pipeline, without a reported cross-validation study against human ratings, leaves open how closely the resulting scores track genuine human perception of colloquial naturalness and acoustic quality.
Evidence: Both the Colloquial Expression scoring (1/3/5 hierarchical rubric) and the Acoustic Interaction scoring (0-10 scale) are performed entirely by Gemini 3 Pro Preview; no human-rater correlation or validation study is reported for either scoring scheme. (§4.2)
- supports: Maintaining consistent paralinguistic style (acoustic delivery) across multi-turn spoken dialogue is substantially harder for current end-to-end models than maintaining semantic content coherence.
Evidence: In implicit dialogue, multi-turn text-content scores improved for all five models (e.g., Qwen3-Omni reaching 4.88/10) while multi-turn audio-style scores fell to 1.04-1.25/10, compared to 2.47-3.50/10 in single-turn settings. (§4.3.5, Table 2, Panel E)
- supports: Fine-grained prosodic attribute perception (pitch, volume) remains substantially weaker than coarse categorical attribute perception (language identification, emotion) in current end-to-end spoken dialogue models.
Evidence: Across all five evaluated models, explicit-understanding accuracy on Language (83.5-97.0%) and Emotion (75.4-92.9%) far exceeds accuracy on Pitch (23.1-34.2%) and Volume (31.1-50.7%). (§4.3.3, Table 2, Panel C)
Limitations and Open Questions
The paper's own scoring methodology relies entirely on an LLM judge (Gemini 3 Pro Preview) for both the Colloquial Expression and Acoustic Interaction sets, with no reported correlation study against human ratings to validate that the automated scores track genuine human perception of naturalness, colloquialism, or paralinguistic accuracy.
Human involvement in the pipeline is limited to filtering and quality-verifying the underlying data (five annotators screening 11,000 colloquial-expression samples; ten annotators verifying acoustic-interaction audio), not to scoring or validating model outputs during evaluation, so the benchmark’s reliability as a proxy for human judgment of the five evaluated systems is not directly established. The Colloquial Expression Set’s audio is synthesized entirely through IndexTTS2 zero-shot cloning, and the Acoustic Interaction Set relies on IndexTTS2 and GPT-4o-mini-TTS for attribute-specific conditioning; any systematic biases or limitations of these underlying TTS systems (e.g., narrower prosodic range than natural speech) could propagate into the benchmark’s stimuli. Only five models are evaluated, all closed checkpoints from major labs, limiting how broadly the reported model-comparison conclusions generalize across the wider space of end-to-end spoken dialogue systems.
Wiki Connections
- Spoken Language Model — WavBench is designed specifically to evaluate end-to-end spoken dialogue models that consume external spoken input through an LLM-based backbone, providing a comprehensive test bed for this model class.
- Speech-to-Speech — The benchmark targets the dialogue sub-paradigm of speech-to-speech interaction, requiring models to produce spoken responses directly from spoken queries without intermediate text.
- Evaluation Metrics — WavBench contributes a specific evaluation methodology, a hierarchical colloquialism scoring rubric and an LLM-judge-based paralinguistic scoring scheme, alongside a large new benchmark dataset.
- IndexTTS2 — Used as the primary zero-shot TTS backend to synthesize nearly all of WavBench’s audio stimuli, including pitch/speed/volume/gender/language-conditioned generation.
- Qwen3-Omni — One of five end-to-end spoken dialogue models evaluated on WavBench; the strongest open-source performer on several subsets.
- Kimi-Audio — One of five end-to-end spoken dialogue models evaluated on WavBench, showing weaker performance on Logic and Accent tasks.
- MiMo-Audio — One of five end-to-end spoken dialogue models evaluated on WavBench, notably weak on Audio Event Detection.
- Step-Audio-2 — One of five end-to-end spoken dialogue models evaluated on WavBench; the strongest model on explicit acoustic understanding.
- VStyle — Directly contrasted as a prior paralinguistic benchmark; WavBench argues VStyle does not evaluate colloquial listenability or paralinguistic stability under complex reasoning.
- OpenOmni — Cited as a representative end-to-end spoken dialogue model in the related-work discussion of the field WavBench evaluates.
- Fun-Audio-Chat — Cited alongside other end-to-end spoken dialogue models as part of the model class WavBench is designed to benchmark.