arXiv · 2025 · Preprint
Zhang et al. (WeChat AI, Tencent) · → Paper · Demo: ? · Code: ✓
WildSpeech-Bench introduces the first benchmark to evaluate end-to-end speech LLMs directly on their generated speech, using real conversational queries, acoustic noise augmentation, and query-aware checklist scoring rather than text-proxy evaluation.
Problem
Existing benchmarks for end-to-end speech-to-speech (S2S) LLMs are largely adapted from speech-to-text (S2T) evaluation: they score the transcribed text of a model’s spoken response rather than the speech itself, even though prior work has shown a substantial gap between a model’s text and speech outputs. Their queries are typically drawn either from text-only instruction datasets (which include queries unrealistic for spoken interaction, such as long or adversarial prompts) or from existing speech corpora like LibriSpeech, which contain declarative read-aloud statements rather than authentic user intents. These benchmarks also synthesize audio with clean, high-quality TTS, giving no signal on robustness to background noise, speaker diversity, or paralinguistic phenomena such as prosodic ambiguity, stuttering, or near-homophones that are unique to spoken communication.
Method
WildSpeech-Bench curates 1,100 single-turn queries from real user-ChatGPT interactions in the WildChat corpus, filtered to English, non-toxic, short (under 50 words) queries, then de-duplicated per user. From an initial 34,736 candidates, two LLMs (LLaMA3.3 and Qwen2.5-72B) jointly score commonality and difficulty; a filtered subset of 2,000 is manually reviewed down to 1,412 queries, then rebalanced across four intent categories (Information Inquiry, Solution Request, Opinion Exchange, Text Creation) to 1,000 queries. A further 100 Paralinguistic-Featured (PF) queries are constructed to specifically stress pause, stress, tone, stutter, and near-homophone phenomena, generated with task-specific GPT-4o prompts or sourced from English corpora, with human review throughout.
For speech synthesis, queries are voice-cloned with CosyVoice2 across 16 speaker profiles spanning two genders, four age groups, and two timbres each, to test model robustness to speaker demographics. A subset of the PF queries is instead recorded directly by two trained native-English speakers, since current TTS cannot reliably render the target prosodic effects (e.g., a pause shifting who a sentence’s audience is). All synthesized audio is further augmented with two categories of noise at a controlled, audibility-checked ratio: human background speech mixed in from LibriTTS, and non-linguistic environmental sounds from ESC-50 (animals, weather, domestic/urban noise).
Evaluation transcribes the model’s generated speech response with Whisper-large-v3, repeating ASR three times and averaging the resulting scores to reduce transcription-noise variance. The transcribed text is then scored 1-10 by GPT-4o-mini. Rather than a single generic rubric, WildSpeech-Bench uses query-aware evaluation: each open-ended query is paired with a hand-authored checklist of the specific facts, structural elements, or critical nuances a correct response must capture, and evaluation prompts are explicitly designed to counteract the tendency of LLM judges to favor longer responses. Acoustic quality is scored separately with UTMOS. The naive cascaded baseline pipeline combines Whisper-large-v3 (ASR), Qwen2.5 7B (LLM), and MeloTTS (TTS).
Key Results
Across five tested end-to-end speech LLMs plus the cascaded baseline, GPT-4o-Audio scores highest overall (6.29 average) and leads on every category, including the paralinguistic-featured subset (6.01 PF average). Among open-source models, Qwen-2.5-omni attains the best overall score (6.01) but its PF average (4.46) trails GLM-4-Voice (4.51) and MiniCPM (4.78), indicating that its “thinker-talker” joint reasoning-and-speech architecture does not translate into stronger paralinguistic understanding. Kimi-Audio scores lowest overall (4.54) despite strong reported results on S2T benchmarks; a targeted follow-up (Appendix G) shows its text-quality score on VoiceBench’s CommonEval subset drops from 3.97 in text-only mode to 3.00 in dual-modal (speech+text) mode, indicating the joint audio-text decoding itself degrades output quality rather than speech synthesis alone.
The naive cascaded pipeline reaches an overall score (5.24) comparable to GLM-4-Voice and MiniCPM, consistent with prior findings that pipeline systems are competitive with early end-to-end models, but it scores markedly worse specifically on the stress (2.95) and tone (4.12) paralinguistic subcategories, where meaning carried by prosody is lost during ASR transcription. Noise-robustness experiments show that both environmental and human background-speech noise cause significant performance drops across every tested model, with human background speech causing the larger degradation; even GPT-4o-Audio is affected. WildSpeech-Bench’s automatic scores correlate more closely with human expert judgments than the VoiceBench baseline, and ablating either the multi-round ASR-and-scoring procedure or the query-aware checklist reduces this correlation, indicating both components contribute independently to evaluation reliability.
Novelty Assessment
The contribution is a benchmark and evaluation methodology, not a new architecture. What is genuinely new: sourcing queries from authentic conversational logs (WildChat) rather than adapting text-only instruction sets or reading from existing speech corpora; systematically injecting speaker-demographic and acoustic-noise diversity with an audibility-checked mixing procedure; supplementing synthetic speech with human-recorded paralinguistic-hard cases that current TTS cannot render; and replacing generic LLM-judge rubrics with per-query, hand-authored evaluation checklists. What is not new: the underlying tools (CosyVoice2 for voice cloning, Whisper for ASR, GPT-4o-mini as judge) are all off-the-shelf, and the “curate from real logs, keep the set small and high-quality” design philosophy directly follows precedents set by AlpacaEval, MT-Bench, and WildBench in the text-LLM evaluation literature, applied here to the speech domain for the first time. The dataset scale (1,100 queries) is a deliberate design choice the authors justify by analogy to these smaller, precision-focused benchmarks rather than a limitation they attempt to hide.
Field Significance
High
WildSpeech-Bench shows that speech-to-text dialogue performance does not reliably predict speech-to-speech quality and exposes a universal noise-robustness gap across leading proprietary and open speech LLMs, while its query-aware checklist evaluation offers a concrete alternative to generic LLM-judge scoring for generative audio assessment.
High - the paper’s cross-model comparison surfaces a specific, previously undemonstrated failure mode (dual-modal decoding degradation) in a widely-used model and a robustness gap that affects even the strongest proprietary system, both of which are directly actionable findings for future speech LLM development. Its contribution is evaluation methodology and dataset construction rather than a new model.
Claims
- supports: Strong performance on speech-to-text dialogue benchmarks does not reliably predict a model’s speech-to-speech generation quality.
Evidence: Kimi-Audio’s text-response quality on VoiceBench’s CommonEval subset drops from 3.97 (text-only mode) to 3.00 (dual-modal speech+text mode), and its overall WildSpeech-Bench score (4.54) trails several models it is known to outperform on S2T-only benchmarks. (§3.2, Appendix G)
- complicates: End-to-end speech LLMs remain broadly vulnerable to background acoustic interference despite strong performance in clean conditions.
Evidence: Adding either environmental noise (ESC-50) or human background speech (LibriTTS) causes a significant performance decline across every evaluated model, including the top-performing GPT-4o-Audio, with human background speech producing the larger drop. (§3.4, Figure 5)
- refines: Cascaded ASR-LLM-TTS pipelines can match dedicated end-to-end speech LLMs on general conversational quality, but the gap re-emerges specifically on tasks requiring paralinguistic understanding.
Evidence: The naive Whisper + Qwen2.5-7B + MeloTTS pipeline scores comparably to GLM-4-Voice and MiniCPM overall (5.24 vs. 5.03/5.08) but scores markedly worse on the stress (2.95) and tone (4.12) paralinguistic subcategories, consistent with prosodic information being lost during ASR transcription. (§3.2, Table 2)
- supports: Query-aware, checklist-based evaluation improves an LLM judge’s alignment with human judgment over generic scoring rubrics for speech-to-speech systems.
Evidence: An ablation removing either the query-aware checklist or the multi-round ASR-plus-scoring procedure reduces the Pearson correlation between automatic and human expert scores relative to the full evaluation method. (§3.3, Figure 4)
Limitations and Open Questions
Warning
Automatic scoring depends on a proprietary judge (GPT-4o-mini) and a proprietary ASR model (Whisper-large-v3) in the evaluation loop; while multi-round ASR and query-aware checklists mitigate transcription and judge noise, results remain tied to the behavior of these external models and are not independently reproducible if either changes or is deprecated.
The benchmark is deliberately restricted to English, non-toxic, single-turn queries by design, deferring multi-turn evaluation to future work on the grounds that a reliable multi-turn benchmark requires a solid single-turn foundation first (§Appendix J). The authors also flag, without resolving, a “teaching to the test” risk: as any standardized benchmark gains adoption, developers may optimize systems to match its checklist criteria rather than improve general conversational ability, potentially decoupling benchmark score from real-world performance (§Appendix I).
Wiki Connections
- Spoken Language Model — evaluates five end-to-end speech LLMs that consume real synthesized and human-recorded audio queries and produce audio responses, directly testing this class of systems rather than a text-only proxy.
- Speech-to-Speech — argues that S2T-adapted evaluation frameworks miss critical S2S-specific failure modes and builds the first benchmark scoring generated speech output directly rather than its transcript.
- Evaluation Metrics — introduces query-aware, hand-authored evaluation checklists as an alternative to generic LLM-judge rubrics, explicitly correcting for length bias in automatic scoring.
- Subjective Evaluation — validates its automatic scoring pipeline against human expert ratings collected under a two-stage annotation-plus-QC protocol, reporting higher correlation with human judgment than a prior baseline.
- VoiceBench — used directly as the comparison baseline for correlation-with-human-judgment analysis and critiqued for relying on text-benchmark-derived queries and text-modality scoring.
- GLM-4-Voice — one of five end-to-end speech LLMs evaluated on the benchmark, notably outperforming Qwen2.5-omni on paralinguistic-featured queries despite a lower overall score.
- Qwen2.5-omni — evaluated as the strongest open-source model overall, but shown to lag on paralinguistic-featured queries relative to smaller open models.
- Kimi-Audio — evaluated model whose dual-modal output mode is shown to substantially degrade text and speech quality relative to its text-only mode, the paper’s central S2T-to-S2S generalization finding.
- GPT-4o System Card — GPT-4o-Audio is evaluated as the top-performing system across every category and subcategory in the benchmark.
- CosyVoice 2 — used as the voice-cloning TTS engine to synthesize the benchmark’s speaker-diverse query audio.
- Baichuan-Audio — its OpenAudio Bench is discussed as related work that improved answer-matching over exact-match scoring but still evaluates only the text modality.
- MiniCPM-V — one of five end-to-end speech LLMs evaluated on the benchmark, scoring competitively on paralinguistic-featured queries relative to larger models.