arXiv · 2025 · Preprint
Yueqian Lin et al. (Duke University / Adobe) · → Paper · Demo: ✗ · Code: ✓
Introduces VERA, a 2,931-episode benchmark that directly compares text and voice modalities on identical multi-step reasoning tasks under real-time latency constraints, revealing a large and consistent “Voice Reasoning Gap” that persists across architectures, thinking-time budgets, and cascade designs.
Problem
Prior voice benchmarks evaluate either acoustic-content understanding (SUPERB, AudioBench, MMAU, MMAR) or conversational mechanics such as turn-taking and dialogue state tracking (SLUE, Spoken SQuAD, full-duplex benchmarks), but none evaluate general-purpose, multi-step reasoning under the specific real-time constraints that voice interaction imposes. As a result, no existing evaluation combines (1) multi-step reasoning difficulty, (2) explicit real-time latency constraints, and (3) a direct, same-task text-versus-voice comparison, leaving the severity of reasoning degradation in production voice assistants unquantified.
Method
VERA adapts five established text reasoning benchmarks (AIME competition math, BrowseComp web-navigation, GPQA Diamond graduate science, MRCR long-context co-reference, SimpleQA factual recall) into a voice-native form via a four-stage pipeline: a voice-suitability filter that screens out visually dependent or memory-heavy items; TTS-aware rewriting that verbalizes numbers and symbols and segments text at prosodic boundaries; structured quality validation, in which a held-out LLM validator scores each episode on TTS readiness, conversational naturalness, and reasoning preservation and retains only episodes above a per-track quality threshold; and speech generation, rendering validated text to 24kHz audio with Higgs-Audio v2, whose built-in variation in timbre, tone, and emotion is used to avoid systematic acoustic bias (confirmed via WeSpeaker speaker-embedding analysis). The pipeline filters roughly 22,000 source items down to 2,931 retained episodes with a mean validator quality score of 9.0, cross-checked by a manual audit of 200 episodes.
Evaluation combines automatic speech-fidelity scoring (WER between ASR transcripts and ground truth, after LLM-based normalization to canonical mathematical notation) with LLM-as-a-judge accuracy grading (GPT-4o, three independent judgments per prediction, majority vote), calibrated against 1,000 human-rated predictions (97.8% agreement) and cross-validated against a second judge model (Gemini-2.5-Flash, 98.7% agreement). A 16-category failure taxonomy, applied via GPT-5 classification, separates transcription artifacts from genuine reasoning or knowledge errors. The paper evaluates 12 voice systems spanning commercial realtime APIs (GPT-realtime, Gemini-2.5-Flash-audio, Nova Sonic), open voice models (Qwen2-Audio, UltraVox, Audio Flamingo 3, Phi-4-multimodal), and end-to-end speech-to-speech architectures (Moshi, Freeze-Omni, Qwen2.5-Omni), against text-only upper bounds (GPT-4o, GPT-5, Gemini-2.5 Pro/Flash) and a custom cascade baseline, LiveAnswer, that pairs a GPT-5 reasoner with a Llama-3.3-70B-Instruct narration synthesizer to test whether decoupling reasoning from speech narration closes the gap.
Key Results
Across 12 voice systems, VERA finds an average accuracy drop of 40.4 percentage points relative to text models from the same family, widening from moderate on factual recall (GPT-5 text 48.3% vs. GPT-realtime voice 27.4%) to near-total collapse on competition math (GPT-5 74.8% vs. GPT-realtime 6.1%). Macro-averaged across all five tracks, the best text models reach 54.0% versus 11.3% for the best voice system, a difference confirmed significant by McNemar’s test (p < 0.001). A latency-accuracy frontier shows a “low-latency plateau”: every voice system with sub-1.5s time-to-first-response scores below 11% accuracy, and no system achieves both fast response and text-level accuracy. Within-modality variance in Math scores (σ² = 3.66) is 171× smaller than between-modality variance (σ² = 625.92), indicating the gap is a property of the voice paradigm rather than of any single system’s implementation.
Diagnostic ablations rule out the simplest explanations. Giving Audio Flamingo 3 a “thinking mode” increases latency by 530% (2.40s → 15.14s) but accuracy falls slightly (1.7% → 1.5%). The LiveAnswer cascade, which uses the same GPT-5 reasoner as the text upper bound, closes much of the gap but still trails it by 15.7 points on Math (59.1% vs. 74.8%) and collapses on the Context track (0.2%) due to narration-induced logical inconsistencies. Speech clarity is not the bottleneck either: WER ranges from 7.9% (Gemini-2.5-Flash-audio) to 19.8% (Freeze-Omni) with no corresponding spread in reasoning accuracy.
Novelty Assessment
The contribution is a dataset and evaluation-methodology paper, not an architectural one: VERA’s value lies in being the first benchmark to combine multi-step reasoning difficulty, explicit latency constraints, and matched text-voice comparison on identical items, filling a gap the paper documents clearly against six prior benchmarks (Table 1). The diagnostic experiments (thinking-time ablation, decoupled cascade, WER-vs-accuracy analysis) are the paper’s strongest original contribution beyond the dataset itself, since they systematically eliminate the most obvious engineering explanations for the gap rather than merely reporting it. The LiveAnswer cascade is a diagnostic instrument assembled from existing off-the-shelf models (GPT-5, Llama-3.3-70B), not a proposed architecture, and the paper does not claim it as a novel system design. LLM-as-a-judge grading is validated with human calibration data, which strengthens the empirical claims, though the study explicitly disclaims causal proof given the heterogeneity of the commercial systems under test.
Field Significance
Tip
High — provides the first systematic, cross-architecture quantification of a large text-voice reasoning gap in contemporary voice-interactive systems, together with diagnostic evidence that the gap is not explained by insufficient thinking time, acoustic quality, or lack of architectural decoupling.
The paper contributes a reusable, reproducible benchmark and a clear empirical finding relevant to any system claiming general-purpose reasoning through a spoken interface: current voice architectures reason substantially worse than their text counterparts, and none of three common mitigation strategies (more thinking time, a cascade that separates reasoning from narration, or higher speech fidelity) closes the gap. It also surfaces architecture-specific failure signatures (fluent-but-incorrect completions from native streaming models versus grounding/consistency errors from cascades) that give the field a concrete diagnostic vocabulary for describing how voice reasoning fails.
Claims
- supports: A substantial reasoning-accuracy gap exists between text and voice modalities within the same model family, and the gap widens with the number of reasoning steps a task requires.
Evidence: GPT-5 text reaches 74.8% accuracy on competition math versus 6.1% for GPT-realtime voice (a 68.7-point gap), while the same model pair differs by only ~21 points on factual recall (48.3% vs. 27.4%); macro-averaged across five tracks, text models reach 54.0% vs. 11.3% for voice. (§5.1, Table 3)
- complicates: Increasing inference-time deliberation (“thinking time”) before a voice system speaks is not, by itself, sufficient to close a voice-native reasoning gap.
Evidence: Enabling Audio Flamingo 3’s thinking mode raises time-to-first-response by 530% (2.40s → 15.14s) but overall accuracy decreases from 1.7% to 1.5%, with Context-track accuracy dropping from 3.8% to 1.8%. (§5.2)
- complicates: Decoupling reasoning from speech narration in a cascade architecture narrows a voice-native reasoning gap but introduces new grounding and logical-consistency failures that prevent full parity with text.
Evidence: A cascade pairing a strong text reasoner with a separate narration model still trails the text-only upper bound by 15.7 points on Math (59.1% vs. 74.8%) and collapses to 0.2% accuracy on the Context track, with elevated rates of UNSUPPORTED_FACT and LOGICAL_CONTRADICTION errors attributed to the narration stage. (§5.2, Figure 5)
- supports: Speech output intelligibility is not the primary bottleneck limiting reasoning performance in voice-interactive systems.
Evidence: Word Error Rate across evaluated voice systems ranges from 7.9% (Gemini-2.5-Flash-audio) to 19.8% (Freeze-Omni) with no corresponding difference in reasoning accuracy; all systems in this WER range perform uniformly poorly on VERA’s reasoning tracks. (§5.2, Table 3)
- refines: Different voice system architectures fail on reasoning tasks in systematically different, architecture-linked ways rather than through a single uniform error pattern.
Evidence: Native streaming models (GPT-realtime, Gemini-2.5-Flash-Audio) under-produce NO_FINAL_ANSWER and OFF_TARGET errors, favoring fluent-but-incorrect completions, while the LiveAnswer cascade over-produces UNSUPPORTED_FACT (+0.27), OFF_TARGET (+0.31), and LOGICAL_CONTRADICTION (+0.22) deviations, and end-to-end models (Moshi, Qwen2.5-Omni) show yet another bimodal pattern of extreme over- or under-completion. (§5.3, Figure 6)
Limitations and Open Questions
Warning
The study is an observational, cross-architecture diagnostic rather than a controlled experiment: because the 12 evaluated voice systems differ in training data, objectives, and proprietary implementation details, the authors explicitly state they cannot isolate the causal impact of modality alone, only document a consistent cross-model pattern.
Additional caveats: accuracy grading relies on an LLM-as-a-judge protocol (validated at 97.8% human agreement, but with track-dependent reliability as low as 84.3% on Science, where answers require more nuanced interpretation); several evaluated systems are closed commercial APIs, limiting exact reproducibility of their internal behavior even though the benchmark and prompts are released; and the LiveAnswer cascade, while informative as a diagnostic, is a single hand-built configuration rather than a systematic sweep over possible decoupled designs.
Wiki Connections
- Spoken Language Model — evaluates 12 contemporary voice-interactive systems (commercial, open, and end-to-end speech LMs) and finds a large, consistent reasoning deficit relative to their text counterparts.
- Speech-to-Speech — includes end-to-end speech-to-speech architectures (Moshi, Freeze-Omni, Qwen2.5-Omni) among the systems benchmarked, showing them among the weakest on complex reasoning tracks.
- Streaming TTS — the paper’s central finding is a latency-accuracy trade-off: real-time, low-latency voice generation plateaus around 10% reasoning accuracy, and closing the gap to text requires sacrificing real-time interaction.
- Evaluation Metrics — introduces the VERA benchmark and a formal Voice Reasoning Gap metric, plus a 16-category failure taxonomy for diagnosing architecture-specific error signatures.
- Subjective Evaluation — validates its LLM-as-a-judge accuracy grading against 1,000 human-rated predictions and a second judge model to establish grading reliability.
- Full-Duplex-Bench — cited as prior work focused on dialogue mechanics (turn-taking, interruption handling) rather than the substantive reasoning VERA targets, motivating VERA’s distinct evaluation gap.