EMNLP · 2025 · Conference
Ramaneswaran Selvakumar et al. (University of Maryland, College Park) · → Paper · Demo: ✗ · Code: ✓
Introduces the first benchmark that requires omni-modal voice assistants to jointly ground paralinguistic speech cues (emotion, tone, background sound, speaker profile) and visual context to produce contextually appropriate text responses, using 1000 professionally recorded, human-spoken questions paired with images or videos.
Problem
Existing omni-modal language model (OLM) benchmarks fall into two disjoint camps. Vision-and-language benchmarks such as OmniBench incorporate images and video but lack spoken dialogue, so they cannot assess a model’s auditory reasoning. Voice-assistant benchmarks such as VoxDialogue, SD-Eval, and S2S-Arena evaluate whether a model can respond appropriately to auditory cues like content, emotion, or speaker demographics carried in a speech instruction, but they do not require integrating that speech information with accompanying visual context. Several of these voice benchmarks also rely on synthetic TTS speech, which the authors argue struggles to render the fine-grained emotional and prosodic variation needed to probe paralinguistic understanding. As a result, no existing benchmark tests whether an omni-modal voice assistant (OVA) can jointly ground non-verbal speech attributes (tone, emotion, background acoustic scene, speaker age/gender) together with visual cues (objects, scene text, video content) to produce a single contextually appropriate response, and existing benchmarks are also vulnerable to shortcut exploitation, where a model can answer correctly using only textual or visual priors without ever attending to the speech signal.
Method
MULTIVOX is a benchmark, not a proposed model. It consists of 1000 human-spoken questions, each paired with either an image (206 unique images) or a video (287 unique videos), and in some cases an overlaid background sound. Construction follows a three-level taxonomy of speech-related skills spanning three domains: acoustic scene understanding (background music, sound events, ambient environment), paralanguage understanding (emotion, voice modulation, pronunciation, volume, pace, stuttering, breathiness), and speaker profiling (biological gender, age group). Eight expert annotators (graduate students in speech and audio-visual research) authored specification cards per sub-category, drafted realistic scenarios plus a “confounder” counterpart that keeps text and visual content identical but flips the targeted speech property so that the expected answer inverts (e.g., a query about room loudness spoken softly versus in a normal tone) (§3.1, §3.2, Figure 2). Spoken queries were recorded by four professional voice actors under delivery guidelines matched to the target attribute, with TTS used only for children’s voices for ethical reasons; two independent annotators verified that each recording conveyed its intended attribute before acceptance (§3.2).
Evaluation is modular: each sample carries a speech hook and/or visual hook plus a reference answer and rationale authored by the annotator. Speech Grounding and Visual Grounding scores test whether the model’s response correctly reflects the hooked modality-specific information, while Contextual Appropriateness (CA), the benchmark’s headline metric, measures how well the response aligns with the intent, context, and modality of the input on a 1-5 scale (§3.3). All three scores are produced by an LLM judge (GPT-4 for CA scoring; GPT-4.1-mini for the validation study), supplied with the reference answer, rationale, and task-level modality metadata, and models are explicitly penalized for producing correct-seeming answers via a shortcut modality rather than the required one (§3.3, §D). The paper validates this LLM-as-judge protocol against three domain-expert human raters on 300 samples, finding human-LLM agreement (Cohen’s κ = 0.78) close to human-human agreement (κ = 0.82) (§D). The ten evaluated OLMs are prompted with the audio and visual inputs directly and are not fine-tuned or otherwise modified by the benchmark; MULTIVOX only measures their existing capabilities.
Key Results
Across 10 evaluated OLMs (Mini-Omni2, VITA-1.5, VideoLLaMA2, Baichuan-Omni, MiniCPM, Intern Omni, Phi-4 multimodal, Qwen2.5-Omni, Qwen2.5-Omni with chain-of-thought, and Gemini 2.5 Flash/Pro), humans reach an average Contextual Appropriateness of 4.35, while the best model, Gemini 2.5 Pro, reaches only 3.56, and the strongest open-source model (Qwen2.5-Omni COT) reaches 3.19 (Table 2). Visual grounding is comparatively strong across all models (e.g., Gemini 2.5 Pro: 91.6-92.0% across domains), but speech grounding lags far behind (e.g., Qwen2.5-Omni: 15-39% across domains versus Gemini 2.5 Pro: 54-77%), establishing paralinguistic and acoustic-scene grounding as the primary bottleneck rather than visual understanding (§5.1, Table 2). Manual error analysis of Gemini’s failures attributes 75.3% of errors to speech-perception failures, versus 7.0% to reasoning failures once perception succeeds (Figure 4). A confounder-pair analysis shows that even the top two models (Gemini 2.5 Pro, Qwen2.5-Omni) fail to flip their answer in 57% and 51% of confounder pairs respectively when the speech attribute is inverted, indicating that much of their apparent accuracy on speech-grounded questions is not actually driven by attending to the speech signal (§5.3, Figure 5). A separate voice-quality validation study (Appendix E) compares professional human recordings against CosyVoice2 and ElevenLabs TTS on 100 paralanguage-focused queries using a 10-rater MOS test: human recordings score 4.6/4.5 (attribute match/naturalness) versus 2.4/2.1 for CosyVoice2 and 3.1/3.3 for ElevenLabs, motivating the benchmark’s choice to use human-recorded rather than synthetic speech.
Novelty Assessment
The contribution is a new benchmark and evaluation methodology rather than a new architecture. Its most distinctive design element is the confounder-pair construction, which pairs each question with a variant that holds text and visual content fixed while flipping only the target speech property, giving a direct diagnostic for shortcut exploitation that prior voice-assistant benchmarks (SD-Eval, VoxDialogue, S2S-Arena) lack. The combination of jointly-required visual and paralinguistic-speech grounding, human-recorded (rather than TTS-synthesized) queries, and a validated LLM-judge protocol is also new relative to prior omni-modal benchmarks (Table 1 in the paper positions this directly against OmniBench, SD-Eval, VoxDialogue, S2S-Arena, and Lyra SVQA). The taxonomy itself (acoustic scene, paralanguage, speaker profile) largely consolidates categories already explored individually by prior benchmarks rather than introducing fundamentally new speech-understanding skills.
Field Significance
moderate — MULTIVOX contributes a diagnostic evaluation resource and a confounder-based methodology for testing whether omni-modal voice assistants genuinely ground paralinguistic speech cues rather than exploiting textual or visual shortcuts. It demonstrates a concrete, reproducible speech-grounding bottleneck across ten current OLMs and provides a validated LLM-as-judge protocol (benchmarked against human raters) that other evaluation efforts in this space can reuse or adapt.
Claims
- supports: Professional human speech recordings preserve fine-grained paralinguistic attributes (emotion, tone, prosody) substantially better than current text-to-speech systems, as judged by human listeners.
Evidence: A 10-rater MOS study on 100 paralanguage-focused queries rated professional human recordings at 4.6/4.5 (attribute match/naturalness) versus 2.4/2.1 for CosyVoice2 and 3.1/3.3 for ElevenLabs TTS. (Appendix E, “Voice Quality Assessment”)
- complicates: Omni-modal voice assistants that consume both speech and visual input tend to under-utilize non-verbal speech cues relative to visual or textual cues when producing a response, even when the speech cue is necessary for correctness.
Evidence: Across all ten evaluated models, visual grounding scores (e.g., Gemini 2.5 Pro: 91.6-92.0%) are consistently far higher than speech grounding scores (e.g., Gemini 2.5 Pro: 54.75-77.42%), and manual error analysis attributes 75.3% of Gemini’s failures to speech-perception errors versus 7.0% to downstream reasoning errors. (§5.1.2, §5.2, Table 2, Figure 4)
- complicates: Aggregate accuracy on speech-grounded evaluation tasks can substantially overstate a model’s genuine reliance on the speech signal, because models can arrive at a correct-seeming answer through modality shortcuts rather than by attending to the intended cue.
Evidence: Confounder-pair analysis shows Gemini 2.5 Pro and Qwen2.5-Omni fail to change their answer in 57% and 51% of pairs respectively when the underlying speech property is flipped while text and visual content are held constant, despite both models showing roughly 50% raw accuracy on these items. (§5.3, Figure 5)
- supports: An LLM-as-judge protocol, when grounded in a reference answer, rationale, and modality-specific metadata, can approximate expert human judgment for scoring the contextual appropriateness of multimodal dialogue responses.
Evidence: On a blind 300-sample validation, human-LLM inter-rater agreement (Cohen’s κ = 0.78) approached human-human agreement (κ = 0.82), with the LLM’s rating falling within one point of the expert average in 92% of cases. (Appendix D, “Human vs. LLM Evaluation Experiment”)
Limitations and Open Questions
The benchmark explicitly evaluates only the textual content of model responses, not the quality, naturalness, or prosody of any generated speech output; the authors state this is left as an "orthogonal direction" for future benchmarks.
Additional limitations stated by the authors: MULTIVOX is restricted to English-language questions, leaving multilingual generalization of omni-modal voice assistants untested (§Limitations). The benchmark also depends on an LLM-as-judge for large-scale scoring; although validated against human raters on a 300-sample subset with strong but imperfect agreement (κ = 0.78 vs. 0.82 human-human), residual judge noise or bias on the full 1000-sample set is not separately quantified.
Wiki Connections
- Spoken Language Model — evaluates ten omni-modal spoken language models (Mini-Omni2, Qwen2.5-Omni, Baichuan-Omni, and others) on their ability to jointly ground speech and visual cues, surfacing a systematic speech-grounding bottleneck across the class.
- Evaluation Metrics — introduces a validated LLM-as-judge Contextual Appropriateness metric plus confounder-pair analysis as a diagnostic against modality-shortcut exploitation in multimodal dialogue evaluation.
- Subjective Evaluation — runs a 10-rater MOS listening test comparing professional human recordings against TTS systems, and validates its LLM judge against three domain-expert human raters on 300 samples.
- Mini-Omni2 — evaluated as one of the open-source OLMs on MULTIVOX; shows one of the weakest speech-grounding scores among tested systems.
- VITA-1.5 — evaluated as one of the open-source OLMs on MULTIVOX.
- Baichuan-Omni-1.5 — evaluated as one of the open-source OLMs on MULTIVOX.
- Qwen2.5-Omni — evaluated (including a chain-of-thought variant) as the strongest open-source OLM on MULTIVOX, though it still trails proprietary Gemini models.
- Phi-4-Mini — evaluated as one of the open-source OLMs on MULTIVOX.
- VoiceBench — cited and compared against in Table 1 as a prior voice-assistant benchmark that converts text benchmarks to speech but omits non-verbal paralinguistic information, motivating MULTIVOX’s design.
- MMAU — cited and compared against in Table 1 as a prior foundational audio-understanding benchmark lacking chat-style, jointly multimodal questions.
- CosyVoice 2 — used as one of two TTS systems in the voice-quality validation study, scoring lower than professional human recordings on attribute match and naturalness.