arXiv · 2026 · Preprint
Seunghee Kim et al. (Hanyang University / Seoul National University / KAIST AI / NAVER Cloud) · → Paper · Demo: ? · Code: ?
Introduces a benchmark testing whether omni-modal models can generate speech that is not only content-correct but acoustically appropriate given multimodal context, and shows that current systems fail this test even when they excel on text-output evaluations.
Problem
Omni-modal models have progressed from understanding text, vision, and audio to generating speech responses directly, but nearly all existing omni-modal benchmarks still judge output quality through text: they ask whether a model interprets multimodal input correctly and produces a semantically right textual answer. This leaves open whether models can deliver a spoken answer with acoustic realization appropriate to the situation, such as whispering in a library scene or speaking faster in an emergency. Prior speech-generation benchmarks (URO-Bench, VocalBench, S2S-Arena, ParaS2SBench) evaluate broad speech-interaction or instruction-following behavior but do not isolate the specific capability of inferring an acoustic target from context distributed across modalities and realizing it faithfully in generated speech. The paper terms this capability context-grounded acoustic control and argues no existing benchmark measures it directly.
Method
OmniACBench evaluates a model on a task with three separated input roles: a text script supplies the linguistic content to be read aloud, a spoken instruction specifies which acoustic dimension to control (e.g., emotion, accent), and an image supplies the situational cue that determines the target value along that dimension. Separating content, control signal, and contextual grounding across three modalities is meant to prevent shortcut solutions and force genuine multimodal integration, distinguishing the task from prosodic rendering or simple explicit style conditioning.
The benchmark covers six acoustic features chosen for multimodal groundability (each admits a natural visual cue) and evaluation diversity (some are objectively measurable, others abstract): Speech Rate, Phonation, Pronunciation, Emotion, Global Accent, and Timbre. Construction follows a three-stage pipeline: (1) Acoustic Feature Selection defines target values and associated image keywords per feature; (2) Tri-Modal Generation synthesizes the text script (via LLM, kept acoustically neutral to avoid shortcut learning), the spoken instruction (via LLM paraphrasing of instruction templates, then synthesized with a TTS model), and the image (via LLM-expanded meta-prompts fed to image generation models); (3) a Quality Control Protocol applies LLM-based and human filtering for semantic preservation, text neutrality, and image-keyword alignment, retaining 3,559 of 3,640 generated instances (97.78%). To reduce single-model bias, each generation stage draws from a pool of 3-4 models (e.g., three LLMs for text/paraphrasing/image-prompt generation, four image-generation models, three TTS models), with model selection randomized per instance.
Evaluation combines measurable and abstract metrics. Semantic fidelity is scored via WER from Whisper-large-v3 transcription. Measurable features use ∆WPM (speech-rate contrast between fast/slow targets), Phoneme Error Rate for pronunciation (heteronyms), and a Voiced Frame Ratio threshold (VFR@0.3) for whisper-like phonation, validated against the Expresso dataset. Abstract features (Emotion, Global Accent, Timbre) are scored by WavLM-Large-based classifiers fine-tuned on the paper’s own label spaces, since off-the-shelf classifiers (emotion2vec, ECAPA-TDNN and wav2vec2 accent classifiers) do not match the target label space or achieve comparable accuracy. Eight omni-modal models that take text, image, and speech and generate speech (MiniCPM-o 4.5, InteractiveOmni 8B/4B, Qwen3-Omni 30B, Qwen2.5-Omni 7B/3B, Uni-MoE-2.0-Omni, MGM-Omni 7B) are evaluated against this protocol, alongside Reference Scores computed from human annotators or the trained evaluators on held-out data.
Key Results
All eight models perform far below the Reference Scores across nearly every metric (Table 2). Semantic WER, the easiest axis, is still 1.04-5.96 for the models versus 0.05 for the reference, meaning speech content itself degrades once generation is conditioned on multimodal context, not just acoustic delivery. Measurable acoustic control is weakest overall: ∆WPM values cluster near zero (some even negative) against a reference gap of 65.87 WPM, and whisper-detection (VFR@0.3) is at or near 0% for all but MiniCPM-o 4.5 and Uni-MoE-2.0-Omni, versus a 96.78% reference. Abstract features fare somewhat better: MiniCPM-o 4.5 clears the 16.7%/33.3%/25.0% random baselines on Emotion (21.44%) and Global Accent (39.34%), but most other models stay near chance, and no model meaningfully controls Timbre (all near the 25% random baseline versus a 96.67% reference).
Diagnostic ablations (Table 3) show script-only reading, spoken-instruction conditioning, and isolated visual-cue inference are all handled with high accuracy in isolation, so the failure is not explained by any single elementary skill. Controlled input decomposition (Figure 3, textualizing inputs progressively toward an Oracle condition with the target value stated explicitly) reveals three distinct failure patterns: some features (Timbre, and Global Accent/Phonation for most models) stay near chance even under Oracle, meaning direct acoustic control is missing; others (Speech Rate for several models) jump only at Oracle, meaning implicit inference from context fails; and MiniCPM-o 4.5’s Speech Rate specifically degrades between the textualized (All-to-T) and multimodal (S-to-T/I-to-T) conditions, indicating a distinct multimodal-grounding failure. Linear probing of hidden states (Figure 4) shows MiniCPM-o 4.5, which tightly couples modality encoders/decoders with the LLM backbone, retains decodable context information into its TTS decoder, whereas Qwen3-Omni 30B’s decoupled Thinker-Talker architecture drops to chance decodability in the Talker.
Novelty Assessment
The contribution is primarily a new dataset and evaluation protocol rather than a new model or architecture. Its novelty lies in the specific task formulation, separating content (text), control signal (spoken instruction), and context (image) across three distinct modalities so that acoustic target inference cannot be solved by any single-modality shortcut, and in pairing measurable and abstract acoustic features under one benchmark with quantitatively verified construction quality (paraphrase diversity, speech intelligibility, image diversity all reported with explicit metrics). The diagnostic methodology, three-way input decomposition plus linear probing of hidden states, goes beyond static benchmark scoring to attribute failure to specific mechanisms (direct control, implicit inference, multimodal grounding), which is a genuine methodological contribution on top of the dataset itself. The task-specific evaluator classifiers for abstract features are an engineering necessity for the benchmark rather than a novel modeling technique.
Field Significance
High
The paper identifies and operationalizes a capability gap, context-grounded acoustic control, that existing omni-modal and speech-generation benchmarks do not isolate, and demonstrates with controlled ablations that this gap is not reducible to any single component skill. Its three-way failure taxonomy (direct control, implicit inference, multimodal grounding) and its architecture-linked probing result give the field concrete diagnostic tools rather than only a leaderboard, providing a foundation for follow-up work on acoustically controllable omni-modal generation.
Claims
- supports: Strong performance on text-output multimodal understanding benchmarks does not transfer to context-grounded acoustic control in speech generation.
Evidence: Eight evaluated omni-modal models, including ones reported to perform strongly on prior omni-modal benchmarks, score far below Reference Scores on every OmniACBench metric. (§4.3, Table 2)
- complicates: Context-grounded acoustic control cannot be attributed to a single failure mode; direct-control, implicit-inference, and multimodal-grounding failures are separable and require different fixes.
Evidence: Controlled input decomposition shows Timbre stays at chance even in the Oracle condition (missing direct control), Speech Rate improves only at Oracle for several models (failed implicit inference), and MiniCPM-o 4.5’s Speech Rate score drops specifically when inputs are distributed across modalities rather than textualized (failed multimodal grounding). (§5.2, Figure 3)
- supports: Architectures that tightly couple modality encoders/decoders with the language backbone preserve context-relevant information for controllable speech generation better than architectures that decouple high-level reasoning from speech decoding.
Evidence: Linear probing of hidden states shows MiniCPM-o 4.5 (end-to-end hidden-state integration) keeps acoustic-attribute information decodable into its TTS decoder, while Qwen3-Omni 30B (Thinker-Talker MoE) drops to near-chance decodability in the Talker. (§5.3, Figure 4)
- complicates: Conditioning speech generation on multimodal context degrades content fidelity itself, independent of any acoustic-style requirement.
Evidence: All eight models show substantially higher semantic WER on OmniACBench than the human Reference Score (0.05), despite achieving uniformly low WER when reading the same scripts without multimodal grounding in the script-only ablation. (§4.3, §5.1, Table 2, Table 3)
- refines: Automatic evaluation of abstract, non-measurable acoustic attributes requires classifiers trained on the target label space rather than generic off-the-shelf speech-attribute classifiers.
Evidence: Off-the-shelf emotion2vec, ECAPA-TDNN, and wav2vec2-based accent classifiers underperform (51.83%-84.57% accuracy) relative to WavLM-Large classifiers fine-tuned on the paper’s own six-way emotion, three-way accent, and four-way timbre label spaces (89.43%-97.29% accuracy), and human validation on a class-balanced subset confirms the trained evaluators track majority-vote human judgments. (§4.1, Appendix C)
Limitations and Open Questions
The authors note two scope limitations. First, each OmniACBench instance evaluates only a single target acoustic feature, whereas real speech generation may require jointly controlling multiple features at once (e.g., a fast rate and an angry emotion together); this compositional setting is left to future work, particularly since current models already struggle in the single-feature case. Second, the benchmark restricts audio input to spoken instructions and excludes other audio context such as environmental sounds or background music that could also inform appropriate acoustic delivery.
Beyond the authors’ stated limitations, the benchmark’s synthetic construction pipeline (LLM-generated scripts and instructions, generated images, TTS-synthesized speech) means every instance passes through several generative models before reaching the evaluated systems; the quality-control protocol mitigates but cannot fully rule out systematic artifacts specific to the model pools used for construction (e.g., a shared bias across the three TTS models used to synthesize spoken instructions).
Wiki Connections
- Spoken Language Model — benchmarks whether LLM-backed omni-modal models that consume an external spoken instruction alongside text and image can generate acoustically appropriate speech, directly probing a core spoken-language-model capability under multimodal grounding.
- Evaluation Metrics — contributes new measurable metrics (∆WPM for speech rate, phoneme error rate for pronunciation, VFR@0.3 for whisper-like phonation) alongside trained classifier-based scores for abstract acoustic attributes.
- Subjective Evaluation — validates its automatic evaluators against real human listening judgments on a class-balanced subset before relying on them at benchmark scale.
- MGM-Omni — evaluated as one of the eight omni-modal baselines on OmniACBench, where its brain-mouth dual-track architecture is analyzed alongside the other models’ varying degrees of acoustic control.
- Qwen3-Omni Technical Report — evaluated as a Thinker-Talker baseline and singled out in the linear-probing analysis, where its decoupled Talker module loses decodable acoustic-context information that the more tightly integrated MiniCPM-o 4.5 retains.
- S2S-Arena (2503.05085, accepted but not yet ingested) — discussed in Related Work as a prior speech-to-speech paralinguistic instruction-following benchmark that OmniACBench differentiates itself from by targeting context-grounded acoustic control specifically, rather than general instruction-following behavior.