arXiv · 2024 · Preprint
S Sakshi et al. · → Paper · Demo: ✓ · Code: ?
MMAU is a 10,000-question multiple-choice benchmark for large audio-language models that spans speech, environmental sounds, and music, requiring 27 distinct expert-level skills across information extraction and complex reasoning tasks.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
MMAU fills a gap left by existing audio benchmarks that focus on foundational tasks such as ASR or acoustic scene classification: it evaluates whether large audio-language models can apply domain-specific knowledge and multi-step reasoning grounded in perceived audio. The benchmark is organised into three domains (speech, sound, music) with questions divided into information extraction (3,499 items) and reasoning (6,501 items), and expert-annotated difficulty levels (easy, medium, hard). Its speech domain specifically probes capabilities relevant to spoken conversational agents and speech LMs, including phonemic analysis, emotional tone interpretation, and multi-speaker role mapping. Evaluation against 18 open-source and proprietary models shows that even state-of-the-art systems achieve only around 53% accuracy (versus 82% for humans), providing a diagnostic signal on where current audio-language models fall short and enabling systematic comparison of audio understanding approaches.
Wiki Connections
- evaluation-metrics — MMAU introduces accuracy-based evaluation using micro-averaged multiple-choice scoring, extending the evaluation-metrics landscape to expert-level audio reasoning.
- spoken-language-model — the benchmark’s speech domain targets capabilities directly relevant to speech LMs, including reasoning over speaker role, emotion, and phonemic structure.
- 2303.08774 (GPT-4 Technical Report) — GPT-4o is evaluated as a text-only baseline on MMAU using audio captions, achieving the strongest cascaded result (59% with strong captions).
- 2301.11325 (MusicLM) — cited as context for music domain audio understanding that MMAU extends with reasoning-focused tasks beyond basic music generation evaluation.
- 2311.07919 (Qwen-Audio) — evaluated on MMAU; Qwen-Audio-Chat achieves 41.86% average accuracy, establishing it as a reference model for audio understanding benchmarking.
- 2407.10759 (Qwen2-Audio Technical Report) — Qwen2-Audio-Instruct is the top open-access model on MMAU with 52.50% average accuracy, making MMAU a key evaluation checkpoint for this model family.
- 2407.21783 (The Llama 3 Herd of Models) — Llama-3-8B-Instruct is evaluated as a text-only LLM baseline with caption inputs, reaching 53.57% with strong captions.
- 2306.12925 (AudioPaLM) — cited as a representative speech audio-language model in the context of domain-specific LALM development that MMAU was designed to evaluate.
- 2310.13289 (SALMONN) — evaluated on MMAU at 32.77% average accuracy; noise-replacement experiments reveal that SALMONN does not reliably attend to audio input.
- 2302.13971 (LLaMA) — LLaMA-2 is noted as the LLM backbone for GAMA, one of the evaluated open-source audio-language models.
- 2402.07729 (AIR-Bench) — compared directly against MMAU in the benchmark comparison table; AIR-Bench covers multiple domains but focuses on foundational tasks without complex reasoning.