arXiv · 2024 · Preprint
Chu et al. (Alibaba Group) · → Paper · Demo: ✓ · Code: ✓
Qwen2-Audio is an 8.2B-parameter large audio-language model that accepts arbitrary audio inputs (speech, sound, music, mixed audio) and produces text responses, supporting both a voice chat mode and an audio analysis mode within a single unified model trained without mode-switching system prompts.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
Qwen2-Audio provides an open-source audio-language foundation that TTS and SCA researchers use as a reference for audio-conditioned instruction following and multimodal speech understanding. The model pairs a Whisper-large-v3 audio encoder (with a 2x temporal pooling layer, yielding approximately 40ms per frame) with the Qwen-7B language model, trained through three stages: pre-training with natural language prompts, supervised fine-tuning, and direct preference optimization (DPO) for factuality alignment. Its unified treatment of voice chat and audio analysis in a single model, without separate system prompts to switch modes, offers a design pattern that SCA papers reference when discussing end-to-end spoken dialogue systems. The AIR-Bench evaluation results, where Qwen2-Audio outperforms Gemini-1.5-pro on audio-centric instruction following, provide SCA and speech understanding researchers with a publicly available strong baseline for cross-system comparisons.
Wiki Connections
Related concepts: spoken-language-model
In-corpus papers cited by this work: 2301.11325 · 2309.16609 · 2311.07919 · 2007.10310 · 2305.11000
Cited by: 2504.18425 (Kimi-Audio, Qwen2-Audio as reference audio-language model)
2501.06282 — MinMo: A Multimodal Large Language Model for Seamless Voice Interaction 2502.11946 — Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction 2502.17239 — Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction