arXiv · 2024 · Preprint
An Yang et al. (Alibaba Group) · → Paper · Demo: ? · Code: ✓
Qwen2 is an open-weight series of dense and Mixture-of-Experts large language models (0.5B to 72B parameters) from Alibaba Group, trained on over 7 trillion tokens with strong multilingual, coding, and reasoning capabilities.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
Qwen2 provides a family of open-weight decoder-only LLMs at multiple scales (0.5B, 1.5B, 7B, 57B-A14B MoE, and 72B), offering TTS and spoken conversational agent systems a range of backbone options from edge-deployable to server-scale. The architecture uses Grouped Query Attention, Dual Chunk Attention with YARN for long-context handling, and SwiGLU activations with RoPE positional embeddings, making it well-suited for autoregressive token prediction tasks such as speech token generation. Its multilingual pre-training data spanning approximately 30 languages and instruction-tuning via supervised fine-tuning and direct preference optimisation (DPO) can provide spoken conversational agent systems with text understanding and instruction-following capabilities across diverse linguistic contexts. The smaller variants (0.5B and 1.5B) are explicitly designed for deployment on portable devices, enabling on-device spoken conversational agent applications.
Wiki Connections
- 2309.16609 (Qwen) — direct predecessor in the Qwen model family; Qwen2 extends its architecture with improved GQA and long-context capabilities
- 2311.07919 (Qwen-Audio) — the audio-extended variant that builds on the same LLM backbone; cited as a related multimodal extension
- 2303.08774 (GPT-4) — cited as a closed-source performance ceiling in the instruction-following and reasoning benchmarks
- 2302.13971 (LLaMA) — cited as a representative open-weight baseline in the benchmark comparison