arXiv · 2023 · Preprint
Bai et al. (Alibaba Group) · → Paper · Demo: ? · Code: ?
Qwen introduces a series of decoder-only transformer LLMs at 1.8B, 7B, and 14B scales, trained on up to 3 trillion tokens with supervised fine-tuning and RLHF alignment, plus specialised variants for code and mathematics.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
Qwen is a decoder-only transformer family built on a modified LLaMA-style architecture with RoPE positional embeddings, RMSNorm, SwiGLU activations, and an extended 152K-token vocabulary optimised for Chinese and multilingual text. The base models and chat variants (Qwen-Chat, Qwen-Chat-RLHF) provide a capable open-weights language model backbone that spoken conversational agent systems and end-to-end TTS pipelines can use for text understanding, response generation, and instruction following. The series covers a practical range of scales (1.8B to 14B parameters) and offers both base and instruction-tuned checkpoints, making it accessible as a language backbone in resource-constrained speech generation systems. The Qwen-VL multimodal extension, noted in the report, broadens the family toward audio-language applications that integrate visual and textual grounding.
Wiki Connections
- spoken-language-model — Qwen’s decoder-only autoregressive architecture and instruction-tuned chat variants can serve as the language model backbone in spoken conversational agent pipelines that separate language understanding from speech synthesis.
- 2302.13971 (LLaMA) — Qwen adopts the LLaMA open-source training paradigm and architectural choices (RoPE, SwiGLU, pre-norm) as its design foundation, extending them with a larger multilingual vocabulary and modified QKV biases.
- 2307.09288 (LLaMA 2) — Qwen benchmarks against LLaMA 2 across MMLU, GSM8K, and coding tasks, demonstrating competitive or superior performance at matched parameter counts.
- 2303.08774 (GPT-4 Technical Report) — Qwen uses GPT-4 as the primary upper-bound reference in its evaluations, contextualising where the open-weights series stands relative to proprietary models.
- 2002.05202 (GLU Variants) — Qwen adopts SwiGLU as its feed-forward activation function, following the finding in this paper that GLU-based activations outperform standard GeLU in transformer language models.
- 1412.6980 (Adam) — Qwen uses AdamW optimisation for pretraining, drawing on the Adam method introduced in this work.
- 1711.05101 (Decoupled Weight Decay) — Qwen uses AdamW, the decoupled weight decay variant introduced in this paper, as its pretraining optimiser.
- 1607.06450 (Layer Norm) — Qwen replaces standard layer normalisation with RMSNorm, building on the normalisation framework introduced here.
- 1810.04805 (BERT / Devlin et al.) — Cited as part of the foundational LLM lineage that contextualises Qwen’s place in the transformer pretraining paradigm.