arXiv · 2025 · Preprint
Jin Xu et al. (Qwen Team (Alibaba)) · → Paper · Demo: ✓ · Code: ✓
Qwen3-Omni is an end-to-end Thinker-Talker multimodal foundation model that adds real-time, multilingual, low-latency speech generation to a text/image/audio/video model without degrading any single modality relative to matched unimodal baselines.
Problem
Prior omni-modal systems, including the authors’ own predecessor Qwen2.5-Omni, tend to trade off performance across modalities: adding audio and vision training signals to a language model has typically come at some cost to text or vision quality, and speech generation in such systems has depended on comparatively heavy vocoding stages (block-wise diffusion transformers) that limit streaming responsiveness. There was also no rigorous, controlled comparison isolating whether joint multimodal training degrades unimodal capability, as opposed to simply reporting aggregate scores against different-sized baselines. On the speech side specifically, systems supporting real-time spoken dialogue faced a tension between representational capacity (richer codebooks for expressive, high-fidelity speech) and first-packet latency (the time before audio starts streaming to the user).
Method
Qwen3-Omni retains the Thinker-Talker decomposition introduced in the predecessor Qwen2.5-Omni but upgrades both components to Mixture-of-Experts (MoE) transformers. The Thinker (30B-A3B MoE) consumes text, audio, and vision/video, producing text output and passing multimodal hidden representations directly to the Talker; the Talker (3B-A0.3B MoE) no longer receives the Thinker’s high-level text representations, conditioning only on audio/visual features so that a decoupled system-prompt design and external interventions (RAG, function calling, safety filters) can sit between the two. Audio input is encoded by a newly trained Audio Transformer (AuT), an attention encoder-decoder trained from scratch on 20 million hours of supervised audio (80% Chinese/English ASR, 10% other-language ASR, 10% audio understanding) with Conv2D downsampling to a 12.5 Hz token rate and dynamic-window flash attention for real-time prefill caching; this replaces the Whisper encoder used previously. Position information across modalities is handled via TM-RoPE, a Multimodal RoPE extension that anchors audio and video tokens to absolute wall-clock time rather than fixed-size chunks, allowing arbitrary-duration streaming.
For speech generation, the Talker predicts a multi-codebook (RVQ) sequence autoregressively: a linear head predicts the zeroth codebook for the current frame from the backbone’s aggregated codebook features, and a multi-token-prediction (MTP) module then generates the remaining residual codebooks for that same frame. Because this hierarchical scheme captures fine acoustic detail directly in the discrete tokens, the paper replaces the computationally heavy block-wise diffusion transformer (DiT) vocoder used in prior systems with a lightweight causal ConvNet (Code2Wav, 200M parameters) that can synthesize waveform immediately after each frame’s tokens are produced, rather than waiting for block-level context. Combined with chunked asynchronous prefilling between Thinker and Talker and an MoE design that keeps KV-cache IO low under concurrency, this streaming multi-codebook pipeline yields a reported theoretical end-to-end first-packet latency of 234 ms in cold-start, single-request conditions.
Talker training proceeds in four stages: large-scale multimodal-to-speech mapping on hundreds of millions of utterances, continual pretraining on higher-quality data to reduce hallucinated speech and extend long-context handling, Direct Preference Optimization (DPO) over multilingual preference pairs to improve generalization and stability, and finally speaker fine-tuning for specific target voices. Thinker post-training separately combines supervised fine-tuning, a strong-to-weak (off-policy then on-policy) distillation pipeline from larger Qwen3 teacher models, and GSPO reinforcement learning with rule-based and LLM-judge rewards. To isolate the effect of joint multimodal training, the authors additionally pretrain matched-parameter-count text-only, vision-only, and Omni models on identical data, learning-rate schedules, batch sizes and FLOPs, varying only the presence of audio/audio-visual training signal.
Key Results
On zero-shot TTS (SEED test set), Qwen3-Omni-30B-A3B reports the lowest (best) content-consistency error among all compared systems on the English subset (1.39), ahead of CosyVoice3 (1.45), Seed-TTS-RL (1.94), F5-TTS (1.83), and its own predecessor Qwen2.5-Omni-7B (2.33); on the Chinese subset it trails Seed-TTS-RL (1.00) and CosyVoice3 (0.71) at 1.07. In multilingual zero-shot voice cloning against MiniMax-Speech and ElevenLabs Multilingual v2, it leads clearly on Chinese, English, and French but is only competitive, not best, on several other languages (e.g., Portuguese, Russian) for speaker similarity. In cross-lingual voice cloning against CosyVoice2/3, it beats CosyVoice3 on any-to-English and any-to-Korean directions and is roughly comparable on any-to-Japanese despite no explicit text normalization. On the audio-to-text side, it reports open-source state-of-the-art ASR and lyric-ASR results on LibriSpeech, WenetSpeech, Fleurs, CommonVoice, Opencpop-test and MIR-1K, an overall VoiceBench score of 89.5 (Thinking variant), trailing only Gemini-2.5-Pro (89.6), and state-of-the-art results on the RUL-MuchoMusic and several MTG/MagnaTagATune music-understanding tasks against both generalist and specialist baselines. The paper reports open-source SOTA on 32 of 36 audio/audio-visual benchmarks and overall SOTA on 22. In the controlled non-degradation study, the matched-size Omni-pretrained base model matches or slightly exceeds text-only and vision-only baselines on the reported text, vision, and video benchmarks trained under identical compute.
Novelty Assessment
The overall system is best characterized as an engineering-integration and scaling effort: the Thinker-Talker decomposition, MoE transformers, RVQ speech tokens, and preference optimization are all established techniques, assembled here at a larger scale with a newly trained audio encoder. The more genuinely architectural contribution is narrower and specific to the speech-generation path: replacing a block-wise diffusion vocoder with a causal-ConvNet Code2Wav stage driven by a multi-codebook autoregressive Talker plus MTP module, which is a concrete design choice for trading vocoder expressiveness against streaming latency. The controlled non-degradation experiment (matched-size, matched-compute text-only vs. vision-only vs. Omni baselines) is a genuine methodological contribution distinct from simply reporting benchmark numbers, though its scope is limited to the specific model sizes and data mixtures tested and the authors themselves note they could not sweep other scales due to cost. The breadth of the empirical claim (open-source SOTA on 32/36 benchmarks) is the paper’s most emphasized result, but many of the individual comparisons rely on the authors’ own reported baseline numbers rather than independently reproduced runs.
Field Significance
Tip
High — Qwen3-Omni provides a controlled demonstration that joint multimodal pretraining, when unimodal and cross-modal data are mixed from the earliest pretraining stage, need not degrade a model’s text or vision performance relative to matched single-modality baselines, while simultaneously delivering competitive zero-shot multilingual speech generation and sub-250ms streaming latency in an openly released model.
Beyond its own benchmark numbers, the paper’s controlled ablation methodology (matched parameter count, data, and compute across text-only, vision-only, and Omni variants) offers a reusable experimental template for testing modality-degradation claims in future omni-modal systems, and its streaming multi-codebook Talker plus causal-ConvNet vocoder design is a concrete alternative to diffusion-based vocoding for latency-sensitive speech generation.
Claims
- supports: Replacing a block-wise diffusion vocoder with a lightweight causal convolutional decoder, driven by a multi-codebook autoregressive token predictor, can substantially reduce first-packet latency in streaming speech generation without sacrificing competitiveness on content-consistency metrics.
Evidence: The Talker’s multi-codebook AR scheme plus a 200M-parameter causal ConvNet Code2Wav stage achieves a 234ms end-to-end first-packet latency at 1x concurrency and the lowest reported content-consistency error on SEED test-en (1.39) among all compared zero-shot TTS systems including flow-matching and diffusion-based baselines. (§2.4, §2.5, Table 1, Table 13)
- supports: Mixing unimodal and cross-modal training data from the earliest stage of pretraining allows a language model to add new input/output modalities without degrading its original text, vision, or audio-specific capabilities relative to matched unimodal baselines.
Evidence: A controlled comparison of parameter-matched text-only, vision-only, and Omni models trained on identical corpora, schedules, and compute shows the Omni model matches or exceeds the unimodal baselines on general, math/STEM, coding, and multilingual text benchmarks as well as vision and video benchmarks. (§6, Table 16)
- complicates: Strong zero-shot voice-cloning performance in one or two conditioning languages does not guarantee comparable speaker-similarity performance uniformly across all supported languages.
Evidence: Against MiniMax-Speech and ElevenLabs Multilingual v2 on a 10-language multilingual test set, the system leads by a substantial margin on Chinese, English, and French but reports only “competitive,” non-leading, speaker-similarity or content-consistency scores on several other languages such as Portuguese and Russian. (§5.2.2, Table 14)
- supports: A large-scale, purpose-built supervised audio encoder trained from scratch for a multimodal LLM’s audio pathway can outperform reusing a general pretrained ASR encoder (e.g., Whisper) as the perceptual front-end for both speech understanding and downstream speech generation.
Evidence: Replacing the Whisper-based audio encoder from the predecessor system with AuT, trained from scratch on 20 million hours of supervised audio at a 12.5 Hz token rate, is cited as a key driver of gains across ASR, lyric-ASR, and voice-interaction benchmarks relative to Qwen2.5-Omni. (§1, §2.2, Table 6, Table 7)
Limitations and Open Questions
Warning
Speech generation quality is evaluated exclusively with automatic metrics (WER/CER-style content consistency, embedding-based speaker similarity, BLEU for translation); no human MOS or listening-test results are reported anywhere in the paper for the Talker’s synthesized speech, so claims of “stable, naturalistic speech synthesis” in the conclusion are not directly supported by subjective evidence in this report.
The paper acknowledges suboptimal performance on long-video benchmarks, attributed to limited positional extrapolation and restricted context length, as an explicit architectural limitation left for future work. The reported 234ms first-packet latency is described as “theoretical,” measured under a specific vLLM/torch.compile/CUDA-Graph deployment configuration rather than as an end-user-measured figure across arbitrary hardware or network conditions, and latency degrades substantially under higher concurrency (up to 1172ms at 6-way concurrency in the audio case). Several baselines used for comparison (ElevenLabs, MiniMax-Speech, Gemini-2.5-Pro, GPT-4o variants) are closed proprietary systems, so exact reproduction of the comparative numbers is not possible outside the authors’ own evaluation pipeline. The non-degradation ablation study, while methodologically rigorous, was run at limited model scales due to computational cost, and the authors explicitly caution that they could not sweep across all model sizes.
Wiki Connections
- Zero-Shot TTS — the Talker performs zero-shot voice cloning evaluated on the SEED, MiniMax multilingual, and CV3-Eval cross-lingual benchmarks against dedicated zero-shot TTS systems.
- Multilingual TTS — speech generation is supported natively across 10 output languages with explicit multilingual and cross-lingual voice-cloning evaluation.
- Streaming TTS — the core architectural contribution is a multi-codebook autoregressive Talker paired with a causal-ConvNet vocoder specifically designed to minimize first-packet latency for real-time streaming.
- Spoken Language Model — the Thinker consumes external speech input via the AuT audio encoder and is adapted through multimodal pretraining and post-training to reason over and respond to spoken content.
- Speech-to-Speech — the Thinker-Talker pipeline supports real-time spoken dialogue where audio input can drive streamed audio output within a single end-to-end model.
- Neural Audio Codec — the Talker operates directly on multi-codebook RVQ tokens, and the paper argues that richer codebook representations let a lightweight ConvNet replace a heavier diffusion-based decoder.
- Qwen2.5-Omni — Qwen3-Omni is presented as a direct successor, replacing its Whisper encoder, single-track codec Talker, and DiT-based vocoder with AuT, a multi-codebook MoE Talker, and a causal ConvNet.
- CosyVoice 3 — used as the strongest reported zero-shot TTS and cross-lingual voice-cloning baseline, with Qwen3-Omni matching or exceeding it on several test-en and cross-lingual conditions.
- CosyVoice 2 — used as a zero-shot and cross-lingual voice-cloning baseline, outperformed by Qwen3-Omni on the SEED and CosyVoice cross-lingual test sets.
- F5-TTS — used as a flow-matching zero-shot TTS baseline on the SEED test set.
- Seed-TTS — SEED test set and its ICL/RL variants serve as the primary zero-shot TTS benchmark and baseline systems.
- MaskGCT — used as a non-autoregressive zero-shot TTS baseline on the SEED test set.
- Spark-TTS — used as an LLM-based single-stream-token zero-shot TTS baseline on the SEED test set.
- VoiceBench — used as the primary benchmark for evaluating voice-chatting and spoken-instruction-following capability.