arXiv · 2026 · Preprint
Qwen Team (Alibaba) · → Paper · Demo: ✓ · Code: ✗
Scales the Thinker-Talker omni model to a Hybrid-Attention MoE Talker that streams multi-codebook RVQ speech tokens through a single interleaved text-speech stream (ARIA), reaching sub-1.3 Seed-TTS WER, competitive multilingual and cross-lingual voice cloning across 29 languages, and 235 to 435 ms first-packet latency in spoken dialogue.
Problem
End-to-end omni models that speak back to the user have to stream speech while the language model is still producing text. In the earlier Qwen3-Omni design the Talker consumed text and speech on two parallel tracks, and the mismatch between the encoding efficiency of the text tokenizer and the speech tokenizer caused streaming instabilities: skipped words, mispronunciations, and ambiguous rendering of numbers (§2.1). Alignment approaches based on forced alignment (MFA) or fixed interleaving rates do not transfer well across languages whose text tokenization is much less efficient than English or Chinese. At the same time, the report aims to expand speech output from a handful of languages to 36 languages and dialects, add zero-shot voice cloning and controllable generation to a conversational Talker, and keep latency low at hundreds of billions of parameters.
Method
Qwen3.5-Omni keeps the Thinker-Talker split introduced in Qwen2.5-Omni. The Thinker is a Hybrid-Attention Mixture-of-Experts LLM (initialized from Qwen3.5, with Gated Delta Net layers to reduce KV-cache I/O over long contexts) that reads text, images, video, and audio and produces text; it supports a 256k-token context (more than 10 hours of audio). Audio enters through AuT, an attention encoder-decoder audio encoder trained from scratch on 40 million hours of audio-text pairs pseudo-labelled by Qwen3-ASR, which downsamples 128-channel mel spectrograms 16x to a 6.25 Hz token rate (§2.2-2.3). Timestamps are inserted as formatted text strings in seconds rather than relying solely on TM-RoPE absolute temporal IDs (§2.3). Two variants are reported, Plus and Flash; exact parameter counts are not given.
The Talker is also a Hybrid MoE Transformer and is conditioned on hidden states extracted from the Thinker’s middle layers, historical text tokens, multimodal context, and the streamed text of the current turn, which lets it modulate prosody, loudness, and emotion with the conversation (§2.4, Figure 2). It autoregressively predicts the first codebook of a multi-codebook RVQ representation (Qwen3.5-Omni-Audio-Tokenizer, inherited from Qwen3-Omni), and a dense multi-token prediction (MTP) Transformer outputs the residual codebooks for the same frame. A causal ConvNet (Code2Wav) decodes each frame incrementally, so playback can begin after a single frame (§2.4-2.5, Table 1). The report does not describe a new codec, its frame rate, or reconstruction results.

Two Talker changes are new relative to Qwen3-Omni (§2.4). First, a dedicated Talker system prompt specifies target voice characteristics and can contain textual descriptions and codec sequences from a reference clip, replacing a conventional speaker embedding and enabling zero-shot voice cloning and controllable generation. Second, ARIA (Adaptive Rate Interleave Alignment) collapses the dual-channel text/speech input into a single interleaved stream under a monotonic constraint: for every prefix of the generated sequence, the cumulative speech-to-text token ratio may not exceed the item-level global ratio. This requires no external alignment and accommodates languages with low text-encoding efficiency, and it lets an arbitrary text prefix be followed by coherent speech continuation.
Talker training has four stages (§4.2): general pretraining on more than 20 million hours of multilingual speech paired with multimodal context (including instruction-following speech generation tasks); a long-context continual pretraining stage on quality-stratified data, extended to 64k tokens and aided by Qwen3-Omni-Captioner; an RL stage combining DPO on human-annotated multilingual preference pairs with rule-based rewards optimized by GSPO; and lightweight speaker fine-tuning for preset/custom voices. The Thinker is post-trained with specialist distillation, on-policy distillation that uses text-conditioned responses as targets for the matching audio-conditioned queries, and an interaction-aligned RL stage targeting code-switching, persona consistency, and long-context instruction following (§4.1). Speech output covers 29 languages and 7 Chinese dialects (Table 3).
Key Results
Zero-shot TTS (Table 8). On Seed-TTS eval, Qwen3.5-Omni-Plus reaches 0.99 (test-zh) and 1.26 (test-en) WER. The test-en figure is the lowest in the table, ahead of Qwen3-Omni-30B-A3B (1.39), CosyVoice 3 (1.45), and MiniMax-Speech (1.65); on test-zh CosyVoice 3 (0.71) and MiniMax-Speech (0.83) remain better. Only content consistency is reported; speaker similarity and naturalness are not given for this benchmark.
Multilingual voice cloning (Tables 9-10). On the MiniMax TTS multilingual test set (20 languages), Qwen3.5-Omni-Plus has the lowest WER in 13 languages against MiniMax and ElevenLabs, and the highest speaker similarity in 15. For example, English WER/SIM is 0.631/0.833 versus 2.164/0.756 (MiniMax) and 0.756/0.613 (ElevenLabs). Losses are concentrated in Arabic, Vietnamese, Turkish, Polish, Hindi, Dutch, and Czech. The 9 additional languages in the internal FLEURS-based set are compared only with ground-truth recordings (transcribed by GPT-4o-transcribe), where the model’s WER is below the ground truth WER in every language; the “22 of 29 languages” headline in §5.2.2 combines these two different comparison types.
Cross-lingual cloning (Table 11). Against Qwen3-Omni, CosyVoice 3, and CosyVoice 2 on CV3-Eval, Qwen3.5-Omni-Plus has the lowest mixed error rate in 11 of 12 directions (losing only Japanese-to-Chinese, 3.55 vs. 3.05 for CosyVoice 3). Chinese-to-Korean drops from 14.4 (CosyVoice 3) to 4.03.
Custom voice (Table 12). After speaker fine-tuning on monolingual data, the model is compared with ElevenLabs Multilingual v2, Gemini-2.5 Pro-Preview-TTS, GPT-Audio, and MiniMax-Speech-2.8-HD using preset voices across 29 languages; it has the best WER in 10 languages (for example Japanese 3.306, Korean 1.309) and is behind in others such as Danish (7.183 vs. 2.604 for ElevenLabs) and Czech.
Spoken dialogue and audio understanding (Table 5). Against Gemini-3.1 Pro, Qwen3.5-Omni-Plus scores 93.1 vs. 88.9 on VoiceBench, is mixed on URO-Bench-pro (66.3/86.3/99.8 vs. 69.1/84.0/99.2 for understanding/reasoning/oral), and slightly lower on SpeechRole (123.5 vs. 124.2) and WildSpeech-Bench (75.4 vs. 76.3). ASR WER is lower across all listed sets, e.g. LibriSpeech clean/other 1.11/2.23 vs. 3.36/4.41.
Latency (Tables 1-2). First-packet latency for audio input at one concurrent stream is 435 ms (Plus) and 235 ms (Flash), rising to 955 ms and 352 ms at eight streams. Talker time-to-first-chunk is 54-95 ms (Plus), codec decoding takes 3-5 ms, and generation RTF stays between 0.178 and 0.334 across settings. The authors note that Plus and Flash use different deployment resources, so the two should not be compared directly.
Non-speech capabilities (Tables 4, 6, 7, 13-15). Text and vision scores are close to Qwen3.5-Plus-Instruct (e.g. MMLU-Pro 85.9 vs. 86.8), video understanding is slightly higher, and audio-visual QA is comparable to Gemini-3.1 Pro (DailyOmni 84.6 vs. 82.7, WorldSense 62.8 vs. 65.5). Appendix tables report FLEURS multilingual ASR (average WER 6.6 vs. 7.3 for Gemini-3.1 Pro) and speech translation BLEU.
All baselines in Tables 9, 11, and 12 are other systems’ reported results or commercial APIs; Table 8 numbers for prior systems are quoted from their papers. No subjective listening test is reported for any speech-generation result.
Novelty Assessment
The speech-generation contribution is mostly a scaled-up engineering integration of the Qwen3-Omni Talker: the RVQ tokenizer, MTP residual-codebook head, causal ConvNet decoder, and chunked prefill are inherited, and the Hybrid MoE backbone comes from Qwen3.5. The genuinely new mechanism is ARIA, a simple prefix-ratio constraint that replaces dual-track or fixed-rate text/speech interleaving with an adaptive single stream; it is conceptually lightweight but addresses a real failure mode of streaming LLM-based TTS in languages with poor text-token efficiency. The prompt-based voice specification (text plus codec tokens in a Talker system prompt) is a reasonable design but similar in spirit to in-context prompting in existing codec LMs. Most of the evidence is scale: more than 20 million hours of Talker training data, 36 output varieties, and an RL stage on human preferences. Importantly, no ablation isolates ARIA, the system prompt, or the DPO/GSPO stage, so their individual contributions cannot be separated from data and model scale.
Field Significance
Tip
High — The report demonstrates that a single omni model with a streaming codec-LM Talker can match or beat dedicated zero-shot TTS systems on content accuracy and speaker similarity across roughly 30 languages while serving spoken dialogue at a few hundred milliseconds of first-packet latency. ARIA provides a concrete, alignment-free recipe for single-stream text/speech interleaving in streaming generation, although the paper offers no ablation of it.
Claims
- supports: Constraining the cumulative speech-to-text token ratio over every generated prefix allows text and speech tokens to be interleaved in a single autoregressive stream without forced alignment or a fixed interleaving rate.
Evidence: ARIA replaces Qwen3-Omni’s dual-channel Talker input with a single interleaved stream in which the prefix speech-to-text ratio may not exceed the utterance-level ratio; the authors motivate it by skipped words and number errors under mismatched tokenization rates, and the resulting model improves Seed-TTS test-en WER from 1.39 (Qwen3-Omni-30B-A3B) to 1.26, though model scale and data also change. (§2.4, §2.5, Table 8)
- supports: A conversational speech LM Talker trained at large multilingual scale can reach content accuracy and speaker similarity comparable to or better than dedicated commercial zero-shot TTS systems in many languages.
Evidence: On the 20-language TTS multilingual test set, Qwen3.5-Omni-Plus has the lowest WER in 13 languages and highest speaker similarity in 15 against MiniMax and ElevenLabs, and the lowest cross-lingual error rate in 11 of 12 CV3-Eval directions. (§5.2.2, §5.2.3, Tables 9 and 11)
- complicates: Gains in multilingual speech generation from scale are uneven across languages, with lower-resource or less-tokenization-efficient languages still trailing specialised systems.
Evidence: Qwen3.5-Omni-Plus trails MiniMax or ElevenLabs on WER in Arabic, Vietnamese, Turkish, Polish, Hindi, Dutch, and Czech, and its custom-voice WER remains above 10 for Urdu, Icelandic, and Persian. (§5.2.2, §5.2.4, Tables 9 and 12)
- supports: Streaming speech generation with a multi-codebook codec, a lightweight residual-codebook predictor, and a causal convolutional decoder keeps speech synthesis off the critical latency path in large omni models.
Evidence: For audio input on the Plus variant, the Talker’s time-to-first-chunk is 54-95 ms across one to eight concurrent streams and codec decoding takes 3-5 ms, with generation RTF between 0.178 and 0.334 across all settings, while first-packet latency is dominated by the Thinker’s time-to-first-token (162-260 ms for audio input on Plus). (§2.5, Table 2)
- complicates: Speaker fine-tuning on monolingual data can transfer a custom voice across many languages, but intelligibility after transfer varies widely by language.
Evidence: The speaker-fine-tuned model generates all 29 languages, achieving the best WER among five systems in 10 languages, yet is clearly behind commercial preset voices in Danish (7.183 vs. 2.604) and Czech (3.769 vs. 1.766). (§5.2.4, Table 12)
Limitations and Open Questions
The speech-generation evaluation is entirely objective: WER/CER and embedding-based speaker similarity. There is no MOS, CMOS, or preference test, so the claimed improvements in prosody, naturalness, and “human-like emotional nuance” from ARIA and the DPO stage are not measured. No ablation isolates ARIA, the Talker system prompt, or the RL stage, and the model weights and training data are not released (only API and demo access), so the results cannot be reproduced independently.
Several comparisons need care. The “22 of 29 languages” figure combines a 20-language comparison against MiniMax and ElevenLabs with 9 internal-set languages compared only against ground-truth recordings. The text of §5.2.3 says Qwen3.5-Omni is best in 10 of 12 cross-lingual directions, while Table 11 marks it best in 11 of 12. Prior-system Seed-TTS numbers are quoted rather than re-run. Custom-voice comparisons use different preset voices per commercial system. Emotion, volume, and speed control, semantic interruption, and voice cloning from user samples are described as capabilities but have no dedicated evaluation, and the codec’s frame rate, bitrate, and reconstruction quality are not reported.
Wiki Connections
- Autoregressive Codec TTS — The Talker is a Hybrid MoE autoregressive LM over multi-codebook RVQ tokens with an MTP head for residual codebooks.
- Spoken Language Model — The Thinker is an LLM adapted to consume user speech through the AuT encoder and is post-trained with on-policy distillation to close the audio-versus-text query gap.
- Speech-to-Speech — Provides an end-to-end spoken dialogue system that answers spoken queries with streamed speech, evaluated on VoiceBench, URO-Bench, SpeechRole, and WildSpeech-Bench.
- Streaming TTS — Contributes ARIA single-stream interleaving and frame-level streaming decoding with reported first-packet latency and RTF under concurrency.
- Zero-Shot TTS — Adds prompt-based zero-shot voice cloning to the Talker and reports Seed-TTS, multilingual, and cross-lingual cloning results.
- Multilingual TTS — Trains the Talker on more than 20 million hours of multilingual speech, supporting 29 languages and 7 Chinese dialects for output.
- Qwen3-Omni — Direct predecessor; Qwen3.5-Omni inherits its RVQ tokenizer, MTP head, and ConvNet decoder and replaces its dual-track Talker input with ARIA, comparing against it on Seed-TTS and cross-lingual benchmarks.
- Qwen2.5-Omni — Introduced the Thinker-Talker architecture this report builds on; included as a Seed-TTS baseline.
- Seed-TTS — Supplies the Seed-TTS eval benchmark used for zero-shot content consistency and serves as a baseline.
- CosyVoice 3 — Strongest baseline on Seed-TTS test-zh and the main cross-lingual competitor on CV3-Eval, whose benchmark it introduced.
- CosyVoice 2 — Baseline on Seed-TTS and cross-lingual cloning.
- MiniMax-Speech — Source of the TTS multilingual test set and a commercial baseline for multilingual WER and speaker similarity.
- VoiceBench — Spoken dialogue benchmark on which the model reports its largest margin over Gemini-3.1 Pro.
- URO-Bench — Pro-track benchmark used to evaluate understanding, reasoning, and oral-conversation quality of spoken responses.