arXiv · 2026 · Technical Report
Junbo Cui et al. (OpenBMB) · → Paper · Demo: ✓ · Code: ✓
A 9B open omni-modal model that listens, watches and speaks at the same time by serialising video, audio and its own text/speech output into one time-chunked token stream (Omni-Flow), with a time-aligned text/speech interleaving scheme (TAIL) that keeps the spoken reply from lagging behind what the model currently perceives.
Problem
Most speech LMs and omni-modal assistants still alternate between a perception phase and a response phase: while the model is speaking it cannot take in new audio or video, and it only speaks when explicitly asked. Full-duplex speech LMs address the first problem for audio alone, but extending simultaneous listening, watching and speaking to a vision-plus-speech model raises a second, speech-specific issue. In streaming speech generation the text produced in a given interval can take much longer than that interval to vocalise, so the audio being played drifts further and further behind the model’s current state. The two common streaming strategies, generating a long text lead before synthesis or interleaving text and speech tokens at a fixed ratio (the paper groups Mini-Omni, Qwen3-Omni, Step-Audio 2 and CosyVoice 2 under these two strategies), do not tie speech playback to wall-clock time and therefore produce stale speech in a live interaction. The authors also aim to avoid having the LLM backbone itself emit speech tokens (about 25 per second), which they argue costs efficiency and degrades language ability.
Method
The system has three parts. A Whisper-Medium encoder (0.3B) processes input audio in streaming chunks at 50 frames per second, and a two-layer MLP projector compresses this 5x to 10 audio tokens per second for the backbone. Video frames go through a SigLIP ViT (0.4B) with a resampler that compresses each image slice from 1,024 to 64 tokens (448x448 maximum resolution in full-duplex mode). The backbone is Qwen3-8B, which only generates text tokens, so in live interaction it needs roughly 3-4 decoding steps per second.
Speech output is handled by a separate ~0.3B Llama-style speech token decoder (20 layers, hidden size 768). For every text token, the backbone hidden state is projected by an MLP and summed with the decoder’s own text embedding, and the decoder autoregressively produces S3 supervised semantic tokens (25 Hz, one codebook of 6,562 entries), so prosody decisions are made by the backbone and the small decoder only models speech. A streaming flow-matching decoder (following CosyVoice 2) converts the tokens to waveform, conditioned on the reference audio in a multimodal system prompt, which is how voice cloning is exposed. The 9.34B learnable parameters run from the encoders through the backbone to the speech token decoder and are trained end to end; the flow-matching decoder is not among the learnable components listed in the configuration table.
Omni-Flow treats the interaction as three time-aligned streams: environment video, environment audio (which carries the user’s speech when present), and the model’s output stream. Time is cut into chunks of length t; each chunk is serialised as visual tokens, then audio tokens, then output tokens, and the chunks are concatenated into one causal sequence. When the model should stay silent it emits a [listen] token, so turn-taking and proactive speech are decided by the model itself, with no external voice activity detector. The ablation on text/instruction benchmarks finds that 1.0 s chunks work best (0.2 s and 0.1 s degrade sharply), that explicit boundary tokens between chunks help, and that predicting a separate listen/speak control token before content (LS) beats putting [listen] in the text vocabulary (LT).

Time-Aligned Interleaving (TAIL) decides how much text to generate in each chunk based on accumulated playback progress: if earlier chunks have pushed speech playback behind the chunk boundary, the model writes fewer text tokens in the current chunk so the speech can catch up. Supervision comes from full-duplex training data with per-token start and end times, where text tokens whose start time falls in chunk k (and their speech tokens) are assigned to chunk k. A bounded look-ahead defers the speech tokens of the last few text tokens to the next chunk, so that pronunciation and prosody can depend on the following word without letting text run far ahead.

Training starts from a MiniCPM-V 4.5 pretraining checkpoint. Stage one freezes the pretrained parts and trains only the new audio projector, backbone-to-speech projector and speech decoder on millions of hours of pseudo-labelled speech (zero-shot TTS, ASR and multi-speaker dialogue). Stage two unfreezes everything for joint pretraining on vision-language, speech, omni-modal and full-duplex data with time-aligned text, speech and video. Supervised fine-tuning follows, including a smaller set of spoken dialogues that voice actors re-recorded in a conversational style. Finally, GRPO with a smoothed length reward and RLAIF-V are applied, both aimed at reasoning and visual hallucination rather than speech.
Key Results
Speech generation (Table 5, SeedTTS test sets, LongTTS, Expresso, ESD). MiniCPM-o 4.5 reports 0.86 CER on SeedTTS test-zh and 2.38 WER on test-en, lower than CosyVoice 2 (1.45 / 2.57) and Qwen3-Omni (1.41 / 3.39). Speaker similarity (SIM-o) is on par with CosyVoice 2 rather than better (74.5 vs. 74.8 zh, 64.9 vs. 65.2 en); Qwen3-Omni has no voice cloning and is listed N/A. On LongTTS the English WER is much lower than both baselines (3.37 vs. 14.80 and 17.33), while Chinese CER (6.58) is worse than CosyVoice 2 (5.27) and much better than Qwen3-Omni (18.99). On Expresso and ESD, both evaluated with a neutral reference clip, the model scores 29.8 and 82.1 against CosyVoice 2’s 17.9 and 53.4. The paper does not define the emotion/style control metric. All speech generation numbers are automatic; no human listening test is reported.
Which mode produced the headline numbers (Table 10). The Table 5 SeedTTS figures match the fixed-ratio interleaving mode, not TAIL. In the full-duplex TAIL mode, Chinese CER is 1.04 (still below both baselines’ 1.45 and 1.41) but English WER rises to 3.93, which is worse than CosyVoice 2 (2.57) and Qwen3-Omni (3.39). SIM-o barely moves (74.1 zh, 65.1 en, the latter the best of the three modes). Non-interleaved generation, where speech is synthesised after the full text, is worse than fixed-ratio interleaving on intelligibility (1.44 CER, 2.70 WER), which the authors read as chunked streaming helping pronunciation. The authors present TAIL’s intelligibility cost as the price of temporal alignment.
Speech understanding (Table 4). Compared with Kimi-Audio and Qwen3-Omni-30B-A3B, the model is close on ASR (LibriSpeech test-clean/other 1.4/2.8 vs. 1.2/2.5 for Qwen3-Omni; best on GigaSpeech 8.5 and VoxPopuli 6.2), leads on CoVoST 2 en-to-zh (49.9), MELD (60.2), VoiceBench AlpacaEval (4.81) and Speech TriviaQA (75.5 vs. 62.9), and trails Qwen3-Omni on CoVoST 2 zh-to-en (26.4 vs. 29.4), MMAU (76.9 vs. 77.5) and Speech Web Questions (70.2 vs. 74.9), and Kimi-Audio on SpeechCMMU (59.2 vs. 67.0).
Full-duplex and omni-modal (Tables 7-8). The only full-duplex benchmark reported is LiveSports-3K-CC, which is audio-free: a 54.4 win rate against LiveCC (41.5) and StreamingVLM (45.6). The model leads on five of seven audio-visual understanding benchmarks in turn-based (simplex) mode against Qwen3-Omni-30B-A3B and Gemini 2.5 Flash. Spoken full-duplex behaviour (turn-taking, barge-in, backchannel timing, response latency) is only shown in qualitative demos.
Vision and text, briefly (Tables 2, 3, 6). OpenCompass averages are 77.6 (instruct) and 78.2 (thinking), against 78.5 and 79.9 for Gemini 2.5 Flash, with large gains in document parsing (OmniDocBench). On text benchmarks the average (82.1) is slightly above the Qwen3-8B backbone (81.6), with gains on BBH and IFEval offset by drops on MMLU (77.0 vs. 81.7) and MATH-500 (77 vs. 84).
Efficiency (Tables 11-12). With vLLM on one RTX 4090, the model runs in BF16 at 154.3 tokens/s and 19 GB, where Qwen3-Omni-30B-A3B runs out of memory; in INT4 it reaches 212.3 tokens/s, 0.58 s first-token latency (64-frame visual input) and 11 GB, against 147.8 tokens/s, 0.98 s and 20 GB. The custom llama.cpp-omni runtime brings the real-time factor to 0.21 (INT4, RTX 4090) and 0.20 (INT4, DGX Spark) at 11 GB, against 1.26-1.27 for INT4 PyTorch. These are system-level throughput and latency figures; speech-specific latency such as time to first audio or response delay after the user stops speaking is not measured.
Novelty Assessment
Most of the stack is assembled from known parts: a Whisper encoder with a projector, a Qwen3 text backbone, a CosyVoice-style S3 token and flow-matching speech path, and a separate small speech decoder driven by backbone hidden states, similar in spirit to the Thinker-Talker split of Qwen2.5-Omni. Chunked serialisation with a silence or listen token is a familiar device in audio-only full-duplex speech LMs; what is new is applying it to simultaneous video, audio and speech output in one model, with ablations on chunk size, boundary tokens and listen/speak control. TAIL is the clearest speech-generation contribution: it ties the amount of text per chunk to cumulative playback time and learns this from timestamped data, as opposed to a fixed text-to-speech token ratio. Its benefit is argued qualitatively. The paper reports only the intelligibility cost of TAIL, not a measurement of the lag it removes, so the method’s central claim has no quantitative support in the paper. The speech-side headline numbers come from the non-full-duplex fixed-ratio mode.
Field Significance
Moderate — This report shows an open 9B model that runs full-duplex listening, watching and speaking on a single consumer GPU, and it describes a concrete time-aligned interleaving scheme for keeping streamed speech synchronised with live perception. Its speech evidence is mostly turn-based TTS and understanding benchmarks against two baselines; the full-duplex speech behaviour that motivates the design is not evaluated quantitatively.
Claims
- supports: Delegating speech-token generation to a small decoder conditioned on LLM hidden states lets an omni-modal model reach dedicated-TTS-level intelligibility and speaker similarity while the backbone only emits text tokens.
Evidence: A ~0.3B speech token decoder fed with projected Qwen3-8B hidden states reaches 0.86 CER / 2.38 WER on SeedTTS test-zh/en against 1.45 / 2.57 for CosyVoice 2, with SIM-o on par (74.5 vs. 74.8, 64.9 vs. 65.2); the backbone needs only 3-4 decoding steps per second. (§2, §6.3, Table 5)
- complicates: Aligning streamed speech output to wall-clock time for full-duplex interaction costs intelligibility relative to fixed-ratio text/speech interleaving.
Evidence: On SeedTTS test sets, time-aligned interleaving (TAIL) gives 1.04 CER / 3.93 WER versus 0.86 / 2.38 for fixed-ratio interleaving, with SIM-o essentially unchanged; the English WER in full-duplex mode is worse than the CosyVoice 2 and Qwen3-Omni baselines. (§6.6, Table 10, Table 5)
- supports: Generating speech in chunks interleaved with text improves pronunciation accuracy over synthesising speech only after the full text response is complete.
Evidence: Fixed-ratio interleaving lowers CER from 1.44 to 0.86 and WER from 2.70 to 2.38 compared with non-interleaved generation, with SIM-o unchanged or slightly higher. (§6.6, Table 10)
- refines: In time-chunked full-duplex serialisation, chunk granularity trades responsiveness against output quality, and too-short chunks break stable listen/speak decisions.
Evidence: With explicit boundaries and a separate listen/speak control token, 1.0 s chunks give MMLU 0.65 and AlpacaEval 3.56, while 0.2 s chunks drop to 0.45 and 1.22 and 0.1 s chunks to 0.32 and 2.40; explicit boundaries and a decoupled control token both beat their alternatives at 1.0 s. (§3.3, Table 1)
Limitations and Open Questions
Warning
The full-duplex speech capability is not quantitatively evaluated: the only full-duplex benchmark (LiveSports-3K-CC) is audio-free, and there are no turn-taking, interruption, backchannel or speech response-latency measurements. The SeedTTS headline numbers come from the fixed-ratio mode, not the TAIL mode used in full-duplex interaction.
- All speech generation results are automatic (CER/WER, SIM-o, an undefined emotion/style control score); there is no MOS or other human listening test, and naturalness of the streamed speech is unmeasured.
- TAIL is meant to reduce the lag between perception and speech playback, but no lag or staleness metric is reported, so the benefit is asserted rather than shown.
- Speech baselines are limited to CosyVoice 2 and Qwen3-Omni for generation and Kimi-Audio and Qwen3-Omni for understanding; no dedicated full-duplex speech LM is compared.
- The authors note that speech in omni-modal streaming mode can be unstable, with mispronunciations and unintended English/Chinese code-mixing, and that proactive behaviour is still simple.
- Speech and full-duplex training data are in-house (millions of hours of pseudo-labelled speech, actor-recorded dialogues, web audio-video), and data scale and composition are only partly described.
- The speech path covers Chinese and English only; other languages are not evaluated.
Wiki Connections
- Spoken Language Model — Streams Whisper features at 10 tokens per second into a Qwen3-8B backbone that answers spoken queries and runs ASR, speech translation and audio understanding.
- Speech-to-Speech — Provides an end-to-end spoken dialogue system with full-duplex listening and speaking, extended to simultaneous video input.
- Zero-Shot TTS — Clones voices from reference audio in a multimodal system prompt and reports SIM-o on SeedTTS test sets on par with CosyVoice 2.
- Multilingual TTS — Trains and evaluates bilingual Chinese/English speech generation, with code-mixing listed as a residual failure mode.
- Emotion Synthesis — Reports emotion/style control on Expresso and ESD with neutral reference audio, ahead of CosyVoice 2.
- CosyVoice 2 — Source of the S3 token and streaming flow-matching speech path, and the main speech generation baseline.
- Qwen3-Omni — Main omni-modal baseline for speech generation, speech understanding and efficiency, which this 9B model matches or beats on most reported speech metrics.
- Mini-Omni — Representative of the fixed text/speech interleaving and backbone-generated speech tokens that TAIL and the separate speech decoder are designed to replace.
- Step-Audio 2 — Cited alongside Mini-Omni as a design where the backbone generates speech tokens directly, the approach this paper avoids.
- MiniCPM-V — Predecessor in the same model family; MiniCPM-o 4.5 is initialised from the MiniCPM-V 4.5 checkpoint and keeps its visual encoding scheme.
- Seed-TTS — Supplies the SeedTTS test-zh/en sets used for the headline intelligibility and speaker similarity results.
- MGM-Omni — Source of the LongTTS benchmark used to assess long-form speech generation stability.
- Whisper — The Whisper-Medium encoder is the model’s streaming audio encoder.