arXiv · 2024 · Preprint
Xie et al. (Inspirai / Tsinghua University) · → Paper · Demo: ✓ · Code: ✓
Mini-Omni2 extends the Mini-Omni speech-text agent to handle three input modalities (vision, audio, text) and adds a command-based semantic interruption mechanism, providing an open-source approximation of GPT-4o’s vision-speech-duplex capabilities in a 0.5B parameter model.
Problem
Building open-source counterparts to proprietary multimodal systems like GPT-4o is difficult because integrating vision, speech, and text modalities into a single end-to-end model requires aligning heterogeneous encoders, managing data complexity across modalities, and enabling low-latency streaming output with real-time duplex interaction. Prior work tended to focus on individual modality pairs (vision-text or audio-text) or relied on cascaded TTS pipelines that do not support end-to-end audio generation.
Method
Mini-Omni2 builds on the Mini-Omni framework by adding a visual modality to an existing audio-text spoken conversational agent. The backbone is Qwen2-0.5B, augmented with two pretrained encoders: CLIP ViT-B/32 for vision (producing a 50-element feature sequence) and Whisper-small for audio input (operating as a continuous feature extractor rather than a discrete tokenizer). Both encoders are connected to the language model via lightweight LlamaMLP adapters. Audio output uses the SNAC codec with a seven-layer RVQ codebook; during generation, the model produces eight tokens per step (one text token plus seven audio tokens across codebook layers) using a delayed parallel decoding scheme inherited from Mini-Omni, which enables real-time streaming speech responses.
Training proceeds in three stages. Stage 1 trains only the adapter weights on speech recognition and image captioning data, aligning multimodal feature distributions to the text embedding space without modifying the language model. Stage 2 unfreezes the language model and trains on multimodal question-answering tasks with text-only outputs, establishing cross-modal reasoning capability. Stage 3 adds audio output targets and introduces the interruption mechanism training. Throughout all stages, a text-to-text question-answering task is included to prevent forgetting of the language model’s base capabilities.
The duplex interaction mechanism uses a command-based approach rather than full-duplex VAD: the model receives streaming SNAC-encoded tokens of the user’s audio in real time and emits frame-level irq (interrupt) or n-irq (no interrupt) state tokens. Training data for this task pairs noise-augmented audio segments containing the phrase “Stop Omni” (synthesised with CosyVoice in varied voice timbres) with appropriate state labels; the model learns to stop audio generation and return to listening when it detects a semantically appropriate interrupt cue.
Key Results
The paper reports WER on LibriSpeech as the primary quantitative evaluation. Mini-Omni2 achieves 4.8 / 9.8 WER on test-clean / test-other, compared to Whisper-small at 4.4 / 10.1 and Mini-Omni at 4.5 / 9.7 (Table 2). Adding the visual modality causes a slight WER increase relative to Mini-Omni, which the authors attribute to the reduced proportion of audio training data. On the test-other split, Mini-Omni2 slightly outperforms the Whisper-small baseline, suggesting some robustness benefit from the broader training distribution. No naturalness MOS or speaker similarity scores are reported for the speech output; no vision benchmark results are included in this version (“will be updated shortly”). Qualitative evaluation is provided via case studies in Figure 6.
Novelty Assessment
The contribution is engineering integration. All core components (CLIP, Whisper, Qwen2, SNAC, delayed parallel decoding) are established; Mini-Omni2’s contribution is assembling them into a functional tri-modal end-to-end spoken conversational agent at a very small model scale. The three-stage training recipe follows the same modality expansion logic as Mini-Omni and other alignment-then-generation pipelines. The semantic interruption mechanism is the most distinctive element: it replaces VAD-style duplex triggering with a learned token-level intent classifier, but it is demonstrated only on a single-command (“Stop Omni”) synthetic dataset. The paper is openly released and positioned as a reference implementation rather than a state-of-the-art system; the authors explicitly defer comprehensive evaluation of vision and speech generation quality to future updates.
Field Significance
Moderate — Mini-Omni2 provides a compact, fully open-source baseline that unifies vision, audio, and text modalities with streaming speech output and a semantic interruption mechanism. It demonstrates that a 0.5B model can be extended to three-modal understanding and audio generation through staged adapter training on a relatively small dataset, which can serve as a reference point for resource-constrained multimodal SCA research. The evaluation is limited and the quantitative claims are narrow, so its primary value is as an engineering template and reproducibility reference rather than as evidence of a new capability frontier.
Claims
- Staged adapter training (encoder alignment before language model fine-tuning) enables tri-modal extension of a compact language model with minimal data without catastrophic forgetting of the base model’s text capabilities. (§3.3)
- Using continuous encoder features (Whisper) rather than discrete audio tokens for speech input yields more stable and semantically consistent representations, reducing ASR loss instability during training. (§3.1, “Audio Encoder”)
- Adding a third modality (vision) to an audio-text spoken conversational agent modestly degrades ASR performance, likely due to diluted training data proportion rather than architectural interference. (§4.4, Table 2)
- Command-based semantic interruption (intent token classification) provides a viable alternative to VAD-based full-duplex detection, with the advantage of robustness to noise and unrelated background sounds. (§3.4)
Limitations and Open Questions
Warning
Evaluation coverage is incomplete: no naturalness MOS, SMOS, or intelligibility metrics for speech output are reported in this version, and vision benchmark results are explicitly deferred. Claims about speech quality and vision understanding capability cannot be independently verified from this paper alone.
The interruption mechanism is demonstrated on a single synthesised phrase (“Stop Omni”) with a narrow distribution of noise conditions. Whether the approach generalises to arbitrary semantic interrupt commands or real conversational interruption patterns is an open question. The model is trained and evaluated exclusively on English data despite using multilingual Whisper; cross-lingual transfer to speech output is not evaluated. The 0.5B model scale is deliberately small, and the authors note that scaling data and compute would likely yield substantial capability gains, but this is not demonstrated. Synthetic data is used for spoken question-answering and interruption training, and the effect of synthetic-to-real domain mismatch on deployment robustness is not assessed.
Wiki Connections
- spoken-language-model — end-to-end audio-in audio-out conversational agent; extends Mini-Omni with vision modality
- speech-to-speech — takes Whisper-encoded audio input and produces SNAC-decoded audio output end-to-end
- neural-codec — uses SNAC (seven-layer RVQ) for discrete audio output generation
- streaming-tts — delayed parallel decoding scheme produces streaming speech responses in real time
- autoregressive-codec-tts — AR generation of SNAC codebook tokens from text and audio context
- 2408.16725 (Mini-Omni) — direct predecessor; Mini-Omni2 adds vision modality and semantic interruption
- 2410.00037 (Moshi) — concurrent full-duplex speech-text model; contrasted on duplex interaction design
- 2409.06666 (LLaMA-Omni) — concurrent speech interaction work using a different backbone scale
- 2408.02622 (LSLM) — concurrent duplex listening-while-speaking model
- 2308.16692 (SpeechTokenizer) — codec used in related SpeechLM systems
- 2210.13438 (EnCodec) — foundational neural codec referenced for SNAC lineage
- 2407.05407 (CosyVoice) — used to synthesise “Stop Omni” interrupt training data
- 2301.02111 (VALL-E) — foundational codec LM establishing discrete token generation for speech
- 2305.11000 (SpeechGPT) — early speech-in speech-out LLM; contextualises the SpeechLM lineage
- 2412.15649 (SLAM-Omni) — cites Mini-Omni2 as a same-scale SDM baseline for comparison
- 2502.06490 (Discrete Speech Tokens Review) — reviews Mini-Omni2 as an SpeechLM system using discrete audio tokens