arXiv · 2025 · Preprint
Fu et al. · → Paper · Demo: ✓ · Code: ✓
VITA-1.5 is a multimodal LLM that integrates vision, language, and speech through a three-stage training strategy, enabling end-to-end speech-to-speech interaction alongside image and video understanding without relying on separate ASR or TTS modules.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
VITA-1.5 demonstrates an end-to-end architecture for joint vision-language-speech understanding and generation, directly relevant to the design of real-time spoken conversational agents. The system combines a 350M-parameter speech encoder (outputting at 12.5 Hz), a vision encoder (InternViT-300M), a Qwen2-7B LLM backbone, and an audio decoder built around TiCodec, a single-codebook neural codec operating at 40 Hz and 24 kHz. Speech output is generated through a non-autoregressive followed by an autoregressive speech decoder appended to the LLM, avoiding cascaded ASR-TTS pipelines that introduce latency. The three-stage training recipe (vision alignment, audio input tuning, audio output tuning) provides a pattern for progressively adding speech capability to a vision-language model while minimising cross-modal training conflicts, which systems targeting combined visual and speech interaction can adopt.
Wiki Connections
- spoken-language-model — VITA-1.5 extends the speech LM paradigm to include visual input and end-to-end speech output, providing an approach for integrating all three modalities without cascaded modules.
- speech-to-speech — the paper introduces a NAR+AR speech decoder chain that enables direct speech token generation from the LLM, supporting real-time speech-to-speech dialogue.
- 2408.05211 (VITA-1.0) — VITA-1.5 is the direct successor to VITA-1.0, replacing its external TTS module with an end-to-end speech decoder and adopting a refined three-stage training strategy.
- 2305.11000 (SpeechGPT) — cited as prior work on empowering LLMs with intrinsic speech interaction; VITA-1.5 extends this direction to include visual modality alongside speech.
- 2302.13971 (LLaMA) — VITA-1.5 builds on LLaMA-family foundation LM research, contextualising its Qwen2-7B backbone within the broader open-source LLM lineage.
- 2307.09288 (LLaMA 2) — cited as part of the foundation LM stack that enabled multimodal extension work of this type.
- 2410.11190 (Mini-Omni2) — compared as a competing open-source omni model lacking visual capability; VITA-1.5 is compared against it on ASR benchmarks.
- 2409.06666 (LLaMA-Omni) — cited as a contemporaneous speech-enabled LLM without vision; serves as a baseline and contextual reference for the speech interaction direction.
- 2410.00037 (Moshi) — cited as another open-source real-time speech dialogue system; VITA-1.5 distinguishes itself by adding vision understanding.
- 2312.11805 (Gemini) — used as a proprietary closed-source baseline for image and video benchmark comparisons.
- 2411.00774 (Freeze-Omni) — compared on ASR and used as an architectural reference for the NAR+AR speech decoder training strategy.
- 2106.06909 (GigaSpeech) — used as part of the Mandarin speech evaluation benchmarks in the ASR evaluation section.