arXiv · 2024 · Preprint
Fu et al. · → Paper · Demo: ✓ · Code: ?
VITA is the first open-source multimodal LLM to simultaneously process video, image, text, and audio inputs while supporting real-time duplex interaction including non-awakening and audio-interrupt capabilities.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
VITA introduces an omni-modal architecture built on Mixtral 8x7B that integrates a 341M-parameter audio encoder (CNN + Transformer) with visual encoders and a bilingual instruction-tuned LLM via modality-specific connectors, enabling end-to-end audio query understanding across Chinese and English. The system introduces state tokens to classify incoming audio as query audio, noisy audio, or text query, which enables the model to filter environmental noise without a wake-up word, a design pattern relevant to spoken conversational agent research on always-on audio interaction. At deployment, VITA uses a duplex scheme where two model instances run concurrently: one generates responses while the other monitors for new audio queries, allowing audio interrupt interaction without blocking the dialogue flow. Speech generation papers cite VITA as a reference point for open-source omni-modal interaction architectures, particularly as a precursor to end-to-end speech output pipelines where TTS is currently handled externally but is identified by the authors as a target for integration.
Wiki Connections
- spoken-language-model — VITA’s duplex deployment and state-token approach to audio classification directly addresses challenges in building always-on spoken dialogue systems.
- speech-to-speech — VITA currently relies on an external TTS tool to produce audio responses and identifies end-to-end speech output as a key open problem, situating it at the boundary of speech-to-speech interaction research.
- 2407.21783 (LLaMA 3) — VITA cites LLaMA 3 as part of the foundation LLM landscape it builds upon, extending the Mixtral 8x7B base rather than LLaMA but positioning itself relative to this family.
- 2408.02622 (E2 TTS / related) — cited as part of the audio modality and speech generation context VITA situates itself within.
- 2312.11805 (Gemini) — cited as the closed-source proprietary model VITA benchmarks its multimodal performance against.
- 2302.13971 (LLaMA) — cited as foundational LLM work underpinning VITA’s language model starting point.
- 2307.09288 (LLaMA 2) — cited as part of the LLM lineage VITA builds on.
- 2106.06909 (WenetSpeech dataset) — used for audio alignment training, providing Chinese ASR data for VITA’s audio encoder.