Speech Generation Research Wiki

VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model

4 min read

arXiv · 2025 · Preprint

Long et al. · → Paper · Demo: ? · Code: ✓

VITA-Audio introduces a Multiple Cross-modal Token Prediction (MCTP) module that generates 10 audio tokens in a single LLM forward pass, achieving zero audio token delay in streaming speech dialogue and a 3-5x inference speedup at 7B scale.

Citation Stub

This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.

Context in Speech Generation

VITA-Audio addresses a core latency bottleneck in end-to-end speech-language systems: prior interleaved-paradigm models must complete multiple LLM forward passes before producing even the first decodable audio chunk, making real-time dialogue impractical. The paper introduces lightweight MCTP modules, each a small transformer block initialized from the LLM’s final layer, that map LLM hidden states directly to subsequent audio tokens without additional LLM forward steps. By exploiting the observed monotonic alignment between text hidden states and audio tokens (a single text token’s hidden state contains sufficient context to generate corresponding audio), these modules can produce a complete audio chunk in the first forward pass, reducing first-audio latency from approximately 236 ms to 53 ms. The four-stage progressive training strategy (audio-text alignment, single-module training, multi-module expansion, supervised fine-tuning) enables the MCTP modules to converge without destabilizing the underlying LLM’s language capabilities, making the architecture applicable to backbone models ranging from 0.5B to 72B parameters.

Wiki Connections

  • spoken-language-model — VITA-Audio adopts the interleaved text-audio token paradigm, extending it with parallel MCTP prediction to remove sequential audio generation bottlenecks.
  • streaming-tts — the MCTP design directly targets streaming deployment, enabling the first decodable audio chunk to be produced during the initial LLM forward pass.
  • speech-to-speech — the model supports end-to-end speech-in, speech-out dialogue, evaluated on spoken question answering benchmarks alongside ASR and TTS tasks.
  • autoregressive-codec-tts — audio tokens are generated autoregressively at the LLM level and via MCTP modules, using discrete codec tokens (GLM-4-Voice tokenizer) decoded to waveforms.
  • 2410.00037 (Moshi) — compared directly as a parallel-paradigm end-to-end speech model; VITA-Audio adopts the interleaved paradigm instead to better preserve LLM language capabilities.
  • 2411.00774 (Freeze-Omni) — compared as a model that leverages LLM hidden states for audio generation but requires additional decoding steps; VITA-Audio achieves zero audio token delay by contrast.
  • 2408.16725 (Mini-Omni) — compared as an interleaved end-to-end model with batch parallel decoding; VITA-Audio’s MCTP approach targets the same latency problem with a different mechanism.
  • 2412.02612 (GLM-4-Voice) — VITA-Audio uses GLM-4-Voice’s speech tokenizer and decoder as its audio codec backbone in the initial model version.
  • 2407.10759 (Qwen2-Audio) — cited as a representative audio-LLM baseline in ASR evaluations; VITA-Audio-Plus-Vanilla achieves competitive WER scores against it.
  • 2407.04051 (FunAudioLLM / SenseVoice) — VITA-Audio-Plus replaces GLM-4-Voice’s tokenizer with SenseVoiceSmall from this system to improve audio encoding quality.
  • 2501.06282 (CosyVoice 2) — referenced as a state-of-the-art TTS system; VITA-Audio-Turbo achieves competitive or lower WER on Seed-TTS and LibriTTS benchmarks.
  • 2406.02430 (Seed-TTS) — the Seed-TTS benchmark serves as the primary TTS evaluation set for measuring intelligibility of VITA-Audio’s generated speech.
  • 2409.06666 (LLaMA-Omni) — compared on spoken question answering; represents an alternative end-to-end architecture using non-autoregressive audio decoding.
  • 2408.05211 (VITA-1.5) — the direct predecessor system from the same team; VITA-Audio substantially improves its inference speed and speech generation capabilities.
  • 2501.01957 (Long-VITA) — the long-context training strategy from this predecessor informs VITA-Audio’s sequence packing and positional embedding approach.
  • 2502.11946 (Step-Audio) — compared in ASR evaluations as a large-scale proprietary speech model; provides a competitive reference point on WenetSpeech and LibriSpeech benchmarks.
  • 2503.20215 (Qwen2.5-Omni) — represents the strongest open-source baseline on ASR benchmarks; VITA-Audio-Plus-Vanilla achieves competitive performance trained on open-source data only.
  • 2502.17239 (LUCY) — compared on both SQA and ASR benchmarks; represents another interleaved end-to-end speech model with batch parallel decoding for latency reduction.
  • 2412.10117 (CosyVoice 2 variant) — listed in TTS benchmark comparisons alongside Seed-TTS and GLM-4-Voice.
  • 2501.15907 (Emilia dataset) — Emilia (~96,700 hours) is the largest TTS data source used in VITA-Audio’s training pipeline.
  • 1808.10583 (AISHELL-2) — used as an ASR evaluation dataset for Mandarin speech recognition performance measurement.

Backlinks

Graph View