arXiv · 2024 · Preprint
Fang et al. (ICT/CAS) · → Paper · Demo: ? · Code: ✓
LLaMA-Omni is an end-to-end speech interaction model that attaches a non-autoregressive CTC-based streaming speech decoder to Llama-3.1-8B-Instruct, enabling simultaneous text and speech response generation from speech instructions with a minimum latency of 236 ms — trained in under 3 days on 4 GPUs.
Problem
Enabling real-time speech interaction with LLMs has two competing failure modes. Cascaded ASR-LLM-TTS pipelines add latency from sequential processing and introduce error propagation between modules. Fully end-to-end speech-token LLMs (SpeechGPT, AudioPaLM) require large-scale speech-text pretraining and either sacrifice response quality (when generating units directly without intermediate text) or sacrifice latency (when generating intermediate text before speech). Neither approach achieves low latency, high quality, and efficient training simultaneously using open-source LLMs.
Method
LLaMA-Omni consists of four components chained together: a frozen Whisper-large-v3 encoder, a trainable speech adaptor, Llama-3.1-8B-Instruct, and a streaming speech decoder. The speech adaptor applies 5x temporal downsampling followed by a 2-layer MLP to project Whisper representations into the LLM’s embedding space. The LLM then autoregressively generates a text response directly from speech input, bypassing transcription. The speech decoder runs in parallel with the LLM: it is a non-autoregressive (NAR) Transformer (2 layers, 4096 hidden dim, 425M parameters) that takes the LLM’s output hidden states, upsamples them by a factor of 25, and uses CTC to predict HuBERT discrete units. A unit-based HiFi-GAN vocoder with a duration predictor then converts discrete units to waveforms. Because the speech decoder uses causal attention and operates on hidden states produced token by token, it can begin emitting unit chunks before the full text response is complete — enabling streaming synthesis.
Training follows two stages. In stage 1, only the speech adaptor and LLM are trained (speech encoder frozen) on text response generation from speech input using cross-entropy loss. In stage 2, only the speech decoder is trained on CTC loss with the rest frozen, learning the alignment from LLM hidden states to HuBERT unit sequences. To provide suitable training data, the authors construct InstructS2S-200K: 200K speech instruction-response pairs derived from Alpaca and UltraChat text data by rewriting instructions for spoken style (via Llama-3-70B-Instruct) and synthesising responses to speech using VITS trained on LJSpeech. The full training takes approximately 65 hours on 4 NVIDIA L40 GPUs.
Key Results
On the InstructS2S-Eval benchmark (199 instructions from AlpacaEval), LLaMA-Omni achieves a ChatGPT Score of 3.99 on S2TIF (speech-to-text instruction-following) and 3.47 on S2SIF (speech-to-speech), outperforming SpeechGPT (2.98/2.19), SALMONN+Orca (3.44/3.40), and Qwen2-Audio+Orca (3.47/3.38) in the offline scenario. UTMOS for the proposed system is 3.93, marginally above all baselines. ASR-WER is 10.82%, better than SpeechGPT (45%) but higher than the industrial TTS cascades (3.78% for SALMONN+Orca), reflecting the gap in scale between InstructS2S-200K (~1K hours of response speech) and industrial TTS training data.
In the streaming scenario, LLaMA-Omni achieves a minimum latency of 236 ms (unit chunk size Omega=10), below GPT-4o’s reported average audio latency of 320 ms. Critically, unlike cascaded systems where reducing the word chunk size degrades speech rate and naturalness, LLaMA-Omni maintains stable prosody across latency settings because streaming unit generation is end-to-end — only the final vocoder step is cascaded. Human evaluation (5 raters, 20 instructions per pair) confirms higher win rates for LLaMA-Omni over both cascaded baselines in both helpfulness and naturalness.
Novelty Assessment
The key architectural novelty is the CTC-based NAR streaming speech decoder that runs in parallel with autoregressive LLM decoding, consuming the LLM’s hidden states directly without requiring explicit text-speech pre-alignment. This allows the system to bypass the full transcription-intermediate-text pipeline and enables streaming without the prosody fragmentation seen in word-chunked TTS cascades. The component choices (Whisper encoder, Llama backbone, HuBERT units, HiFi-GAN vocoder) are all established; the contribution is in the integration architecture and the specific CTC alignment approach between LLM hidden states and speech units. The InstructS2S-200K dataset is a useful methodological contribution — the careful reformulation of text instruction data into a speech-appropriate style addresses a real gap in training data for this paradigm. The training efficiency claim (4 GPUs, 3 days) is genuine and meaningful for the open-source community, though it relies on the capabilities already encoded in Llama-3.1-8B-Instruct.
Field Significance
Tip
High — LLaMA-Omni demonstrates that an open-source, resource-efficient SCA system can match or approach proprietary systems like GPT-4o on response latency while maintaining speech quality, using only 200K training examples and commodity hardware. The CTC-based parallel speech decoder design establishes a practical template for LLM-speech integration that decouples text reasoning quality from speech generation throughput.
Claims
- End-to-end speech LLMs with parallel text and speech generation can achieve lower response latency than cascaded ASR-LLM-TTS pipelines without sacrificing prosody coherence under low-latency streaming conditions. (§4.5)
- Aligning LLM output to speech interaction conventions through targeted instruction data rewriting substantially improves response style suitability, independently of model architecture. (§3, §4.4)
- Non-autoregressive CTC decoding from LLM hidden states enables streaming speech synthesis whose speech rate and naturalness are robust to chunk size variation, unlike word-level streaming TTS cascades. (§4.5, Table 4)
- Training an end-to-end speech interaction model on a small, carefully curated speech instruction dataset is sufficient to significantly close the gap with models trained on orders of magnitude more data, provided the LLM backbone is sufficiently capable. (§4.4, §5)
Limitations and Open Questions
Warning
The ASR-WER of 10.82% is notably higher than cascaded baselines (3.78% for SALMONN+Orca), reflecting that the speech decoder is trained on only approximately 1K hours of response speech — far below industrial TTS scale. Intelligibility limitations restrict applicability in domains requiring precise spoken content.
The evaluation benchmark (InstructS2S-Eval) is derived from AlpacaEval with math and code questions removed, which skews toward conversational helpfulness and may not represent more demanding speech interaction tasks. The speech encoder relies on Whisper, which is optimised for ASR rather than general speech understanding, potentially limiting response to prosodic or para-linguistic cues in the user’s speech. The current architecture does not support full-duplex interaction (interruption, turn-taking) — speech responses are generated after the full instruction is received. The training data is synthesised from text corpora, which may not capture the naturalness and variability of real spoken dialogue.
Wiki Connections
This paper’s architecture directly interfaces with neural-codec (HuBERT discrete units as the speech representation backbone) and spoken-language-model (LLM-native speech generation). The streaming synthesis component connects to streaming-tts. The speech-to-speech instruction-following paradigm is central to speech-to-speech and instruction-conditioned-tts. The use of HuBERT for unit extraction makes self-supervised-speech a core dependency of the proposed system.
In-corpus related papers: 2301.02111 (VALL-E, foundational neural codec LM for TTS), 2407.05407 (CosyVoice, used for instruction speech synthesis in InstructS2S-200K), 2407.10759 (Qwen2-Audio, cascaded baseline), 2408.16725 (Mini-Omni, contemporary parallel work on omni-modal LLMs), 2306.12925 (AudioPaLM, prior speech-text LLM).
Cited by in-corpus papers: 2411.00774 (Freeze-Omni cites LLaMA-Omni as a key prior work for speech interaction with frozen LLMs) · 2410.17196 (VoiceBench evaluates LLaMA-Omni as a representative end-to-end audio LLM)
2501.06282 — MinMo: A Multimodal Large Language Model for Seamless Voice Interaction 2502.11946 — Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction 2502.17239 — Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction 2507.16632 — Step-Audio 2 Technical Report