arXiv · 2024 · Preprint
Wang et al. (Tencent Youtu Lab) · → Paper · Demo: ✓ · Code: ?
Freeze-Omni connects a streaming speech encoder and an autoregressive speech decoder to a text LLM whose weights remain entirely frozen throughout training, achieving low-latency duplex speech dialogue while preserving the backbone’s full question-answering intelligence.
Problem
Building speech-to-speech dialogue agents around existing text LLMs requires aligning the speech modality with the LLM’s representation space. The common approach of fine-tuning the LLM parameters risks catastrophic forgetting, particularly when only small amounts of spoken Q&A data are available. Existing open systems such as Moshi, LLaMA-Omni, and Mini-Omni2 all update LLM weights to varying degrees, and several show measurable degradation in question-answering accuracy compared to the underlying text model. A complementary limitation is latency: cascaded ASR + LLM + TTS pipelines introduce substantial interaction delay relative to end-to-end systems.
Method
Freeze-Omni is built from three independently trained modules connected to a frozen backbone LLM (Qwen2-7B-Instruct). Speech input is handled by a chunk-wise streaming speech encoder (24-layer Transformer with 4x downsampling convolutions, ~350M parameters, 12.5 Hz output frame rate), followed by an adapter that projects encoder outputs into the LLM’s embedding space. Speech output is produced by a two-stage token-based decoder (~120M parameters): a non-autoregressive (NAR) prefix speech decoder that converts LLM hidden states into a conditioning context, and an autoregressive (AR) speech decoder that generates discrete speech tokens from that context. A single-codebook TiCodec (1024 entries, 40 Hz) converts speech tokens to 24 kHz waveforms. The backbone LLM’s parameters are never updated at any point during training.
Training proceeds in three stages for each modality. For speech input: (1) CTC-based ASR pretraining of the speech encoder; (2) connecting the encoder to the frozen LLM via the adapter, trained with ASR-style supervised objectives on 110k hours of Chinese/English speech; (3) prompt-embedding fine-tuning on 60k synthesised multi-round Q&A pairs with all components except trainable prompt embeddings frozen. For speech output: (1) training a single-codebook TiCodec on speech data alone; (2) training the NAR and AR speech decoders on ~3k hours of text-speech pairs using LLM token embeddings as semantic input (LLM embedding layer frozen); (3) prefix kv-cache fine-tuning of the NAR prefix decoder on Q&A data to close the distribution gap between LLM output text tokens and the training text used in stage 2. Only the NAR prefix decoder is trainable in this final stage.
Duplex capability is achieved by adding a chunk-level state-prediction classification head to the LLM’s output layer during stage-3 training. This head predicts one of three states per chunk: continue receiving speech, interrupt and generate (state 1), or end-of-speech without interruption (state 2). Voice activity detection triggers the pipeline at the start of each utterance. A server-side scheduling strategy keeps encoder and LLM KV-caches isolated per user, allowing any model replica to serve any chunk of any active session without coordination overhead.
Key Results
On ASR benchmarks, Freeze-Omni’s speech encoder (trained with dynamic chunk) achieves 3.24% WER on LibriSpeech test-clean and 7.68% on test-other, outperforming Mini-Omni2 (4.7% / 9.4%) on both (Table 1). On Mandarin benchmarks it reaches 2.15% CER on AISHELL-1.
On spoken question answering (Table 3), Freeze-Omni (7B) achieves 44.73% on Web Questions, 72.0% on LlaMA Questions, and 53.88% on Trivia QA. Compared to Moshi (7B), which scores 26.6% / 62.3% / 22.8% on the same sets, Freeze-Omni outperforms it substantially. The gap between Freeze-Omni and its text-only backbone (Qwen2-7B-Instruct: 45.13% / 77.67% / 63.93%) is smaller than the gap between Moshi and its backbone (Helium: 32.3% / 75.0% / 56.4%), supporting the paper’s claim that the frozen-LLM approach better preserves reasoning ability.
End-to-end statistical latency (Table 4) averages 745 ms from LLM interruption to first PCM chunk output, with a 90th-percentile of 1020 ms. Adding network latency, the paper estimates real-scenario average latency at around 1.2 seconds.
Novelty Assessment
The core innovation is architectural: demonstrating that a speech-to-speech dialogue system competitive with fine-tuned alternatives can be built by keeping the LLM completely frozen. The three-stage training protocol, particularly the prefix kv-cache fine-tuning step that adapts the speech decoder to the LLM’s output distribution without touching the LLM itself, is a concrete technical contribution. The chunk-level state prediction for duplex dialogue extends an approach used in other streaming SCA systems but integrates it without altering the LLM.
The broader framing as an “engineering integration” is also apt: the speech encoder, AR/NAR decoder, and TiCodec are themselves not novel architectures. The VALL-E-inspired AR/NAR decoder pattern and the adapter-based LLM alignment are both established. The novelty lies in the training recipe that makes frozen-LLM alignment work well enough to close the intelligence gap with fine-tuned systems. Data requirements are modest by the standards of the field (60k Q&A examples, 3k hours TTS data, 8 GPUs), which is a meaningful practical contribution.
Field Significance
Tip
High: Freeze-Omni provides strong evidence that keeping the LLM backbone frozen is a viable alternative to fine-tuning for speech dialogue systems, one that preserves reasoning quality at substantially lower data cost. The frozen-LLM paradigm addresses a real constraint for researchers without access to large spoken Q&A corpora, and the question-answering results demonstrate a tighter intelligence gap than competing open systems at the time of publication. The multi-stage modular training recipe is reusable across LLM backbones.
Claims
- Freezing the LLM backbone during speech-modality alignment reduces the intelligence gap between spoken and text question-answering performance compared to fine-tuned approaches. (§3.4, Table 3)
- A three-stage training curriculum, using large ASR corpora for encoder pretraining followed by small-scale multi-modal Q&A fine-tuning, is sufficient to achieve competitive spoken dialogue quality without updating backbone LLM parameters. (§2.2.2, §2.3.2)
- Chunk-level state classification integrated into the LLM’s prefill stage enables duplex interruption detection without requiring a separate monitoring model or additional LLM context. (§2.4)
- Decoupling encoder and LLM KV-cache per user session allows a server-side pool of model replicas to handle concurrent users with chunk-granular scheduling. (§2.4)
Limitations and Open Questions
Warning
The spoken Q&A benchmarks used for intelligence comparison (Web Questions, LlaMA Questions, Trivia QA) were synthesised from text using edge-tts rather than collected from real speakers. Results on naturally spoken or noisy input are not reported, limiting generalisability claims about real-world speech understanding.
Speech output quality is evaluated primarily through CER on 1000 utterances using a single speaker, not through subjective MOS ratings or speaker naturalness benchmarks. It is therefore difficult to assess voice quality relative to other systems. The system supports a limited number of output speakers and does not yet support style or voice instruct-following, which the authors flag as future work. Emotion understanding and audio captioning are also deferred to a planned encoder upgrade. The duplex state classifier operates at chunk boundaries (approximately 160-320 ms non-statistical latency), which may be perceptible in fast-paced dialogue.
Wiki Connections
Freeze-Omni is an instance of the spoken-language-model paradigm and contributes to speech-to-speech dialogue research. The streaming, chunk-wise design connects to streaming-tts considerations. The AR/NAR speech decoder over a single-codebook codec relates to autoregressive-codec-tts and neural-codec design choices.
Related in-corpus papers: 2410.00037 (Moshi, primary comparison), 2409.06666 (LLaMA-Omni), 2410.11190 (Mini-Omni2), 2408.16725 (Mini-Omni), 2305.11000 (SpeechGPT), 2407.10759 (Qwen2-Audio), 2406.05370 (VALL-E 2), 2408.05211 (VITA).
2501.06282 — MinMo: A Multimodal Large Language Model for Seamless Voice Interaction 2502.11946 — Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction 2502.17239 — Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction 2507.16632 — Step-Audio 2 Technical Report