arXiv · 2025 · Preprint
Li et al. (Baichuan Inc.) · → Paper · Demo: ? · Code: ?
Baichuan-Omni-1.5 is an omni-modal model that processes text, image, audio, and video inputs and produces both text and speech outputs end-to-end, built around a custom 8-layer RVQ audio tokenizer (Baichuan-Audio-Tokenizer) and a flow-matching-based audio decoder.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
Baichuan-Omni-1.5 demonstrates one approach to integrating speech input and output into a multimodal large language model without cascading separate ASR and TTS modules. The model introduces the Baichuan-Audio-Tokenizer, an 8-layer RVQ codec operating at 12.5 Hz that jointly captures semantic and acoustic information by supervising both a Whisper-based Mel reconstruction objective and a transcript prediction objective. The audio decoder refines tokenizer outputs using a flow-matching model following the design of Matcha-TTS and CosyVoice, then converts Mel spectrograms to waveforms via HiFi-GAN, enabling controllable bilingual (Chinese and English) speech generation. The training strategy progressively incorporates modalities across four stages, specifically to mitigate the modality conflicts that arise when audio and vision are jointly trained alongside text, a design challenge of direct relevance to spoken conversational agent research.
Wiki Connections
- neural-codec — Baichuan-Audio-Tokenizer is an 8-layer RVQ codec designed to balance semantic and acoustic information for integration with a large language model backbone.
- flow-matching — the audio decoder uses a flow-matching model (U-Net architecture) to generate Mel spectrograms from audio tokens, following established flow-matching vocoder designs.
- spoken-language-model — Baichuan-Omni-1.5 extends a decoder-only LLM to interleave text and audio token generation, placing it within the speech language model paradigm.
- speech-to-speech — the system supports end-to-end speech-in, speech-out interaction as an alternative to cascaded ASR+TTS pipelines.
- 2303.08774 (GPT-4 Technical Report) — cited as the proprietary reference point that motivates omni-modal capability benchmarking.
- 2210.13438 (EnCodec) — prior RVQ codec work that informs the Baichuan-Audio-Tokenizer design.
- 2210.02747 (Flow Matching for Generative Modeling) — foundational flow-matching method used in the audio decoder.
- 2407.05407 (CosyVoice) — the audio decoder U-Net architecture follows CosyVoice’s flow-matching decoder design.
- 2410.00037 (Moshi) — a competing speech-text foundation model; Baichuan-Omni-1.5 pursues a similar end-to-end speech dialogue capability.
- 2409.06666 (LLaMA-Omni) — related work on seamless speech interaction with large language models; AlpacaEval audio benchmark setup follows this paper.
- 2408.05211 (VITA) — used as a baseline omni-modal model in comparative evaluations.
- 2501.01957 (VITA-1.5) — used as a primary baseline in image, video, audio, and omni-modal benchmarks.
- 2408.16725 (Mini-Omni) — related work on real-time speech input and output integrated with a language model.
- 2410.11190 (Mini-Omni2) — related work on open-source GPT-4o-style omni-modal dialogue.
- 2407.10759 (Qwen2-Audio) — open-source audio language model used for context and comparison.
- 2311.07919 (Qwen-Audio) — earlier Qwen audio model cited as related work in audio language modelling.
- 2407.08551 (VALL-E 2 / LauraGPT) — cited in the context of audio generation head design.
- 2412.02612 (GLM-4-Voice) — used as a baseline in audio understanding evaluations.
- 2411.17607 (Scaling Speech-Text Pre-training) — cited in relation to speech-text interleaved training strategies.
- 2309.16609 (Qwen Technical Report) — foundational LLM series underlying the Baichuan model family comparisons.
- 2412.15115 (Qwen2.5 Technical Report) — cited in the introduction as a reference for LLM progress.
- 2408.01800 (MiniCPM-V) — compared as a vision-language baseline.