arXiv · 2025 · Preprint
Microsoft et al. (Microsoft) · → Paper · Demo: ? · Code: ?
Introduces Phi-4-Mini (3.8B language model) and Phi-4-Multimodal (5.6B), which extend the Phi-4 base via modality-specific LoRA adapters for vision and speech/audio inputs without modifying the frozen language model backbone.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
Phi-4-Multimodal provides a compact speech-and-audio-capable language model through a “Mixture of LoRAs” design: a 460M-parameter conformer-based audio encoder projects log-Mel features into the text embedding space, and a separate 460M-parameter LoRA module applied to the language decoder enables speech instruction following, ASR, translation, summarisation, and audio understanding — all without touching the frozen Phi-4-Mini weights. The speech audio modality trains on approximately 2M hours of speech-text pairs for encoder-language alignment, followed by roughly 100M curated SFT examples covering ASR, translation, spoken question answering, and speech summarisation. This architecture enables spoken conversational agents and speech LMs to adopt a small yet capable language backbone for tasks that require tight integration of speech comprehension and language reasoning, with the frozen-backbone design offering a practical strategy for maintaining text quality while adding audio capabilities.
Wiki Connections
- spoken-language-model — Phi-4-Multimodal’s speech/audio LoRA stream enables spoken instruction following and spoken QA, making it a usable backbone for speech LM research.
- self-supervised-speech — the audio encoder is initialised from a pre-trained attention-based encoder-decoder ASR model rather than a self-supervised model, so this slug applies only marginally; included because the conformer encoder architecture overlaps with SSL encoder designs commonly referenced in the TTS/SCA community.
- 2308.11596 (SeamlessM4T) — Phi-4-Multimodal is compared against SeamlessM4T v2 on multilingual ASR and translation benchmarks as a strong open-source baseline.
- 2407.10759 (Qwen2-Audio) — cited as a contemporary multimodal speech-language model; Phi-4-Multimodal is benchmarked alongside it on speech and audio understanding tasks.
- 2407.21783 (LLaMA 3) — referenced for the LlamaVision cross-attention approach that Phi-4-Multimodal’s LoRA design is contrasted against.
- 2501.12948 (DeepSeek-R1) — used as a reasoning benchmark target; the experimental Phi-4-Mini reasoning variant is compared against DeepSeek-R1-Distill models of similar scale.
- 2410.21276 (GPT-4o) — used as a closed-source upper-bound comparison on both vision-language and speech/audio benchmarks.
- 2312.11805 (Gemini) — used as a closed-source comparison on vision-language, vision-speech, and speech/audio evaluation tasks.
- 2410.11190 (Mini-Omni2) — cited as a contemporary open-source multimodal model with speech and duplex capabilities.
- 2412.02612 (GLM-4-Voice) — cited as a related speech-language model in the multimodal architecture discussion.
- 2412.15115 (Qwen2.5) — referenced in the context of competing small language models for language benchmark comparison.
- 2410.19168 (MMAU) — used as an audio understanding evaluation benchmark for Phi-4-Multimodal.