arXiv · 2025 · Preprint
MiniCPM Team et al. · → Paper · Demo: ? · Code: ✓
MiniCPM4 is a family of ultra-efficient LLMs (0.5B and 8B parameters) designed for on-device deployment, combining a trainable sparse attention mechanism (InfLLM v2), high-quality pre-training data pipelines (UltraClean), and a custom CUDA inference framework (CPM.cu) to match larger models’ performance at a fraction of the compute cost.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
MiniCPM4 provides a compact, deployable LLM backbone for systems that require language understanding and generation within tight memory and compute budgets, such as on-device spoken conversational agents. Its 0.5B and 8B variants can serve as the language model component in TTS or speech LM pipelines running on edge hardware, where cloud-based LLMs are impractical. The InfLLM v2 sparse attention mechanism enables efficient 128K-token context processing, which is relevant for speech LMs that must handle long conversational histories. The model’s quantization-aware training variant (BitCPM4, using ternary weights) further reduces the storage footprint, enabling deployment on extremely resource-constrained devices where speech generation must run locally.
Wiki Connections
- 2412.19437 (DeepSeek-V3) — MiniCPM4 draws inspiration from DeepSeek-V3’s multi-token prediction and FP8 mixed-precision training objectives, adopting both techniques in its pre-training engineering.
- 2501.12948 (DeepSeek-R1) — MiniCPM4.1, the hybrid reasoning variant, uses reinforcement learning training methods demonstrated by DeepSeek-R1, adapting chunk-wise rollout for load-balanced RL.
- 2407.21783 (Llama 3) — MiniCPM4’s efficient data verification strategy is inspired by Llama 3.1’s approach to data quality evaluation, and Llama 3 serves as a comparison baseline in evaluations.
- 2001.08361 (Scaling Laws) — MiniCPM4 applies predictable scaling via ModelTunnel v2, extending the scaling law paradigm with ScalingBench to predict downstream task performance from small-scale experiments.
- 2303.08774 (GPT-4) — GPT-4o is used to generate reasoning steps for ScalingBench construction; the paper benchmarks MiniCPM4 against GPT-4V-class systems for on-device capability.
- 2402.03300 (DeepSeekMath) — MiniCPM4’s RL post-training for reasoning uses mathematics data curation approaches consistent with DeepSeekMath’s findings on math-specific RL training.
- 2503.19786 (Gemma 3) — Gemma 3 serves as an open-source baseline in MiniCPM4’s standard evaluation comparisons for similar parameter-count models.
- 2505.09388 (Qwen3) — Qwen3-8B is the primary benchmark target: MiniCPM4-8B achieves comparable performance using only 22% of Qwen3-8B’s training tokens.
- 2408.01800 (MiniCPM-V) — MiniCPM-V is a prior multimodal model in the same MiniCPM family; MiniCPM4 extends the efficiency-focused design lineage to pure language modeling with stronger on-device inference.