arXiv · 2024 · Preprint
DeepSeek-AI et al. (DeepSeek-AI) · → Paper · Demo: ? · Code: ✓
DeepSeek-V3 is a 671B-parameter Mixture-of-Experts language model that achieves performance comparable to leading closed-source models at a total training cost of approximately $5.6M, through innovations in auxiliary-loss-free load balancing, multi-token prediction, and FP8 mixed-precision training.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
DeepSeek-V3 is a decoder-only MoE language model in the Transformer family, combining Multi-head Latent Attention (MLA) for KV-cache-efficient inference with the DeepSeekMoE architecture that activates only 37B of its 671B parameters per token. Its 128K context window and strong instruction-following capability after supervised fine-tuning and reinforcement learning alignment make it suitable as a language backbone for autoregressive speech generation pipelines that rely on a foundation LM to predict speech tokens or interleave text and audio sequences. The model’s cost-efficient training recipe and open-source weights lower the barrier for speech generation researchers to use a frontier-class LLM as a language-side component without relying on proprietary APIs. Its multi-token prediction training objective additionally enables speculative decoding, which can accelerate inference in latency-sensitive speech generation settings.
Wiki Connections
- spoken-language-model — DeepSeek-V3’s decoder-only MoE architecture and open weights make it a candidate backbone for autoregressive speech LMs that interleave text and audio tokens