arXiv · 2020 · Preprint
Kaplan et al. (OpenAI) · → Paper · Demo: ? · Code: ?
Empirical power-law scaling relationships between language model loss and three factors — model size, dataset size, and compute budget — spanning more than seven orders of magnitude, providing a principled framework for allocating training resources.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
This paper establishes that Transformer language model performance follows predictable power-law relationships with model parameter count, dataset token count, and training compute, independently of most architectural hyperparameters such as depth or width. A key practical finding is that compute-optimal training requires training much larger models for fewer steps than convergence would demand, scaling model size faster than data size. Speech generation systems that use autoregressive language models over discrete audio tokens — including codec-based TTS and spoken language model architectures — can draw on these relationships to guide decisions about model scale, training data size, and compute allocation. The framework also provides a principled basis for the trend toward very large language model backbones in both TTS and spoken conversational agent pipelines.
Wiki Connections
- autoregressive-codec-tts — scaling law predictions for LM loss directly inform how autoregressive codec TTS systems choose model size and training compute budgets.
- spoken-language-model — spoken language models operating over discrete speech tokens inherit the LM scaling regime this paper characterises, making its compute-optimal training prescriptions directly applicable.