arXiv · 2020 · Preprint
Brown et al. · → Paper · Demo: ? · Code: ?
GPT-3, a 175-billion-parameter autoregressive language model, demonstrates that scaling dramatically improves few-shot task performance without any gradient updates or task-specific fine-tuning.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
GPT-3 establishes that a single large autoregressive language model can perform diverse tasks from natural language instructions or a handful of demonstrations at inference time, without fine-tuning. This in-context learning capability provides the conceptual foundation for conditioning TTS and spoken conversational agent systems on free-form text instructions: if a sufficiently large language model can follow task prompts zero-shot, the same mechanism can drive instruction-conditioned speech style, prosody, or speaker adaptation. Spoken conversational agent architectures that integrate a frozen or lightly adapted LLM backbone as a reasoning and response-generation component inherit this few-shot generalization property. The 175B decoder-only transformer architecture of GPT-3 also serves as the reference point against which smaller but more efficient LLM backbones used in real-time speech systems are positioned.
Wiki Connections
- spoken-language-model — GPT-3’s decoder-only autoregressive LM paradigm and in-context learning capability provide the conceptual basis for speech LM architectures that generate speech tokens via next-token prediction
- 1810.04805 (BERT) — cited as the dominant language representation model that GPT-3’s few-shot approach contrasts with and extends beyond
- 1711.05101 (AdamW) — optimizer used for GPT-3 training