arXiv · 2020 · Preprint
Shazeer · → Paper · Demo: ? · Code: ?
This paper introduces GLU variants (GEGLU, SwiGLU, ReGLU) as drop-in replacements for the standard ReLU/GELU feed-forward sublayer in Transformers, demonstrating consistent perplexity and downstream task improvements at matched parameter and compute budgets.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
This paper introduces gated linear unit variants for the position-wise feed-forward sublayers of Transformer models, replacing the standard two-matrix FFN (with ReLU or GELU activation) with a three-matrix gated formulation where one projection acts as a multiplicative gate. The key variants — GEGLU (gating with GELU) and SwiGLU (gating with Swish) — yield lower pre-training perplexity and higher scores on GLUE, SuperGLUE, and SQuAD benchmarks at identical parameter and computation counts, achieved by reducing the hidden dimension by a factor of two-thirds to offset the extra weight matrix. Large language model backbones used in TTS and spoken conversational agent systems can incorporate SwiGLU or GEGLU FFN layers as a structural choice that provides a quality improvement without increasing model size or inference cost. The technique requires no changes to attention mechanisms or training objectives, making it straightforwardly applicable to encoder-decoder and decoder-only architectures alike.
Wiki Connections
- No in-corpus papers cited.