arXiv · 2016 · Preprint
Ba et al. · → Paper · Demo: ? · Code: ?
Introduces layer normalization, which computes mean and variance statistics across all hidden units within a single layer and training case rather than across a mini-batch, enabling stable training of recurrent neural networks without batch-size constraints.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
Layer normalization addresses a fundamental limitation of batch normalization: its dependence on mini-batch statistics makes it ill-suited for recurrent neural networks and small-batch training regimes. By computing normalization statistics over all neurons in a layer at a single time step, the method provides identical behaviour at training and inference time and applies naturally to sequence models where input lengths vary. Transformer-based TTS and VC architectures, which process variable-length text and audio sequences through stacked self-attention layers, incorporate layer normalization as a standard component of each sublayer. The technique’s invariance to re-scaling of individual training cases and its implicit learning rate stabilisation also benefit the deep acoustic models used in neural codec and speech language model training pipelines.
Wiki Connections
- 1412.6980 (Adam optimizer) — Layer Normalization uses Adam as the optimizer in its RNN experiments, and both techniques are now standard components of transformer-based speech generation architectures.