arXiv · 2017 · Preprint
Loshchilov and Hutter (University of Freiburg) · → Paper · Demo: ? · Code: ✓
This paper demonstrates that L2 regularization and weight decay are not equivalent for adaptive gradient optimizers such as Adam, and proposes AdamW, which decouples the weight decay step from the gradient-based update to substantially improve generalization.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
AdamW introduces a modification to the Adam optimizer that separates the weight decay penalty from the adaptive gradient scaling step: rather than folding regularization into the gradient before the second-moment estimate adjusts it, AdamW applies weight decay directly to the parameters at a fixed rate after each update. This decoupling means all parameters are penalized uniformly regardless of their historical gradient magnitudes, restoring the regularization behavior that Adam with L2 failed to provide. TTS, VC, and SCA systems train large transformer and flow-matching models with AdamW as a default optimizer, relying on the improved generalization and more decoupled learning-rate/weight-decay hyperparameter search space it provides. The paper also demonstrates that cosine annealing combined with AdamW (AdamWR) yields strong anytime performance, a schedule pairing that appears broadly across neural speech synthesis training recipes.
Wiki Connections
In-corpus papers cited by this work: 1412.6980
Cited by: 2010.05646 (HiFi-GAN, uses AdamW optimizer) · 2410.00037 (Moshi, uses AdamW for training)