arXiv · 2014 · Preprint
Kingma and Ba · → Paper · Demo: ? · Code: ?
Adam is a first-order stochastic optimization algorithm that computes per-parameter adaptive learning rates from exponential moving averages of the gradient and squared gradient, with bias-correction to stabilise early training.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
Adam introduces adaptive moment estimation (from which its name derives): it maintains running estimates of the first moment (mean) and second moment (uncentered variance) of the gradients, applies bias correction to counteract the zero-initialization of these estimates, and uses the ratio of the corrected moments to scale each parameter’s update. The design inherits the sparse-gradient handling of AdaGrad and the non-stationary robustness of RMSProp, while adding bias correction that improves stability when the second-moment decay rate is close to 1. TTS and SCA papers cite Adam as the optimiser used to train their models, treating its default hyperparameters (alpha=0.001, beta1=0.9, beta2=0.999) as a reliable starting point for large-scale neural model training. It appears across the corpus in acoustic models, vocoders, codec encoders, speaker encoders, and language model fine-tuning, reflecting its role as the default gradient-based optimiser for neural speech generation research.
Wiki Connections
No in-corpus papers cited by this paper.
Cited by: 2006.04558 (FastSpeech 2, uses Adam for training)