Follow-the-Regularized-Leader

This paper offers a different understanding of the Adam optimizer, a popular choice in deep neural network training. Despite its practical success, the theoretical understanding of Adam’s algorithmic components has…