Enhancing Stochastic Gradient Descent: A Unified Framework and Novel Acceleration Methods for Faster Convergence

This article presents a unified framework to address the challenging convergence analysis under non-convex conditions in stochastic gradient descent (SGD). It interprets the updated direction as the sum of the stochastic subgradient and an additional acceleration term. The authors propose two new acceleration methods: Reject Accelerating and Random Vector Accelerating. They demonstrate that these methods can directly lead to an improvement in convergence rate.

Publication date: 2 Feb 2024
Project Page: https://arxiv.org/abs/2402.01515
Paper: https://arxiv.org/pdf/2402.01515