This study proposes a method to improve optimization by coupling the gradients of private and public data via a weighted linear combination. It provides an optimal solution for the optimal weight in the convex setting, proving the acceleration in the convergence of non-convex loss and the effects of various hyper-parameters. The research supports its findings with empirical experiments across language and vision benchmarks, providing a guideline for choosing the optimal weight of the gradient coupling.
Publication date: 2 Oct 2023
Project Page: https://arxiv.org/abs/2310.01304v1
Paper: https://arxiv.org/pdf/2310.01304
