The article studies the impact of overparameterization on the complexity and sensitivity of the function represented by neural networks. It discusses the ‘double-descent’ phenomenon, where highly overparameterized models escape overfitting and achieve good test performance. The focus is on the Boolean mean dimension (BMD), a metric used in Boolean function analysis. The authors use a theoretical analysis based on the replica method to understand the behavior of BMD in the high dimensional regime. They find that as the degree of overparameterization increases, the BMD reaches a peak at the interpolation threshold, which corresponds with the generalization error peak, and then slowly approaches a low asymptotic value.
Publication date: 23 Jan 2024
Project Page: https://arxiv.org/abs/2401.12610
Paper: https://arxiv.org/pdf/2401.12610
