The paper presents an in-depth analysis of the joint effect of task similarity and overparameterization on catastrophic forgetting in continual learning. The researchers focus on two-task continual linear regression, where the second task is a random orthogonal transformation of the first task. The study found that in highly overparameterized models, intermediate task similarity causes the most forgetting. However, near the interpolation threshold, forgetting decreases with the expected task similarity. The findings were validated using linear regression on synthetic data and neural networks on established permutation task benchmarks.
Publication date: 2024-01-23
Project Page: https://arxiv.org/abs/2401.12617
Paper: https://arxiv.org/pdf/2401.12617
