The paper ‘Unlearnable Algorithms for In-context Learning’ focuses on the concept of machine unlearning, a process that modifies a model to behave as if it were trained without including a certain datapoint. This is particularly useful when data of unknown provenance is involved. The researchers propose an efficient unlearning method for the task adaptation phase of a large language model (LLM). The ERASE algorithm is introduced, which enables efficient exact unlearning of task adaptation training data. The cost of unlearning operation using ERASE is independent of model and dataset size. The paper also discusses the trade-offs between in-context learning and fine-tuning approaches.
Publication date: 1 Feb 2024
Project Page: https://arxiv.org/abs/2402.00751v1
Paper: https://arxiv.org/pdf/2402.00751
