The paper presents a study testing the portability of task-specific knowledge encoded in modules trained by parameter-efficient finetuning (PEFT) techniques. Using sentiment analysis as an example, the authors conduct 1,440 training/testing runs, comparing the performance of ported modules with modules trained from scratch or from parameters sampled from the same distribution. The results indicate that ported modules outperform the alternatives, but the success rate varies depending on the type of PEFT and differences between the original and receiving pretrained models. The authors conclude that task-specific knowledge in the form of structurally modular sets of parameters produced by PEFT techniques is highly portable.
Publication date: 2023-04-01
Project Page: Not provided in the text
Paper: https://arxiv.org/pdf/2401.14228
