This article delves into the realm of model editing in Large Language Models (LLMs). The authors seek to understand its strengths and limitations, thus facilitating robust, realistic applications of communicative AI. The research tackles three key questions: Can edited LLMs behave consistently in realistic situations? To what extent does rephrasing prompts lead LLMs to deviate from the edited knowledge memory? Which knowledge features are correlated with the performance and robustness of editing? The findings reveal a substantial disparity between existing editing methods and the practical application of LLMs, showing that more popular knowledge is easier to recall and more challenging to edit effectively.
Publication date: 9 Feb 2024
Project Page: https://github.com/xbmxb/edit_analysis
Paper: https://arxiv.org/pdf/2402.05827
