This research paper focuses on the resilience of Large Language Models (LLMs), particularly GPT-4, when subjected to scrambled text. The authors introduce the ‘Scrambled Bench’ to test the LLMs’ ability to recover scrambled sentences and answer questions given scrambled context. The results indicate that GPT-4 almost flawlessly processes inputs with unnatural errors, even under extreme conditions, and can almost perfectly reconstruct original sentences from scrambled ones, reducing the edit distance by 95%. This ability of GPT-4 is counter-intuitive given the severe disruption to input tokenization caused by scrambled text.
Publication date: 1 Dec 2023
Project Page: https://github.com/ccqq77/unnatural-error-correction
Paper: https://arxiv.org/pdf/2311.18805
