The paper evaluates the rationality and cognitive biases of seven Large Language Models (LLMs) using tasks from cognitive psychology. The LLMs display irrationality in these tasks, often providing incorrect answers that differ from human-like biases. The paper also highlights the significant inconsistency in the LLMs’ responses. The study aims to provide a methodology to assess and compare the capabilities of these models, particularly with respect to rational reasoning. The authors argue for species-fair comparisons between humans and machines, suggesting that different approaches should be taken to evaluate cognitive and foundation models.
Publication date: 14 Feb 2024
Project Page: https://arxiv.org/pdf/2402.09193v1.pdf
Paper: https://arxiv.org/pdf/2402.09193
