The study presents a systematic evaluation of how social roles in system prompts influence the performance of large language models (LLMs). The researchers curated a list of 162 roles spanning 6 types of interpersonal relationships and 8 types of occupations. They analyzed the performance of three popular LLMs over 2457 questions. The study revealed that incorporating interpersonal roles in prompts consistently improves the models’ performance. However, predicting the role that leads to the best performance remains a challenge. The findings can inform the design of system prompts for AI systems.
Publication date: 16 Nov 2023
Project Page: https://github.com/Jiaxin-Pei/Prompting-with-Social-Roles
Paper: https://arxiv.org/pdf/2311.10054
