This article presents a detailed study on the factors influencing social biases in Masked Language Models (MLMs) and their performance in downstream tasks. The researchers studied 39 pretrained MLMs covering various factors like model size, training objectives, tokenization methods, training data domains, and languages. They found that factors often neglected in previous studies, like tokenization or model objectives, have a significant impact on the social biases learned by MLMs. They used Gradient Boosting to consider dependencies among factors and used the coefficient of determination (R2) to analyze the importance of factors. The study also highlights the need for debiasing methods to ensure fair and unbiased MLMs, pointing out the trade-off between social bias and downstream task performance.
Publication date: 20 Oct 2023
Project Page: Not provided
Paper: https://arxiv.org/pdf/2310.12936
