This research paper discusses the Backpack, an alternative to the Transformer model, and its application in Chinese language modeling. The Backpack improves interpretability in English language modeling by breaking down predictions into a weighted sum of token sense components. This study trains, evaluates, interprets, and controls Backpack language models in character-tokenized Chinese, where words are made up of many characters. The Chinese Backpack language model performs comparably to a Transformer and learns rich character-level meanings. The study also shows that gender bias can be localized to specific character senses and intervened to reduce the bias.
Publication date: 19 Oct 2023
Project Page: https://arxiv.org/abs/2310.12751v1
Paper: https://arxiv.org/pdf/2310.12751
