The research presents a novel tool called ‘time vectors’ for customizing language models according to specific time periods. This is achieved by finetuning a language model on data from a single time period and then subtracting the weights of the original pretrained model. The resultant vector specifies a direction in the weight space that improves performance on text from that time period. The researchers found that time vectors for adjacent time periods are positioned closer together, and used this structure to create new models that perform better on intervening and future time periods, without additional training. The results were consistent across different tasks, domains, model sizes, and time scales, suggesting that time is encoded in the weight space of finetuned models.
Publication date: 20 Dec 2023
Project Page: https://arxiv.org/abs/2312.13401
Paper: https://arxiv.org/pdf/2312.13401
