This research proposes using small pretrained foundational generative language models as a general learning framework for sequence-based tasks. The approach overcomes challenges associated with training neural networks and language models from scratch. The paper suggests creating small, highly specialized models that can execute challenging tasks that the base model cannot. The study shows that these language models can be fine-tuned with instruction examples to achieve near state-of-the-art results on challenging cheminformatics tasks. The research also highlights the importance of data formatting and pretrained foundational language model selection for successful fine-tuning.
Publication date: 9 Feb 2024
Project Page: https://arxiv.org/abs/2402.05616v1
Paper: https://arxiv.org/pdf/2402.05616
