The paper introduces CoLLiE, a library that assists in the collaborative training of large language models. It uses 3D parallelism, parameter-efficient fine-tuning (PEFT) methods, and a range of optimizers like Lion, Adan, Sophia, LOMO, and AdaLomo. CoLLiE is designed to be efficient, user-friendly, and customizable. It has shown better training efficiency compared to other solutions in pre-training and fine-tuning scenarios. The paper also presents an evaluation of the correlation between model size and GPU memory consumption under different optimization methods, along with an analysis of the throughput. Lastly, the paper provides a detailed comparison of various optimizers and PEFT methods within the instruction-tuning context.
Publication date: 4 Dec 2023
Project Page: https://github.com/OpenLMLab/collie
Paper: https://arxiv.org/pdf/2312.00407
