CoLLiE: Collaborative Training of Large Language Models in an Efficient Way
The paper introduces CoLLiE, a library that assists in the collaborative training of large language models. It uses 3D parallelism, parameter-efficient fine-tuning (PEFT) methods, and a range of optimizers like…
Continue reading