The article presents a study on the issue of model collapse in peer collaborative learning (PCL). The authors analyze the high homogenization between the student and teacher as the cause and propose a solution – a decoupled knowledge for online knowledge distillation generated by an independent teacher. This design increases the diversity between networks and reduces the possibility of model collapse. The authors also present an initialization scheme for the teacher and a decaying ensemble scheme to improve the teacher’s supervisory resilience. The method’s effectiveness is demonstrated through experiments on CIFAR-10, CIFAR-100, and TinyImageNet.
Publication date: 18 Dec 2023
Project Page: https://github.com/shaoeric/Decoupled-Knowledge-with-Ensemble-Learning-for-Online-Distillation
Paper: https://arxiv.org/pdf/2312.11218
