The research introduces LumiNet, a new knowledge-transfer algorithm designed to improve logit-based knowledge distillation. It aims to boost the student model’s learning depth by recalibrating logits based on the model’s representation capability. LumiNet reconstructs inter-class relationships, allowing the student model to gain a broader knowledge scope. The algorithm’s effectiveness is demonstrated through testing on benchmark datasets like CIFAR-100, ImageNet, and MSCOCO. LumiNet outperforms leading feature-based methods and exhibits robustness in diverse settings.
Publication date: 5 Oct 2023
Project Page: https://arxiv.org/abs/2310.03669v1
Paper: https://arxiv.org/pdf/2310.03669
