The article provides an in-depth analysis of In-Context Learning (ICL) and Supervised Fine-Tuning (SFT), two predominant methodologies in machine learning. The authors highlight a common problem with these methods: overconfidence and miscalibration, especially in low-resource scenarios. To address this, they explore the potential of self-ensembling techniques at different modeling stages. The study finds that it’s difficult to achieve simultaneous gains for both task performance and calibration, and the problem of miscalibration exists across all learning methods in low-resource scenarios. They conclude that self-ensembling can make predictions more calibrated and have comparable or even better performance.
Publication date: 21 Dec 2023
Project Page: https://arxiv.org/abs/2312.13772v1
Paper: https://arxiv.org/pdf/2312.13772
