The article ‘Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification’ discusses a common issue in multi-label classification (MLC) tasks, the ‘Sigmoid Bottleneck’. This refers to the occurrence of ‘unargmaxable’ label combinations in tasks where the number of possible labels exceeds the number of input features. The authors propose a solution by introducing a Discrete Fourier Transform (DFT) output layer, which ensures all sparse label combinations are argmaxable. This DFT layer is found to train faster and be more parameter efficient, matching the F1@k score of a sigmoid layer while using up to 50% fewer trainable parameters.
Publication date: 17 Oct 2023
Project Page: https://github.com/andreasgrv/sigmoid-bottleneck
Paper: https://arxiv.org/pdf/2310.10443
