The article introduces Structured Probabilistic Coding (SPC), a novel supervised representation learning framework that extracts compact and informative representations from input related to a target task. The SPC uses an encoder-only probabilistic coding technology with structured regularization from the target label space. It encodes the hidden representation into a Gaussian distribution space, maximizing the prior entropy of latent representations concerning label space. SPC can perform information encoding and task prediction simultaneously in one module, utilizing effective information from input data. It also uses variational inference in the output space to reduce randomness and uncertainty. The article reports that SPC can significantly improve the performance of pre-trained language models for various classification and regression tasks.
Publication date: 22 Dec 2023
Project Page: Not provided
Paper: https://arxiv.org/pdf/2312.13933
