The study presents the Associative Transformer (AiT), a model that uses sparse interactions instead of the conventional pairwise attention mechanism, aligning more with biological principles. The AiT model is based on the Global Workspace Theory and associative memory, inducing low-rank explicit memory that guides bottleneck attention in a shared workspace. The model fosters competition among inputs for writing information into memory, making it a sparse representation learner. AiT outperforms methods like the Set Transformer, Vision Transformer, and Coordination in various vision tasks.
Publication date: 22 Sep 2023
Project Page: https://arxiv.org/abs/2309.12862v1
Paper: https://arxiv.org/pdf/2309.12862
