Attention Bottleneck

The study presents the Associative Transformer (AiT), a model that uses sparse interactions instead of the conventional pairwise attention mechanism, aligning more with biological principles. The AiT model is based…