Focused Transformer

The Focused Transformer (FOT) is a new approach designed to tackle the challenge of scaling the context length in language models. Large language models have the capability to incorporate new…