Dilated Attention

LONGNET is designed to address the challenge of scaling sequence length in large language models. Traditional methods struggle with either computational complexity or model expressivity, limiting the maximum sequence length….