> using a global attention mask and a local chunked attention mask.
Would you mind expanding on this? Or point to a reference or two? Thanks! I am trying to understand it.
Would you mind expanding on this? Or point to a reference or two? Thanks! I am trying to understand it.
There is a wealth of literature to catch up on to understand the performance motivations behind those choices, but you can think of it as essentially a balancing act. They want to extend the context length, which is limited by conventional attention compute scaling. RoPE on the other hand is a trick that helps you to scale attention to longer context, but at the cost of poor retrieval across the entire context window. This approach is a hybrid of those two things. The recent Cohere models employ a similar methodology.
Otherwise yes there are lots of papers on this and related topics, a few dozen in fact. But here are some notable ones, a couple of them are linked in their blog post.
RoFormer: Enhanced Transformer with Rotary Position Embedding - https://arxiv.org/abs/2104.09864
Scaling Laws of RoPE-based Extrapolation - https://arxiv.org/abs/2310.05209
The Impact of Positional Encoding on Length Generalization in Transformers - https://arxiv.org/abs/2305.19466
Scalable-Softmax Is Superior for Attention - https://arxiv.org/abs/2501.19399
A colleague from a discord I spend time in threw together this video a year or so ago, might be helpful as a first watch before a deep dive: https://www.youtube.com/watch?v=IZYx2YFzVNc
Covers positional encoding as a general concept first, then goes into rotary embeddings.