Hacker News Reader

Comment by spmurrayzzz | Hacker News Reader

I'd pump the brakes a bit on the 10M context expectations. Its just another linear attention mechanism with rope scaling [1]. They're doing something similar to what cohere did recently, using a global attention mask and a local chunked attention mask.

Notably the max sequence length in training was 256k, but the native short context is still just 8k. I'd expect the retrieval performance to be all over the place here. Looking forward to seeing some topic modeling benchmarks run against it (ill be doing so with some of my local/private datasets).

[1] https://github.com/meta-llama/llama-models/blob/eececc27d275...

EDIT: should be fair/complete and note they do claim perfect NIAH text retrieval performance across all 10M tokens for the Scout model on their blog post: https://ai.meta.com/blog/llama-4-multimodal-intelligence/. There are some serious limitations and caveats to that particular flavor of test though.

View on HN

> using a global attention mask and a local chunked attention mask.

Would you mind expanding on this? Or point to a reference or two? Thanks! I am trying to understand it.

The source file I linked in my initial comment is honestly the most succinct way to understand how this works, but the TL;DR is that there is a NoPE layer interval parameter passed to the transformer block implementation. That defines how frequent a "no positional encoding" layer is used. The NoPE layers use the global attention mask, which is a traditional application of attention (attends to all tokens in the context window). The other layers use RoPE (rotary positional encodings) and a chunked local attention mask, which only attends to a fixed set of tokens in each chunk.

There is a wealth of literature to catch up on to understand the performance motivations behind those choices, but you can think of it as essentially a balancing act. They want to extend the context length, which is limited by conventional attention compute scaling. RoPE on the other hand is a trick that helps you to scale attention to longer context, but at the cost of poor retrieval across the entire context window. This approach is a hybrid of those two things. The recent Cohere models employ a similar methodology.

Thanks. Is there a summary paper out there? I get that I need to read a lot and am willing to do so..

Not sure what you mean by summary paper, its a pretty dense topic that assumes a fair amount of prior knowledge of the fundamentals. But maybe the Meta blog post may suffice for that?

Otherwise yes there are lots of papers on this and related topics, a few dozen in fact. But here are some notable ones, a couple of them are linked in their blog post.

RoFormer: Enhanced Transformer with Rotary Position Embedding - https://arxiv.org/abs/2104.09864

Scaling Laws of RoPE-based Extrapolation - https://arxiv.org/abs/2310.05209

The Impact of Positional Encoding on Length Generalization in Transformers - https://arxiv.org/abs/2305.19466

Scalable-Softmax Is Superior for Attention - https://arxiv.org/abs/2501.19399

Thanks! I am familiar with attention, linear attention, flash attention etc... just not up to speed on how it is scaled to 1M or 10M context windows.

Ah, got it. Yea, then I'd focus on learning how RoPE works first. That will at least help you understand how the retrieval in current long context implementations is so limited.

A colleague from a discord I spend time in threw together this video a year or so ago, might be helpful as a first watch before a deep dive: https://www.youtube.com/watch?v=IZYx2YFzVNc

Covers positional encoding as a general concept first, then goes into rotary embeddings.

Thanks! That was super helpful.

Reply on news.ycombinator.com