Hacker News Reader

Not training full attention might score nicely in benchmarks but humans will instantly notice the whole spectrum is not represented. What you are proposing is basically get rid of infrequent combinations but those happen in the real world and will be missing from whatever your LLM will produce.

View on HN

Your typical LLM benchmarks simply do not test or use large context sizes.

We need benchmarks for tasks that requiere large context sizes (like recalling facts and understanding them in long stories). I'm sure OpenAI have internal benchmarks for these tasks.

https://github.com/google-research/long-range-arena

The attention is sparse in a single attention head in a single layer, but the model still has a way to route information where it needs it given multiple attention heads and multiple layers.

Similarly, your computer is not physically connected to mine but I'm still able to read your comment.

Reply on news.ycombinator.com