What is a long context window?

blog.google

13 pointsnojito6 comments

That's a pretty weak article from Google, No real substance, Don't really explain or even hint at how they achieved such a long context window. It's just kind of a self-congratulatory pat on the back of an article :/

The article doesn't even profess to answer the "how" only the "what" as evidenced by this quote:

> we asked the Google DeepMind project team to explain what long context windows are, and how this breakthrough experimental feature can help developers in many ways.

At this time we can only speculate on how they achieved it. The best guess so far is RingAttention. See https://github.com/lucidrains/ring-attention-pytorch

> The article doesn't even profess to answer the "how" only the "what"

Even the "what" isn't explained well enough if you're not already familiar with the topic. I'm not really sure who is meant to be the audience of this article.

Big congrats to the team though, great competitive leap for them.

tldr: 1M/128K = 7.8x longer

The window was 32k before. 128k was their goal for the new version, and is the new standard window size. 1M is in private preview. 10M has been demonstrated internally.

1M/32k = 31x longer

Reply on news.ycombinator.com