That's a pretty weak article from Google, No real substance, Don't really explain or even hint at how they achieved such a long context window. It's just kind of a self-congratulatory pat on the back of an article :/
> we asked the Google DeepMind project team to explain what long context windows are, and how this breakthrough experimental feature can help developers in many ways.
At this time we can only speculate on how they achieved it. The best guess so far is RingAttention. See https://github.com/lucidrains/ring-attention-pytorch
Even the "what" isn't explained well enough if you're not already familiar with the topic. I'm not really sure who is meant to be the audience of this article.
Big congrats to the team though, great competitive leap for them.