Hacker News Reader

Comment by matsemann | Hacker News Reader

Could you (or someone) explain what this means?

It's how much text it can consider at a time when generating a response. Basically the size of the prompt. A token is not quite a word but you can think of it as roughly that. Previously, the best most LLMs could do is around 32K. This new model does 1M, and in testing they could put it up to 10M with near perfect retrieval.

As the other comment mentions, you can paste the content of entire books or documents and ask very pointed question about it. Last year, Anthropic was showing off their 100K context window, and that's exactly what they did, they gave it the content of The Great Gatsby and asked it questions about specific lines of the book.

Similarly, imagine giving it hundreds of documents and asking it to spot some specific detail in there.

Awesome explanation, thanks for the comparison

Great explanation. I was amazed when I started using Claude because I could find a recently-transcribed novella, upload it, and ask specific questions. I'm downright giddy to try a 1M+ model.

The input you give it can be very long. This can qualitatively change the experience. Imagine, for example, copy pasting the entire lord of the rings plus another 100 books you like and asking it to write a similar book...

I doubt it’s smart enough to write another (coherent, good) book based on 103 books. But you could ask it questions about the books and it would search and synthesize good answers.

I just googled it, and the LOTR trilogy apparently has a total of 480,000 words, which brings home how huge 1M is! It'd be fascinating to see how well Gemini could summarize the plot or reason about it.

One point I'm unclear on is how these huge context sizes are implemented by the various models. Are any of them the actual raw "width of the model" that is propagated through it, or are these all hierarchical summarization and chunk embedding index lookup type tricks?

For another reference, Shakespeare’s complete works are ~885k words.

The Encyclopedia Britannica is ~44M words.

Reading Lord of the Rings, and writing a quality book in the same style, are almost wholly unrelated tasks. Over 150 million copies of Lord of the Rings have been sold, but few readers are capable of "writing a similar book" in terms of quality. There's no reason to think this would work well.

I mean, Terry Brooks did it with the Sword of Shannara. (/s)

Reply on news.ycombinator.com