Hacker News Reader

>> hard to deny that their LLM models aren't really, really good though.

The context window of Gemini 1.5 pro is incredibly large and it retains the memory of things in the middle of the window well. It is quite a game changer for RAG applications.

View on HN

Bear in mind that a "1 million token" context window isn't actually that. You're being sold a sparse attention model, which is guaranteed to drop critical context. Google TPUs aren't running inference on a TERABYTE of fp8 query-key inputs, let alone TWO of fp16.

Google's marketing wins again, I guess.

It looks like long context degraded from 1.5 to 2.0 according to the 2.0 launch benchmarks.

Those benchmarks are for the Flash 2.0 model, showing it improved over the 1.5 Pro model minus for that one benchmark

Reply on news.ycombinator.com