HNHacker News
TopNewBestAskShowJobs

quelladora

2 karma · joined November 13, 2013

submissionscomments
quelladora··on Making 2.5 Flash and 2.5 Pro GA, and introducing Gemini 2.5 Flash-Lite
Right, the way I see it, MRCR isn't a retrieval task in the same vein as RULER. It’s less about finding one (or multiple) specific facts and more about piecing together scattered information to figure out the ordering of a set of relevant keys. Of course, it’s still a fairly simple challenge in the grand scheme of things.

LongProc looks like a fantastic test for a different but related problem, getting models to generate long answers. It seems to measure a skill the others don't. Meanwhile, RULER feels even more artificial than MRCR, since it's almost entirely focused on that simple "find the fact" skill.

But I think you're spot-on with the main takeaway, and the best frontier models are still struggling with long context. The DeepMind team points this out in the paper with that Pokemon example and the MRCR evaluation scores themselves.

quelladora··on Making 2.5 Flash and 2.5 Pro GA, and introducing Gemini 2.5 Flash-Lite
MRCR does go significantly beyond multi-needle retrieval - that's why the performance drops off as a function of context length. It's still a very simple task (reproduce the i^th essay about rocks), but it's very much not solved.

See contextarena.ai and the original paper https://arxiv.org/abs/2409.12640

It also seems to match up well with evals like https://fiction.live/stories/Fiction-liveBench-Feb-21-2025/o...

The other evals you mention are not necessarily harder than this relatively simple one..