Open-LLM performances are plateauing

I feel like the post doesn't relate to the title?

The post seems to be about changing the Leaderboard and doesn't comment too much about whether the actual real-life performance of LLMs is plateauing and what can be done about it.

Just to be sure: the post does not say that the performance of Open LLMs is plateauing (because that would be false, e.g. Google just released a Gemma2 that blows out of the water all previous open models from the same sizes).

Its true title is "Performances are plateauing, let's make the leaderboard steep again", which means "on the Open LLM leaderboard, top models have basically reached a point where they've all grokked the benchmarks which makes it harder to distinguish them, so let's change the benchmarks for harder ones to make a difference again."

LLM performance in general is plateauing. That is natural for any technology. Progress slows down as it reaches maturity.

Even if they’re plateauing there’s still a lot of value to be had in what they already do. I think the mistake so far has been aiming too high or too low—ie, products that require AGI-like LLMs or unimaginative “low-hanging fruit” product ideas which are obvious from the function of an LLM. The former have been wishful thinking, and the latter have no moat. The Goldilocks area is in understanding what current LLMs can do in a way that you can either do something complicated that we couldn’t do reliably without LLMs or do something simple that wasn’t worth doing without LLMs. And in both cases the products need to be built in a way that naturally incorporates the expected failure modes of the tech. (For example, I don’t need it to write all my code; there’s a lot of value in just using ChatGPT to help me write one-off bash scripts.)

Meta question, does anyone know why arrow keys (and vimium J and K) don't work for scrolling this page until it is clicked?