The difficulty of the task and human performance on that task hasn't changed. LLMs performance on that task has changed dramatically.
its in the training data, probably many times, and it was then, too
As the generations increase, so too has the complexity of the games it can one-shot.
I'm still waiting for any evidence of LLM economic impact in the real world.
Those demos are put out by non-engineer AI-bros who start from a blank slate and just test raw one-shot performance of the model.
In comparison I'm aware of how much additional performance you can extract out of models with a team of professional engineers that are balls-deep in harness engineering.
That delta is very large.
It's exciting.