Hacker News Reader

This is a sight-reading test. If a musician practices a piece for thousands of hours, it would no longer be an effective sight reading / creativity test. The purpose of the test was to see how models would compose something novel requiring the ability to compose orthogonal, normally unrelated, components into a coherent image.

We do. People who, for example, memorize question banks to pass certification tests without knowing the underlying material are equally frowned upon for not having the problem solving skills that they purport to. I'll leave the contrasts between LLMs and people to the well-written sibling comments.

More like “This artist won the drawing competition because someone told her the theme in advance and the specifically practiced drawing pelicans for hundreds of hours.”

Except, of course, LLMs are not humans, and they do not learn or "reason" in a way which even remotely resembles humans.

Plus obviously humans can still overfit to a specific style of test.