Hacker News Reader

Comment by freediver | Hacker News Reader

On a similar note, I just updated LLM Chess Puzzles repo [1] yesterday.

The fact that gpt-4.5 gets 85% correctly solved is unexpected and somewhat scary (if model was not trained on this).

[1] https://github.com/kagisearch/llm-chess-puzzles

Given that o3 is trained on the contents of the Internet, and the answers to all these chess problems are almost certainly on the Internet in multiple places, in a sense it has been weakly trained on this content. The question for me becomes: is the LLM doing better on these problems because it’s improving in reasoning, or is it simply improving in information retrieval.

And then there's the further question of where we draw the line in ourselves. One of my teachers -- a philosopher -- once said that real, actual thought is incredibly rare. He's a world-renowned expert but says he can count on one hand the number of times in his life that he felt he was thinking rather than remembering and reorganizing what he already knew.

That's not to say "are you remembering or reasoning" means the same thing when applied to humans vs when it's applied to LLMs.

> One of my teachers -- a philosopher -- once said that real, actual thought is incredibly rare

Probably should listen to psychologists and neuroscientists about this, not philosophers tbh

When it comes to describing what the brain physically does, sure. When it comes to naming and interpreting, the philosophers are the ones to trust.

It's getting incredibly difficult to find anything on the internet that these models weren't trained on, which is why recent llm tests have used so much secrecy and only shows a few sample questions.

Oh cool, I wonder how good 03 will be. While using 03, I noticed something funny: sometimes I gave it a screenshot without any position data. It ended up using Python and spent 10 minutes just trying to figure out where the figures were exactly.

Reply on news.ycombinator.com