Thats just absolutly not true.
A human being has general intelligence and needs A LOT of training and finetuning to become good in chess.
And there is a relevant and significant difference between the expectation of an AGI and an ASI system.
A human being has general intelligence and needs A LOT of training and finetuning to become good in chess.
And there is a relevant and significant difference between the expectation of an AGI and an ASI system.
An intelligent adult could simply read a short summary of the rules of chess and then, if they were careful, play a very bad game of chess without making illegal moves.
An LLM that has not been trained on any chess data cannot do that, at present. If you doubt it, take a current model and tell it that you want to play it at a variant of chess where, say, knights can also move diagonally like bishops. A human can easily adapt to this new ruleset (even if they make tactical mistakes, not having practiced with this variant of the rules).
Nothing is forcing the LLM to play 'blind'. If it's smart, it should be able to create its own representation of the chess board and update it with every move, just like a human would. Any chess engine that's sensitive to how the moves are formatted is clearly not very capable.
The LLM would only be playing 'blindfolded' if you somehow forbade it from making notes (as you effectively do by literally blindfolding a human, given how limited human working memory is). But you are not doing that. The LLM is free to keep track of the game state via whatever means it chooses.
None of this is about superhuman ability. Any human who understands a given chess notation can convert it to a visual representation of a chess board and then use that representation to choose their next move, with their usual level of performance.
> I just think this isn't a very good thing by which to evaluate LLM capabilities
I don’t think any single task is a good way to evaluate LLM capabilities, but I don’t see why chess is worse than a lot of other tasks. (Of course it is of no practical consequence whether LLMs can play chess, so if you are just making that point, then yes, I agree.)
> If there's no argument you'll accept
It’s a little unfair to suggest that I wouldn’t accept any argument whatever for your position just because I haven’t been convinced by your very brief comments so far. I could equally well say the same thing to you!
It absolutely still makes mistakes if you ask it to draw the board each turn, which should be equivalent to giving it the position because it only has to update one move at a time and then it has the position in the context window.