Is this a joke making fun of the common way people dismiss other ChatGPT successes? This makes no sense with respect to chess, because every game is unique, and playing a move from a different game in a new game is nonsensical.
1. e4 e5 2. Bc4 Bc5 3. Qh5? Nf6?? 4. Qxf7++
The game Go has a claim to every game being unique. But not chess. And particularly not if both players follow a standard opening which there is a lot of theory about. Opening books often have lines 20+ moves deep that have been played many times. And grandmasters will play into these lines in tournament games so that they can reveal a novel idea that they came up with even farther in than that.
All games were provided in the article. None of them were 4 move checkmates; nearly every one is longer than 20 moves and some are 40 or longer. There is simply no possible way that ChatGPT is regurgitating the exact same 40-move-long game it's seen before. You can check a chess database if you'd like; virtually all games longer than 20 moves are unique.
1. It definitely regurgitates opening theory, much more than can reasonably be calculated at its strength.
2. It might be regurgitating tactical sequences that appear in a lot of positions but remain identical in algebraic notation. Famous example:
1. Nxf7+ Kg8
2. Nh6++ Kh8
3. Qg8+ Rxg8
4. Nf7#
There's some qualitative evidence for this in the games.
In one of the games it has a bishop on f6 as white. It plays Qxh6?? Kxh6 and then resigns due to illegal move. I'd bet good money that illegal move was Rhx# where x is 1-4. So it seems like in some these positions it's filling in a tactical sequence that often occurs in the vicinity of recent moves, even when it's illegal or doesn't work tactically.
Many many games follow the same moves(1 move = 2 plies) for a long time, up to 30 moves in some cases, 20 moves is downright common and 10 moves is more common than not.
These series of moves are referred to as opening theory and are described at copious length in tons of books.
This is because while the raw number of possible paths to take is immense, the number of reasonable paths for 2 players of a given strength gets smaller and smaller.
If I went over the 300 or so classical tournament games I've played I would ballmark that maybe just one or two would deviate from all known theory in the first 10 moves.
So the criticism is valid in my view. The existence of copious chess literature can't simply be ignored here.
EDIT: I checked and it left the lichess database after 9 moves. The lichess db has probably 5 orders of magnitude more chess games in it than chatGPT has in its training data.
In theory if I was playing a 1200 player I would almost always win, but let's say they have some extremely devious preparation that I fell into due to nonchalance and by the time we're both out of book I'm down a queen. It might not matter that I'm 600 points stronger at that point. If they don't make a sufficient amount of errors in return I will lose anyway.
So it would be interesting to eliminate all opening knowledge and that way be able to qualitately get at which aspects of chess it's actually good at, which is sucks out, and how much of its strength can be attributed to opening knowledge.
I'm still impressed by this btw. I did not expect this to be possible at all really. But being impressed is not an excuse to ignore methodological flaws. :)
It's clearly following some opening theory in all the games I've looked at so far. So yes, it is regurgitating opening moves. That's clearly not all it's doing, which is very impressive, but these are not mutually exclusive.
From this, I take it that the question is if ChatGPT is repeating existing games, or not. All you need is a single game where it's not repeating a single game to prove it definitively. You can hardly play 60 moves without an error by accident.
I believe you're responding to a different question, something like "does ChatGPT fully understand the game of chess".
As someone very clever once said, welcome to the end of the thought process.
We've established that:
1. It doesn't repeat entire games when the games go long enough
2. It does repeat a lot of opening theory
3. It seems to repeat common, partially position independent tactical sequences even when they're illegal or don't work tactically.
GP did say "sequence of moves", and if it matches what it has seen from the first move on, including the opponent, it will be in a valid "sequence of moves".
then, even midgame or endgame, if a sequence is played on one side of the board, even though the other side of the board may be different, the sequence has a great chance of being good (not always of course, but a 1400 rating is solid (you know the rules and some moves) but not amazing
Classical Markov chains played chess at some rate of success. ChatGPT is probably a lot better but not fundamentally different - It's predicting which moves to play based on sets of past games, not by memorizing it but by memoizing it.
I wouldn't be surprised if the relevant state in a typical beginner's chess game also excluded many units in the sense that yes, you could move them, but a beginner is going to just ignore them in any case.
I wonder how well it could perform in Go, there are way more permutations there so finding an identical state should be more difficult.
But the answer is insanely unlikely, past a certain number of moves. The combinatorial explosion is inescapable. Even grandmaster games are often novelties in <10 moves.
So, it has a to have some kind of internal representation of board state and what makes a reasonable move and such that enables it to generalize (choosing random legal moves is almost unbelievably bad, so it’s not doing that).
I also doubt that it has been trained on the full (massive) database of Lichess games, but that would be an interesting experiment: https://database.lichess.org/
Very low. On lichess when you analyse your games you can see which positions have been reached before, and you almost always diverge in the opening.
The lichess db has orders of magnitude more games of chess than the chatGPT training data does, so there is absolutely no way that chatGPT could reach 1400 purely based off positions in its training data.
Though I think you're overestimating how many positions have occured. Frequently, by move 20-25 you have a unique position that's never been played before (unless you're playing a well known main line or something)
The problem is not a failure to understand the rules. It is just not very good at maintaining the state.