Hacker News Reader

Comment by elif | Hacker News Reader

> had Chess-GPT play against Stockfish with this random initialization. Its performance plummeted. The larger 50 million parameter model’s win rate dropped from 70% to 17%.

Beating stockfish 17% of the time is still incredible for any engine.

and being trained on human moves rather than engine moves will make this so much more annoying to analyze cheating...

View on HN

I updated the article to also mention this in the second blog post. All games were against Stockfish level 0, with nodes searched limited to 100,000 rather than a time based search limit to limit variability due to different processors.

I googled stockfish "level 0" and it came up with no results. What do you mean by level 0?

The Stockfish program can be set to play at strength level 0-20. Estimates of the levels' Elo is provided here: https://github.com/official-stockfish/Stockfish/commit/a08b8...

It means it plays several times worse than a 1960s chess engine.

Have you tried performing a search using Chess-GPT ?.

Article says: ‘Chess-GPT also played chess well, with the best model playing at approximately 1500 Elo.’

So I’m guessing this wasn’t full strength stockfish.

It notes in the blog post that it was SF level 0.

The previous blog post [0] mentioned they were using stockfish with 0.1 seconds per move.

[0] https://adamkarvonen.github.io/machine_learning/2024/01/03/c...

i think i dont understand what stockfish does then. doesn't it play the basically perfect move at each step? how is it possible to beat stockfish 70% of the time? with a naive GPT type thing?

Chess is not a fully "solved" game, so the actual perfect move is not generally known. Like any other engine, Stockfish just tries its best and is not infallible.

That’s not the right answer. The real answer is it was playing against a stockfish at a low difficulty setting.

well that's much more palatable, haha

Technically, the tablebases for 7-piece chess are fully solved, and since we are talking about games starting from partial board states, you could trivially compare both engines to the "perfect" move for a huge number of board states.

> Doesn't it play the basically perfect move at each step?

"the basically perfect move" has a lot of implications here.

Because chess isn't a solved game a seemingly "perfect move" in a game involving two 2000 ELO rated players might not seem perfect to a 3000 ELO rated player. (To be clear: the only actual perfect moves are ones that all trees lead to a checkmate, but in general use "perfect move" means roughly the best possible move to play)

In this case Stockfish was set to a lower setting which corresponds to 1320 ELO.

It would be impressive if it was playing Stockfish at a grandmaster level. But it looks like it's lower-level. Still impressive that it can learn to play decently at some level. Doubt it will ever be competitive with top engine levels.

Reply on news.ycombinator.com