Hacker News Reader

Comment by qq66 | Hacker News Reader

Making a computer play like a 1300-rated human is harder than making a computer beat Magnus Carlsen.

This is really interesting because i ran into a pokemon bot the other day were its training led to calibration of 50% winrste at all levels of play on Pokémon showdown. It was a complete accident.

But that doesn’t imply that that bot played like an average human.

Making a computer have a 50% score against a 1300-rated human is way easier than making it play like a 1300-rated human.

For the former, you can take a top-of-the-line program and have it flip a coin in every game whether to make a random move every move or not.

Playing a chess bot that works this way feels like playing a Magnus Carlsen who's trying to let you win.

It's not hard to make a chess bot that plays at a 1300 strength, i.e. its rating would converge to 1300 if it were allowed to compete. But it will not play like a 1300-rated human. It would play like a superhuman genius on most moves and then make beginner-level blunders at random moments.

Making one that realistically plays like a human is an unsolved problem.

ah that makes sense. thanks!

Of course, you are right. But (the linked site) at least has a bot that plays the opening like a human of chosen rating perfectly. It stops working after the opening-stage (since it just copies moves from humans in the lichess game database), but it is still very impressive. For later game stages, some other method would have to be used (unless we play multiple orders of magnintude more games on lichess).

Now that i think about it, i remember the people in the alphago documentary talking about the bot giving its moves percentage scores in both how high winning % the move had and how high % chance that a human would have made the same move that it just played. I wonder why they never showed what a full game of the most human-like moves from alphago would look like. Maybe it actually worked, by feeding it all the pro games in existence, and training it to play the high human % instead of the higest win probability moves like they did in the end.

https://www.chessassess.com/openings

I think this can be achieved with some ease with a machine learning model. You will have to train it on games between 1300-rated players and below. A transformer model might work even better in terms of the evenness of play (behaving like a 1300 rated player throughout the game).

> I think this can be achieved with some ease with a machine learning model.

What evidence lead you to think that, and how surprised would you be to be wrong?

Ahem https://lichess.org/@/lichess/blog/introducing-maia-a-human-...

So like a 1500 rated human?

Definitely, but it seems like it's now possible: https://www.maiachess.com/

Take the computer which beats Magnus and restrain it to never make the best move in a position. Expand this to N best moves as needed to reach 1300 rating.

You've identified a potential strategy by which a computer can play like a 1300-rated player, but not one where it will "play like a 1300-rated human". Patzers can still find and make moves in your set of N (if only by blind chance).

Yeah, you would have to weigh the moves based on how "obvious" it is, such as how active the piece has been, how many turns until it leads to winning material, or other such 'bad habits' humans fall for.

This won't work. With that strategy, you can make a computer make play like a 1300 player, but not a 1300 human player.

That's kind of what they do for "training" bots and it produces something which plays NOTHING like a 1300-rated human.

I assume you could just give the computer a large set of 1300 rated games and train it to predict moves from that set :)

I think there's a real difference between "a computer"— in this context meaning an algorithm written by a human, possibly calibrated with a small number of parameters but not trained in any meaningful sense, and a "chess model" which works as you describe.

I think the chess model would be successful at producing the desired outcome but it's not as interesting. There's something to be said for being able to write down in precise terms how to play imperfectly in a manner that feels like a single cohesive intelligence strategizing against you.

Even 1300s sometimes make the best move. Sometimes the best move is really easy to see or even mandatory, like if you are in check and MUST take that checking piece. Sometimes the best move is only obvious if you can look 20 moves ahead. Sometimes the best move is only obvious if you can look 5 moves ahead, but the line is so forcing that even 1300s can look that far ahead.

Despite decades of research, nobody has found a good way to make computers play like humans.

Then I can't refrain from asking: and what's the style of LLMs? For example the ChatGPT which is apparently rated around 1800? That should be completely different from that of a classic chess engine.

I can’t speak for ChatGPT, but your intuition is correct that LLMs tend to play more like “humans” than Stockfish or other semi-brute force approaches.

LLMs can be trained on chess games, but the tree of possible board states branches so fast that for any given position there is simply very little training data available. Even the billions of games played on chess.com and lichess are only a drop in the bucket compared to how many possible board states there are. This would have to be split further by rating range, so the amount of games for any given rating range would be even lower.

This means that the LLM does not actually have a lot of training data available to learn how a 1300 would play, and subsequently does a poor job at imitating it. There is a bunch of papers available online if you want more info.

LLMs already do play at elo ~1400-1800. The question was how does their style feels like to someone who can appreciate the difference between a human player and a chess engine (and the different styles of different human players).

ChatGPT will hallucinate and make impossible/invalid moves frequently, so I don't see how it could have a chess rating

That's not the case. Depending on the version, (Chat)GPT seems to be able to play between ~1400 and ~1800 elo, very rarely making invalid moves.

Reply on news.ycombinator.com