Poor man’s version of this which requires no training would be to evaluate positions at low depth and high depth and select positions where the best move switches.
Training neural nets to model behavior at different levels is also possible but high rated players are inherently more difficult to model.
I've looked at the ~same problem in go instead of chess quite a bit. It turns out that that strategy does remarkably poorly, mostly because weak players don't just have low-depth thinking, they also do random weird shit and miss entire patterns. Eg in go if you run a strong go engine at _zero_ thinking (just take the model's best guess first move), it's already damn close to pro level.
Getting them to play anything like a beginner, or even intermediate player is functionally impossible. You _may_ be able to trick one into playing as weak as a beginner player if you really try, but it would feel _nothing_ like one.
> Training neural nets to model behavior at different levels is also possible but high rated players are inherently more difficult to model.
Maybe more difficult to model (not sure tbh, but granted for the moment), but it's _far_ easier to generate training data for strong players via reinforcement learning approaches.
I do think that positions where the best move at depth N is a terrible move at N+1 are hard, especially if it isn't just recaptures on the same square. Chess engines used to have special heuristics to avoid such "horizon effects", don't know if they still do.
Two reasons. One, strong players are performing tree search, so you need to model their tree search process rather than a simple depth zero prediction. And two, there are far fewer high rated players so there is far less high quality training data. A lot of high rated games are also bullet, which aren’t as useful.
> but it's _far_ easier to generate training data for strong players via reinforcement learning approaches.
Can you say more about this?
yes and no. there's a bit of a bathtub curve here where below ~1000 elo lower ratings are harder to predict because their moves are closer to random.
If you check figure 4, SF at depth 15 matches about 47% of the time which is lower than their NN. So quantitatively, SF isn’t amazing at move matching, and lower than the best NN at its relevant rating range (1900 -> 54%).
Qualitatively, SF is pretty unsatisfying as a model of a pro human because a lot of its moves are counterintuitive.
For example, certain types of piece forks are easy for newer players to spot (knight forking a king+rook on the back rank) and others are harder (bishop forking a king and a pawn in the late game when nobody is on the back rank), but stockfish is going to find both just as easily.
So if you want to get deep into complexity you do need a model of different players and what types of advantages are easy to see for them.
Won’t that bias in favour of common sacrifices in the endgame? The issue with using depth is that humans don’t struggle with depth uniformly. Humans can calculate to a lot more depth with a sequence of obvious moves (recaptures, pawn races) than they can in more complex situations (closed positions in the midgame).
I vibe coded this into a browser app, but the evaluation is slow around depth 5: https://camjohnson26.github.io/chess-trainer/
Cute chess is a pretty GUI where one can do this too by setting the ply parameter. Castling is done by dragging the king on the rook!