Hacker News Reader

Comment by nomilk | Hacker News Reader

Centipawn loss (or simply the engine's evaluation of a position) doesn't take into account how realistically a human could hold a position.

During yesterday's WCC Game 6 the computer evaluation meant little when players were in time trouble. Anything could have happened going into the first time control, despite the game being dead drawn for the first 3.5 hours.

In the final stages the computer again evaluated the game as drawn, but presumed Nepo could defend perfectly for tens of moves without a single inaccuracy. Super GMs can't do that given hours or days, let alone minutes.

Last thought: did anyone else assume this was written in R/ggplot2 at first glance? Seaborn and/or matplotlib look strikingly like ggplot2 now days!

View on HN

I was thinking a bit about this during the game too.

Perhaps alongside centipawn loss (a measure of how many hundredths of a pawn a player loses by making the non-optimal move as determined by a chess AI engine) we could also measure the difficulty of any position.

Stockfish (a popular chess engine) roughly works by constructing a tree of possible moves and evaluating the score according to some heuristic at its maximum depth. The best result at depth n (25 I believe) is considered the best move and incurs 0 centipawn loss.

Perhaps we can define the difficulty of a position by the relative centipawn loss at each preceeding depth in the tree? The difficulty of a position is then determined by the depth at which the best move no longer changes.

This is an interesting thought! A couple of other scattered thoughts I had about this:

- Engine evaluation of a leaf of the tree will always be different and more sophisticated than human heuristics. So there's a problem where a human can't be expected to follow down some lines. Of course, this is always changing, as humans seek to understand engine heuristics better. Carlsen's "blunder" at move 33 was a good example of this, from my memory.

- Maybe there's a difficulty metric like "sharpness", some function of the number of moves which do not incur a significant centipawn loss. Toward the end of game 6, Carlsen faced a relatively low sharpness on his moves, whereas Nepomniachtchi faced a high sharpness, and despite the theoretical draw, this difference will prove to be decisive between humans. This seems like it could interact in interesting ways with your difficulty metric - for example, what does it mean if sharpness is only revealed at high depth?

- It would be interesting to take the tree generated by stockfish, and weight the tree at each node by the probability that a human player would evaluate the position as winning. Then you could give a probability of ending up at each terminal position of the tree. Maybe some sort of deep learning model trained on players previous games? Time controls add such a confounding factor to this, but it would be so interesting to see "wild engine lines" highlighted in real-time.

You could calculate difficulty by having some amount of dials on engine strength (lookahead, eval complexity) and fitting them to a function of approximate strength. You could then try to plot how the position "real outcome" changes. You could also try to use different engines or otherwise introduce variance as this sort of thing can be a tad deterministic for stats

You are definitely right about the evaluation; I switched between several streams and I don't think there was anyone saying that they didn't prefer white after around move 39, despite a 0.0 eval for a lot of those positions. But part of the reason the eval is misleading is because it might be reflecting a sequence of "only moves" - where only one move can hold that evaluation and it may be very hard to find some of those moves for black, while white has lots of good moves in each position. While that is a problem with human interpretation of eval, I do not see how it invalidates use of ACPL which is an average across the entire game.

In my opinion, neural network engines like Alpha Zero or Leela Zero do a much better job of assessing how difficult a position is to hold. They also report the evaluation in a completely different manner, win/draw/loss probability as opposed to centipawn loss.

For example, in yesterday’s game Stockfish was often giving a drawn evaluation (0.00) where Leela Chess gave a win probability of 30%+. I was posting about this during the game.

https://twitter.com/nik_king_/status/1466794534214504454?s=2...

Stockfish also uses a neural network for evaluation.

True, it’s works differently though. And Stockfish does not compute win/draw/loss probabilities as part of its eval. It converts cp to wdl using an exponential fit based stockfish vs stockfish games. So draw percentage in Leela is a lot more interesting and useful.

Yes, but at its heart it's still classical alpha-beta search rather than monte carlo.

> In the final stages the computer again evaluated the game as drawn, but presumed Nepo could defend perfectly for tens of moves without a single inaccuracy.

I agree that it’s not very useful to compare with table bases, especially given the “30 seconds time added per move” regime this was played under by the time they reached the position.

However, I don’t think the table bases even have enough information to indicate how close to losing a theoretically drawn position is. So, i don’t think this required perfect accuracy to defend against (defining ‘inaccuracy’ as any move for black that either makes it take longer to reach a draw or moves to a losing position. That, I think, is the most reasonable definition)

and exhaustion also starts to play a role after playing for like 7 hours, at some point mistakes will start to slip in.

This is merely the seaborn “darkgrid” style option. You need to set it explicitly if you want this effect.

Why isn't the time left of each player an input to the evaluator It shouldn't assume that everyone has plenty of time!

I’ve thought about this, too. Sadly, times only started being recorded in the last 20 years and last time I tried I couldn’t find a large dataset with the times included.

Chess.com did do one study and found a large percentage of mistakes occurs in moves 36-40 because in some time controls additional time is added at move 40.

Reply on news.ycombinator.com