Hacker News Reader

Comment by senkora | Hacker News Reader

It seems dubious to show that engines are sometimes bad at evaluating positions by giving a position with three black bishops on black squares.

View on HN

While unheard of, this is not illegal.

You could theoretically convert two pawns to bishops and have three black bishops. Nobody would do that as it is usual to convert pawns to queens, but it is within the rules for you to choose.

So if you plan to write chess engine it would be pretty stupid of you to not prepare it to face multiple black bishops. If I knew that it would give me a lot of advantage.

I could imagine such a move making sense with the king on the backline such that promotion to queen would trigger a stalemate

In rare cases this might happen, two times in a match very unlikely.

The only situation where this might be worthwhile would be a forced mate in consequence. Otherwise the Queen is worthy enough that you can afford the tempo loss.

If it is in a tournament, it could even happen for reasons not related to the position itself. Let's say you have a winning position, but you are in severe time pressure. You really need a few moments to stop and think to figure out possible stalemate swindles your opponent might have.

You are about to promote to a queen. You realize that both of your bishops are still on the board, and promote to a bishop instead of a queen.

That lets you stop the clock and summon an arbiter to go find another bishop of the right color. While the arbiter if off dealing with that, you get your much needed thinking time.

Underpromotions are one of those "toy problem" things. A human might finish a whole professional career having never once promoted a pawn to anything but a queen, but in theoretical problem positions - like the one you're talking about - they happen "all the time" because it's a fun twist.

So it's understandable for a machine not to even bother modelling these weird cases I think.

edit: This comment is wrong and useless; I'm leaving it up to preserve the logic of the replies.

To quantify, here's raw counts from a representative sample of serious games:

    $ rg -Io '=\w' twic*.pgn | sort | uniq -c
    
        365 =B
       1551 =N
     122386 =Q
       1393 =R

I see knight (N) under-promotions pretty regularly on chess streams. The common pattern is there isn't a free tempo to promote the pawn, unless promoting the pawn creates an immediate threat that holds the attacker's initiative; and a knight at that particular moment creates that opportunity.

This obviously can't be trivially gleaned from those statistics - but would you say most of the bishop/rook promotions are just showboating, or are they done to avoid stalemates?

I, of course, understand why people would promote to a knight.

I'm an 1850 player and when I am playing under extreme time pressure (e.g. < 2 seconds remaining), I occasionally underpromote to a rook because I can execute a rook+king endgame mechanically faster than a queen+king without an accidental misclick causing a stalemate.

That's interesting. Is it just easier because you can't accidentally move to diagonals, or it's easier in your mind because you've seen it more? If you can even tell the difference.

Not OP but with rook and king you make a box and shrink it while keeping your king next to your rook, and stalemate is all but impossible. With queen and rook you have to get the queen to a knight's distance from your opponents king and mirror their moves until the edge of the board, and at the end you need a slightly bigger L (4 squares) to avoid stalemate while your bring your king around. In the former the two pieces work together, in the latter they work somewhat independently (with a quirk at the end) until the final sequence of moves and therefore there is a slightly larger risk of error. In practice you should drill this regularly to mitigate the risk.

>or it's easier in your mind because you've seen it more

I think the point OP is making is that if you need to use your mind at all with <2 seconds left you're going to lose. Pre moving safe legal moves without hitting a stalemate is definitely easier without a queen.

Ahh you're right; I should have thought of that possibility! It turns out my data is more or less garbage; mea culpa.

In fact: all of the =R promotions I inspected were frivolous, and 4/5 of the =N promotions were frivolous. I'm surprised it's that bad. One scenario where this happens is "capture-promote, followed by immediate re-capture".

Props for trying it out even though the data was garbage.

A rook promotion is something regular chess players will need to do to prevent stalemate, but it isn't that common where it is required (as opposed to someone who could win with a queen instead but it is trickery so beginners should go for a rook ). A Knight is the better promotion choice often enough that you should study tactics that lead to it, while a rook it is enough to do a stalemate check before promoting. There are tactics that only a bishop wins, but they are things you study for fun, not because you expect them.

I've seen a game by Alireza where he delivers mate by promoting to a knight. Arguably, the position was winning anyhow and standard promotion would also win, but the knight made it immediate checkmate.

(also this https://www.youtube.com/watch?v=3COS0_p3sfo)

Knight instead of queen offers different moves. Bishop instead of queen offers fewer moves. It's easy to imagine (and demonstrate, as you have) positions where a promotion to knight is advantageous, but it's hard to imagine positions where a promotion to bishop is advantageous except that it may confuse the opponent (computer or otherwise).

Well, on a little bit further reflection, possibly you may want to promote to bishop instead of queen if it helped you get to stalemate when you were confident you wouldn't win. Or as others said, to avoid the opponent having an opportunity to get stalemate.

Avoiding stalemate is a reason to underpromote, but it seems rather improbable that such a situation would come up twice in a single game.

I think it's worth noting the value here. The reason why these positions are interesting is that they're so obvious to humans, even ametuer humans, and yet difficult to even very strong computers. This implies that the way computers "understand" the game is radically different; an engine can't reason logically about a chess puzzle like a human can, for instance.

I would say that the computers are operating in a binary way, they don't have a "spectrum brain" which humans have various components to be able to analyze and see different scenarios. It is like humans have spatial awareness whereas computers are "one-track mind".

The linked article includes a concrete example of such a case wherein a GM underpromotes to a rook to avoid a stalemate against another GM.

https://media.chesscomfiles.com/videos/wide/My_Best_Move_Ale...

Statistics of it is irrelevant.

If I know a player is bad at certain types of situations I will be trying to force the player into those positions.

And chess engines are vulnerable because they don't learn and can be repeatedly exploited if you can find a way.

In poker this approach can cost you a lot of money. "I will try to force the player into those positions" actually means you're now doing things that aren't optimal for you, based on a possibly false assumption about the world.

I suspect but can't prove that for the positions we're talking about here (opponent's pawn should be promoted, but promoting to a Queen is terrible) in Chess you're putting yourself in a very similar situation, ie. to pull this off you're obliged to deviate so far from good strategy that you invariably lose anyway.

For chess engines playing each other we already know it doesn't matter because the chess engines play far more games than it would take to notice, some of them have exactly this "weakness" and others do not, and of course for chess engines playing humans it's a joke, the humans aren't competitive in that scenario.

> So if you plan to write chess engine it would be pretty stupid of you to not prepare it to face multiple black bishops. If I knew that it would give me a lot of advantage.

It would not.

Rybka (formerly the #1 chess engine) doesn't support promoting to bishops. The author chose an efficient move representation that couldn't handle more than four promotion types. Not quite the same situation, but I can be sure the difference is small even against an opponent trying to exploit it.

Isn't that a big advantage... if you can promote to bishop and the other player crashes, wouldn't you win by default or something?

So long as you don't crash the engine just evaluates when it happens .bishop promotions are rare.

Wait, there are only four promotions! Queen, rook, bishop, and knight – you're not allowed to keep a pawn on the back rank.

Yes. It’s possible some engines represent bishop-on-black as a different thing than bishop-on-white, which would make 5 with bishops and 3 without.

Actually now that I think about it, promoting to a second king would be pretty sick. Then you have a spare so losing the original doesn't lose you the game immediately. I could see an incredible game where someone uses this to salvage an otherwise unescapable checkmate.

Good point. If I remember right, the engine used one state to represent "no promotion happening this turn". Of course it's redundant, because you can determine that from the rest of the representation, but apparently it was worth being able to determine that with a single bitwise operation.

Edit:

> each move is easily encoded as a 16 bit value: 6 bits for origin square, 6 for destination square, 1 for en passant, 1 for castling, and 2 for promotion. The 2 promotion bits can only represent 4 options... and since the move encoding is for ANY move, not just promotions, one of the options is "no promotion"[0]

And another tidbit from that page: the same encoding is used for making moves and retracting them (going back up the search tree), so efficient encoding of "no promotion" is important there. E.g. If there's a white rook on e8 and the last move was e7-e8, when retracting it you need to know whether to put a pawn or a rook on e7.

[0] http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?tid=1...

There are other more realistic examples like the Nakamura game sacrificing 2 useless rooks to give the engine a false sense of winning. Then the engine sacrifices a valuable pawn to avoid a 50 move draw. Turn out the pawn was much more valuable than the 2 rooks in the particular closed position and Nakamura goes on to win. Basically, the only way to beat an engine is to try to create an unusual situation that the AI hasn’t practiced before.

But as others other out, AI suffers when it doesn’t have enough experience in a particular situation. So, the author is really just pointing out the extreme edge cases AI hasn’t mastered yet. But over time and getting these examples into the training process there’s no reason to believe that the AI couldn’t learn these situations as well.

Once you tell a human 'there's a tactic, go', given enough time we find it.

Is it though? If humans instantly realise that the three black bishops have some special significance and a chess engine does not then apparently the chess engine lacks some understanding that humans do have.

Sure, but it's 'meta' knowledge.

If, for instance, the name of a chess problem includes the solution in it, written in plain English, no current chess engine is going to pick up on this bit of 'meta' information - yet any English-speaking human trying to solve the problem would be trivially able to use it.

Does that make humans 'better' at chess than chess engines? Not really. Does it make humans 'better' at solving a given chess position, given some metacontextual clues? It does, but it is a bit contrived.

It's not exactly 'meta' knowledge, it's something that anyone with a few hours of experience in chess would be able to pick up on. It doesn't require knowledge of anything other than chess.

If an AI can't pick up on it then that suggests it lacks parts of what to humans would be a basic understanding of chess.

It does, however, seem to be the ‘secret sauce’ to humans’ ability to generalise our problem solving across domains.

It's not 'meta' knowledge, it's a basic rule of the game.

Promoting to a bishop is a rule of the game.

Solving a chess puzzle where there are for some reason three black-square bishops - recognizing that the bishops are important to the puzzle is meta-knowledge.

I misunderstood your point initially, but yes I agree. In a chess puzzle that was constructed yes it is meta-knowledge. But in the course of a normal game it is not. Importantly this meta-knowledge is not strictly required in either situation.

The distinction in my mind, was that the meta-knowledge here is not about the game of chess, but about the opponent. In the case of the puzzle this is meta-knowledge that the puzzle designer gave you a hint. In the case of Nakamura's game, his meta-knowledge is information about his opponent (the chess engine) and the heuristics it uses.

Chess engines only see the board state (at least to my knowledge), while humans have some kind of information about their opponent, or the history that lead to that board state. Humans have extra information, and sometimes that information is exploitable.

I agree. I’m shit at chess and even I can see the solution to the first problem presented.

If we’re just going to say, “no one will ever do that in real life” then the humans will just start finding AI weaknesses and how to maximally exploit them.

I play board games and I definitely “play my opponent”.

If you can force 2 surviving bishop promotions, you can force 2 queen promotions. There's no need for finesse when you can utterly crush instead.

Reply on news.ycombinator.com