How to store a chess position in 26 bytes using bit-level magic (2022)

Storing a chess position in 26 bytes.

Storing a game is also interesting. The number of legal moves varies depending on the position. You could try to define a variable length encoding by giving more likely moves a shorter encoding, but the ordering would need to be deterministic so it could be decoded (running Stockfish for a second isn't).

I played with this in the past (https://news.ycombinator.com/item?id=34461113#34462521), but am willing to take another stab at it.

Store one 64 bit bitboard - a set bit means that a piece is present at that place. An unset bit means that no piece is after that position. After the bitboard, store a list of 32 4 bit integers, where the order of the pieces in the list corresponds to the order of the bits set. If there are less than 16 bits set in the bitboard, ignore the last items in the list.

  0000 - 0x0 - black pawn 
  0001 - 0x1 - black pawn (can be en-passant'd)
  0010 - 0x2 - black knight
  0011 - 0x3 - black bishop
  0100 - 0x4 - black rook (castling unavailable)
  0101 - 0x5 - black rook (castling available)
  0110 - 0x6 - black king
  0111 - 0x7 - black queen
  1000 - 0x8 - white pawn 
  1001 - 0x9 - white pawn (can be en-passant'd)
  1010 - 0xA - white knight
  1011 - 0xB - white bishop
  1100 - 0xC - white rook (castling unavailable)
  1101 - 0xD - white rook (castling available)
  1110 - 0xE - white king
  1111 - 0xF - white queen

I think that covers all possibilities to store a chess position in 64 + 32 * 4 = 192 bits, or 24 bytes exactly.

The starting position would be represented with a bitboard of 0xFFFF00000000FFFF, with a list of [0xD, 0xA, 0xB, 0xF, 0xE, 0xB, 0xA, 0xD, 0x8, 0x8, 0x8, 0x8, 0x8, 0x8, 0x8, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5, 0x2, 0x3, 0x7, 0x6, 0x3, 0x2, 0x5], using the same position-to-number scheme in the blog post

Edit: 32 pieces, not 16. Thanks to the peanut gallery for catching it quickly

Edit2: To store which player is next: do nothing for white. For black, if there are 32 pieces, flip the bitboard upside down. (To check if it's black's turn, verify that black pawns are "below" white pawns, which is illegal before captures are made.) If there are less than 32 pieces and it's black's turn, invert the bitboard. (To check, count the number of set bits.) This is entirely taken from https://news.ycombinator.com/item?id=37526484

There are two more things that should be considered. Both of them unlikely to influence a position, but they can matter - just like the en passant target.

The fifty move rule[0] is quite simple, just store a number, fits in five bits. But the threefold repetition rule[1] is quite a pickle - it basically means that to know everything about a position you need to know every position that occurred before it.

[0] https://en.wikipedia.org/wiki/Fifty-move_rule [1] https://en.wikipedia.org/wiki/Threefold_repetition

An interesting cognitive phenomenon: if you ask chess experts and novices to memorize and recall an arbitrary chessboard, the experts are significantly better than the novices _only if_ the board is a legal board that can be arrived at during play. If it is not a legal board, there's no particular difference between novices and experts. Original ref: Chase & Simon 1973, Perception in Chess.

This is usually taken to mean that the brains (or whatever) of experts see structure that can be used for compression that novices don't, but that compression has assumed invariants you cannot break.

For what it's worth, this definitely isn't the theoretical limit for compressing a chess board position, which I would probably resort to calculating mathematically. In particular, (simplifying to boards with no pieces captured) this scheme uses 24 bytes for representing piece positions, but the constraint that no two pieces are in the same square means you should actually need only log2(64!/32!) = 22.3 bytes for that case, with 13 bits of savings. The scheme proposed here reclaims some of that overhead by granting special meaning when a piece's location overlaps with the king, but still is capable of representing other illegal positions where non-king pieces occupy the same square.

Unfortunately, decoding a mathematically optimal encoding quickly devolves to making a list of all possible board configurations and indexing into it, so I'd definitely believe that the proposal here is close to the smallest practically useful representation.

How to store a legal chess position in log_2(8726713169886222032347729969256422370854716254) ~ 152.6 bits ~ 19.1 bytes using rankings:

https://github.com/tromp/ChessPositionRanking

But the major point of this project is to allow for random sampling of positions with a decent likelihood of getting a legal one, which allows for accurate estimation of the number of legal positions.

When I was building my multiplayer Sudoku app, I had fun experimenting with how small I could pack a sudoku grid into a URL. I came up with a scheme combining bit packing with the knowledge of how the grid works. So as you move through the grid, based on the knowledge of what's been placed where and the remaining digits you have a fewer options.

Was fun, I was playing code golf with myself. Sadly seem to have lost the code and I didn't use it in the end.

It would probably have been possible to go smaller than I got it by combining it with a sudoku solver, so stop packing positions once it's solvable. But life moved on.

Original title is:

> Compressing chess positions for fun and profit

Which unlike that here is correct: you can store a board, the positions, in 26B; not an arbitrary length game!

You can get this even lower, without the need for the extra promotion bits, if you make a note of a few things:

1. There is still redundancy in that there are multiple pieces of the same type, and if you permute their locations you get another representation of the same state. 2. You don't need to say that some pawn's location is the king if it's en passant. You can just use the back row, which pawns can't get to.

Using this, you can use the permutations of the pawns to store extra information. For instance, all of the pawns will be on the board at squares #a, #b, ... . Since pawns themselves have numerical indices, you could say that the "canonical" representation of the board state has all of the pawn locations sorted in ascending order. Then, different permutations of that can carry information about promotions and which of the "pawns" are really queens, etc.

What about bishops?

The bishops are limited to half the board so only need 5 bits for position. This frees up 4 bits, but you lose the capture state (can't use King's position for capture state). Well, you CAN use the king's position for capture state for two of the bishops at any given time. Then for the other two bishops use a bit to store their capture state. This saves 2 bits overall, bringing the total down to exactly 26 bytes.

Gonna have to think that through for awhile, not sure if it works out.

Update: I see a comment below that does this but uses 21 bits (instead of 22) by storing bishop position and capture state as a base-33 number.

Meanwhile the non-comp-sci dev in me is like, "why not just store every position in a simple JSON so everyone can read it and everything can parse it". Yeah, it might be half a kB, but you don't have to run through mental gymnastics just to render the pieces...

A chess position, not a game (still very interesting though!)

Ignoring whose turn, castling & en passant, a static huffman table isn't terrible:

    0 - empty (1*32=32)
    10y - pawn (color) (3*16=48)
    11xxxy - piece (color) (6*16=96)

The initial board takes 176 bits (22 bytes) to describe. In most games, the definition length would decrease. A game position is self-delimiting as it always has exactly 64 entries (no need to store the variable length)

one of the extra 3 non-pawn piece values can be used to encode 'rook (castling unavailable)' without spending extra bits. a scheme for storing en passant without any additional bits is less clear (but doing it with 3 extra bits is, so 22.375 bytes for positions that would regularly be reached in play).

I think it COULD increase in at least one specific circumstances: two promoted pawns (+6 bits total) with only 1 taken pawn (-3 bits). I think the maximum is 12 promoted pawns and 4 taken pawns which would make some hypothetical board take 204 bits (25.5 bytes) in this encoding.

A static arithmetic encoding (rather than a huffman encoding) of the same values should take a hair less space.

People have pointed out that due to rules about no repeats, the full state requires storing previous moves.

I’m fairly convinced that the most efficient encoding would be keeping only the move history and then playing that forward to obtain the board state.

E.g.: there are only 32 distinct pieces so a 5-bit number can select one uniquely. Each piece has a maximum of about 32 positions it can move to. Then the encoding is just 10 bits per ply. Typical games are 40 moves (80 ply) and hence require just 800 bits or 100 bytes for the whole game history.

Then you could get clever with Huffman coding or the like, since some moves are more common than others.

Bishops move on the diagonal and so can only stay on squares of their original color; you could thus encode the positions of the 4 bishops with 5 bits each instead of 6 each, saving a total of 4 bits, but this would preclude the use of the "store our/their king's location" hack to encode things. But you can encode bishop positions as a 4-digit base-33 number (digits 0..31 indicate the square, digit 32 is captured), which can be stored in binary as a 21-bit number. Net savings ~3 bits.

The "no two pieces on the same square" constraint could similarly be used more aggressively.

For my chess tactics trainer at https://www.checkmatechamp.net/ , I tried a lot of different techniques to compress a small set of positions (about 800). I was not concerned with storing the positions in a fixed size, so I was mainly trying things that would make the entropy of each compressed position smaller. I eventually fit these into a file that is about 14 KB in size when gzipped. Each position takes about 17.5 bytes, excluding the HTTP headers. If you go to the page and scroll through the positions, you'll see that each position loads instantly. That's because all the positions are in that 14 KB file which is loaded initially. I didn't include the castling or en passant info in that file, but I suspect it wouldn't add more than one byte per position on average.

I only considered techniques that would not cause the decoder to become overly complicated. One "non-standard" technique that I used was inverting the color of all the pieces on one side of the board. Specifically, on the lower half of the board, I change all the white pieces to black and vice versa. This greatly lowers the average entropy of a typical position, since for most of the game, each player keeps most of his pieces on his side of the board. It is also easy to handle in the decoder in one line (if row < 4, then piece = -piece).

I have meaning to write a blog post about it, but I haven't gotten around to it yet. If you are interested in hearing more, let me know, and I will move it up my to-do list.

Oh, no encoding of the turns? Then the encoding I came up with after following in the link to Forsyth–Edwards Notation is even better.

Version 1: For each position on the board, store if it has a piece or not. 8 bytes for that. Then for each piece in order, 4 bits can encode color and type. 16 bytes for that. Then you can spend 4 bits on castle ability, and 4 bits to pick the pawn that is able to be captured en passant (If there isn't one, pick a pawn that isn't in position. It's impossible for all the pawns to be in a vulnerable rank and have an enemy pawn in position.) So that's 25 bytes, no sweat.

Version 2: Instead of 4 bits per piece, store color|0 for a pawn, and color|1|type for anything else. Encode castle-eligible rooks and castle-ineligible rooks as separate types. This costs 14 bytes at the start of the game, and if you promote a piece at least one pawn has to die so the cost increases by 1 bit at most, up to 8 times. So 15 bytes, and you can always squeeze the bits for en passant eligibility into the 15th byte, because more promotions means fewer pawns to keep track of. You can even say whose turn it is by encoding "active turn king" and "inactive turn king" as different types. That's 23 bytes for the entire board state.

Either way you can add the turn counters with 2 more bytes.

Edit: Okay, this version beats mine solidly: https://news.ycombinator.com/item?id=37526804 I was thinking wrong about the number of captures versus promotions, and you can get much more clever with encoding en passant. So 22 bytes or less is enough for this sort of method.

The state of an individual chess piece can be stored in 6 bits. 3 bits for the X coordinate, 3 bits for the Y coordinate.

A captured piece can use the same coordinates as the King, which would indicate that it's captured.

For promoted pieces, you can use 3 bits per promoted piece type (Bishop, Rook, Knight, Queen) to indicate how many exist. For the case of 8 promoted pawns, you'd need one more bit to indicate that all pawns have been promoted and to treat a 7 as an 8.

You need 1 bit to indicate who's turn it is. You need 1 bit per player to indicate if castling is still possible, and 1 bit to indicate if the last pawn move was two squares.

What gets tricky is tracking for the draw rules.

There's '50 moves with no pawn or capture', '3 move repetition', '5 move repetition', '75 moves with no capture or pawn move'. 50 or 75 are simple enough to count, but tracking identical boards from a previous state would be hard to do with limited bits.

So then we have:

6 bits for piece coordinates of 32 pieces (192 bits)

13 bits for pawn promotion status for each player (26 bits)

1 bit per player for castling allowed (2 bits)

1 bit for whose turn

1 bit for en-passant possible

7 bits for move counter with no pawn movement or captures

That works out to 29 bytes.

Edit: Article did promotions much better by removing impossible promoted piece counts.

'Castling allowed' flag could be tied to 'both rooks in their original positions'. If the pieces look like they're in their original positions, but a king or rook moved, swap the two rooks so they're not in the original position anymore.

En-passent check by swapping pawn order is neat, but pawn order could instead be used to eliminate the 'promoted' flag per pawn instead.

I wonder if you could shave off some more bits by considering the specific movement restrictions of some pieces:

Bishops can only ever occupy squares of their respective colour, so you'd only have to encode ~32 possible positions instead of 64.

Pawns (before promotion) can only move straight forward and diagonally forward, which makes their range of valid positions a sort of upside-down triangle shape, with the "tip" of the triangle at the respective pawn's starting position. (e.g. it's impossible to move the pawn from A2 to H2 - or even to H8 - without a promotion)

Haven't made the exact calculations, but it might be possible to encode both positions with 5 bits each instead of 6 bits.

Reading this I realized chess is not fully Markovian (you cannot know everything about the game by just looking at the board at any point), specifically because whether determining if castling and en passant are valid moves depend on the history of the board.

Any scheme that does not make allowance for illegal moves or invalid positions is not suitable for purpose, IMO. In OTB situations players can make mistakes that no one catches. These illegalities become part of the official record of the game.

Minor point: the FIDE rules[1] state

> Article 9: The Drawn Game

> ...

> 9.2 The game is drawn, upon a correct claim by a player having the move, when the same position for at least the third time (not necessarily by a repetition of moves) [happens and a draw is claimed]

> 9.3 The game is drawn, upon a correct claim by a player having the move, if: ... 9.3.2 the last 50 moves by each player have been completed without the movement of any pawn and without any capture.

> ...

> 9.6 If one or both of the following occur(s) then the game is drawn:

> 9.6.1 the same position has appeared, as in 9.2.2 at least five times.

> 9.6.2 any series of at least 75 moves have been made by each player without the movement of any pawn and without any capture. If the last move resulted in checkmate, that shall take precedence.

This means that, in order to store a full chess game-state, you also need to keep track of how many moves since the last pawn move or capture (for the 50 and 75 move rule (9.3.2 and 9.6.2 respectively)) and also what positions have previously occurred (for the 3 / 5 position repeats, (9.2 and 9.6.1 respectively)).

The maximum number of moves from any chess position is 218 according to this post[2], so each move in the history can be uniquely identified by a single byte. You only need to store the moves since the last capture (since you can't repeat an earlier position), and you _definitely_ can't have more than 112 pawn moves without a capture (because there are only 16 pawns and they can only move forward, except in the case of en-passant which requires a different pawn to move forward twice).

But that gives an upper bound of 8400 moves -- this bound is probably much higher than it needs to be though.

----

[1] https://handbook.fide.com/chapter/E012023

[2] https://www.chess.com/forum/view/fun-with-chess/what-chess-p...

This reminds me of how Oscar Toledo disassembled Video Chess, which ran in just 128 bytes of memory:

https://news.ycombinator.com/item?id=36431917

It uses lower 4 bits of 64 bytes to store the board positions (upper 4 bits is used to store other data). 64 nibbles is 32 bytes, not too far off from the 26 bytes here.

It’s been a minute since I did this seriously so maybe my magic is out of date but I don’t think this is how bits to bytes works:

> 100 bits / ~12 bytes

> 18 bits / ~2 bytes

I didn’t want to spend a whole lot of effort tracking the arithmetic after this. Was it actually 28 bytes? 27 because I can squeeze those extras into one shared byte?

It's as long as my comment

There is also the Forsyth-Edwards Notation which is in text, can be quite brief if there are only few pieces on the board.

oh fuck. I feel like this is going to become an interview question.