Show HN: Scannable Chess Scoresheets | Hacker News Reader

Show HN: Scannable Chess Scoresheets

reinechess.com

117 pointsindy750072 comments

This is definitely a real problem for some people - a chess organizer friend of mine recently spent 4-5 hours entering games from a scoresheet into the computer every night. This solution is pretty nice - but it is possible to imagine that such a solution might work directly on horribly written scoresheets, using handwriting recognition + the kind of constraints mentioned by indy7500 - that would be pretty amazing from both a technical, and usefulness, standpoint.

(I am OP) I thought about doing it that way but found that handwriting recognition was pretty weak as of now and wanted to try something solid. The reason is, usually, handwriting recognition would take into account the following characters (it doesn't recognize individual characters at a time, it looks at a database of handwritten whole words). That would be the ideal if done well, of course.

I have no experience with handwriting recognition, but intuitively it seems like the facts that:

a) the grammar for chess move notation is fairly limited; and

b) each move must be valid relative to the current game state

can be used as contextual clues to resolve ambiguous characters. Of course, this means you can't use an off-the-shelf handwriting recognition package that recognizes whole words like you describe, but requires a custom per-character recognizer. (But then, isn't that what you're doing already?)

Now that I think about it, you're right. I don't know what the accuracy would be like, but it's certainly possible and worth a try.

Also, a slight problem with b) is that it must be in the context of the whole game, since incorrect moves may only arise later on. This makes it harder with characters too ofc

Wait, are you GM Parimarjan? If so, this is truly an honor.

> No recording errors: if it's "Bg4" and you write "Bg5", we can't help :(

You can help by checking the legality of the current move (which you seem to already do) and also the rest of the game?

The way we check legality and adjust is with the character recognition. We use the second highest and third highest confidence characters. So, if you write a 5 and the model recognizes it as a 5 and then a 6 and 7, but really it's a 4, we can't help. This is because we'd add moves exponentially every time we add that many combinations

I don't get it. If Bg5 is an illegal move then the recognition is obviously wrong and you can prune it (I would think?).

Who am I to talk though, I'm not the one programming this thing. That's just what my gut tells me. :)

Don't say that, you're definitely more experienced than me :) I'll try to clarify, but I could be wrong. This is from the chess.com article on it here: https://www.chess.com/blog/ReineChess/scannable-scoresheets-...

The model actually gives us a confidence for each possibility. It might say, N-90%, K-10%, B/R/Q-0% for one of the boxes. Here, we look at how confident our character recognizer is on each character. If the PGN is invalid, then we know some character was recognized incorrectly. We look at the low-confidence characters and change them to the next highest confidence prediction, checking to see which combination of changes delivers a valid PGN. For example, consider the string of moves 1. d4 d2. Black can't play d2 on move 1! But we look at the next most likely predictions, perhaps d8 for white and d5 for black. These are the combinations:

d4 d2

d4 d5

d8 d2

d8 d5

Only the second one is valid, so we choose it and continue ahead.

If you literally wrote "d4 d2," then it's a lot less likely that the correct digit instead of 2 will be in the top 3 predictions.

This all makes really good sense. For now I think this is a workable strategy, but I bet you can do better (take that as a challenge ;) ).

Anyways, this is really cool, and smart idea. I'll be printing these out and bringing them to my next tourney. Excited to see how it goes! How do I send you direct feedback?

We currently don't store scoresheets when uploaded to our website. Would you be able to send the pictures to us at info@reinechess.com? You could add comments about how it went (or not, since we can just run it and see for ourselves). If you think that might be a hassle I could quickly add a feedback form on the website. Also, I'd encourage that you try it once before the tourney with the restrictions—we haven't tested on very varied handwriting. Oh, and thanks so much for offering feedback!

P.S. Did you have something in mind for improving the postprocessing / trying out different iterations? Or was it just a challenge :D

Is there a reason you don't store the images of submitted games? Acquiring real-world training data to test changes with would be very useful. Perhaps make it an opt-in feature people can select when uploading a game so people are "offering" you this information to you rather than you just "taking" it.

One way to choose between two legal candidate moves would be to look forward for the next move by the candidate pieces and see which ones make sense given. If you are trying to choose between e6 and e5, for instance, look forward and see what the next e pawn move is. Plenty of other ways to go about this, of course.

No, not a clear reason other than wanted to get the product out earlier. We'll work on an opt-in, that does sound the best.

Wouldn't other pieces affect this though? For example, what if the move was e6 and the next e-pawn move was e7. What if a piece was already there, Ne7 (wrongly recognized)? I thought about slowly running through it like this, but it felt like I kept running into problems which are obvious for us to recognize but I wasn't sure how to implement it computationally

This is neat. For anyone wondering why not just record the moves on your phone: Digital devices are not allowed at over-the-board tournament games.

Writing moves down on a scoresheet is a requirement in chess tournaments anyway (in classical time control at least).

Not true*

FIDE (and anything under FIDE rules) does certify some purpose built digital recording devices, they are hilariously expensive and hilariously crap but they exist.

* https://www.monroi.com/products/personal-chess-manager.html

I really like Chess Score Pad for iOS, but it's not approved for tournament use.

I like ChessNoteR almost as much, and it's USCF-approved. Not FIDE-approved yet, but they're working on it.

https://www.chessnoter.com

For the curious, the FIDE requirements for electronic score sheets are in this document [1].

The USCF certification requirements are here [2]. This also includes a list of USCF certified electronic scoresheets.

[1] https://www.fide.com/FIDE/handbook/Standards_of_Chess_Equipm...

[2] https://www.uschess.org/docs/gov/reports/eScoresheets/Certif...

As others have pointed out, there are digital scoresheets. But further, digital devices are not banned at OTB tournaments, most players will have a cell phone, and might even text or take calls. It's only in high stakes tournaments that you'll see cell phones banned, but even then it's only for the players. It's still perfectly legal, and extremely common for the game to be video recorded.

I am not aware of what you mean by "high stakes tournaments". All official tournaments, even when only amateurs play, actually ban cell phones since about 15 years ago.

https://www.fide.com/FIDE/handbook/Anti%20Cheating%20Guideli...

11.3.b. During play, a player is forbidden to have a mobile phone and/or other device capable of processing or transmitting chess analysis in the playing venue. If it is evident that a player brought such a device into the playing venue, he shall lose the game. The opponent shall win. The rules of a competition may specify a different, less severe, penalty.

That rule only seems to apply to devices the player controls themselves, so if the tournament had a recording device or if both players jointly sponsored a recording device it doesn't appear it would trigger that rule.

I'd also imagine that this rule's enforcement would be relaxed when, for instance, a player sets up a phone to record the game and has no access to that phone during the game, since any information the phone is recording (and any resulting processing for cheating) would be inaccessible to the user during the game.

> But further, digital devices are not banned at OTB tournaments, most players will have a cell phone, and might even text or take calls.

I guess this may vary by country, but at least in the USA, phones have been banned in every "novice" tournament that I've participated in. Even books are off limits.

Digital devices are allowed, as long as that device is a "Chess Notation Device".

This enables you to record and replace moves (including helping forecasting moves).

If everyone used these devices, then problem solved.

But I've only see someone using it once. Good chess players tend to already be good at visualizing past, present, and future positions.

This is cool. One thing I keep hoping either exists now or will soon is the ability to take a photo of an actual board position and get a FEN string out (or just directly send it to Stockfish and get an evaluation). It'd be really helpful to quickly analyze positions when I'm trying to help my kids out, cos we don't generally keep notation and I'm not always confident that my advice on a position is blunder-free.

I think you'll be pleasantly surprised, this is a thing!! I thought about making it but quickly decided computer vision wasn't up to the task yet. It seems I was horribly wrong: https://github.com/maciejczyzewski/neural-chessboard Haven't tried it myself but it looks amazing

Oh man, I had just blithely assumed nobody had done it and never googled, thanks for this!

This is great! Sadly it does not seem to be open-source.

I have wanted something like this for other data entry tasks like: tracking expenses or tracking body weight. I enjoy working with pen and paper much more than typing on a Phone or PC. Just give me a template I can print, let me write the data to it, and have it scanned.

Is anyone aware of a product (perferably open-source) that lets me do that?

I think I'd be willing to make it open source? I'll talk to a person that worked with me on it.

Scannable templates using some third party ML/AI service like tensorflow could be a neat business to develop.

I'm not sure if you posted it anywhere, but would you mind sharing what technology you used for this? I would be very interested in contributing if it became open source.

https://github.com/Messier-16/Reine

> Do not indicate checks, mates or promotions at all, i.e. leave out the "+," "#," or "=Q".

Some of your screenshots include the +, is this accurate?

If you don't include the =Q you can't encode all games?

I took the screenshot with the "+" before I realized it'd be easier for now to ignore the "+". This is my fault for being lazy, I'll change it. I looked at a database of lichess games (40,000) to convince myself that non-Queen promotions were worth ignoring for the beta. If promoting to a different piece is clearly the better move, I could run a Stockfish check anytime a promotion move is made to assume the player chose the best non-Queen promotion (since otherwise you'd just promote to a Queen in a serious game)

Makes sense, congrats on shipping!

I suppose it's unlikely to happen in a game serious/slow enough that people are bothering to record it on paper and upload it, but I've tricked someone into stalemating me by promoting to a queen before. Long term I don't think assuming people made the best move is the right decision.

Thanks! Definitely, the check and mate things should also be fixed long term. The nice thing is, I was able to put in tons of constraints to improve processing (for example the last two boxes must be a square like e4) if I ignore checks.

Note that promotions do fit in 5 characters if you omit the equal sign, for example "e8Q" (or "dxe1N")

You could also 'assume queen until proven otherwise.' eg, if all remaining moves are diagonal, it stays a queen (even if it was really a bishop). And switch to an =K if you see a knight move (or the checkmate requires it to be a knight).

I also thought of that. Sometimes it is possible by the following moves to know what the promotion is, although sometimes that is not the case, such as if the piece hasn't moved after that, then you might not be able to know what promotion it is if you do not write it down. (Trying to guess isn't very good unless it can be proven from the sequence of moves.)

We have decided to make it open source! Will publish in another article perhaps, when it has enough comments added in the next two days!

That's great! Maybe try to integrate it with a popular library like python-chess (https://github.com/niklasf/python-chess)?

We did use that for the postprocessing actually, the part where moves are cross-checked for lower probability predictions.

Awesome job! If anyone else is working on a project like this or is interested in learning more about applied machine learning we've got a helpful Slack community over at Heartbeat (https://bit.ly/heartbeatslack)

I just posted this project on #share over on that Slack. Their #help channel helped me a ton (especially Jameson and hart!) Fully endorse this

Another way to make them to scan would be that there is one space you can just write free text and then next to them you can fill in the circles to indicate rank moved from, file moved from, rank moved to, file moved to, and promotion (if applicable). You can indicate castling by the king's move.

That would be an awful lot of bubbles per move, no?

Yes, although there are ways to reduce it (perhaps in a way similar to POSTNET; this would reduce the space taken up but require more fill in at once). But you do not have to fill it in right away you can do so after the game is finish

Would you mind quickly explaining what you guys mean? Sorry, I'd love to improve it but I'm not sure I get it. If you had a bunch of bubbles for each square and piece etc. wouldn't you be taking up much more space than 5 boxes? And it's also a huge notation change for everyone

P.S. Here's a chess.com article with a video demo (if you don't want to try yourself) and tons of info about the story / how it works. Anyone that has advice on the website design or the software, please help! I'm new.

Looks like you forgot your link?

https://www.chess.com/blog/ReineChess/scannable-scoresheets-...

Nice idea, but doesn't the fact that each letter / digit has to be in a separate box (if I understand this right) defeat the purpose?

Normally at tournaments they would hand you a score sheet that doesn't have this crossword-like layout.

Hmm... I see why it's a hassle. How does it defeat he purpose though?

Because the scoresheet you bring home from a tournament won't be in the format this tool expects? So you'd still have to rewrite it onto the "crosswords" sheet before OCR-ing it? (In which case one extra step gets automated away thanks to the tool... but at the cost of another one taking its place).

Unless I'm missing something

The idea is you'd write down the moves during the game on the Scoresheet. Ideally, if people don't think it's too big of a change, we could just have tournament directors hand them out. Otherwise, print, bring to game, scan and throw away

It's been a while since I've played in a tournament, but I think I'd like your score sheets better than the traditional. I've been in situations where I tried to analyze a game I played a while ago, and some of the moves were illegible. Your grid forces legibility, which is something that you probably had to compromise on for the OCR, with an unexpected benefit. Really nice product and congrats on the launch!

The USCF rules state that the TD can require players to use a specific scoresheet. It's not generally used in most casual tournaments, but almost mandatory for tournaments that matter.

I don't know about the US, but eg. scoresheets with carbon coy were handed out in the last tournament I played (Italy).

I'm surprised that handwriting recognition is such a problem for you. Wonder if you can improve accuracy by customizing the training for each user as you grow a larger corpus of examples in each users' hand?

Yes, that is the plan. That should heavily increase accuracy. Do you mean handwriting recognition as opposed to character recognition? It wasn't that we tried it and decided it wouldn't work, I just couldn't find previous instances of accurate enough recognition with handwriting. Accuracy was my priority. My sense was also that people would be turned off if they had to still manually input some games, and I had no idea how many manually inputted games it would take to reach good enough accuracy. This was when I hadn't come up with constraining characters and such, but now I can see that (with some attempts at handwriting recognition) that may be possible / is the next step after an app.

It is open sourced now if you or anyone would like to contribute. https://github.com/Messier-16/Reine/

Now open sourced. https://github.com/Messier-16/Reine/

Pretty cool, will definitely save me some time, though I guess I used the time converting it by hand to think through some of the moves.

Or you could use ChessNoteR

https://www.chessnoter.com/

This is very cool. Analyzing your own games is the single best way to improve your play.

are chess players required to write down their moves?

or could the event organizers set up some overhead cameras

or make a smart board

and automate the entire thing that way?

The answer is yes to all 3! There are cameras and smart boards at the highest level, but chess players are still required to write down their moves. Archaic for sure, but tradition is strong with these old games.

>are chess players required to write down their moves?

Not at fast time controls, or with little time left.

"If a player has less than five minutes left on his clock at some stage in a period and does not have additional time of 30 seconds or more added with each move, then for the remainder of the period he is not obliged to [record moves]." http://www.fide.com/FIDE/handbook/LawsOfChess.pdf

And smart boards are expensive.

Yes that's all possible, just a lot more expensive.

That's really cool. Thanks for sharing.

Oooh! Finally!!

Reply on news.ycombinator.com