RNN-Based Handwriting Recognition in Gboard

For industry/real world work, TensorFlow is best in class. It is far superior to any other existing framework. I agree that there are always areas for improvement, but the way you worded your comment makes it almost sound like TF is pretty subpar compared to other offerings.

The reality is more, TensorFlow is really the only option you have if you don’t want to build everything from scratch again. Whether that’s a good or bad thing, well at least it’s because TensorFlow is actually a good product and not because Google is preventing others from building their own / pushing others down.

That is an important point, Google is the master of releasing things in half, this was a common practice in Tensorflow since the initial release, they basically removed a lot of things to release it and it became a Frankenstein base of code. Bazel is another example, inside Google it works amazingly, but the open source project is a pain in the ass.

I don't know ML, but I was surprised too, since converting a series of points into Bezier paths of a certain degree seems arbitrary...

Just a side comment: take into account that (as per the paper) there is temporal input in Gboard (i.e. the timestamp of each stroke is important).

You do not have that for ing, so the software does not know that the dot is “independent”).

The reason is that online OCR (this particular case) is entirely different from offline OCR.

Online OCR is when you input the strokes directly on the tablet/phone, so it becomes a sequence of XY coordinates with an associated timestamp. It takes into account where you start and where you end the stroke on the canvas, along with the intermediate points (information galore).

Offline OCR is when you take a photo of your handwriting in your notebook, so you just get the raw pixels of a image. In offline OCR, you'd also have to properly segment and binarize the image before the OCR step.

With that being said, tesseract (version 4) uses an LSTM.

I just tried tout for refirsttime, and although the keyboardspace on my phone is barely large enough to cran five characters in there, the input scrolls sideraysautomatically if you litt your finger long enoy gh. So longer words can be entered as well. It doesn't seem to reevaluate previously decoded segments baselon what follows, though, so you can end up with weird misspellings at the beyinning of words. I dont think! Im going to use it from now on, beaurthere cognition is balenow ghto requiresigniti cant editing and the frictonisatittooncomfortable vihout a stylus.

Edited with the QWERTY keyboard:

I just tried it out for the first time, and although the keyboard space on my phone is barely large enough to cram five characters in there, the input scrolls sideways automatically if you lift your finger long enough. So longer words can be entered as well. It doesn't seem to reevaluate previously decoded segments based on what follows, though, so you can end up with weird misspellings at the beginning of words. I don't think I'm going to use it from now on, because the recognition is bad enough to require significant editing and the friction is a bit too uncomfortable without a stylus.

Only if you're writing words in the dictionary, and you're using the English alphabet.

Handwriting recognition is way more impactful for users in, say, Chinese.

It does have multiple languages. Clicking the settings icon right between the button to switch to numbers and the emoji button switches languages. You can also hold it down to go to settings.

these features are not missing: check settings

Handwriting recognition has been available on Android for a while (especially for Chinese). This article is "only" about incremental improvements.