Back to the Future of Handwriting Recognition
jackschaedler.github.io
jackschaedler.github.io
Lots of people care, specially in Asia(Chinese and Japanese). It is just that the problem is incredible hard.
We put 5 very smart people working for a year on that, and it was totally impossible meeting people's expectations, specially people like doctors taking notes fast(and ugly).
We thought that the market was in creating mindmaps or something instead as people could write slower and better.
But people write a double u and expect the computer to see an "m". With deep learning is possible but extremely flimsy.
> The program was efficient enough to run in real-time on a IBM System/360 computer, and robust enough to properly identify 90 percent of the symbols drawn by first-time users.
I just want to point out that 90% accuracy is, from a user's point of view, awful handwriting recognition performance. It means you will be correcting on average about 10 words per paragraph! Even 99% percent accuracy is not nearly good enough to give people a sense that the computer is good at handwriting recognition.
I also want to point out the difficulty and danger in interpreting strokes when doing handwriting recognition.
In the last demo box, try writing a capital Y without lifting the pen. You'll have to go "up and down" one or both upper branches. Because of this, the recognizer will call it a K, A, or N even though it is obviously a Y when you're done.
This demo is constrained to only using one stroke per letter, but systems that permit multiple strokes still get into trouble when the strokes don't match what they are expecting--for example if you draw an X using 4 individual strokes outward from a central point.
This also happens with words. In Microsoft's handwriting recognition in Office in the early 2000s, writing the letters of a word out of order completely borked the recognition. For example writing "xample" and then going back and adding an "e" at the beginning would not produce a recognized word of "example."
My point with all of this is that there is a reason you probably don't do all your computing with natural handwriting. It's a surprisingly difficult problem. Users do not expect it to matter how they form letters and words on the page. And they have very low tolerance for correcting computer mistakes.
you could also keep multiple interpretation of a word pending (and a text search for all of them would take you there) and eventually ask the user to disambiguate if the user wants to. I assume this would be an acceptable solution for non dictionary words too...
That is to say, people have a higher tolerance for things that are within expected norms of their environment. Ideally, we want no corrections. But, having to do them constantly for a time will quickly desensitize people to this. (And yes, this is currently just an assertion of mine, I don't have data backing it. Just some anecdotes.)
I hadn't considered that it was intended to act as a warning that the content might be more error-prone.
Arguably, an X drawn this way should NOT be recognized as an X--that's not how an X is spelled.
If the task is communicating with the computer, then recognition of the gesture is a valid approach. Just as there are conventions regarding the spelling of words, there are conventions involved in the formation of letters. Why not use them? It would even seem incorrect to leave these out.
A computer that interprets the behavior of writing, rather than the final symbols, is going to violate user expectations at some point.
Why? Because people do not always write as linearly as you might expect, especially when writing fast. They might drop or mis-write letters or words, then go back and fix it. Or quickly jot down just enough letters to remind themselves of what they heard, then go back and fill the rest in. A routine that interprets actions in order is going to have a hard time with actions that the user completes out of order.
I completely concede that it is possible to get the same results with other stroke orderings. However, there is a reason when teaching children how to write, we often get fairly prescriptive with stroke ordering, as well.
If you truly want to write fast, you go with a shorthand system. I don't know many folks that have tried reading other's shorthands. It probably isn't as tough as you'd imagine, but most of those systems are more demanding on stroke order, not less. If only because the speed is gained by being very prescriptive.
Not exactly. E.g. Japanese handwriting and the order of strokes etc (also in traditional caligraphy/penmanship)
But are you saying that Japanese writing is only readable if you observe the writer during the act of writing? Because that's what some stroke recognition engines do.
Here's a tangible example. Imagine I write "h l o", pause, then go back and place an "e" in the first space, and an "l" in the second space, then hold it up to you. You're going to see "hello," right?
But an algorithm that tries to interpret the act of writing itself, might see "hlo el", because that's the order in which I wrote the characters.
Not exactly, but the situation, as I understand it, is somewhat related: Japanese writing is better (and thus more readable) if the writer observes (respects) a specific stroke ordering.
So, one doesn't have to observe a writer while he is writing to be able to better read what they wrote. But the ordering of strokes can have impact on readability, even when one sees the written words after they've been completed.
Not that I don't think "meet me halfway" type approaches (like the Graffiti system) aren't worth using, but in this case we're talking about recognizing writing (the artifact), not writing (the verb).
I am reminded of the Graffiti handwriting notation used by Palm OS. That was single stroke, and devices came with a card depicting all the characters.
I was never able to become fluent.
The trick was that they treated "Graffiti" as a new alphabet that users had to learn. Thus, when the recognition engine failed, many users blamed themselves (i.e. their Graffiti fluency) rather than the product.
In contrast, when products that promised to recognize natural handwriting had a recognition failure, the users tended to blame the products.
It's a good lesson for product development--user satisfaction will depend in part on user expectations.
Wait, what? Doesn't that imply that a paragraph needs to have 100 words in it, in order for 10 of them to be recognized wrong at 90% success rate? That seems super-long, anyway.
My stats are really rusty, perhaps that's just one of those unintuitive cases that confuse people like me.
For reference, the above paragraph is 78 words long.
Edit: I think where the Afterword says "inputting text with a stylus is likely slower than touch typing", they're forgetting that we still don't have a really acceptable way of inputting text on mobile devices. Swype and its ilk are close, but still hamfisted at times.
Reality check: Our machines do not yet accurately manage simple reading tasks.
It was (yay!) published as recognit.bas (VB) and I'd be really happy if someone still has a copy.
It recognized just numbers but the basis of operation was similar to the linked article.