Does a small startup have a chance on hard problems? AtD vs. The Spell Checker Poem

A non-real word error is what most spellcheckers look for. They check if a word is in a dictionary and if it isn't they generate a set of potential words from that error and then rack and stack them. AtD's non-real word spelling corrector uses a dictionary gleaned from the intersection of many wordlists and a corpus of text. The rating of each correction is handled by neural networks. It's all fun and I have carefully measured, tweaked, and remeasured this part of the technology. I also reproduced an experiment conducted by some Polish researchers in 2005 and used their data to find out how I compare to Office 2003. I learned my dictionary needs more work but my accuracy is on par with Office here.

The real word errors are just as serious but somehow ignored by most spelling correctors. A real word error is where you use a word that is in the dictionary but really meant some other word. You can imagine why this is difficult. I use a statistical model to try to pick the best word from pre-made confusion sets. While I was impressed with the accuracy of this I had to bias heavily against false positives making it less than what I'd like to see. I'm currently working to have the grammar checker take over or augment some words where this technology does poorly. That said, this poem is all about the real word detection and correction and I was quite delighted to see how well my code did against it anyways (even after I bias heavily against false positives).

So in short, yeah I'm just having fun.