Why It’s So Hard to Catch Your Own Typos

You've tried the command line spell checkers, but they all irksomely need to be taught to ignore various tokens emanating from your markup language, requiring you to drag a custom dictionary from account to account, system to system, or else do it all over again.

Plus all spell checkers of all kinds choke on the technical language you use, especially the acronyms and strange names: OSPF, ISDN, XML, Yacc, GCC, C++, ... All have to be painstakingly marked as valid, to be ignored (as long as you're on the same computer with the same dictionary for the same spell checking application).

Spell checking technical writing is a pain in the butt, and so sometimes it doesn't get done.

1) break the original text file up into about 200 chunks of a few hundred words each (always breaking on paragraph boundaries)

2) randomize the chunks

3) display each chunk with the font faded to almost invisible in a UI with a "play" button that revealed one... word... at... a... time at an appropriate reading pace (more on which below.)

I kept the original document open in an editor and made changes as I found them (there was also a "pause" button, obviously!) The application kept track of what chunks had already been proofed.

The randomization of chunks helped ensure I never got caught up in the story, but the word-at-a-time reading was what really made it powerful. There were still a few typos that crept past (there always are) but only two or three have been found by readers.

The important thing is maximizing the degree of surprise you have at discovering the next word. By tuning the reading rate appropriately I could digest one word and have a few tens of milliseconds pause before the next one showed up. This forced my attention to focus on what that word would be, and brought stylistic problems to the fore as well as simple typos.

The process is not fast: there is a "speed of light" for proof-reading that seems to be about 10 pages per hour (which is about what professional proof-readers do) and I found I could only reasonably do a few hours per day, but that was still only a couple of weeks to proof a 140,000 word novel.

This was also done after running the thing through various automated grammar checkers (LanguageTool was the most useful: https://www.languagetool.org/) which caught a lot of more mundane errors.

I also had a decent editor and excellent feedback from first readers, which is indispensable, but much more of the process can be automated than is generally believed.

It is a truism that authors can't proof their own work, but with the right attitude, reasonable tools and a modest amount of human backup it appears to be practical.

A lot of companies employ manual input of information, my experience being in finance. Often a hand-written copy of a form in input to a system. Mistakes can involve financial loss. An often used 'solution' (often applied in finance) is to use a 'maker / checker' system where an individual (maker) inputs a field, and a checker checks what the maker input. [A checker is often ranked an order of corporate seniority over a maker.] For large value or sensitive transactions, a string of checkers may be applied (or suggested as remedies where accuracy is seen as a problem).

But... adding checkers rarely improves error rates. Checkers tend to trust Makers, and Makers tend to trust Checkers. More Checkers means more trust / less care - adding checkers can often make the chance of a mistake occurring higher.

A HNer may now suggest incentive structures, independent makers, technological solutions including drawing analogies with reCaptcha. I have done the same. Some operations companies do tried varied approaches, but many, including the largest of financial companies, do not for varying reasons. Not least as Operations departments of large multinationals have been squeezed by budget cuts of 10% per year for more than half a decade - some leads to outsourcing or offshoring, some moving the effort to the 'creator of cost' (i.e. changing the paper form to an input form).

I feel it would be an interesting challenge to sell something to enterprises seeking very high levels of accuracy (by which I mean, something in the order of 99.9% for non-standardised poorly scanned hand-written forms often including numbers, names and addresses).

I find a context change helps when reviewing my code for bugs. I like to publish my commits to a web-based code-review (I use Gerrit, but it doesn't matter) even if I'm not adding a reviewer. Just looking at the diff in my terminal doesn't seem to be enough of a context change (perhaps because I use the same font/color scheme in my terminal and editor). Reading the diff in Gerrit, I often catch something I'd missed. (Embarrassingly, I've more than once caught a mistake after I've sent a patch to a mailing list.) I suppose I could just use gitx for the context change.

I suspect same cognitive information bias is part of the reason most people believe themselves to be more intelligent than average.

1. When proof-reading in print rather than on-screen, I catch three times more typos. Not fully sure why.

2. Giving gaps between writing and reading (even as small as 24 hours especially if you spend time doing something else in the middle) helps quite a bit, and not only with typos but also with the message itself.

I'm not going to lie: knowing this makes my hours spent daily in front of a keyboard feel just a little bit more badass.

The brain transmits information between parts of it. A -> B -> C -> ..., with A, B, and C being sets of neurons (so one arrow is not just one connection, but somewhere between thousands and millions).

Whenever you're doing anything with your brain, you can simplify to this situation : every part of your brain is attempting to provide inputs to every other part. When multiple inputs are coming in, a selection is made (1) it wants correct input data (2) faster input data is better (3) when a "best" set of data is found, attention provided to any other signal will fall off exponentially.

This is normally very useful. Say you're jumping over a rock. You've done this before, so the part of the brain controlling your legs wants information ... "what's happening". And 2 parts of the brain will provide. One part will ultimately come from the sensors in your leg. But it's too late : by the time the sensors say you've hit the ground, it'll be ~0.05-0.2 seconds after you've hit the ground, potentially too late to compensate without falling over.

Fortunately, when you started the jump, a piece of your memory also said "hey, I recognize this, here's what's going to happen", and your leg control system will compare the values of the sensors for a while, agree with the memory, and start responding to memory input instead of sensor input.

Meaning your brain is responding to events it could not possibly have sensed. If you're looking for this behaviour, you'll see it everywhere. (In practice, of course, your brain also has a "physics simulator" that will provide input).

Assuming you've got good components providing all sorts of inputs, this mechanism is fantastic. You can look at something for a bit, have a simulation routine in your brain "synchronize" to it, look away to something else, synchronize to it, and then respond to both stimuli simultaneously despite your sensors (say, eyes) not actually capable of observing both stimuli.

Unfortunately, when you're re-reading a text you've written, the spell checker in your brain will switch to what you remembered writing down, and after a short while won't care what your eyes are actually reading any more. So unless the reading method calls attention (which it will only do in WTF cases), you'll miss any spelling errors.

Then you have the following 2 parts of your brain :

* a reading method that is providing outputs that contain the spell errors, but isn't actually connected to the spell checker any more, various other pieces of your mind might pay attention to it because they've got nothing better to do, and this might result in capturing attention, but probably won't.

* the memory, providing the correct input, connected to the spell checker

(in fact, shortly after that, the conscious part of the brain that is attempting to spell check will realize that the chance of getting an "error" signal from the spell checker is very small, and will "predict" that the spell checker will return an "OK" for the entire text and simply not listen for other signals anymore).

In short : do NOT count on humans (or animals, for that matter) to find small errors in repeating patterns. Write a short python function to do it for them.

I like to say that human minds respond "faster than light". In a sense, they do. You see this in martial arts championships (e.g. https://www.youtube.com/watch?v=a4lX0fKDTD8 ), where the outcome of an attack is determined before the first signals go out to the muscles, and these fighters are acutely aware of that. You see neither fighter is looking at anything other than his opponents' face. Everything else mostly comes from the predictor. This then often results in the situation "I'm going to go for left top attack, change the balance between my shoulders to X, then step 2 is ... Oh no crap he's seen my plan, abort". Of course, if you haven't fought karatekas before you won't be able to play that game, and you'll be very predictable to them ... But for spell checking this sucks.

I can't believe I actually had to have my spell checker correct "spellchecking".

It also means that, for example, you want to endlessly mesmerise a human being, you have to give it a signal it superficially thinks it understands (so it gets deep into the brain, and gets a lot of different components involved in analysing it), but contains an obvious "core" random component, so the brain will never successfully predict it for any period of time. Ideally it should have random signals at multiple levels of meaning.

Such a signal, however, would be very bad for you. But addictive.