A researcher brought this to my attention years ago with a set of domains in particular I won't name. What was most interesting to me is just how frequent bit errors must happen. According to the research, they'd basically received thousands of emails destined for the correct domain. Really makes you think.
At some point, he made the observation that 1-2% of crashes occur not because of a programmer error or anything, but "chance"; bit flips and the like, in either the app's memory or the phone's services.
So while on an individual basis it's rare, and in practice nobody will really notice an issue, statistically bit flips are significant enough to need attention.
Good quality servers are the top of the reliability standings. The bottom... the bottom is very far down. Very far.
I'm a hardware guy now. I'm still amazed anything ever works at all. The worst are things like DDR SDRAM that work well enough that people think they can get away without error detection. At least modern MLC NAND flash is so bad that it obviously doesn't work at all without ECC. Better that than silent failures.
The first Guild Wars game actually added a Prime95-style iteration before sending a bug report/crash dump back. An astonishing number of clients failed this test; if I'm remembering the conversation I had correctly, something like half of all oddball crashes got filtered out by that test. Half! (To be fair, overclocked cheap gaming machines may actually be the bottom tier of reliability, but the point stands.)
Back in the days of spinning disks, I had a coworker who had previous worked on the SQL Server test team.
One thing they tracked was bit errors. He said they were surprisingly common to encounter (and of course then account for)
Or, to put it another way, there is a reason modern file systems are paranoid.
Also way back when I managed a huge distributed test cluster, we ran ~3 million tests a week. A non-trivial # of test failures (few hundred) were spurious network failures where the test package didn't get copied to the test client correctly, we detected it "something went wrong" and marked the test as failed. I know in theory Ethernet promises the data gets there without issue, but in reality, at scale, it wasn't. Flaky hardware, bad cable, failing switch, a couple machines with unreliable RAM? Who knows, but it happens.
> I could probably count on one hand the number of times we've had to address any issue (always failed hardware)
Another product I worked on, the bus timings for the DRAM were slightly off. And I do mean slightly, I wasn't directly involved but I think it was something stupid like being 1 cycle short on waiting for something.
We had amazing logging. Soon as we had a few hundred thousand units out, we get ~10 stack traces come in from crash dumps, always in the same place. My team spent a month going over every line of code related to that place in memory.
New build goes out, more failures, again tiny tiny %, you'd only notice this at scale. We track it down, pointer being corrupted, it was stored in the same memory address we had been looking at last time.
Bit was being flipped. We only noticed it because it had hit a pointer. Most of the DRAM, the vast vast majority, was capable of operating 1 cycle out of spec, but some chips weren't.
We only detected and fixed this bit flip issue because the principle engineer on that product insisted that we investigate and fix EVERY single product crash. He insisted on 100% reliability.
Do you think every firmware team for every single component of every single motherboard has that same dedication to quality? Because if not, errors are going to slip through, and they won't even be noticed, not to mention investigated.
> Do you think every firmware team for every single component of every single motherboard has that same dedication to quality?
The answer is, of course, no. Build quality and work conditions vary wildly.
So imagine you are sending a mail to jim@mail.example.com. You see it in the address field as correct. You click send, and your client resolves mailnexample.com, gets an address, and delivers it, while you are none the wiser.
This is particularly bad for third level domains, which are more common than you'd expect.
> In fact, out of the 32 valid domain names that are 1-bitflip away from windows.com
"windows" is 7 characters long so how is there 32 1-bitflip combinations?
Also one of the ones mentioned was "windo7s", but "w" is 01111000 and "7" is 0000111, which isn't a single bit flip away unless I'm doing something wrong.