Hacker News Reader

Comment by H8crilA | Hacker News Reader

This problem has no ultimate solution. I've seen all components flip bits, CPUs, networking cards, RAM, most often you just can't know for sure what did it. You can remedy it a bit (like with ECC), but ultimately there will always be corruption if you process hundreds of petabytes of data. Get used to it, your computer executes an instruction with a probability extremely close to 1, but not equal to 1.

Deep in the archives of a well known tech company is a very well documented case of a bit flip that caused the wrong function to be executed in a C++ v-table. The big oof was that this function was the equivalent of an SQL "drop table", and just happened to be 32 bytes off of a very benign function that did something like stat(). Really funny stuff once the crisis is over :)

View on HN

ECC isn't a terribly complicated technology, and can be used in all those cases.

In limited cases, a checksum is good enough. If you checksum outgoing data, and verify it on reception, then it being corrupted in transit whether on the network card or the cable can be detected and transparently compensated for.

Really, we can do much better than to "get used to it".

You are under the impression that CPUs and other chips always perform the same instructions as are written in the code, and only RAM can flip bits because DRAM is DRAM :)

It can (and should! whenever possible) be improved, not fixed. There's always that pesky gamma that can hit a specific transistor, even if it is deep underground. Gamma cannot be fully stopped. At certain scales data corruption becomes directly measurable. And yes, corruption levels vary between pieces of hardware.

Of course not. I'm not saying we can have perfection. I'm saying that we can do much better, using methods and technologies that are very old at this point.

The reason why we don't is laziness and market segmentation, mostly.

The probability of a bit flip depends on the size of the transistor used. RAM tends to pack many small transistors.

Sure but once your registers, cache, data bus and address bus has ECC you have vastly smaller area that can flip.

You can even just buy (well, chipaggedon aside) ARM cores that have 2 chips running in parallel and faulting when the result is different

> You can even just buy (well, chipaggedon aside) ARM cores that have 2 chips running in parallel and faulting when the result is different

See dual-core lock-step Arm chips (used for automotive).

OP said: We can do better

You replied: it should be improved, not (it cannot be completely 100%) fixed

Your error: replying to a strawman

> You are under the impression that CPUs and other chips always perform the same instructions as are written in the code, and only RAM can flip bits because DRAM is DRAM :)

I thought L2/L3 is ECC (at least on Intel, though L1 I think is parity only)

> ultimately there will always be corruption if you process hundreds of petabytes of data.

Your statement is in practical terms false.

Error correction exists, and adding enough redundancy to make any corruption have such a tiny chance of being undetected that the whole history of the universe could go by without it happening is a problem that was solved over 60 years ago with the invention of the Reed-Solomon code,

Is this something that's documented publicly? I'd love to read more.

If this is the same issue at the same well known tech company that I recall, it wasn't a hardware failure but a case of undefined behaviour. The existing code used memset to initialize a struct with zeroes, then someone else added a virtual method (to it? to a base? or member?) which caused the struct to grow a vptr. Now the memset was clobbering the vptr. They compiled this program with some form of link time optimization and the compiler emitted assembly that tested the vptr against one of two known values, and when it didn't match (because it was NULL) it must be the other known value! Either way it didn't use the vtable, it just made direct calls to the appropriate member functions, which led to a "drop table". As part of resolution it also led to https://reviews.llvm.org/rL130299 for compile-time checking and also UBSan's `-fsanitize=vptr` for runtime checking.

One can process petabytes without bit flips if one use proper checksums and error correction codes. While that does imposes overhead, it is not big and, thanks to Shannon theorem, can be made arbitrary small with sufficiently big blocks.

Reply on news.ycombinator.com