Hacker News Reader

Comment by slavik81 | Hacker News Reader

> "you must have ECC in all your computers otherwise they will constantly and silently corrupt your data + crash" and "statistically speaking, most computers in the world do not use ECC". How can both of these things be true?

What makes you say that? Both those things being true simply means that most computers silently corrupt your data and crash. That matches my experience. My programs occasionally crash. My pictures and videos are occasionally corrupted.

Do I know that those events are caused by memory failures? No. Most of them are probably other sorts of software or hardware failures, but some could be memory errors.

View on HN

I've never had a photo or video occasionally corrupted on any computer I've ever owned going back to 1985. Crashes? Sure, who hasn't.

"Some could be.." is computing by coincidence, and I'm not a fan of that logic. You can refer to the 2007, 2009, and 2012 studies for measured data on server farms.

A few bit flips in an image are unlikely to be noticeable (it depends on the format though; uncompressed is obviously the most resilient, JPEG should be reasonably resistant too.

Also, I don't think images stay for too long in RAM. Your HDD definitively has ECC.

Actually jpg is not very resilient, there is artforms doing single-bit changes to jpg-files to end up with very strange images.

This is an adversarial error, I think it's pretty good against random errors. I'll do a test sometime.

I've had plenty of corrupt JPGs - they're certainly noticable since a single error tends to mess up an entire macroblock, often even the entire rest of the image.

Was fun finding a bunch of my own photos like that. Luckily that was on a disposable copy and not primary storage, but it was enough to put me completely off non-checksumming filesystems.

I presume most of the corruption I've experienced is because I stored my data a single hard drive that I used until it completely failed. Of course, that's how most consumers use their systems so it's probably a common experience.

Though, I usually only notice it years after the fact. I have a lot of photos and a lot of videos and only view a few old ones on occasion.

> "Some could be.." is computing by coincidence

That second paragraph of mine is mostly superfluous, and you seem to have read more into it than I intended. Whoops. I should have been more clear. I was not trying to make any sort of claim as to how to do computing. I was only trying to show that there are other possibilities that you would need to investigate and reject before going from your premises to that particular conclusion.

> I've never had a photo or video occasionally corrupted on any computer I've ever owned

How do you know? Just the other day I pushed a TS video stream through the script that accidentally modified some bytes (I didn't even bother to analyze the exact changes). There were occasional artifacts observable, but the stream played nevertheless. Which is how the players are being made: to be able to resynchronize even on bigger errors. Somebody who programs these things can surely tell you much more about that.

I was able to notice that because I played the stream in mplayer while watching the console output. So I saw the "debug messages" detecting "something" wrong, and stopped seeing them once I've corrected the script. It was the messages, not the "watching" that made me sure.

Why worry about it if you don't even notice the corruption ?

For your own videos that nobody ever even wants to watch again, you surely don't have. For the servers for your business...

Well... I can imagine there are enough people who won't care too.

Note that the chance for corruptions increases as long as the data remain in RAM unsaved and only much later get to be saved. We are lucky that, at least in private use, we typically copy the data to the medium without keeping it too long only in the RAM (order of seconds or less). So that also reduces the chances of our pictures being saved with the wrong bits.

An important tenet of the digital world is avoiding error propagation. You pass the video along a few times and the errors accumulate.

Most normal people also don't keep large amounts of data or long running programs in memory for long periods of time, which is the case that is most vulnerable.

is this even true of a typical server workload? I guess it depends on the server and the work, but certainly for a webserver, this doesn't seem true.

It's true of databases or storage servers that rely heavily on caching in RAM (like with ZFS). Also many HPC workloads, which might be storing e.g. huge matrices distributed in memory over many nodes.

Reply on news.ycombinator.com