Hacker News Reader

Comment by codinghorror | Hacker News Reader

I just want to be clear that in the original referenced article, I am not anti-ECC per se, I just found myself caught in the massive cognitive dissonance between "you must have ECC in all your computers otherwise they will constantly and silently corrupt your data + crash" and "statistically speaking, most computers in the world do not use ECC". How can both of these things be true?

The argument for ECC is credible (I personally think rowhammer is the best example of this actually mattering, but ironically a) you can rowhammer ECC memory just fine and b) DDR4 has hardware features to mitigate rowhammer -- which shows how quickly things are changing), but it also seems to hinge on whether you have hundreds to thousands of computers all working together, e.g. the positive effects of ECC only seem to matter enough statistically at a _very_ large scale.

View on HN

> "you must have ECC in all your computers otherwise they will constantly and silently corrupt your data + crash" and "statistically speaking, most computers in the world do not use ECC". How can both of these things be true?

What makes you say that? Both those things being true simply means that most computers silently corrupt your data and crash. That matches my experience. My programs occasionally crash. My pictures and videos are occasionally corrupted.

Do I know that those events are caused by memory failures? No. Most of them are probably other sorts of software or hardware failures, but some could be memory errors.

I've never had a photo or video occasionally corrupted on any computer I've ever owned going back to 1985. Crashes? Sure, who hasn't.

"Some could be.." is computing by coincidence, and I'm not a fan of that logic. You can refer to the 2007, 2009, and 2012 studies for measured data on server farms.

> I've never had a photo or video occasionally corrupted on any computer I've ever owned

How do you know? Just the other day I pushed a TS video stream through the script that accidentally modified some bytes (I didn't even bother to analyze the exact changes). There were occasional artifacts observable, but the stream played nevertheless. Which is how the players are being made: to be able to resynchronize even on bigger errors. Somebody who programs these things can surely tell you much more about that.

I was able to notice that because I played the stream in mplayer while watching the console output. So I saw the "debug messages" detecting "something" wrong, and stopped seeing them once I've corrected the script. It was the messages, not the "watching" that made me sure.

Why worry about it if you don't even notice the corruption ?

For your own videos that nobody ever even wants to watch again, you surely don't have. For the servers for your business...

Well... I can imagine there are enough people who won't care too.

Note that the chance for corruptions increases as long as the data remain in RAM unsaved and only much later get to be saved. We are lucky that, at least in private use, we typically copy the data to the medium without keeping it too long only in the RAM (order of seconds or less). So that also reduces the chances of our pictures being saved with the wrong bits.

An important tenet of the digital world is avoiding error propagation. You pass the video along a few times and the errors accumulate.

I presume most of the corruption I've experienced is because I stored my data a single hard drive that I used until it completely failed. Of course, that's how most consumers use their systems so it's probably a common experience.

Though, I usually only notice it years after the fact. I have a lot of photos and a lot of videos and only view a few old ones on occasion.

> "Some could be.." is computing by coincidence

That second paragraph of mine is mostly superfluous, and you seem to have read more into it than I intended. Whoops. I should have been more clear. I was not trying to make any sort of claim as to how to do computing. I was only trying to show that there are other possibilities that you would need to investigate and reject before going from your premises to that particular conclusion.

A few bit flips in an image are unlikely to be noticeable (it depends on the format though; uncompressed is obviously the most resilient, JPEG should be reasonably resistant too.

Also, I don't think images stay for too long in RAM. Your HDD definitively has ECC.

I've had plenty of corrupt JPGs - they're certainly noticable since a single error tends to mess up an entire macroblock, often even the entire rest of the image.

Was fun finding a bunch of my own photos like that. Luckily that was on a disposable copy and not primary storage, but it was enough to put me completely off non-checksumming filesystems.

Actually jpg is not very resilient, there is artforms doing single-bit changes to jpg-files to end up with very strange images.

This is an adversarial error, I think it's pretty good against random errors. I'll do a test sometime.

Most normal people also don't keep large amounts of data or long running programs in memory for long periods of time, which is the case that is most vulnerable.

is this even true of a typical server workload? I guess it depends on the server and the work, but certainly for a webserver, this doesn't seem true.

It's true of databases or storage servers that rely heavily on caching in RAM (like with ZFS). Also many HPC workloads, which might be storing e.g. huge matrices distributed in memory over many nodes.

The answer is unfortunately: because we don't act like professionals.

I hope that, say, civil engineers wouldn't build bridges the way we throw together computer software and hardware.

Well they do... just look at the fiasco with the replacement for the Bay Bridge in California.

That decision was not made by bridge engineers. It was made by architects, artists, and civilians. Basically, everyone with zero engineering experience. The whole thing was a political rig job, setup to create something iconic, rather than something that worked.

San Franciscans insist on always doing things differently, not necessarily doing them well.

Well lots of people come into the picture when you build any engineering system. This includes software and hardware. There are many cost tradeoffs and compromises.

In this case, it was purely non technical reasons and not many tradeoffs. It was all political: http://www.sfchronicle.com/bayarea/article/Bay-Bridge-s-trou...

> "statistically speaking, most computers in the world do not use ECC". How can both of these things be true?

... most computers aren't particularly reliable.

I'm curious - are you not monitoring or at least keeping a vague eye on ECC correction events with your existing hardware? If so, are you just not seeing any?

I've never really operated at any sort of "large" scale - a handful of racks at most - but I've always found correction events to be about as routine as IO errors, and certainly way more common than outright disk failures.

Neither of our (very good IMO) sysadmins bothered to check ECC counters on our existing server memory as far as I know. We did check SMART counters and SSD wear levels, though.. It is certainly a very good point that unusual numbers of ECC errors (corrected) are a bad sign and that machine should probably have its memory replaced at a minimum.

I.. would probably have done that before concluding I didn't have any need for it. Even just do a quick one-off mcelog run to see what events have happened recently - you don't have to do some big statistical analysis.

I'd estimate about half the machines I've used with ECC have had at least one correction a year, maybe a quarter had one every few months. I've seen one-off weird bursts that never happen again, and I've had quite a few cases where ongoing corrections have indicated a DIMM needed reseating. An actual blatantly faulty DIMM's been quite rare.

You aren't concerned about cosmic rays?

https://en.wikipedia.org/wiki/Cosmic_ray#Effect_on_electroni...

>> Studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month.[76] To alleviate this problem, the Intel Corporation has proposed a cosmic ray detector that could be integrated into future high-density microprocessors, allowing the processor to repeat the last command following a cosmic-ray event.[77]

Is there a real-world practical rowhammer attack against ECC memory? I haven't seen one and the test tools I ran never triggered anything on servers with ECC.

If you read the comments to the referenced article, someone had a rowhammer problem even with ECC memory. ECC does indeed reduce the chance of a memory error a lot, but the chance is far from zero (at scale of hundreds/thousands of servers), even with ECC. Another paradox about ECC, it's not a guarantee, so you still need to build systems that can tolerate / mitigate statistically rare memory error states.

This commenter seemed quite credible to me, here's where it starts:

http://discourse.codinghorror.com/t/to-ecc-or-not-to-ecc/377...

> Yep, these are what we had -- uncorrectable errors with ECC memory caused by row hammer. Luckily there are mitigations. Sandy Bridge allows you to double...

Uncorrectable errors with ECC memory caused by row hammer is just ECC working as designed.

The scary thing about rowhammer isn't just the potential a fault, it's the potential for a security vulnerability. Yeah, a DOS attack isn't nice either, but it's a million times less worse than a privilege escalation caused by memory corruption.

For rowhammer to be even the same magnitude of problem with ECC memory would require not an uncorrectable error, but an error that gets past the error detection mechanism entirely and produces data either seen as correct or correctable, but which is not the original data.

Rowhammer might get through eventually, but it will blow the corrected error count through the roof on the way there.

The job of ECC is to prevent transient failures and to warn you about less-transient failures. It's not a paradox that it can't paper over things that are actually broken.

Reply on news.ycombinator.com