Comment by ksec | Hacker News Reader

Comment by ksec | Hacker News Reader

Parent Full thread

>I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory."

Did they ( Google ) or He ( Craig Silverstein ) ever officially admit it on record? I did a Google search and results that came up were all on HN. Did they at least make a few PR pieces saying that they are using ECC memory now because I dont see any with searching. Admitting they made a mistake without officially saying it?

I mean the whole world of Server or computer might not need ECC insanity was started entirely because of Google [1] [2] with news and articles published even in the early 00s [3]. And after that it has spread like wildfire and became a common accepted fact that even Google doesn't need ECC. Just like Apple were using custom ARM instruction to achieve their fast JS VM performance became a "fact". ( For the last time, no they didn't ). And proponents of ECC memory has been fighting this misinformation like mad for decades. To the point giving up and only rant about every now and then. [3]

[1] https://blog.codinghorror.com/building-a-computer-the-google...

[2] https://blog.codinghorror.com/to-ecc-or-not-to-ecc/

[3] https://danluu.com/why-ecc/

Your [3] has a footnote quoting a Google book that reads "Modern DRAM systems are a good example of a case in which powerful error correction can be provided at a very low additional cost... The following machine generation at Google did include memory parity detection, and once the price of memory with ECC dropped to competitive levels, all subsequent generations have used ECC DRAM"

Well They made a big deal out of it and lots of PR out of not having use ECC, and only quietly adding back ECC and listed it out in a book that barely any one reads.

The fact that ECC isn't the default across everything is a failure of human cognition and Capitalism.

I have always been angry that ECC was treated as an "enterprise" feature and increased the price way more than it should have.

It's treated as enterprise because most regular users had no realistic reason to care about it. The practical consequences of not having this on your phone or home PC are completely invisible for the vast majority of people. Because of this and the fact that in the consumer space prices matter most the situation is quite expected.

Users most definitely want it. They just aren't technically literate enough to know about it.

Hello user, for an extra 10 dollars, would you like to guarantee that cosmic radiation never affects your computing experience, including having to reformat, reinstall, or otherwise reboot your phone/computer randomly when one day something just doesn't work anymore?

Was it an update? Was it cosmic radiation? Was it a bad capacitor? Who knows!

Why do you think so many problems are solved by rebooting? Sure 99 out of 100 might be software bugs, but the other 1 out of 100 is straight up cosmic radiation, and that 1 percent is growing more and more every year as software becomes more robust and bug free with better tooling.

Indeed, users don't know enough to care but more than that, for most users this is simply not a problem because they haven't been practically affected. Most may never experience a bit flip, and most of the flips won't be obvious anyway (e.g. no visible effect).

So your hypothetical dialogue will sound more like a scam to a regular user. You're charging money to tackle a problem they don't have or see, only addressing a single one of the root causes that trigger that same result, and in the end you're not even completely fixing it, just reducing the already infinitesimal odds it happens.

It will become mainstream when manufacturers just decides to include it everywhere and not really give the user a choice. Apple is a prime candidate for a company with enough clout to afford this.

My point is, they have been affected. I think most people have seen bitflips.

>2009 Google's paper "DRAM Errors in the Wild: A Large-Scale Field Study" says that there can be up to 25000-75000 one-bit FIT per Mbit (failures in time per billion hours), which is equal to 1 - 5 bit errors per hour for 8GB of RAM after my calculations. Paper says the same: "mean correctable error rates of 2000–6000 per GB per year".

https://stackoverflow.com/questions/2580933/cosmic-rays-what...

I believe bitflips are far more common than people realize. I see weird shit all the time from my users that is only explainable due to bizarre software bugs or bitflips.

Bit flips are reasonably common, but mean is of very little use when one stick can throw no errors and another stick can throw thousands. A more useful quote is "About a third of machines and over 8% of DIMMs in our fleet saw at least one correctable error per year."

But behringer, regular users don't see bitflips, they see "a random error", a random reboot, a random green pixel in a photo. That's if they even consciously observed anything at all. You yourself said those could have a dozen other root causes. I for one have not actually observed a bitflip (so something that I can actually consciously detect as an issue) in a decade. No unexpected reboots, no unexpected file corruption, etc., nothing that made me say "weird, must be some memory corruption".

Don't get me wrong, I'm all for ECC memory, I practically showed the value of ECC after intentionally running the same multi-day Matlab job several times on a non-ECC machine with different results just to prove this very point. But you use your deeper knowledge to assume what someone with shallow knowledge and interest wants or needs, and that will almost always be off mark. Your car does not have a roll cage. Normal computers do not have ECC memory.

"Your car does not have a roll cage" No but the crash structures are one of the areas where we are absolutely trying to recreate the effect of a rollcage without the weight. The weight putting constraints on fuel efficiency is the big issue there.

For ECC, there's no reason to really expect that adding ECC to everything would really make things too expensive or slow long term. For a long time the reason why desktops did not have ECC was pretty much because intel wanted people who really need ECC to buy Xeons.

> No but the crash structures are one of the areas where we are absolutely trying to recreate the effect of a rollcage without the weight

Exactly like how we try to recreate the effect of ECC hardware but without its "weight" (cost, complexity). For a problem that's not nearly as visible or life and death.

> too expensive or slow long term

Not "too" expensive long term still means more expensive especially short term, like when the person buys it. For an issue no average consumer is actually frustrated about. Why are we still debating why those consumers don't care about ECC? How is it different from hot swappable RAM for the average Joe? They'll upgrade or expand their RAM more often than they'll get frustrated about the effect of bitflips. So why not have hot swappable memory? Because as much as I'd love hot swappable anything, regular consumers neither care about it nor want to pay for it.

>So your hypothetical dialogue will sound more like a scam to a regular user. You're charging money to tackle a problem they don't have or see,

I don't know the same argument could be made and ecc be swapped with surge protectors and people still purchase those.

I feel this way about all software.

Reply on news.ycombinator.com