Hacker News Reader

Comment by kimixa | Hacker News Reader

> Gamers don't really care that ECC prevents one crash in years because they are used to frequent crashes already.

I work on GPU drivers for one of those companies.

We regularly get reports and backtraces that cannot be reproduced, or "Cannot Happen" without some external factor (e.g. some other bit of code poking around our memory space). Often they're just silently dropped or ignored on the long tail of issues that nobody can get any traction on.

My understanding is the stats from hyperscalers is that ECC correction events happen a lot more than "Common Knowledge" may imply - I wonder just what proportion of things that are blamed on software may actually be due to hardware issues like this?

Again, without a significant change in the market (IE enough gamers start using ECC to actually be statistically relevant and comparing stability) this cannot really be tested, but I've wondered.

View on HN

Except that anyone using Intel desktop CPUs pretty much can't use ECC thanks to marketing deciding that ECC is a market segmentation feature.

The real way to make ECC happen industry wide is for OS vendors like Microsoft to make it a platform requirement. A no ECC, no boot policy would change things overnight. Sadly, we can't even get DRAM manufacturers to fix row hammer properly, so the likelihood of this happening is pretty much nil.

You can use ECC by buying the Xeon version which is only slightly more expensive.

Though the motherboards and ram modules are also harder to find and with far less choice. (and actually the processors are too, depending on your market)

Yes, sure, motherboards and CPUs have far less choices. But generally plenty for any normal use case. Motherboards and CPUs have a crazy number of flavors, none of which matter one whit for normal use cases. It's so bad motherboards are differentiating on the manliness of the heat sink design, lighting options, and the color of the motherboard.

If people cared, they would buy ECC capable chips. In fact my desktop is a Xeon e3-1230v5, which as cheaper and slightly slower (3.4 vs 3.6 GHz or something) then the equivalent i7. It was $50 more for the motherboard and $100 more for the ram. I'm sure if the market flooded to ECC capable chips (the silicon is the same) Intel would sell them.

So many people grumble, but I'm not really sure Intel should push ECC if desktops users aren't willing to pay a modest premium for it.

Many cheer AMD, which does not disable ECC on desktop chips, but neither do they promise ECC will actually work. It's a confusing mess between physical capacity (ram increases by 16GB when you add a 16GB dimm), and the actual correction of errors and telling the OS about the event. Only on the EPYC does AMD test and certify that ECC will work.

Your comments about AMD are not entirely true: AMD officially supports ECC in the Pro variants of their desktop CPUs / APUs. I'm currently using a Ryzen 3800X with ECC memory and EDAC reports that it's working. ASRock lists qualified ECC DIMMs for virtually all of their motherboards. It's more that it's a feature motherboard vendors can support if they want to rather than being a platform requirement. But yes, it is sad that AMD doesn't make it an officially supported feature of the platform.

> AMD officially supports ECC in the Pro variants of their desktop CPUs / APUs

ECC is only officially supported on the Ryzen Threadripper PRO, and that's largely because it uses registered memory which will practically always have ECC capability.

Regular Ryzen PRO CPUs have the same ECC support as non-PRO Ryzen CPUs: functional, but not validated by AMD.

Ryzen PRO APUs have (non-validated) support for ECC, whereas non-PRO APUs don't support ECC at all.

Right, but sadly AMD Ryzen Pro aren't sold directly to consumers (well unless you buy grey market on ebay), even if they did there's no such thing as a Ryzen Pro motherboard, or a Ryzen Pro BIOS. Not to mention AMD has been slow to upstream various EDAC related kernel patches.

It's not really a supported config, there's no guarantee it will work, no promise from AMD (other than it's not disabled), and you can't return a Ryzen CPU because the ECC doesn't work.

Various tests from various reddit posts show that some vendors "qualify" ECC dimms as working (an inserted Dimm increases ram available), some correct bits but don't correctly tell the kernel, and others actually do correct and report. So it's basically just a big mess. I'd buy an Epyc, but sadly unlike Intel the premium for a "server" chip is huge, if you can find them anywhere near MSRP. I looked for a Epyc 7313p near MRSP (announced in March 2021) without luck and finally gave up and bought a Ryzen.

With all that said, ASRock is from what I can tell the best AMD motherboard company to support ECC with AMD. I've got a ASRock X570D4U, other than it refusing to allow sharing a IPMI/BMC interface with the system, I'm pretty happy with it.

> If people cared, they would buy ECC capable chips.

People don’t care, because people don’t know. RowHammer-vulnerable RAM is like canned food with botulinum and gasoline with lead. In 100 years, we’ll all be flabbergasted that everyone in the industry wasn’t arrested for letting it happen.

You don't fix rowhammer with ECC, you fix it by tracking row activation counts and then refreshing nearby rows. Without any cheap-outs like using small hash tables.

https://comsec.ethz.ch/research/dram/blacksmith/

I've read that paper, yes. It works by bypassing target row refresh, because all those chips are trying to be clever about their row tracking and failing.

Track row activation counts for real, no estimates and no hash tables, and you're not vulnerable to this.

Heh, dunno, the rates I've seen likely are 100x or lower than other common cause of mistakes, corruption, and confusion.

Do I want ECC, sure. Would bitflips ECC could prevent be in any in the top 10 reasons in year for file corruption, application crashes, and OS crashes etc... unlikely.

You’re not wrong, but that’s not really the point. The rest of the computer industry needs serious reform, too. That doesn’t take away from needing ECC in “consumer” products.

> I'm not really sure Intel should push ECC if desktops users aren't willing to pay a modest premium for it.

$50 more for the motherboard and $100 more for the RAM is not a "modest premium" for most of the desktop market. But aside from that, the bigger problem is that consumers (and especially gamers) will always prioritize the advantages that are more easily quantified. When forced to choose between ECC support and a few hundred MHz plus the option to overclock further, consumers will pick the faster processor.

If Intel offered a part at the high end of the product line that had both ECC and overclocking enabled, it would be guaranteed to sell quite a few units, because a lot of consumers will always want to own the top of the line part. And such a part would be an opportunity for a better experiment to determine how much premium users are willing to pay for ECC, if Intel also kept offering the current -K overclockable parts that don't have ECC.

Well the CPU was $50 cheaper so that offset the ECC premium. I cared about ECC much more than the 166 MHz difference or whatever it was. For people buying an i7 (this was before the i9 I believe) + 16GB ram in 2015, $100 is a pretty small premium (less than 10%). Decent mid range CPUs were $300 (gtx 1070). There were lower end Xeons equivalent to the i5 as well (lacking SMT/hyperthreading).

> My understanding is the stats from hyperscalers is that ECC correction events happen a lot more than "Common Knowledge" may imply - I wonder just what proportion of things that are blamed on software may actually be due to hardware issues like this?

What is the order of magnitude of ECC correction events?

https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf

Looks like a good paper, but transistors have gotten quite a bit smaller since 2009, and voltages have gotten lower as well. So the energy required to flip a bit has shrunk quite a bit since 2009.

Reply on news.ycombinator.com