Hacker News Reader

Comment by pedrocr | Hacker News Reader

Finding a motherboard that supports it is relatively easy. The Asus Prime X370 Pro for example seems like a good choice for a simple home server with ECC and 8 SATAs. The problem is actually finding reasonable ECC RAM. Unregistered/Unbuffered ECC RAM is an unusual configuration that most manufacturers don't provide. It's hard to find, expensive and much slower which Zen is supposedly sensitive to.

Shouldn't we have moved to ECC RAM everywhere a long time ago? With economies of scale would it actually be any more expensive or slower? There's no place where the extra safety is a negative, is there?

View on HN

ECC RAM not being in consumer PCs is largely a market segmentation pushed by Intel[1]. It is in fact quite ridiculous if you consider that essentially every other bus, interconnect and storage in your computer has error correction except _main_ memory (and the main memory bus). If it weren't so normalized we'd go like "Dude, do you even realize how lol it is to have no parity on all of the most important data the computer is working with?!".

Also the lost productivity due to main memory errors not being detected probably easily goes into the billions. Thanks, Intel.

[1] There was a time when consumer systems genuinely didn't support ECC for lack of hardware support. This hasn't been the case for many, many years.

Intel: Fuck You, Pay Me.

I don't believe any CPU cache has ECC, either. Which is where the memory you're actually working with lives.

Also no other interconnect in your system is as much of a bottleneck as the one to main memory is. It's worth keeping that in mind before entirely blaming this on "market segmentation". ECC RAM does actually slow down the part of a system that is already the bottleneck in most common situations.

> I don't believe any CPU cache has ECC, either. Which is where the memory you're actually working with lives.

I'm not aware of any desktop CPU that doesn't have ECC caches. CPU internal busses use ECC and external interconnects (e.g. PCIe, DMI, QPI/UPI) use it as well.

> ECC RAM does actually slow down the part of a system that is already the bottleneck in most common situations.

ECC invariably introduces some additional latency in the memory controller, but I don't see a persuasive argument why it would reduce throughput. It would surprise me if this additional latency is measurable, given that the ECC logic is already in the core and in the data path anyway, and the system configuration (AMD, Intel) / CPU fuses (Intel) only decide whether it is active.

That being said buffered ECC modules are usually not the fastest. I don't think that this is due to any technical limitation per se, but rather market demand (cost, perf per Watt).

> I'm not aware of any desktop CPU that doesn't have ECC caches. CPU internal busses use ECC and external interconnects (e.g. PCIe, DMI, QPI/UPI) use it as well.

I believe all of those are just parity checked and not ECC?

> ECC invariably introduces some additional latency in the memory controller

Latency is a non-trivial factor here, too, though.

> That being said buffered ECC modules are usually not the fastest. I don't think that this is due to any technical limitation per se, but rather market demand (cost, perf per Watt).

Poking around it looks like ECC RAM tops out at DDR4 2666 @ 1.2v. By comparison there's no shortage of DDR4 3200+ options at 1.2v. Whether or not this is purely market demand or not, it doesn't seem like there's a purely power reason for it.

But you also can't solely blame Intel for a lack of market demand. Even when the choice is there nobody seems to be making ECC memory for high-end desktop usages. Where's the DDR4 3200 for Threadripper or Xeon-W workstations, for example? They surely benefit from the improved bandwidth, or else they wouldn't have triple & quad channel memory. And they'd surely pay the price of admission, because we're talking $3,000+ entry points for builds.

Caches are definitely ECC, because the corresponding MCEs can tell the system both that an error was detected and corrected, and that an uncorrectable error occured (which by default leads to a kernel panic iirc).

Actually, all modern CPUs have ECC protected caches and they have MCA events to report any problems with them too.

Are you confusing parity with ECC?

No.

I haven't bought new RAM lately, but not that long ago it was often much cheaper to buy an old server and outfit it with used ECC DDR3 than to buy equivalent consumer RAM, simply because there wasn't much demand for previous-generation ECC RAM.

Right now DDR4 is fairly new, but as old servers get rotated out I expect a good market for cheap ECC DDR4 sticks that come from used servers but are too small to get reused in new servers. (unregistered/unbuffered is still a problem though)

The main advantage of ECC is reliability so buying used RAM doesn't seem like a great option. If you're willing to buy used a good way to get a nice workstation is to just buy a used workstation machine (e.g., HP Z400/Z600/Z800 line) and just upgrade the storage and GPU. But if you're gaming or trying to upgrade a home server what you want instead is a nice Motherboard/CPU/RAM combination. Right now Ryzen would be a great option for that if there were some good ECC UDIMM options.

Server RAM would be RDIMMs or LRDIMMs while consumer processors mostly can support only UDIMMs.

ECC RAM by design is more expensive. Usually every 8 bits need a 9th parity bits, so you need 9/8ths more RAM to support ECC. Plus you need to make the parity calculation for everytime you go to RAM.

Whether it's needed or not...that's use case dependent.

I know ECC is more complex, my hypothesis was that in volume the difference would be small enough to not care and that we're only stuck with non-ECC RAM because of path dependence.

>Whether it's needed or not...that's use case dependent.

Even for gaming, having your game crash/misbehave because of bit flips is at the very least annoying. But maybe this is uncommon enough that it doesn't make enough difference?

Gaming systems are often overclocked and aren't really stable often enough in the first place.

> Even for gaming, having your game crash/misbehave because of bit flips is at the very least annoying. But maybe this is uncommon enough that it doesn't make enough difference?

My gaming system with non-ecc memory will run memtest all day long never spotting a single error.

I know these errors can & do happen, but I'm fairly certain the number of crashes I've experienced that would have been prevented using ECC memory is single digits at most.

But ECC being in general slower than non-ECC? Well that's very noticeable.

ECC is useless until it is not. Just like insurance, backups, etc.

The failure risk of ECC is significantly lower than a lack of insurance of backups, though.

Drive fails and no backups? Potentially terabytes of data vanishes. House burns down and no insurance? Hundreds of thousands of dollars to repair. No ECC and a bit flips? Nothing happens, program crashes, or maybe a single file gets corrupted in an unrecoverable way.

And maybe that file happens to be an important encryption key. Or maybe a whole filesystem gets corrupted if some important metadata is. Or maybe your system develops a hardware problem over time, like an oxidized CPU pin or a contact in a memory slot and then you are getting flaky bits on a regular basis. ECC is useless until you experience a problem of those kinds. And many people will never do. But some will. I have.

It's not just 9/8 more RAM, but 9/8 more memory bandwidth on the bus. So, yes, a pretty nasty performance hog for typical use.

I wonder how feasible it would be to build ECC ram where you could toggle the ECC part and just use the extra capacity if you so wished.

ECC is a hardware level implementation usually.

But why does it have to be implemented in the DIMMs? Why not in the memory controller, such that any RAM could be used with or without parity?

If you do that, you're moving the cost on to the motherboards and increasing the cost for something most consumers won't get any advantage out of.

ECC logic is implemented in the memory controller (in the CPU these days). The ECC DIMM just provides extra chips to store ECC bits. And the motherboard, if it supports ECC, provides just extra data lanes that connect the extra DIMM chips to the appropriate pins on the CPU.

Is your typical performance limited by memory bandwidth? Or rather by latency?

Once again...that's use case. But ECC will affect both. You have 9/8 more bits you have to get from RAM, then you have to doo the ECC calculation for each byte

The bus is also 8 bits wider, so it’s a wash.

In other words, a 32GB ECC memory stick has to include an extra 4GB for parity.

That's actually a price I would be willing to pay.

So you will need to pay 360$ instead of 320$. Many people would choose cheaper memory. I guess, almost everyone except some PC enthusiasts (last time I discussed it, the majority of PC enthusiasts thought that ECC on desktop is not needed, so they wouldn't want to pay for it). I agree that ECC is nice to have, but price is real.

RAM prices already vary by more than 12%, and it seems like there is no shortage of people paying premium for brand recognition, different board colors (green is "out"), useless heat sinks etc. I think there would be plenty of people willing to pay extra for ECC (some for peace of mind, some because they need it, some just to feel superior)

The market segment of "people who are willing to pay a premium for computing devices" is pretty vast, isn't it?

Most people would be fine with a $200 Chromebook, $500 bare-bones Windows notebook, or a garden-variety PC from 2007 but millions of us choose to pay more because we value the additional things that newer, more powerful computing devices give us.

Lots of professionals and enthusiasts gladly pay large premiums for higher-spec gear, even when the improvements are quite small, because those small improvements are enjoyed over many thousands of hours of lifetime use.

Perhaps more to the point, you already see gamers paying premiums for higher-specced memory to enable their overclocking and tweaking endeavors.

So I definitely think there's a market of people who'd pay more for ECC...

I guess they will pay for higher frequency, but for ECC? Not so much. It won't help to achieve higher geek rating after all. I might be totally wrong about it, of course.

Just like I paid extra for a nice PSU instead of the very cheapest, I would pay more for ECC.

The cost difference for ECC amortized over the life of the hardware is negligible compared to the annoyance and time spent trying to work out what's causing those random bluescreens/reboots/corruption.

Pretty sure my next PC build is going to be Zen 2 (3xxx series) and I'm going to try and get ECC memory for it. Even if it costs a little more/a little slower I think the knowledge that my data hasn't been corrupted is worth it.

Same. I've been through two computers that have randomly been flaky. It turned out, after 2 years of debugging, that the PSU didn't really like suddenly having load on it (like when the CPU turbo-boosts 8 cores) and that that was leading to crashes. (It was a 1000W PSU, too; I thought overprovisioning would solve all my problems, but I guess not.) I suspected the memory the whole time, though, and having ECC would have at least been one less thing to worry about.

The other thing that annoys me about PC hardware is that the motherboard tries as hard as possible to make your system unstable. I don't want overclocking. I don't want to run the memory at XMP speeds. Just give me a button for "run everything at its conservative spec". (With that in mind, I'm not sure memory manufacturers test anything other than their XMP timings, leaving you to guess whether the non-XMP profile has the right voltage/latency numbers. It's infuriating!)

> It's hard to find,

This wasn't my experience either on memory.net or crucial.com

> expensive

Yes it is, much more than the 1/8th more RAM would require.

> much slower

to quantify, the fastest currently is DDR4-2666 while non-ECC goes to the crazy DDR4-4700.

>This wasn't my experience either on memory.net or crucial.com

As far as I can tell crucial.com shows a total of 2 options. Both 16GB UDIMMs. One tall, one short. I think last time I checked it had none. I didn't see a single option on memory.net. Remember that you need Unbuffered/Unregistered ECC (UDIMMs with ECC) and there aren't many options of those. RDIMMs will not work.

It's also hard to verify the DIMMs will work because the motherboard's QVL doesn't list any of the ones I've found so far. Not all retailers will have these either, so that seems hard enough to find for me.

People have mentioned better luck buying Samsung-branded ECC UDIMMs. I heard, but couldn't verify, some of their modules are even on the latest QVLs.

> It's hard to find, expensive and much slower which Zen is supposedly sensitive to.

Hard to find -- perhaps. Although these days, with internet and so many sellers, even hard to find things are quite easy to find.

Expensive -- just a little bit more than non-ECC RAM. Probably because of the couple more chips ECC RAM uses.

Much slower -- slower for sure, but again, I think it's not much slower.

Kingston, Samsung, Crucial / Micron all sell UDIMMs.

There are only a few modules of 2666Mhz DDR4 ECC RAM by any of those manufacturers. Non-ECC goes to almost 2x that speed and there is plenty of choice. In the places I've seen it's around 40-50% more expensive for 12.5% more RAM chips. These are not small differences.

Reply on news.ycombinator.com