Options for genuine ECC RAM on the desktop in (early) 2024
utcc.utoronto.ca
utcc.utoronto.ca
Torvalds went on his lengthy post to say, "The "modern DRAM is so reliable that it doesn't need ECC" was always a bedtime story for children that had been dropped on their heads a bit too many times. Yes, I'm pissed off about it. You can find me complaining about this literally for decades now. I don't want to say "I was right". I want this fixed, and I want ECC. And AMD did it. Intel didn't."
https://www.phoronix.com/review/amd-ryzen9-ddr5-ecc
"When plotting the benchmarks with a statistically significant difference, there was just a handful of benchmarks within the 2~3% faster range when ECC was disabled. As shown earlier, the Graph500 HPC benchmark was the outlier with a ~8% difference toggling ECC. PostgreSQL and Hadoop were among the few real-world workloads with the 2~3% difference, which is a small price to pay for the benefits of ECC. It's great that the desktop AMD Ryzen processors continue to support ECC memory while still it's ultimately up to the motherboard vendors whether ECC is to be supported. We're at least seeing more AMD Ryzen motherboards come to market with a server focus which in turn is great for allowing more motherboard options with ECC memory support. My testing with the ASRock Rack 1U4LW-B650/2L2T 1U server (ASRockRack B650D4U-2L2T/BCM motherboard) continues working out very well and hasn't yielded any platform troubles with all of my hundreds of hours of Linux testing thus far."
I wish more motherboards bothered to validate ECC working - it's a nice benefit in an affordable platform.
"It works" as in it boots, or "it works" as in it checks and reports single-bit errors and reports double-bit errors?
What OS? If Linux, are you using rasdaemon? Does it confirm that the appropriate EDAC drivers are loaded and working?
Or is that data only available in the BMC's event log?
For Linux support you need a very recent kernel - I forget how recent, but some reporting on it: https://www.phoronix.com/news/AMD-EDAC-Ryzen-7000-Series
The new Asrock Ryzen 7xxx motherboards unfortunately don't have that same ECC support.
Otherwise, organizations such as Microsoft (windows readyness logo) or RISC-V (application profiles such as RVA23) could make it a requirement.
It is clear that we can't just rely on the hardware vendors to do it, and practically nobody should be running computers w/o ECC.
Why? I’ve done so my whole life with zero problems.
Here's the basic set of problems:
1) ECC is almost free if done at scale (e.g. every computer). The only reason it costs extra is that it is only done for high-paying customers and in limited volume.
2) Computer crashes cost lives. If e.g. your doctor's device has an issue mid-surgery, you potentially pay.
3) It's good for the economy. In either case, the cost savings of not having a call center rep say "Wait a minute. My computer just crashed." far outweighs the costs.
4) It's critical for resilience. Disasters (solar flares, EMP, etc.) are rare, but if every computer destroys data, we have a systemic problem.
5) The issue primarily is one of transparency. Most people don't realize it's an issue, and don't shop on it.
6) That's not to mention dual-market uses. E.g. if we need to ramp up military production, medical production, etc. next crisis, our infrastructure should be ready for it.
Classical economic theory states government interventions do well for disaster resilience, for externalized costs, and for lack of transparency. Those are all places where markets fail. That means this should be regulated. We should have error correction in all storage and in all computation.
Best intervention, though, isn't a ban but a tax. Sales tax should be on a sliding scale from 1-10%, with:
- 1% for parts / support / service manuals being available for some extended period
- 1% for all software being open-source
- 1% for all hardware being open
- 1% for use of standard, interchangeable parts (e.g. batteries, power connectors, etc.)
- 1% for environmental factors (CO2, toxic materials, etc.)
- 1% for reusable infrastructure (e.g. being able to repurpose manufacturing equipment for crisis use)
- 1% for workplace factors (e.g reasonable benefits, audits/compliance, non-discrimination for target populations like ex-convicts, etc.)
... and so on
Those sorts of gentle incentives accomplish the same thing via market mechanisms without limiting freedom. For a commodity product, a 10% edge far outweighs costs. If Apple wants to make a proprietary magnetic screen connector and save 1mm of space, a 1% extra tax won't even slow them down.
This is simply not true. ECC means you need to store a minimum of 11% more bits, and at scale this means that you are paying a minimum of 11% more for memory.
Note that I am very pro ECC, and think that most computers should use ECC ram.
That +10% may become +5% after adding CPU, cooler, MB, PSU, case and it may become +3% after adding SSDs, a high-quality monitor and other usual peripherals.
While at work I have seen plenty of problems with computers without ECC, for my desktops and servers and even for some laptops, when I could, I have used only ECC memory, for the last few decades.
That has never increased the cost as much as I have saved by choosing wisely other components in the computers that I have built myself.
The main problem has never been the extra cost, but the difficulty to find them exactly when I needed them, because at retail the availability of ECC modules may be sporadic and many distributors appear and disappear, so if I upgrade systems after some years I seldom can find the same source.
I'd also expect RMA costs to be lower since you can be sure which DIMM is bad, and that it's not CPU, CPU socket, motherboard, powersupply, etc.
A ECC DRAM module contains 9 chips rather than 8. I'm not sure how you can claim that's "almost free" unless you have a very loose definition of "almost free".
>2) Computer crashes cost lives. If e.g. your doctor's device has an issue mid-surgery, you potentially pay.
Sounds like a good reason to regulate medical devices and force them to undergo certification as well. Surely you wouldn't want the the computer to have ECC memory, but it decides to reboot for a 2 hour update?
>3) It's good for the economy. In either case, the cost savings of not having a call center rep say "Wait a minute. My computer just crashed." far outweighs the costs.
If this is true, why do we need the government to step in? Why aren't businesses doing this themselves out of self interest?
>4) It's critical for resilience. Disasters (solar flares, EMP, etc.) are rare, but if every computer destroys data, we have a systemic problem.
ECC protects against solar flares and EMP? That's new.
>6) That's not to mention dual-market uses. E.g. if we need to ramp up military production, medical production, etc. next crisis, our infrastructure should be ready for it.
1. DRAM modules aren't made in the US, they're made in asian countries with cheap labor. If there's a military crisis, we'd be screwed regardless
2. As mentioned above, ECC DRAM modules are basically the same as non-ECC modules but with an extra chip. The only work to retool a line from non-ECC to ECC is to change the PCBs. Thus the supposed benefits is negligible, unless for whatever reason the military needs a massive amount of desktops.
I do. Let's do the math here. A typical computer has 8GB RAM. That's under $20 at my local retail store. We're talking a $2 price difference. That's with the methods used today. The overhead can go arbitrarily low if we adopt techniques used in nonvolatile storage and work in larger blocks of bits.
$2 is less than the cost of a tiny fraction of a crash.
> Surely you wouldn't want the the computer to have ECC memory, but it decides to reboot for a 2 hour update?
A broader regulatory regime would certainly be good.
> If this is true, why do we need the government to step in? Why aren't businesses doing this themselves out of self interest?
Mostly, because businesses can focus on a finite number of things. A $20 benefit which costs $2 to implement stops making sense if it means your local clinic needs to have someone on-staff who can do an ROI computation on adopting ECC. It's cheaper to buy something sub-optimal.
Again, that's where centralization and regulation helps.
> ECC protects against solar flares and EMP? That's new.
That's literally what they're there for. ECC primarily protects against bit flips, which are the result of random radiation. Mild flares will flip a few bits which ECC will correct. Severe ones will flip a lot of bits, which ECC will detect, and the computer can shut down before destroying more data.
> 1. DRAM modules aren't made in the US, they're made in asian countries with cheap labor. If there's a military crisis, we'd be screwed regardless
I'm not sure what you mean by "modules," but DRAM PCBs are easy to assemble. They can be made in the US overnight. DRAM chips are primarily made in the US (Micron), Korea (Samsung), and Taiwan (TSMC).
https://www.micron.com/manufacturing-expansion
And not all crises are military.
> The only work to retool a line from non-ECC to ECC is to change the PCBs. Thus the supposed benefits is negligible, unless for whatever reason the military needs a massive amount of desktops.
If all you need is memory, that's true. If you also want motherboards and processors which will take those chips, that's harder. If you don't want bit flips in your computation, that's impossible without redesigning all CPUs. Academic CPUs can do error-correcting computation at nominal cost, but there isn't a market incentive to bring that to market.
>Sounds like a good reason to regulate medical devices and force them to undergo certification as well. Surely you wouldn't want the the computer to have ECC memory, but it decides to reboot for a 2 hour update?
Did you know that medical reports are frequently viewed on consumer or consumer-grade hardware? Or that there is software that is classified as a class II medical device that runs on iOS devices and commodity desktops? Or that class III devices like pacemakers now often use phone apps for things like managing telemetry? It's all well and good to make sure the hardware in the OR is reliable, but that's woefully insufficient.
The power consumed by the relatively few gates that generate and verify the ECC code is dwarfed by the power consumption needed to transport bits over the PCB traces.
The extra memory cell arrays for ECC bits add some power consumption for refreshing their content and the extra PCB traces for the ECC bits add some power consumption, but this extra power is in the milliwatt range, so it is negligible in comparison with the CPU and GPU power even in a small laptop, not only in a desktop.
Most gaming desktops use overclocked memory modules that have an extra power consumption many times greater than the DDR5-4800 or DDR5-5600 ECC UDIMMs.
The ECC UDIMMs never need heatsinks, while many of the commonly used overclocked DIMMs will overheat without heatsinks.
So the claim that using ECC will increase the power consumption is completely bogus. Something like using a power-save profile instead of a high-performance profile, for clock management, or using JEDEC timings for memory, instead of high-performance timings, would diminish the power consumption by orders of magnitude more than not using ECC memory.
Right now, I power rails set to where billions of bits to never flip. If I can tolerate a one-in-a-billion bit flip, my tolerances go up a lot, and power consumption goes down by a similar factor.
Properly implemented, ECC would likely cut power while increasing reliability.
"Properly implemented" only happens in volume. That sort of optimization makes no sense for the very limited volumes ECC has today. Neither do a lot of other optimizations.
Still, those sorts of tweaks are more important for battery-powered devices. I'm not sure the impact on server power consumption would be important. RAM is not power-hungry relative to a Xeon or an H100.
Given that software can also crash, I'd go with a different approach: Redundant Array of Inexpensive Computers. For most situations, RAIC1 is good enough.
The manufacturer has RMA'd the whole set of memory sticks, but ECC would've detected this earlier and I'd have saved a couple of hours of my time.
And folks experienced zero file corruption… until they started using ZFS checksums and regular scrubbing.
I am happy with the point in the price/reliability/power usage tradeoff that non-ECC ram represents. Why force me to have fewer choices and use something that is worse for my needs in order to solve a theoretical problem that I don’t care about?
In general don't worry that for very important calculations, they are run a couple of times, at least just to verify.
Citation needed, as this is unheard of in software we trust regardless to do banking or sign legally binding documents with.
For such a minimal cost I'd rather have DIMM 3 died with a non-correctable error.
Measures are in the works there, such as the switch to a simpler and cleaner ISA.
The value of simplicity is often overlooked; RISC-V will lead to higher quality software and hardware.
But I digress, as this has no weight in whether or not lack of ECC is acceptable.
* https://arstechnica.com/gadgets/2009/10/apple-abandons-zfs-o...
* https://web.archive.org/web/20091103163642/http://mail.opens...
There's no reason why any newly created file system shouldn't have checksumming, and if grandmothers don't know what's happening underneath the hood it's no worse than them not knowing how virtual memory or the TCP/IP stack work.
Some years ago I built a small N3150 based home server with 16GB RAM. All was fine. But there was this one process that would always act weird after running for a couple of days. It was a Python server with long-running WebSocket connections which I had configured to use `select()` instead of `epoll()`, and for some reason it would eat up file handles until it crashed or froze, can't remember what it was, very odd stuff. I think it was running inside a Docker container and that container then behaved very strangely.
Turns out that when I then made a MemTest86 it had faulty RAM. All the data which passed through that area of the RAM and got written to disk was assumed to be corrupted.
After replacing the RAM (RMA) and doing a MemTest on it for a day to ensure that it was OK, that problem never appeared again.
But keep in mind, this was just one specific process which showed an issue after running for a couple of days, among other long-running processes.
2 >> There's also the chance that you think you had zero problems because you never got notified.
Both are likely true. I.e. Over my 40 years of dealing with computers ,it is likely there were errors which did not become problems.
I have about 20tb of unique files in my personal storage at this point (used to run photography business, plus personal videos and photos,plus movies and tv shows, plus actually important documents which are less than 3% by size).I have about 4-8 computers running around my home doing stuff (plus dozens of computing devices - phones, googles, whatnots).
I understand the overwhelming chance there were many errors over time in running computers or raid transfers or usb backups over decades. They just haven't manifested themselves as problems.
And yes, the corruption is there. DRAM is alarmingly unreliable. Because no ECC, we just do not notice it most of the time, or misattribute the problems it causes.
See https://www.truenas.com/community/threads/ecc-vs-non-ecc-ram...
Also, when you download large files which you can run checksun against such as steam games, it is pretty common to find minor corruption (this i verified across many computers)
See also: PDF https://media.defcon.org/DEF%20CON%2021/DEF%20CON%2021%20pre...
Them not having ECC is insanity.
There are of course niche cases where memory reliability does not matter. Exceptions can be made for those few.
But think about the general case first. Got to protect the non-technical from non-ECC memory.
I guess the computing world has not survived up to this point then?
freedom TO vs freedom FROM
Every time when a program crashes or remains stuck or data corruption is discovered, the user automatically assumes that this is a software bug.
While it is indeed true that software bugs are really much more frequent than memory errors before the memory modules happen to become defective, in a computer without ECC the user can never know whether it has been a software bug or it was a hardware memory problem.
So many memory errors remain hidden by the abundant software bugs, which in most commercial programs, especially in those from Microsoft, almost never generate error messages from which the cause of the error can be determined.
Example devices are: LattePanda Sigma [0] and AsRock Industrial NUCS BOX-1360P [1].
Unfortunately they are quite expensive and enabling in-band ECC lowers performance significantly. So my next server rig will likely have an AMD PRO CPU instead.
[0] https://www.servethehome.com/lattepanda-sigma-review-the-ras... [1] https://www.anandtech.com/show/18732/asrock-industrial-nucs-...
I'm hopeful that this is going to help fix problems with occasional bit flips from cosmic rays and similar. Since it is used to keep marginal chips in the product stream, I'm not sure if it is really beneficial.
The Raspberry Pi 4B claims DDR4 ECC RAM and I'm pretty sure that's on die ECC.
I'd really prefer full blown ECC RAM and would pay a reasonable increase for it. I recently upgraded my desktop and sticking with ECC RAM would have cost hundreds of dollars more so I skipped it.
[0] https://en.wikipedia.org/wiki/Reliability,_availability_and_...
https://www.supermicro.com/en/products/motherboard/h13sae-mf
The DRAM bandwidth per core is identical for Ryzen 7950X and for 96-core Genoa Epyc CPUs. On the other hand, the Epyc CPUs with high-core count have a better performance per watt.
So the initial cost for a cluster of Ryzen servers is many times less. Depending on the cost of electricity, if an Epyc server is used 24/7, after some years the expenses with it may become lower than with Ryzen.
If the server is used intermittently, the total cost of ownership may remain lower with Ryzen until the end of life.
The only certain advantages of Epyc are the ability of aggregating a higher amount of memory, especially if it is preferable to have it inside a single box, and the faster inter-core communication for applications that use all cores (as opposed to the case when the cores are partitioned between weakly-coupled applications, e.g. between different virtual machines).
The prices of the Epyc CPUs have increased a lot since their first generation until now.
With Zen 1, a server with any Ryzen would not have been competitive with a server with Epyc. Meanwhile, the ratio between the prices per core of Epyc and Ryzen has increased a lot, while the ratio between the performance per core of Epyc and Ryzen has decreased a lot, because the clock frequency of Ryzens has become much greater while that of Epycs has increased only a little.
These two evolutions combined have made that now the servers with Ryzen have become preferable to servers with Epyc in many cases.
AMD has realized that they no longer have a solution for cheap servers, so in theory they have introduced the Siena CPUs for this purpose. Nevertheless, those remain somewhat too expensive and moreover they are nowhere to be seen.
Please, I don't want to hear "you don't need ECC", I just want some recommendations.
Of course the benefit of ECC can be debated. But it seems such a waste to bother debating it when Intel could just make the whole thing a non-issue by a flick of a switch.
If storage drives reported writes accurately and memory didn't occasionally silently corrupt, software falling over would be more likely to imply an error the developer is empowered to fix.
But whenever you get a bug report, is really the first thing that comes to mind "maybe their hardware had a transient error?"? The amount of software vs hardware bugs is still like a thousand to one, and that's _even_ for companies which receive a disproportionate amount of hardware errors (like OS vendors). For regular software, I'd bet it's a couple million to one.
This isn't inherently evil behaviour.
AMD is interesting in this regard. ECC on desktops. Lots of x86 chips have some integrated GPU. Lots of PCIe lanes. The idea seems to be biased much more towards build a great platform, as opposed to build a great profit margin.
(just like AMD also segmented based on SMT up through at least the ryzen 4000 series despite the fans bashing intel for the same, lol)
Moreover, with amd one increasingly has to worry about the platform lock, which is an absolutely obscene e-waste stream AMD has created to reduce the secondhand sales market. This isn’t limited to server CPUs or pro CPUs, notionally it affects all CPUs that might have ever passed through an oem system, including non-pro consumer CPUs etc.
https://youtu.be/bFNJVaO9E-o?t=420
The stated reason is security/antitheft, but the platform lock doesn't lock a CPU to a motherboard, it locks it to a brand. So if someone has physical access to your server rack and wants to steal the CPU, they just have to find a lenovo-branded or hp-branded chip to swap it out with... and there are rivers of those because locked chips are essentially treated as e-waste. What it does do very successfully is throw a whole cloud of confusion and pain over the secondhand market and drive these to be scrapped etc and push sales of new devices instead.
AMD could very easily have implemented a lock/unlock system using non-volatile storage or simple e-fuse even/odd counting (limited number of locks/unlocks, granted, but it carries with the CPU not the mobo, and how many systems is the average CPU used in?). They, naturally, chose the one that is permanent but bricks the cpu for everyone else, but which also doesn't really fully prevent CPU swapping attacks either. Plenty of e-fuses to spare for detecting memory overclocking though - we all know denying warranties comes first ;)
For the type of person who gets wound up about the moral imperative of ecc, that’s not exactly what you want to see from AMD either. First 2 Rs are "reduce" and "re-use", followed by "recycle". Like how about we don't tivoize the cpu in 2018 or whatever? Secondhand sales are just as much of a moral right and AMD is really having the old college try at taking that away, and frankly a lot more impactful to the average user. I strongly wish the EU would step in and ban this practice. This is not a good/healthy path, it would actually be really bad if everyone was doing this on their products.
Also, Intel has ECC support on the upper half of their range. Yes, it requires motherboard support, but you also have to pick and choose with AMD because only a handful of motherboards actually support/validate it anyway. Plus with AMD you have the fTPM issue (and very few of the premium creator boards/etc have external TPM headers). This is the reality as a homelab owner - you have to pick your hardware very carefully regardless of brand, a random B650 board probably is not going to report errors and may not even detect them. And while it may work on any given BIOS, it may stop working at any future point, whether intentionally or not.
I remember news stories about the platform lock but haven't kept up to date, do you know how that played out in practice?
My asrock board doesn't seem to have an option to burn out fuses to stop it working elsewhere but I am slightly less cavalier about changing settings knowing that might be in there.
Yeah the lack of iGPU support (plex encoding, VMs, etc) pushed me to the 7100 last time I did a homeserver build. It's been muddled on intel's line forever - for a while it was only i3s (since they dropped the idea of a 2C xeon there was no reason to keep segmenting it) but then with alder lake they flipped it and it's on everything 12500 and above (except F skus). And while you do need a motherboard that supports it, W680 supports overclocking/XMP/etc - the tier of boards I'm looking at aren't materially different in cost (or actually intel often trends cheaper).
The platform lock has actually affected me quite badly trying to do a ROMED8-2T build, a massive number of the desirable CPUs are platform locked and unless they explicitly say unlocked it's always a question whether any given chip is actually unlocked or just doesn't say/doesn't know. Which is what I see as the whole point - get those machines into the scrapper (zero value means nobody bothers reselling) instead of people buying perfectly good older cpus and drawing extra life from them.
I am not thrilled about the idea of that bleeding into the consumer market at all. So far AMD hasn't gotten much traction with OEMs but that's changing over time, and I see every reason for AMD to incentivize OEM partners to use this functionality across their whole lineups - OEMs want cheaper chips and AMD can sell them cheaper because they know they are junk after the first sale. Why wouldn't ThinkStation SFF PCs etc come with locked processors at some point? AMD clearly has laid the groundwork and all the tooling is there and just is boiling the frog very cautiously.
By the time people see the impact in the consumer market it will be too late. And there are a lot more ThinkStations and Optiplexes sold than DIY PCs.
Correct single bit errors and alert me to them.
The value of the notification is that if most days you get nothing and suddenly 5000 corrected errors pop up, you know something is not quite right as of late. Maybe you're overdoing the overclocking. Maybe the module isn't sitting quite well in the slot. Maybe a solder joint is failing.
If you have a 2 bit memory error, the typical result is a BSOD/kernel oops.
Which is a good thing actually. It means you know for sure your RAM is bad, and it prevents any faulty data from being written anywhere. So rather than possibly propagating the error is just stopped.
I believe there have been systems that try to handle it more elegantly. Kill the process the memory belongs to if it's a non-essential one then permanently blacklist that block to ensure it's not used again.
No I've set it up so I get an email.
> Because if yes, you'd get such a popup daily
I doubt that. There was a study done at Google that reported 8% of DIMMs encountered at least one memory error per year[0].
> I'm not sure what the actual value of it is.
I like my PC to be as deterministic as possible[1] so if my programs crash I don't want to have to turn it off and test the memory which takes hours[2].
[0] - https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf
In my experience the most valuable benefit of ECC is the detection of aged memory modules.
All modern DRAM has a limited lifetime. Hopefully that lifetime is more than 5 years, but there are always some modules that may reach end-of-life much faster, especially on computers that are always on.
While on a new memory module it would be abnormal for memory errors to happen more frequently than a few times per year, on an aged module errors may happen many times per day, or even many times per hour.
I have encountered a few such cases that happened after several years of continuous using. Due to ECC I was notified about the errors and I could replace the offending modules before them becoming so bad as to guarantee data corruption.
Another reason is that even if ECC cannot prevent a RowHammer attack, it makes pretty certain that a RowHammer attack will be detected immediately (because the attacker cannot control the location of the flipped bits, and even if there are many errors, so they will be miscorrected, in most cases they will still be detected and logged and the administrator should always be instantly notified when there are two or more errors per day, as this will never happen normally).
When a similar problem occurred on a PC with ECC RAM, I got a warning message saying memory errors had been detected that repeated a few times, then I got a message saying one stick of RAM had been deactivated and that I could keep running. The Dell T3500 dropped from 24 GB to 20 GB and did not even need to be rebooted.
That is why I love ECC RAM and Xeon processors. I know if there is a problem and can wait to fix it.
I can buy used RAM and processors with confidence that the BIOS self rests will tell me what is good and what is bad.
It seems RDIMM’s are mostly supported on higher end server boards but I haven’t seen a technical comparison of the two types.
The 32 cores is pretty pointless, I very rarely use more than half. And it probably uses way to much power.
You just need to buy ECC UDIMM modules, which, especially for DDR5, are more expensive, up to 50% more expensive for DDR5, even if this price difference is not justified.
Moreover, the ECC modules are never factory overclocked, like most non-ECC modules, therefore ECC UDIMM modules faster than DDR5-4800 are hard to find. Nevertheless, that has not stopped some users to overclock them manually.
There are some embedded computers for industrial applications, with certain models of Intel Atom or Core CPUs, which support the so-called in-band ECC, which can be used with non-ECC memory, like LPDDR5. Only on those computers enabling ECC reduces both the available memory and the performance.