Hacker News Reader

Comment by vbezhenar | Hacker News Reader

ECC support in AMD systems is strange. It's supported theoretically, but practically there are issues, one have to carefully pick motherboard and even then it's some kind of unsupported configuration. Intel sells cheap and fast Xeons with proper ECC support. I'm very interested in AMD CPUs and I hope that ECC story will improve, so I can buy some kind of workstation-branded motherboard and use fully supported ECC configuration.

View on HN

The difference is that AMD doesn't disable ECC support in any model line, while Intel disables it, sometimes without rhyme.

Extra funny when you notice that certain Xeon lines are actually i7 with different branding and ECC left enabled.

The problems with ECC on AMD comes from consumer vendors not putting the time into testing, and possibly not even connecting the ECC lines (remember, ECC requires putting additional traces between memory controller and memory slots). Then you have to deal with whatever customisation the vendor of the motherboard did to firmware - their changes might have resulted in effective disabling of ECC.

With Intel, you either have the same game as above (with the non-Xeon ECC-capable parts), or pay through the nose for comparable performance "workstation/enterprise" gear, as ECC support being used for market segmentation by intel is pretty much an open secret.

You don't need to pay through the nose with Intel, at least for latest generation. 10900K costs $499. 10900K with ECC called Xeon W-1290P and costs $539. That's 8% extra. ASUS Pro WS W480-Ace is $280 which is reasonable cost for a good motherboard.

I picked up a i3-9100 a few months back because its was a low cost processor with ECC (for an edge/embedded solution). The problem then becomes the motherboard, and it seems intel has just shifted the ECC tax from the processor to the motherboard/chipset. That core fits on a lot of low cost motherboards, but to enable ECC requires about another $100 chipset tax.

How do you discover which Xeon is the i7 version of the same chip or vice versa? Is it just spelunking through Intel Ark to find a same generation, clock speeds Xeon or is there more to it?

Any of the Xeon Ws are just rebranded i7s, others might be as well. You can tell because they have the DMI memory bus instead of the Xeon exclusive UPI interconnect.

Something like that, yes. Specs are almost identical. There are CPU families so I picked best models from a family and compared them.

Would be interesting to compare die shots.

I was with you until the mobo cost. Wtf, $280 for a mobo?? A totally decent B550 mobo with ECC support can be had for less than half that price.

The only advantage of Intel Xeon at this point is RDIMM/LR-DIMM support on their Xeon-W platform. AMD supports that only on Epyc and Threadripper Pro.

Yeah, with Haswell you could get an i7-4770 for like $320 or a Xeon E3-1245v3 (includes an iGPU) that ran the same clock speed and spec, generally, but was about $270. If you went for the 1241v3 you lost the iGPU and 4 watts of power usage but gain +100mhz base clock.

And that gen of CPUS had low power T-series chips rated at 65w where today it's 35w. The power drop is part of why I'm happy with my SFF setup as the thermals are quite good on 14nm locked CPUs like my 7700.

ECC support is iffy at the consumer brands. Its a "we won't disable it, but we won't guarantee that it works" sort of deal.

If you want verified ECC support, you need to buy the workstation chips and motherboards: Threadripper Pro or EPYCs.

ECC is supported on the Pro series as well. My home server is running Ryzen 5 Pro 4650G (yay for integrated graphics) and Asrock B550M.

I went through the effort of using qvl memory, but actually testing ECC is a bit more difficult. While ECC is supported & active, memory errors are sadly not reported to the OS. I remember seeing a forum post somewhere of somebody overclocking/undervolting the ram to force errors, but I can't seem to find it right now. There's a fine line between stable, stable with recovered errors, and unstable.

That's what I'm talking about and I wouldn't call it "fully supported". I want to know about ECC statistics. It's important because if I can see that ECC recovers abnormally high number of errors, it's likely that I need to replace RAM right now.

ECC statistics are available for Ryzen if you use Linux (configured to load the appropriate EDAC module).

So on Linux, ECC is fully supported, even with Ryzen.

Do you know for which motherboards this is true? I think ASRock Rack X470D4U should be fine, as they market it as server board.

https://www.servethehome.com/asrock-rack-x470d4u-review-amd-...

I have used an older ASRock MB with the first generation Zen, and it was OK with ECC.

With Ryzen 3xxx, I have used ASUS Pro WS X570-ACE ($315), which is sold as a workstation board, so you definitely should expect ECC to work without problems, and also the Mini-ITX MB ASRock X570 Phantom Gaming-ITX/TB3 ($230), which also worked OK.

I expect that the other ASRock MB's (most of them or maybe all of them specify the support of ECC) also work OK with ECC modules.

The ASrock Rack server board should also not have any problems with ECC. IIRC that server board supports only up to DDR4-2933, but until now faster unbuffered ECC memory modules were not sold anywhere, so that is not a disadvantage.

Edit: I have looked again at the ASRock site and they have updated the memory support specification. If you use only 2 UDIMM modules (i.e. up to 64 GB total), then you can use up to DDR4-3200 (which I have never seen offered anywhere until now; 2666 is easy to find, 2933 is also supported by Intel since March, so it should become available soon).

Sadly, the IPMI implementation in the X470D4U series is awful. The remote console crashes frequently and is generally pretty unreliable. I'm disappointed that Supermicro doesn't have any Ryzen AM4 server boards. AMD getting ignored by many of the server vendors is just a repeat of what happened when the Opteron series was first introduced. At least there are a number of Epyc server boards available.

I don't disagree with you, but when was the last time you ever had to replace a stick of ram?

This year for me.

What operating system? On Linux, I had a memory stick that was not completely inserted, and periodically I saw corrected memory errors reported in the logs until I fixed the issue.

Indeed, forgot to mention this is on Freebsd.

https://forums.freebsd.org/threads/how-to-find-out-if-ecc-is...

The latest BIOS for ASUS Prime X370 Pro has ECC explicitly as a configuration option. Seems to work in Linux. I am using 2x8R ECC 2666Mhz RAM from Kingston.

Interesting! Do you think this BIOS or ECC works also on X370 Crosshair VI Hero? Hard to find info on this...

It's up to the motherboard vendor. They'll guarantee that it works if you use memory from the memory QVL.

Example: https://www.asus.com/no/Motherboards/Pro-WS-X570-ACE/HelpDes...

What Xeon and Xeon motherboard with ECC support are "cheap?"

In that price range, AMD markets Threadripper and Epyc, both with proper ECC support.

ECC support in Ryzen systems is up to the motherboard manufacturer, and some manufacturers advertise support very clearly. E.g., at least a couple years ago, ASRock explicitly supported ECC in all their Ryzen motherboards.

New workstation Xeons (Comet Lake), e.g. Xeon W-1250, W-1270, W-1290 are hard to find online but can be seen, in few places, for 350,500,800 euro. Motherboards for these Xeons (socket LGA1200) seem to be a somewhat pricier at 300 euro, but they positively have ECC. Intel still costs more for same perf/capabilities, but if you need ECC and Intel platform, it isn't crazy.

Most people wanting ECC just for the warm fuzzy enthusiast feeling would prefer Ryzen with good MB that supports ECC. Intel makes more sense at the high-end many-core Xeons, due to supporting huge memory with much higher throughput (LGA3647). But that segment is bonkers-expensive.

That's an interesting kind of enthusiasm. The usual PC enthusiast thing is to overclock memory to hell until the system starts glitching and dial it back just a little and call it a day :)

I don't feel myself confident with flaky hardware. I'm using UPS and ECC, so I can be sure that no bugs are introduced by hardware issues. I would love to use GPU with ECC, but they are ridiculously priced, so I'm out of luck.

That's not a real need, of course, it's just some kind of whim, but if I can pay for it, why not. Just like those crazy overclocker dudes pay for their whims.

You are right, but if you buy a motherboard that claims to support ECC, you will usually not have any problems.

For example I am using an ASUS Pro WS X570-ACE, which is a reasonably priced workstation board ($300) with a Ryzen 7 3700X and ECC memory.

ECC worked OK, without any problems. I have also used a couple of ASRock MB's and ECC also worked OK on them.

I would much prefer more guarantees from AMD, but rather than buying a slow Intel CPU I prefer a little risk with AMD.

AMD guarantees ECC works with their workstation focused Threadripper line. On Ryzen, it only works if you do your homework picking hardware.

It's a shame they made the TR platform much more expensive in the last generation.

Is that a new thing? I have an older Threadripper and my motherboard definitely doesn’t support ECC (even though the CPU does)

As far as I know all Threadripper chips and board were required to support ECC. Mind you this means UDIMM ECC commonly targeted at workstations, not the RDIMM ECC used in servers.

Dumb question. Why would one want ECC in something that isn't a server? How often do bits in memory actually flip by themselves for it to be warranted?

According to Google statistics, bit flips are encountered at a rate of 8% per DIMM per year. https://research.google/pubs/pub35162/

You want ECC in your desktop if you cannot afford your data to become (silently) corrupted. A single bit-flip in a JPEG image can totally ruin it. And you won't notice without opening it, because the thumbnail is unaffected. When you finally notice, the corruption has possibly spread to backups already.

For servers that is especially important, because data will reside cached in memory for days or even weeks. Also ECC is often the last line of defense against Rowhammer.

https://www.cs.virginia.edu/~gurumurthi/papers/asplos15.pdf is the latest study I can find. Some key findings; error rates per DRAM module and per technology cycle (DDR2, DDR3, and so presumably DDR4) have similar failure rates. Higher altitude has higher failure rates.

FIT (failures in time, or failures per billion hours) are about 20-30 per DRAM (18 per DIMM in the study) at a fault level, which may not rise to an error condition. ECC corrects many faults, if present, which don't become errors.

So expect a (4GB in this study) DIMM to have a fault once per 200 years on average. Of course, this is server-class hardware running ideal conditions.

This is, notably, in stark contrast to Google's results (https://research.google.com/pubs/archive/35162.pdf&) of about 20,000-70,000 FIT per Mbit, but closer reading suggests that there were a fairly large number of high-fault DIMMs affecting the mean and Google historically bought the cheapest hardware available. Even at this fault rate an average DIMM wouldn't fail in any given month, for example.

Personal gaming, browsing and light document work PCs do not need ECC. Video/photo editing/development PCs might be on the threshold for not wanting to produce or store corrupted work.

Multiply the risk out for any organization with many people doing the same kind of work where corruption on a single workstation could cause annoying trouble for everyone else.

Good question. Most people don't really need it, as systems work quite well for long enough time without it and no visible corruption happens. But if you are tech enthusiast who knows a thing or two about errors then you want it, because it makes your system "serious"-grade. It boosts your confidence in quality of your system, and thus self-confidence, ego and bragging rights. Also, it makes your system more professional-grade, where you can monitor ECC errors and, if needed, replace RAM modules or motherboard.

https://stackoverflow.com/questions/4109218/do-gamma-rays-fr...

tl;dr: one bit error in 4GB every 72 hours

I wish someone with a larger server farm would count the number of reported ECC errors per GB-hour and give us updated numbers. That StackOverflow question is about 10 years old now, and I think it's relying on data even older than that.

Someone once did a bit-squatting experiment and "estimates that 614,400 memory errors occur per hour globally".

https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-squa...

It would be interesting to repeat this experiment today.

Yes; per the internet archive[1], the data's at least 20 years old.

1. https://web.archive.org/web/20010612184424/http://www.boeing...

A very good way to undercut low end Xeons which were bought solely for their insurance against an out of the blue crash.

You do have to pick a motherboard that supports it, but when you do, it should work without issues as long as your CPU supports it. Not all ryzen CPUs support ECC.

Here is an example: ASUS Pro WS X570-ACE

Aside from that one example, are there others? I'm not a fan of its fan.

Reply on news.ycombinator.com