Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

I really hope the ECC carries through. It irritates me to have to buy a "server" CPU if I want ECC on my desktop (which I do) and it isn't that many gates! Its not like folks are tight on transistors or anything. And on my 48GB desktop (currently using a Xeon CPU) I'll see anywhere from 1 to 4 corrected single bit errors a month.

For things like large CAD drawings which are essentially one giant data structure, flipping a bit in the middle of them somewhere silently can leave the file unable to be opened. So I certainly prefer not to have those bits flip.

View on HN

I have a Ryzen 5 1600 with a pair of Crucial ECC UDIM. It works perfectly on the ASUS Prime B350M-A/CSM uATX board. Stress tested. Errors are logged and halt behavior is confirmed (ubuntu 17.04).

Interesting, is there a list of boards and supported modules that work with ECC?

Interesting, could you possibly explain how "halt behavior" indicates that ECC is fully functioning? I've just read a lot about ECC being compatible but not tests explicitly showing it works.

On my system if an uncorrectable error were to happen the system goes into a 'machine check' state (a halt). In theory, the BIOS supports 'chip kill' which is that it reboots with a bit set to not use that specific chip in the DIMM. I've never seen that happen on my desktop but in the data center when it happened a system that should have 128GB of memory would reboot and come back with less than the full amount (as expected by the DIMMs plugged in) reported by the BIOS.

Interesting, depending on the amount of RAM killed that seems like it could be a bit overkill for what could just be a cosmic ray flip.

Though perhaps the rare frequency of cosmic ray flips makes that acceptable.

Chipkill is an IBM name for the feature that corrects multiple bit errors in DRAM, are you sure about the reboot thing or are there two similarly named ECC techs that do different things?

Intel discussion of Chipkill (https://www.intel.com/content/dam/www/public/us/en/documents...). Servers in question were Supermicro "Jupiter" chassis with the Westmere processors, discussion on Supermicro of their implementation (https://www.supermicro.com/support/faqs/faq.cfm?faq=2642)

When systems rebooted with less memory than the system configuration database said they should have most of the time there would be a multi-bit error detection, machine check, and 'memory update' in the IPMI buffer.

Why kill the DIMM? Cosmic ray bitflips don't harm the hardware, AFAIK.

It's usually not cosmic ray.

The system must be configured to halt the system on detection of uncorrectable multibit errors -- which shows that the detection of uncorrectable errors works.

The specs for the board indicate no ECC support. How did you test it?

BTW, I'm hoping that ECC is there.

Currently on mobile, but the hardware blog Hardware Cannucks has a fantastic post covering how "true" ECC support / functionality is on Ryzen. They also investigate this on both Windows and Linux.

Thanks - just read it. I get it now. The chipset supports it and linux turns it on. Looks like the halt functionality is not there.

Sorry for the uninformed question but:

With RAM sizes having ballooned to very large sizes (16-32GB now fairly common for a workstation) why is non-ECC memory even considered? Other methods used to safeguard large sums of 0s and 1s like hard drives, SSDs, modern filesystems (ZFS, Btrfs) have builtin error-correcting mechanisms. Why is getting hit with a cosmic ray and having a bit flipped in your CSV file any more acceptable than the same thing occurring on a "server" with ECC memory?

Because ECC essentially takes 9 bits per byte for storage - typical implmentation is 9 chips per stick, not 8. You also need an additional controller to actually do the error checking.

So, even if it weren't for the typical "enterprise/industrial" you'd be looking at a minimum of 12.5% parts cost increase.

Multiple that out * billions of ram sticks and you're talking real money for something with dubious relevance for most users.

The controller is already built into the cpu, and cost increases aren't linear like that at all when scale is factored in, especially considering packaging and assembly overhead.

Skylake-X without ECC: $999

Skylake-SP with ECC: $3,000

There's your answer.

I agree with this, Intel uses it as a margin enhancer. My CPU was $150 more than the equivalent desktop processor.

Current Ryzen CPUs support ECC, so presumably so will the Threadripper variant. See Lisa Su's comment(!) on Reddit repro'd on this page: http://www.hardwarecanucks.com/forum/hardware-canucks-review...

That's not an answer. That's just Intel being dicks.

I would say it's more that Intel's business model depends on price discrimination and they don't know how to survive without it.

Silly example. All the below are 4 cores/8 threads.

  Xeon E3-1230 v6 kaby lake 3.5 GHz - 3.9 GHz $250
  Xeon E3-1240 v6 kaby lake 3.7 GHz - 4.1 GHz $272
  Core i7-7700    kaby lake 3.6 GHz - 4.2 GHz $303
  Xeon E3-1270 v6 kaby lake 3.8 GHz - 4.2 GHz $328
  Core i7-7700k   kaby lake 4.2 GHz - 4.5 GHz $339

What's the ECC premium again? Clearly they are on pretty similar price/performance curves.

The premium is in workstation -> server CPUs, not desktop. The E3 line are Xeons in name only.

Threadripper 1950X with ECC, 16 instead of 10 cores, 64 instead of 44 PCIe lanes and without a raid dongle which prevents certain raid modes unless you pay for them (like with Intel): $999.

I love competition. Though I would prefer if there was a third competitior in the x86 CPU space and the GPU space.

And you haven't replaced it? The only time I had ECC memory report errors, I was also experiencing undetected errors as well. System was not stable. Pulled it out, happy ever since. I've always thought of ECC as a warning system (despite the correction ability). Like a spare tire.

Memory errors happen all the time, at least they did several years back when the Google server farm saw "an average of one single-bit-error every 14 to 40 hours per Gigabit of DRAM": http://www.intelligentmemory.com/support/faq/ecc-dram/how-of...

This translates to "a mean of 3,751 correctable errors per DIMM per year": http://www.zdnet.com/article/dram-error-rates-nightmare-on-d...

I'm not sure how things pan out these days with newer memory types. ECC checks and fixes these errors so they're not an issue.

Citing the results of a paper in which they studied the errors observed mostly on DDR1 memory?

A far more recent study by CMU based on the entire fleet of Fb servers shows that correctable error rates dropped dramatically in the past decade.

http://repository.cmu.edu/cgi/viewcontent.cgi?article=1345&c...

[Unless it's obvious to others,] paper doesn't conclude that DDR1 to DDR3 discrepancy correlates to error rate. It only concludes from empirical measurements that memory density scales with error rate (2).

Here is a better paper to cite: https://www.cs.rochester.edu/~kshen/papers/usenix2010-li.pdf

I don't consider it broken so I haven't replaced them, the system is rock solid stable.

Your statement though was interesting, how do you have both ECC memory report errors and 'undetected errors'. At least from a memory perspective, with ECC an 'undetected' error is a multi-bit error that both flips bits and leaves the ECC bits in a legal configuration. That seems like it would be pretty rare.

That said, I've seen motherboards (in our data center) where the memory slots themselves were unreliable (probably bad or weak solder joints on the DIMM sockets or missing terminator resistors). They appeared as a machine with a lot of ECC errors but the same DIMM in another motherboard gave no errors.

The patient first presented with classic memory corruption symptoms, like random segfaults. Ran memtest. No errors reported, but a flurry of corrections logged in the BIOS. Pulled a pair of dimms, everything cleared up. Swapped slots too, so I feel confident blaming the RAM.

It did fall outside my expectation of how ECC works. One bit errors and three bit errors, but not two? Some access pattern that memtest strides don't hit? I didn't really need the extra RAM, so I just moved on without it.

SECDED - single error correction, double error detection. Your Duff memory module was flipping more than two bits.

The typical PC-building online communities have been mindlessly drumming the line "Only servers need ECC" into each other's heads for so long that they'll probably end up refusing it, even when it's offered to them on a plate.

OEMs will probably still ship systems without ECC too. Does not mean it should not be an option. I wonder as a side note on Bristol Ridge for example, for 8GB of RAM would you rather take dual channel or ECC? (the fun thing is that 4Gbit DDR4 is cheaper per bit on DRAMeXchange spot prices than 8Gbit DDR4 again, but...)

Well, when people are shocked that someone is experiencing 1-4 single bit errors a month across an expanse of 48GB, it's not hard to see why people think it's not something they need.

A single bit error is enough to create a misspelling in a txt file. There's a low probability of it happening, but if it does, it's unnerving to think that it will not be detected.

There's a very low probability!

- In consumer equipment, much of your RAM is often unused at any given time

- Most lines in a cache are eventually thrown away, never used

- Did you even save your text file, or just open it to read it?

- What about all the space occupied by read-only information, like executables, media files, game files, and libraries? You might crash, but nothing will be written to disk

You might get an error a month, but the odds you'll get an error that matters on your average consumer machine with average workloads is much much lower.

The typical pc builder (or user) isn't professional so rarely use data that can be corrupted. I edit photos and play games. The photos have backups so I wouldn't sweat a corrupt bit here and there. The largest structure I use is my file system but I try never to get too attached to data on a machine (machines should be possible to reinstall within an hour - data such as photos should be elsewhere).

People working with sensitive datasets or fragile data structures (large cad files was mentioned) can certainly use ECC with good reason.

But for most home machines? Sure, if it doesnt cost 10 or even 3% extra then I'd recommend it. So it would be great if AMD could pressure Intel towards bringing ECC to consumer chips.

Otherwise for a normal builder just put that $100 towards a better graphics card (if gaming) or a better monitor or whatever, and the lack of ECC will make your game crash once in 3 years (it crashes 99 more times due to bugs and bad drivers...)

"Not minding" data corruption is a perverted mindset that comes from internalising Intel's market segmentation. It's not a law of home computing that we don't deserve the good technology.

Absolutely, but my point was that people don't even have important "data" a lot of the time on home build PCs so the question is how much added cost is actually acceptable for ECC? For me on a gaming machine the acceptable added cost would be in the very low single digits, so while I welcome if AMD goes "ECC for all" I was just trying to rationalize not recommending it to home pc builders with current pricing.

> "Not minding" data corruption is a perverted mindset that comes from internalising Intel's market segmentation.

Disagree. The mindset of "assume my metal box can catch fire at any time" is absolutely the right one to adopt, and the more valuable your data is the more right the mindset becomes.

Intel's nasty market segmentation strategy doesn't make that mindset wrong.

Are you going to get the word out to every owner of Xeon(s) in the world that they're doing it wrong and should be using consumer gear if they truly value their data?

Or, is ECC in fact a good thing that's worth having?

The key here is "truly value" and how much. Intel pricing forces you to think about whether you value what's in your ram.

If you are only gaming on a Xeon you should have put that money on the gpu. If you are doing databases, cad etc without ECC then the converse is true: should have put more money towards ECC. Can't see the controversy in recommending Non-ECC for Intel buyers based on the workload in question.

Are there any Ryzen motherboards that support ECC currently? ("support" defined as "corrects errors", not "can plug in ECC ram and it functions as normal ram without error correction")

Last I am aware there are not.

There is a thread on overclock.net which collects the AM4 motherboards with ECC support [1].

You don't need complete support from the motherboard though. As long as the motherboard and BIOS/UEFI don't sabotage ECC you can at least use it from the OS, even if the motherboard doesn't explicitly support it [2].

[1] http://www.overclock.net/t/1629642/ryzen-ecc-motherboards

[2] http://www.hardwarecanucks.com/forum/hardware-canucks-review...

1-4 corrected errors per month?! This sounds quite bad - which manufactor?

Isn't that more or less in the range expected from background radiation bitflips when having 48 GiB?

I manage many servers with >=384GB RAM. They maybe detect an error once a year, with several never reporting an error in its life. If I noticed a server reporting errors more than twice a year I'd probably swap its memory, and if that didn't fix the problem I'd probably replace the server.

Depends on what you mean by "reporting an error". The default configuration for most machines I've used is to silently correct single bit failures unless they hit some threshold. So you actively have to run an EDAC (or similar) driver and poll the error counters. Its only with uncorrectable errors does it generally rise to the level of a full MCE.

Many server-grade motherboards will report ECC memory faults in the event log through IPMI.

How do these manifest themselves and how do you detect them?

The error events are reported through IPMI in a system event log.

Error rate would probably depend on the mass and quality of shielding above/around the servers.

Crucial (six 8GB sticks). I don't put it into the 'bad' category. When overseeing 1600 machines in a data center with a collective 307TB of RAM suggests its actually on the 'good' side. Bad is multiple errors per day. Really bad is hundreds or thousands per day.

Because they are corrected nothing actually happens on the system (other than what ever accessed them saw on the order of 700nS to access RAM rather than on the order of 100nS. If it gets a double bit error it will machine check so I'll know that my memory has failed me :-).

All ram should be ECC (selective number of bits), then the processes that fab the ram can be made cheaper and sloppier. Zero bit-error-rate non-ECC RAM would actually be more expensive than ECC ram. A chip kill be on a much smaller granularity. I suspect these things haven't been done because the RAM cabal likes their high prices.

Threadripper will have ECC support [0] just like Ryzen but validation depends on the motherboard manufacturer.

[0] https://www.reddit.com/r/Amd/comments/6icdyo/amd_threadrippe...

Xeon E3 pricing is very similar to the equiv i5 and i7, and supports ECC. Plenty of fairly cheap motherboards ($50-$100 more than the i5/i7 boards).

Some i3 models also support ECC when paired with the right chipset. My NAS runs on an i3-2100/Supermicro/ECC DDR3 setup. The i3 was a $100 Microcenter special.

If I ever have to upgrade the compute portion I will definitely consider Ryzen.

Check out the asus prime x370-pro. :)

After reading it had both AM4 support and ECC I started planning a build based on it. Assuming the ECC implementation works as I need it will probably be my next desktop.

Reply on news.ycombinator.com