Monitoring ECC memory on Linux with rasdaemon | Hacker News Reader

Monitoring ECC memory on Linux with rasdaemon

setphaserstostun.org

64 pointszdw37 comments

I ran into this very tool when setting up my home server some months ago (ECC and ZFS being necessary for integrity’s sake). It works, though I’ve yet to see a single error, which has only left me thinking that the fuss over ECC RAM and the risk of bit flips was overblown.

I have some servers that in many years of service have never logged a single error. I have others that pop a cluster of errors over the course of a few hours every year or two. Beyond differences in component brands, no obvious reasons for the different behavior have presented themselves to me.

FWIW I’m a believer in ECC. But give it some time. Maybe you’ll see errors. Or maybe you’ll get lucky and see none. Either way, you’re covered!

> Beyond differences in component brands, no obvious reasons for the different behavior have presented themselves to me.

Google's results show that the error rate is strongly driven by bad modules rather than random errors. A module that throws one error is up to 224x more likely to throw another error, and corrected errors are a strong predictor of future uncorrectable errors.

The major exception is cosmic rays, which can flip bits even in a good module. But unless you're at high altitude they don't really dominate the error rate.

http://www.cs.toronto.edu/%7Ebianca/papers/sigmetrics09.pdf

Obviously without ECC you don't know if you have bad modules, but, if a machine passes a memtest scrub for 24+ hours then it's probably fine.

On-die ECC should also catch the majority of these errors, especially if you are not right at the limits of your memory training. JEDEC type timings should be very forgiving to transmission errors in general.

People are very down on the "on-die ECC" on DDR5, they act like ECC is completely useless unless you protect the whole datapath, and of course it would be preferable to do so. But actually the on-die ECC should catch a large number of the errors, because bad modules (dies) are where the errors mostly happen.

Also, the author of that paper wasn't aware that Google had a stint around that time buying up DRAM chips that had failed manufacturer's QA and sticking them on DIMMs themselves, trying to clock them slow and use ECC to get cheaper, mostly working memory. It was a poor choice and Google stopped the process AFAIK.

My only objection to on-die ECC is that, AFAIK, there's no way to query the number of internal corrections that have occurred. That is important information! Yes, it might freak out some sysadmins that frequent corrected errors are now "normal" (after all, the main reason for on-die ECC is that manufacturers won't have to scrap slightly imperfect DRAM), but those individuals can be re-educated, and I really want to be able to tell if a given module is running within spec yet right at the edge of failure.

Yes, I'd like to see these plotted over time and correlated with machine failures.

yep definitely, it'd be nice to have a way to communicate this, I totally agree. And yeah they probably didn't because people would freak at seeing errors corrected during normal operation.

I agree with the "if a machine passes a memtest scrub for 24+ hours then it's probably fine" in theory. Sadly in practice I regularly find dimms that generate errors plenty of errors, and test find with memtest.

Not sure why, maybe it's the different memory access patterns, maybe bad dimms are more sensitive to repeated accesses below the rowhammer level, that healthy dimms have no problems with.

I find things like make -j for large builds MUCH better at finding memory errors than memtest.

yep also true. maybe you could build some kind of tooling around this, if builds are deterministic (or you do something like pgbench with a deterministic workload/script) then there's no reason you can't check the output in addition to very obvious segfault failures.

Years ago the GCC FAQ had an entry for a strange compiler error, that pretty much was only triggered by overclocking or memory errors. I just looked and couldn't find it.

Oh, absolutely, I wouldn't want to be without ECC, I understand the value of having the additional integrity, and the reassurance that things are OK.

I was just hoping to see some errors on a regular basis, so I could feel vindicated and rave to everyone about how useful ECC is and point to my console and say "look at how many errors you might be missing!". ;)

You're not covered for all types of errors, only single bit flips.

A double bit flip will result in a halt. Ya may not find it convenient, but for my purposes I call that covered.

(I choose not to worry about flips of three bits or more. YMMV.)

> A double bit flip will result in a halt.

This is dependent on the system configuration. For Linux, default behavior is to halt only if the uncorrected error occurs in memory used by the kernel.

It's more accurate to say that double bit flips will be detected.

Thanks for the clarification. My Google Fu apparently isn't up to the task, can you share where the knobs are to control this behavior?

https://docs.kernel.org/admin-guide/ras.html#module-paramete...

Search for "edac_mc_panic_on_ue"

Thanks!

Are there error patterns that could still result in silent failure? I'm not %100 sure how modern ECC RAM is implemented, but I can imagine a simple parity bit would fail silently if different bits fail simultaneously by flipping bits in the opposite direction.

Flipping three or more bits can do this.

"You really DO need ECC Memory" - https://perspectives.mvdirona.com/2009/10/you-really-do-need...

Heh, well there's 2 things I wanted from ECC. Sure I wanted the very occasional random bit flip to be caught and fixed. Making the system more reliable, maybe save a few process crashes or reboots over 10 years.

Although I was much more interested in ability to find dying hardware. Whenever weird (file corruptions, processes dying, reboots, things not working like they did yesterday) could be from numerous causes. Voltage dips, problems in the power supply, software bugs, OS bugs, security compromise, defective CPU/Ram/motherboard/dimms, etc. It can be very hard to track down, you might well spend weeks and significant $$$ playing the game of swap the part and wait for the problem to reappear ... or not. Much like what recently happened to LinusT. I'll happy pay another 10% system price for a system that has a few less crashes and will tell me when it starts seeing memory errors (while fixing the small ones).

I'm typing this from a Xeon E3-1230v5 bought in 2015, that cost LESS then the similar i7. I did spent a bit more on the motherboard ($50) and ram ($75), but it's been rock solid since and if I had a dimm problem it would likely be 10 minutes to troubleshoot and then I'd RMA or buy replacement ram. ECC reduces the chance of writing the wrong data to the right place on disk or, even worse, the right data to the wrong place.

They don't happen often, but when they do, they do.

There's a reason why every datacenter does it. You must be able to trust your machine.

On nine servers (each 512 GiB RAM) I have access to one experienced an ECC error back in november. My local workstation (64 GiB RAM) logged two errors back in december.

Bit flips aren't evenly distributed over time or across RAM. In other words, the vast, vast majority of DIMMs are good and will never have a bit flip. For those modules, the money you spent on ECC will be in a sense "wasted". (Except for the value of knowing that you had a good module in the first place.)

IIRC, Linux doesn't report corrected errors by default. If you enable that, you'll see a warning periodically (I see a few a year).

Are you saying that the tool in the article doesn't report corrected errors? It looks to me like that's exactly what it does, but maybe you know more than I do.

As in it won't send messages to kernel log and you need the tool to ask kernel about it

rasdaemon is presented as the followon to mcelog, but in accordance with current systems development practices, it's not a feature-complete replacement. mcelog can offline bad memory pages instead of merely reporting the failure.

In addition, rasdaemon doesn't seem to have received any commits since April of last year, where mcelog appears to be under active development. Can someone help me understand the recent uptick in interest for rasdaemon, when mcelog seems to remain the superior option?

It could be because mcelog "is not supported in kernels 4.12+" [0].

[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=890301

This looks like it; the /dev/mcelog character device has been deprecated. It looks like we'll have to either port the mcelog software to the kernel-trace interface or provide rasdaemon with a tool to offline bad memory.

It's frustrating that 4.12 is five years old and this hole in functionality remains. I wonder why kernel devs don't consider userspace support for their features when swinging the deprecation axe.

I don't get it either, what was the problem with mcelog that this is trying to solve ?

Pre ECC I know one big bank which republished incoming data feeds onto multiple geographically distinct compute back in early 90's to address this or any such single point of failure ... a different level of confidence is appreciated when running aggregate 12 digit books

Can someone parse the readout at the end? How many errors did he actually experience? It looked to me like there was one non-correctable error. IDK how the OS decides which errors are fatal and which are non-fatal.

The OS is informed by the system firmware which errors are uncorrectable and which are correctable. rasdaemon gets this information via kernel trace, where mcelog reads the machine check exceptions.

Will this work from inside a VM?

The hypervisor can generally pass real or simulated MCE errors to the guest.

See eg https://github.com/qemu/qemu/blob/48804eebd4a327e4b11f902ba8...

No.

Reply on news.ycombinator.com