Hacker News Reader

> But really most of all I live in a world of Git repositories. My entire Emacs config is versioned under Git (I know YMMV but I like it that way). Some people version under Git their entire user dir.

> Tell me how my lack of ECC is going to really make life miserable here?

Git will break if RAM is bad just like anything else. That it checksums everything won't save you from checking in corrupt data, the filesystem itself being corrupt, or some internal git structure becoming corrupt. Losing your repo because something in it was written wrong is very much a possibility.

Having multiple machines involved helps, but it's not a complete fix, because the possibility exists of something damaged being transmitted from a broken machine to a good one, ensuring there's no good copy anywhere.

There's really nothing software can do to operate correctly with bad RAM all of the time. Instructions for the software are in RAM. The OS that the software expects to behave right is in RAM. Various buffers used for disk access and networking are in RAM. An application like git assumes all of that is performing correctly, and can't compensate for every possible malfunction that could happen.

How trustworthy is git-fsck for detecting random-bit corruption?

Depends on how you look at it.

For actual verification, unless there's a bug my understanding is that very trustworthy. But by that point it's already too late. Okay, you know something is broken, but that won't give your good data back.

But that only tells you that Git data is intact and that all the hashes match. If git got a corrupted file to start with, then correctly hashed it, everything will verify 100% and still be broken.

> Seriously: which percentage of developers have ECC on their development machine(s)?

Every single one of my non-mini computers uses ECC ram except for my laptop. If someone would release a framework laptop motherboard that supports ECC ram (and preferably risc-v) I'd finally be able to close to reliability gap. It blows my mind that we say "Well sure, we COULD make infallible ram, but that would cost a tiny bit extra so instead lets just hope nothing bad happens." That's right up there with not wearing a seat-belt because I haven't needed one yet.

"If someone would release a framework laptop motherboard that supports ECC ram..."

This. If I could, I would. The only reason my laptop doesn't have ecc is because the manufacturer doesn't offer the option in any macines I otherwise want.

That comment was very misguided in trying to suggest that there is any valid excuse to tolerate unreliable execution hardware. git and ssh and md5sums do not mean that it's ok if your very brain can't be trusted to deliver data from one part to another within itself, or spit back the same data that was put in a cell. Everything else is built upon that!

I develop on VMs on my server (with ECC and RAIDZ) which i access using SSH or VDI protocols. I would love to have ECC on all my machines but that isn't feasible until the status quo changes, so I stick with the remote approach. To me that's an acceptable tradeoff as the non-ECC desktops/laptops are just used as dumb terminals while the real work happens on the reliable server.

I can't speak for ECC at the moment but ZFS has definitely saved me from data corruption that would have been left to manifset otherwise.

But if I open up an editor and write a document then that could be corrupted in RAM and then the corrupted data saved to disk. The document is likely to be more important than some ripped CDs or a git repo that I can download again.

The CPU, RAM and mobo manufacturers need to get together and make ECC RAM mandatory. It's absurd that we have machines with gigabytes of storage using microscopic (nanoscopic??) capacitors that doesn't have this basic protection. And honestly this should have happened years ago.

(Edit: And before anyone says DDR5 is ECC by default, that's not quite true although the difference is a bit subtle: https://en.wikipedia.org/wiki/DDR5_SDRAM#DIMMs_versus_memory...)

>But if I open up an editor and write a document then that could be corrupted in RAM and then the corrupted data saved to disk. The document is likely to be more important than some ripped CDs or a git repo that I can download again.

Devils advocate: if it is just some bits in character flipped it's entirely recoverable while flipping some bits in compressed stream for video would corrupt more.

Agreed. One problem with RAM errors (which I've actually experienced) is they are insidious. You probably won't notice them immediately so the error can be propagated, and they're very very difficult to diagnose if you do notice them.

Back in the 90s we had a database server which had a stuck bit in memory normally mapped to the page cache. This caused sectors to be written to the backing software RAID which couldn't be read back in (because I think some checksum was corrupted when written and then failed when read back). It took an absolute age to diagnose this. I think I only worked it out by eliminating everything else.

I went out of my way for ECC because losing files can mean losing days of work. With how much a sweng gets paid it's worth the ECC premium if it saves me a few days of work over the life of the machine.

Recently I discovered that one of my SSDs was quietly failing without setting off any warnings; doing a chkdsk showed that some files had already gotten corrupted. One of them was my backblaze backup index!

Even though I have automated backups (backblaze + macrium backups to a NAS), recovering files from them is non-trivial. If I were to lose work to non-ECC ram who knows how long it would take me to reconstruct a known-good work environment and file set. Imagine if you're working on something huge and hard to validate like neural net weights where corruption can occur silently and be hard to detect after it's happened?

My favorite is when data silently corrupts, and then is happily propagated to your off-site recovery :/

Yeah, I don't know when the drive started failing, so in practice my backups are probably all screwed too. My only option would be to get a spare drive, restore a backup to it, then do a block level diff of the current and backup volumes and try to figure out whether any of the differences are file corruption.

Probably I'm saying nothing you haven't thought of, but ZFS is great for preventing this. If you also pair it with ECC, then you've eliminated most ways to cause corruption.

Checksums don't save you from memory corruption. If your data gets corrupted in memory, you will just end up checksumming and committing bad data. Or your checksum could get corrupted, and you commit a checksum that doesn't match your data. Checksums are more useful for safeguarding against disk or network corruption (although you shouldn't have network corruption issues over TLS or SSH).

Apparently Ryzen 7000 cpus can use ECC. I've heard reports that AMD needs to release an AGESA update, though, and ECC DDR5 memory availability is terrible. I'm hopeful that the situation will improve, because I also want to update my desktop. I've been using ECC memory since losing a filesystem on a desktop when a DIMM went bad.

When people complain about these random OS crashes and freezes it's usually RAM corruption at fault.

Per 1gb of RAM, you can expect to see 266 bit errors per month[1-2] if you are using your PC 16h per day. Multiply that by 64GB or 128GB of RAM and it's crazy to think that you won't run into any of the stability issues.

[1-2] https://static.googleusercontent.com/media/research.google.c... [1-2] https://en.wikipedia.org/wiki/ECC_memory#Research

>266 bit errors per month

That seems like an overestimate. How can memtest ever pass on non-ECC RAM if errors are that frequent?

Because memtest only looks at values it wrote out very recently, before they have had a chance to flip.

Memtest is looking for reliable failures, not evanescent one-off events.

SDRAM is continuously refreshing all cells. How long ago data was written doesn't make a big difference (aside from the case where you're reading data immediately after writing or reading that data).

This does not, of course, make any sense. Any given memory cell will be read once per, say, millisecond, and the value written back. Once it has flipped once, the wrong value will then be written back, and after that the wrong value is read back out and rewritten again, indefinitely. Errors are sticky, and accumulate. (Flipping back again is negligible unlikely.)

With ECC in the refresh path, such an error could be corrected and the right value would be overwritten over top of the bad one. Then errors would not accumulate, but would instead be "scrubbed". Mainframe machines scrub their RAM. Disks too.

> SDRAM is continuously refreshing all cells

...so ? we're not talking about slow deterioration (which is why refresh is needed), we're talking about bit flip from cosmic rays where cell changes state completely.

I don't think this rate is reasonable to use in this manner. I've run thousands of servers, with lots of ECC ram (quite a few servers had 768GB of ram, but most were more like 32 or 64). The vast majority ran for years with zero reported errors. A small handful would have major errors of thousands in an hour, but we would replace for hundreds in an hour. A couple servers developed a periodic report of one or two (correctable) errors per day. If 266 bit errors per month per 1Gb was a usable rate, all of our servers would have been throwing ECC correctable errors all the time.

But I didn't have time or desire to publish a study on our experience, so there's no hard numbers.

> Per 1gb of RAM

The Google study you linked says "25,000 to 70,000 errors per billion device hours per Mbit".

Assuming bits, {25000 to 70000}/1e9 * 3016 1000 = 12 to 33.6 bit errors per month. Assuming bytes, it's 96 to 268 bit errors per month.

Apparently you've meant bytes, not bits. (I'm a pedant, but was also just unsure and interested in the numbers.)

FWIW, a comment I ran across that confirms toast0's view[1]: "It's a bimodal distribution - you either have many errors (due to a defect somewhere) or basically zero. If you're on the good side of the distribution, with only extremely rare errors, then you probably don't need ECC. But without ECC, you don't know whether you need ECC!"

[1] https://www.realworldtech.com/forum/?threadid=198497&curpost...

> which percentage of developers have ECC on their development machine(s)?

That question won't help you evaluate the demand for ECC simply because the supply is strangled. Those who want it have to make compromises to get ECC: get a Xeon, or get one of few AMD motherboards with matching CPUs and overpriced RAM without a guarantee that it will end up working.

> ...get one of few AMD motherboards with matching CPUs and overpriced RAM...

I mean, if you want a _guarantee_, then sure, get one of those certified-for-ECC motherboards.

But, like, as far as I know, going _at least_ far back as the Phenom II (released in 2008) AMD desktop processors have always supported ECC RAM. And -as far as I know- ASUS motherboards for said processors have always supported dropping in ECC RAM (and Linux and memtest and friends have always agreed that ECC was enabled and functioning in such a system).

Source: Personal experience with Phenom II, Threadripper, and Ryzen 5 CPUs and ASUS motherboards, and looking-from-a-distance at the rest of the AMD CPUs between the Phenom and the Ryzen 5.

And condoms are only 99.8% effective, so let's not use it? You can always pull out in time or verify checksu^W^W Plan B?

> But really most of all I live in a world of Git repositories

That's great for you, but 99% don't even know what Git is, just like checksums, cryptographic hashes, Merkle trees, digital signatures and reproducible builds.

You just miss the one imortant thing: ECC isn't that helpful where you have the means to check and verify the data. ECC is the only way to at least know what something is happening with data when there is no way to check.

To give you a slight idea I would tell you an anecdote from my L1 life almost two decades ago:

I visited a client who claimed what the PC was working erratically and constantly threw weird error messages.

Welp, the usual deal, just some ugly software or a virus. In the first 3 minutes I got like 5 errors about failing to load a .dll from C:\WINDOWS\SYSTEM33\USER32.DLL, nothing unusual, just need to.. WAIT. Why system33? Stupid virus masking for a well-known folder? Doubt. So I go to C:\Windows and I see:

  System32
  System33
  System34

If you are a smart fellow you probably already understood what that was a bit-flip error which somehow managed to be in the in-memory copy of MFT of the system drive. And this is the only reason the user noticed it - because sometimes programs wouldn't start and sometimes there was weird error messages. If that error was in the data area of some program - it wouldn't be discovered at all.

Hang on... to go from "System32" to "System33" is one bit flip, but to go from "System33" to "System34" is three bit flips at once. Doesn't that seem astronomically more likely to be a malicious program than bad RAM?

It's astronomically more likely to be a manufacturing defect what affected multiple cells than a malicious program what ... wrote to System34? And how the replacement of the DRAM module 'cured' that malicious program?