Hacker News Reader

Comment by agildehaus | Hacker News Reader

Unfortunately bit rot is not something APFS addresses as it only checksums metadata. Wonder if this is something Apple can add to APFS in the future?

View on HN

I hope. It wouldn't be such a big issue if Intel stopped stripping ECC from desktop CPU's. Traditionally bad ram causes more bit rot than damaged drives. Ram get less reliable every die shrink, everything should be ECC by now.

CPU ECC is for RAM, not storage - checksums/parity is not the concern of the CPU at all.

Though we're talking about Optane, then that's different. In the future (with solid OS support), Optane will be both RAM and non-volatile storage - and you really don't want data corruption there. Has Intel announced anything regarding ECC, parity or other systems in-place to protect against bitrot in Optane?

I don't think you get what I'm saying. The bit rot that happens to your data usually happens when it's in the RAM. The CPU likes it there when you're doing anything with it and once you're done editing a file it will stream happily corrupted data right back to disk. Hard drive and SSD error detection and correction is quite good, in regular ram its non existent.

This is a major danger with database servers, as regular bit rot can be detected or corrected above the filesystem level, but no systems exist that can garantee protection to data in ram. Since databases try to do most/all operations in ram before flushing to disk there's a huge risk of silent data corruption.

Adding bitrot protection to the filesystem only gives you more protection if the drive has crappy error correction. Adding protection to ram adds checks to a system that usually has none

> The bit rot that happens to your data usually happens when it's in the RAM.

Would love to read a source on this.

Disks have their own block checksumming so they can catch errors.

Just check your disk's SMART indicators for unrecoverable errors. Drives are using pretty high quality checksums on data at rest; the chance of corruption silently slipping past both the BCH and LDPC codes used by a typical modern SSD is far smaller than the chance of your non-ECC RAM getting corrupted by cosmic rays.

Your statement is correct, but doesn't tell the rest of the story.

The disk has an onboard data buffer, usually implemented in DRAM. That DRAM chip is likely sourced by the lowest bidder, and it is likely not to have either parity or ECC. Also the CPU used in the drive can introduce its own data errors and corruption. It doesn't even have to be hardware. A firmware bug can also corrupt data.

There are numerous buffers and storage elements involved in transferring the data to/from the disk's onboard memory to/from the computer's RAM. Many/most of those elements are not checked for parity. Data is subject to corruption in transit.

That's what makes a filesystem like ZFS so interesting. It checks the entire data path. It doesn't care where the corruption occurs.

I never considered DRAM errors on the drive... Maybe that's one of the big cost differentiators of those super expensive commercial grade SSDs.

I was under the impression that without ECC RAM, you should't be using an auto-repairing file system anyway because it will cause the bit rot to spread when it incorrectly calculates checksums.

This is incorrect. The point of ECC RAM is that if you are worried about bit-rot, you'll still get it without ECC RAM. Strictly speaking something like ZFS improves the situation - since a bit-flip which hits the checksum and not the data at the very least will raise an alert that something went wrong. But this will also happen with any other filesystem - it'll just be silent.

But the point is no filesystem can protect you from unreliable RAM.

Why not just go with AMD? ECC is supported on all Ryzen SKUs.

But that does not mean that all the boards support it, even if the chip works at all. What good ECC RAM does, if the errors are not logged?

Sure probability of errors increases, but RAM reliability seems to have improved

Bit rot isn't addressed by any filesystem without device redundancy, which no Apple device has had as an option since the Mac Mini server was discontinued.

ZFS can tell you which files are corrupt, and you can then manually restore those files from an offline backup. No online device redundancy required.

Unless the backup files are also corrupt.

Sure, that's a small possibility, but are we trying to get to 0% chance of data corruption?

There is no point to go in the probability of data corruption below than a probability of an event that makes stored data irrelevant. Like, for example, Earth hit by an asteroid, global nuclear war or, for personal data, a chance of death.

Everything is backed up in the Akashic records anyway, it's fine.

You can tell zfs to store N copies of the files (you do it at file system level, but it doesn't have to be for the whole disk) instead of one, so you can recover from bit rot even with a single disk. At least in theory, I have never tried it.

I assume bit rot isn't an issue on devices running APFS. They're designed to be continuously backed up. EDIT: never mind; didn't realize iCloud didn't keep historical copies for long.

Backup doesn't do anything about bitrot. Worse, it just propagates and compounds it over time.

Backup will not fix your bitrotted files by itself. It will just propagate silently into the future where you don't have access to historically correct files anymore.

Reply on news.ycombinator.com