Hacker News Reader

Comment by JackMcMack | Hacker News Reader

The issue is not a loudly failing disk. If you get read errors, or if the disk is simply offline, it's easy to recalculate the data.

But what if you have silent data corruption: a single bit? Even if you have parity (raid5) or a duplicate (raid1), how do you know which part is correct, and which part to repair? Without checksums.

I've had to restore 1TB from Crashplan. They apparently had checksums in their software, and something like 10-15 files were damaged.

Without checksums, you simply don't know if your data is damaged, even on a single hard drive. ZFS gives you data integrity with checksums, and redundancy with mirrors or raid-z.

This is why a regular scrub is recommended, you verify the checksums and repair if possible.

View on HN

Silent data corruption is not the typical failure mode for HDDs. Sure, it happens. Much more likely is that drive's internal ECC detects the bad sector, attempts to reread it several times, and if it fails, you get EIO back. Given this model, the various kinds of RAID know which drive had the erroneous sector, because it isn't silent.

This sort of gets into it but isn't a fantastic source: https://www.enterprisestorageforum.com/storage-hardware/driv...

Notably, Apple's new filesystem introduced in 2017 does not detect data corruption; it only checksums metadata. https://en.wikipedia.org/wiki/Apple_File_System#Data_integri... Apparently Apple Engineers are of the opinion that Apple SSD and HDD media simply don't have silent data corruption. They have some pretty smart engineers, so maybe they're on to something. It does beggar the question, though: why have metadata checksums at all? http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/

Anyway, I'm not exactly supporting threat models that ignore silent data corruption. Just pointing out that they're not exactly as ridiculous as they might seem on their face.

> Silent data corruption is not the typical failure mode for HDDs. Sure, it happens. Much more likely is that drive's internal ECC detects the bad sector, attempts to reread it several times, and if it fails, you get EIO back. Given this model, the various kinds of RAID know which drive had the erroneous sector, because it isn't silent.

Linux md will do the completely wrong thing with this: it will treat that as a drive error, and if you accumulate a lot in a short time (such as rebuilding an arrary) then it will conclude that this is a failed drive and kick it out of the array.

It could always be worse — it could kick out the other drive!

Well, with modern disk sizes it will almost certainly kick out all the drives sooner or later, making it essentially impossible to ever rebuild an array after a disk failure if you weren't scrubbing regularly before the disk failure (and dicey even if you were).

I totally believe you but this has never actually been a problem for me with Linux md. I tend to replace drives when the very first error is detected, and for spinning rust I usually get a very small number of errors initially. Also, md on my system seems to automatically run scrubs weekly. Currently observing 0 for Raw_Read_Error_Rate and Reallocated_Sector_Ct for all drives, with 24k Power_On_Hours (>2.5 years).

Weekly scrubs put you in a much better situation than not having them. I suspect your distribution must have done something to close the gap, because vanilla md does not put them in place and the tutorials do not always make it clear that setting up scrubs is not a nice-to-have thing but something that's absolutely critical if you want to have a hope of avoiding data loss.

How many array rebuilds have you done? You only see the read errors when you go to actually read the sector in question, and even in a week I wouldn't want to rely on not having any sectors go bad.

Given enough writes, any SSD will eventually fail.

As a related anecdote, I did some stress-testing on a ZFS filesystem with two Kingston NVMe SSD-s in mirrored configuration and at one point I got two random read errors on one drive, which subsequently got corrected by ZFS.

Silent failures might not be the best phrase to use for these situations, but ZFS does let you know that corruption has taken place and you have to acknowledge that message for it to disappear.

If the SSD controller isn't bugged, lifetime write limits don't result in silent data corruption; they just result in errored writes and inability to persist new data. Apple may have decided to only qual and sell SSDs that they've tested to failure, rather than throwing darts at random.

It's not clear to me if you're describing silent data corruption or detected data corruption (at the controller level), but either way Apple isn't selling Kingston NVMe SSDs; they have their own SSD controller and vendor the raw flash.

I was won over to btrfs for my new NAS (~6 years ago!) when I saw a demo (which I expect is also valid on zfs) where the presenter starts with a mirrored setup, finds the exact block offset location of the file's contents and then overwrites a byte in it with a zero using /usr/bin/dd against the raw disk, /dev/sdX.

He then reads the file, uncorrupted, with a kernel warning about a bad checksum that was fixed (from the mirror) during the read.

This demonstration is why I switched to btrfs (and ECC) for my new home server/NAS.

That, along with compression sealed the deal. I was disappointed about not having dedupe at the time, but it probably really wouldn't have saved me that much space. (I keep meaning to run the dedupe tool to see what it might save me now.)

Reply on news.ycombinator.com