Hacker News Reader

Comment by anon9001 | Hacker News Reader

I use ECC and mdadm+LVM+ext4. I haven't had any issues yet, but this thread is making me wonder if there's some data integrity problem that I'm setting myself up for.

With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing? If the concerns about "data integrity" are really about time until a disk is replaced, then it makes sense. Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems.

View on HN

The issue is not a loudly failing disk. If you get read errors, or if the disk is simply offline, it's easy to recalculate the data.

But what if you have silent data corruption: a single bit? Even if you have parity (raid5) or a duplicate (raid1), how do you know which part is correct, and which part to repair? Without checksums.

I've had to restore 1TB from Crashplan. They apparently had checksums in their software, and something like 10-15 files were damaged.

Without checksums, you simply don't know if your data is damaged, even on a single hard drive. ZFS gives you data integrity with checksums, and redundancy with mirrors or raid-z.

This is why a regular scrub is recommended, you verify the checksums and repair if possible.

Silent data corruption is not the typical failure mode for HDDs. Sure, it happens. Much more likely is that drive's internal ECC detects the bad sector, attempts to reread it several times, and if it fails, you get EIO back. Given this model, the various kinds of RAID know which drive had the erroneous sector, because it isn't silent.

This sort of gets into it but isn't a fantastic source: https://www.enterprisestorageforum.com/storage-hardware/driv...

Notably, Apple's new filesystem introduced in 2017 does not detect data corruption; it only checksums metadata. https://en.wikipedia.org/wiki/Apple_File_System#Data_integri... Apparently Apple Engineers are of the opinion that Apple SSD and HDD media simply don't have silent data corruption. They have some pretty smart engineers, so maybe they're on to something. It does beggar the question, though: why have metadata checksums at all? http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/

Anyway, I'm not exactly supporting threat models that ignore silent data corruption. Just pointing out that they're not exactly as ridiculous as they might seem on their face.

> Silent data corruption is not the typical failure mode for HDDs. Sure, it happens. Much more likely is that drive's internal ECC detects the bad sector, attempts to reread it several times, and if it fails, you get EIO back. Given this model, the various kinds of RAID know which drive had the erroneous sector, because it isn't silent.

Linux md will do the completely wrong thing with this: it will treat that as a drive error, and if you accumulate a lot in a short time (such as rebuilding an arrary) then it will conclude that this is a failed drive and kick it out of the array.

It could always be worse — it could kick out the other drive!

Well, with modern disk sizes it will almost certainly kick out all the drives sooner or later, making it essentially impossible to ever rebuild an array after a disk failure if you weren't scrubbing regularly before the disk failure (and dicey even if you were).

I totally believe you but this has never actually been a problem for me with Linux md. I tend to replace drives when the very first error is detected, and for spinning rust I usually get a very small number of errors initially. Also, md on my system seems to automatically run scrubs weekly. Currently observing 0 for Raw_Read_Error_Rate and Reallocated_Sector_Ct for all drives, with 24k Power_On_Hours (>2.5 years).

Weekly scrubs put you in a much better situation than not having them. I suspect your distribution must have done something to close the gap, because vanilla md does not put them in place and the tutorials do not always make it clear that setting up scrubs is not a nice-to-have thing but something that's absolutely critical if you want to have a hope of avoiding data loss.

How many array rebuilds have you done? You only see the read errors when you go to actually read the sector in question, and even in a week I wouldn't want to rely on not having any sectors go bad.

Given enough writes, any SSD will eventually fail.

As a related anecdote, I did some stress-testing on a ZFS filesystem with two Kingston NVMe SSD-s in mirrored configuration and at one point I got two random read errors on one drive, which subsequently got corrected by ZFS.

Silent failures might not be the best phrase to use for these situations, but ZFS does let you know that corruption has taken place and you have to acknowledge that message for it to disappear.

If the SSD controller isn't bugged, lifetime write limits don't result in silent data corruption; they just result in errored writes and inability to persist new data. Apple may have decided to only qual and sell SSDs that they've tested to failure, rather than throwing darts at random.

It's not clear to me if you're describing silent data corruption or detected data corruption (at the controller level), but either way Apple isn't selling Kingston NVMe SSDs; they have their own SSD controller and vendor the raw flash.

I was won over to btrfs for my new NAS (~6 years ago!) when I saw a demo (which I expect is also valid on zfs) where the presenter starts with a mirrored setup, finds the exact block offset location of the file's contents and then overwrites a byte in it with a zero using /usr/bin/dd against the raw disk, /dev/sdX.

He then reads the file, uncorrupted, with a kernel warning about a bad checksum that was fixed (from the mirror) during the read.

This demonstration is why I switched to btrfs (and ECC) for my new home server/NAS.

That, along with compression sealed the deal. I was disappointed about not having dedupe at the time, but it probably really wouldn't have saved me that much space. (I keep meaning to run the dedupe tool to see what it might save me now.)

>I use ECC and mdadm+LVM+ext4. I haven't had any issues yet, but this thread is making me wonder if there's some data integrity problem that I'm setting myself up for.

Nothing in your setup there protects you from bitrot.

>With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing?

Drive rebuild time is primarily a function of a variety of things - speed of the disks, processing power involved, etc. There's nothing magic about any solution here. ZFS does get a leg up in that it knows what blocks are in use, but hardware raid and mdadm are quite fast when they find empty blocks too, so it's not some overwhelmingly massive improvement.

>Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems.

I think you're fundamentally confused about what journaling provides a filesystem. A journal helps you in situations where your system crashes, loses power, etc. and you had not finished committing your changes to the filesystem. It does not protect you from data integrity issues that are unrelated to this sort of situation - a bit flipping in an HDD from electromagnetic radiation or an SSD from electron leakage will not be caught by a journal.

> a bit flipping in an HDD from electromagnetic radiation or an SSD from electron leakage will not be caught by a journal

Isn't that what the scrubbing process is for? My understanding was that RAID is keeping parity bits to make sure bitrot hasn't happened and it periodically scrubs and repairs when bits do rot. Is the difference that ZFS scrubs in real-time?

Sorry if I'm being dense here, but I did a fair amount of reading about this, and I'm trying to understand what I'm missing.

>Isn't that what the scrubbing process is for? My understanding was that RAID is keeping parity bits to make sure bitrot hasn't happened and it periodically scrubs and repairs when bits do rot.

Nope. Parity bits aren't used to detect bitrot. If you get a URE when you go to read, the parity bits will be used to get you data. If you read successfully, nothing happens at all. If you scrub, and the data is found to be inconsistent, md assumes that the parity is wrong and overwrites it.

https://linux.die.net/man/4/md

>If a read error is detected during this process, the normal read-error handling causes correct data to be found from other devices and to be written back to the faulty device. In many case this will effectively fix the bad block.

>If all blocks read successfully but are found to not be consistent, then this is regarded as a mismatch.

>If check was used, then no action is taken to handle the mismatch, it is simply recorded. If repair was used, then a mismatch will be repaired in the same way that resync repairs arrays. For RAID5/RAID6 new parity blocks are written. For RAID1/RAID10, all but one block are overwritten with the content of that one block.

Thanks for the explanation. I didn't realize md didn't use the parity checks unless there's a URE.

Is it correct to say that ZFS parity is used to reconcile on each read even without UREs, but with md parity is only used when there are UREs? If that's true, it makes much more sense why ZFS is considered the right choice.

Also, do you know if adding luks in the mix changes that?

My data goes md->lvm->luks->ext4 and I hadn't considered which part of that is supposed to catch the error if it wasn't a URE.

It makes sense that md would update parity if there was no URE. It also makes sense that LVM wouldn't know or care about data integrity.

I would think luks would require data integrity to work properly, but I'm not sure what happens if there is bitrot in a luks block.

I'm also surprised ext4 doen't have checksumming, but apparently it's only for able to checksum for metadata and it's disabled by default: https://serverfault.com/questions/456319/how-to-get-checksum...

Dm-integrity sounds worth a look, but https://raid.wiki.kernel.org/index.php/Dm-integrity doesn't inspire much confidence.

The other replies answered most of your questions so I won't retread those.

>Dm-integrity sounds worth a look, but https://raid.wiki.kernel.org/index.php/Dm-integrity doesn't inspire much confidence.

That wiki page is pretty yikes - I was under the impression dm-integrity was farther along.

Some misc other stuff I like about ZFS:

l2arc/zil on ssd/optane devices for fast read/write caching Easy NFS/SMB sharing zdb ( https://manpages.debian.org/testing/zfsutils-linux/zdb.8.en.... )

I don't think you would be disappointed if you checked it out.

I don't think ZFS checks the parity constantly, but it verifies the checksums of every block it does read.

luks won't change that at all, it'll still be at the md level overall. The main encryption modes used by luks will lose a block to bitrot and such but won't lose the rest of the drive. They essentially generate a new key (really a new IV/nonce I think?) for every block through various means. This is the only way to allow for random access to any part of the disk without having to decrypt the previous blocks, and rewriting the rest of the disk whenever you change a part in the middle. I think you can setup an authenticated luks mode, but I've not looked at it and that should catch errors but that's all it would do, catch them.

ZFS has native encryption now that is solid. So you can even avoid luks if you can live with some higher meta data leakage. There was a great talk about the details you can find on yt.

On the last round of zpool scrubs of the 13PB-used set of zfs filesystems here (raidz3, mostly SAS drives, all hosts with ECC RAM), 50MB of data was repaired.

50MB isn't much compared with 13PB, but silent corruption is a real thing.

That makes sense, but mdadm also scrubs weekly on a cronjob. Is there something about ZFS that makes corruption more recoverable than mdadm's RAID?

The core ZFS feature being references here is storage checksums. The general argument is that as storage volumes have increased in size and/or workload, the conventional checksum methods have not been changed to increase their reliability in a way that prevents real world corruption on real world storage during their real world lifetimes.

A RAID system can only reliably recover when an IO error is reported. In this case, it can assume which element of the stored data is no longer trustworthy and use the other parts to reconstruct it. A RAID scrub operation that finds inconsistencies, without a reported IO error, cannot really determine why it is inconsistent. It doesn't know which element is corrupt and which other elements to use to reconstruct it.

In effect, ZFS adds another data integrity layer over the physical storage, so you can have a lower probability of silent failures where the disk controller or IO channels do their own "parity" or other checks and report a successful read when they actually returned corrupt values.

Something like mdadm or LVM would need to add a new layer of checksums over the physical devices to provide this level of protection. Each block/chunk/stripe written to disk would need another integrity checksum written as well. During operation, these checksums would be verified before deciding that a read operation was actually successful. If the verification fails, a virtual IO error is synthesized to flag the read as bad, so that the other RAID recovery techniques can then be applied while knowing which block/chunk/stripe is invalid.

One thing is that ZFS tells you exactly what file is corrupted. I recently took over a ZFS setup that was built with plain ZFS on top of a 24-drive wide RAID6. Drives failed, rebuilt the RAID6, nothing appeared to be wrong. ZFS scrub revealed two corrupted files. I was able to pull backups of those two files, replace them on the FS, and rerun the scrub.

Obviously I moved off the RAID6 soon after, but this is a powerful example of why the checksuming of ZFS is so useful.

RAID5 corruption is non-recoverable - you can't tell which is right, the data or the parity. RAID6 is recoverable - you can get consensus between the data and the two parity bits. ZFS stores hashes of each block, so as long as you have two copies, you can just pick the data+hash that's correct and restore the corrupt block, even with only two copies of the data.

The journal contains information that can be replayed on a crash to hopefully avoid an file system check and repair to make the file system consistent. It's intended to address how long fsck can take as file systems get bigger. It has nothing to do with data integrity per se. Recently ext4 adds journal and metadata checksums (feature metadata_csum) to detect corruption, but data is not checksummed by the file system. Each drive sector is checksummed. The expectation by ext4 (and XFS) when it comes to data integrity is that it's the responsibility of the hardware or some other layer, like dm-integrity.

The data integrity problem is you don't have a way of unambiguously determining data integrity. You have to trust the hardware. Obviously that's good enough most of the time, but it can transiently misstep without outright failing: bit flips, torn or misdirected writes.

By the way, make sure you check for timeout mismatches if you're using consumer drives. Affects Btrfs, mdadm and LVM raid, and I assume ZFS on Linux. https://raid.wiki.kernel.org/index.php/Timeout_Mismatch

I went ~2 years without running a scrub when I first started using ZFS many years ago; when I ran my first scrub, it found 2 file corruptions, one in a random picture back when I still copied pictures from a digital camera to a hard drive, one in a random file in the browser cache.

(If I'd been scrubbing regularly, these almost certain would have been found before corruption crept into the mirrored copy of the file, and been recoverable.)

This is a great point about how pernicious corruption is: it replicates. Detection is more important than correction, at least you can stop the corruption from spreading. It's typical for people to reuse storage hardware, and each derivative inherits prior corruption as well as contributing its own. Most people don't keep a long term unchanging archive, and even that archive likely has no means of unambiguous verification of data integrity.

To be honest I'm not sure if the corruption was replicated directly, or if it's just that the same file got unlucky on two different drives -- I can't recall if this was before or after a drive failure.

> With RAID6 I can have 2 drives fail without data loss.

If you don't have checksums on your data, how do you know it's good (even without drive failures)? Your RAID rebuild could be recreating junk.

> Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems.

Journaling helps with speeding up post-crash reboots: instead of doing a full fsck, it just replays the log.

"Your RAID rebuild could be recreating junk.."

From experience i can tell this can happen. nervous trauma twitch

Your setup won't save you from e.g. bad cables silently corrupting data, or drives simply returning bad data, etc. etc.

Btw, journaling has nothing to do with integrity, though I believe it is checksummed these days.

> Your setup won't save you from e.g. bad cables silently corrupting data, or drives simply returning bad data, etc. etc.

I thought this was what RAID was designed to prevent. I don't know what a cable failure would look like in my setup in practice, but I would think the disk would start failing parity checks against the rest of the array and be flagged as bad.

RAID is primarily about redundancy and speed, not really end-to-end integrity. (Though some setups will certainly have some measures against corruption.)

ZFS is really designed for end-to-end integrity. Once you're stored data on ZFS, you're basically guaranteed that you'll always get the exact same thing back (or an error)... no matter where in the "chain" of I/O the failure occurred.

Well, each RAID implementation has different designs and goals in mind, but the original design was simply meant to replace expensive single drives.

Using more spindles you could get more performance out of a bunch of cheap disks, and to help with the fact that these cheap disks failed more often, you made them redundant.

In general, dealing with data corruption that does not generate an error is not within the purview of RAID implementations. ZFS and btrfs handle it because they're more than just RAID implementations. mdadm can do similar things in a RAID setup if dm-integrity is used on all of the drives (but it is not used by default when just setting up mdadm/lvm/etc.)

Or a power supply problem, which was a source of great pain for me at one point.

Reply on news.ycombinator.com