Why ZFS is not good at growing and reshaping pools, or shrinking them
utcc.utoronto.ca
utcc.utoronto.ca
For a similar trade off compare hard links on Windows vs. ext4 and co. on Linux. On Linux there is no command to list all hard links to a file while Windows (NTFS) has one (FindFirstFileName etc.). This is why creating hard links on Windows is much slower and there is a limit of 1024 hard links to a file. To find all hard links to a file on Linux one would need to iterate over all files and check if they have the same inode number, however.
It's been a long time since I took that filesystems class, but I recall it's something like that?
https://en.wikipedia.org/wiki/Inode_pointer_structure has a summary
https://docs.microsoft.com/en-us/archive/blogs/askcore/the-f... talks about the NTFS equivalent
Are you actually sure that's why? Usually the Windows bottleneck is in the I/O subsystem overhead rather than the file system itself. Note that FindFirstFileName requires opening the file first, which is the operation that incurs a lot of overhead irrespective of what you're doing after. I read that on Linux this is avoided through the use of a cache (hash table?), which Windows doesn't have. Once the file is opened, the I/O performance on Windows and Linux is comparable.
I mean... you just described a scrub. I'd gladly sit through a scrub if it meant that I could grow a raidz vdev.
I'm suspecting there's some other hurdle.
Which is totally fine. Growing an array is a big and rare event. I'd be totally fine if I have to take the array offline or read-only for the duration. I just want the option to do so at all.
Lots of software is not as CoW-friendly as it could be, offline dedup fixes that with relatively low overhead.
Running a dedupe based on file rather than block would do the same thing. You don't gain much if anything in your example using btrfs.
What is being argued here?
* fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions
* fdupes will not let you reconcile duplicates in snapshots on ZFS
* ZFS deduplication is block-based
Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done.
>fdupes will not let you reconcile duplicates in snapshots on ZFS
Why not? The files are kept in a hidden folder in the base directory.
>ZFS deduplication is block-based
It is also live. You don't gain Mich with offline block deduplication over file, though it is better. Live deduplication is a feature even if unnecessary for most.
For files that are used by anything, they might be written to. I can't have a write to one file appear in a completely different file just because they had the same contents at one point. And what if my duplicate files have different owners or permissions?
> Why not? The files are kept in a hidden folder in the base directory.
Those are read-only views, not real folders.
> You don't gain [much] with offline block deduplication over file, though it is better.
In general, perhaps not. But for ZFS specifically you gain a lot by not using its deduplication system. It needs tons of memory and explodes files into fragments.
The act of saving should destroy the link, though I know that isn't always the case. Still, this is a benefit of a CoW filesystem, not offline dedupe. Permissions may be a problem, I'm not great with them.
>Those are read-only views, not real folders.
Then mount them writable,
>But for ZFS specifically you gain a lot by not using its deduplication system. It needs tons of memory and explodes files into fragments
Live deduplication is valuable in some instances, otherwise there are still offline alternatives available.
It's one method of gaining CoW. And ZFS does not support CoW between files. But whatever feature you lump it under, it's an advantage Btrfs has.
> Then mount them writable,
You can't.[1] And even if you could, you can't make a hard link between subvolumes.
[1] You can 'clone' a snapshot to make a writable subvolume, but the original read-only snapshot can never be deleted while the clone exists, so this doesn't help you fix duplicates.
That's bullshit. hard links are the same exact inode, with all that entails. reflinking is at the data block level. Totally different.
Reflink has its own fs metadata including inode, with (initially) shared extents. Those shared extents can have their blocks individually and independently modified, per file. The point at which there are no more shared blocks, they're not reflinks.
It's not an intrinsic property of file-dedupe vs block-dedupe. It's just how it's conventionally done.
Inodes that happen to share blocks are not the same file. ie totally different.
Of course this only works if the data is read-only.
It won't reclaim any space on ZFS if you're using snapshots.
I think that block-level deduplication has more predictable performance though: for example if you write some blocks to a deduplicated file, with block-level deduplication you just write the new blocks, with file-level deduplication you have to rewrite the whole file.
The current implementation of dedup in zfs was the biggest disappointment I had in setting up my current storage box. Even with plenty of horsepower, I quickly turned it off and accepted the net reduction in capacity.
[1] I put the wrong command here originally.
Or actually detects duplicate blocks? Because that would be still be something, knowing how much deduplication would benefit you, if implemented.
Any filesystem with data checksumming should only need to build an index of the checksums to find duplicates, which sounds far cheaper. Does btrfs do that?
The utilities can be written in any language, there's several popular ones called dupremove, bedup, btrfs-dedupe, bees, and dduper.
Bedup specifically mentions "It integrates deeply with btrfs so that scans are incremental and low-impact."
For my use I'd likely ignore all files less than a few weeks old for deduplication purposes. Any virtual machine images I'd use block level deduplication and everything else I'd use file level deduplication.
Doesn't seem much different overhead wise than the normal scrub that linux software RAID, ZFS, and btrfs does anyways. I'm all for not storing old files twice, but would like to avoid the overhead of checking for deduplication on every write.
If they did, the dedup process for a multi tb drive would just be a few seconds, and the overhead would be so small it could be run on the whole drive every minute all day long.
Yes you can deduplicate 1TB quickly if you already have the data and metadata data checksums.
ZFS (with dedupe on) checks the checksum of every written block against every existing block in the pool, thus the slow writes and being very ram intensive.
Btrfs instead calculates data and metadata checksums for integrity then allows using them afterwards for deduplication. You could of course check for block level checksums for 100% of your storage. Or you could decide to dedupe only old files, only VMs, or only non-VMs. Or even a hybrid of block level for VMs, but file level for non-VMs.
As a result ZFS with deduplication on uses a ton of ram and has slow writes. This is so big an impact that most folks just turn it off, despite the tempting lure of "free" disk space.
This allows the dedup table (which stores the hashes of existing data to find identical blocks) to be limited in size to whatever you want, and especially prevent it from eating all the RAM. The tradeoff is that the smaller it is, the deduplication hit rate drops.
Here's a talk by Matt Ahrens about it, which also includes some other tricks to improve dedup performance:
I do use zfs in preference to ufs on my pfSense boxes running non RAID on mSATA type storage but that is not a scientific choice but based on some quite iffy hearsay.
I have my opinion and you have yours. When it comes to filesystems then I want simple (and good backups.) Fast is nice but reliable is better. If I want faster I buy a bigger RAID controller or whatever.
Their backup options are drastically more complex because there's no way to take an atomic snapshot.
The complexity of data-integrity and backups are pushed a layer up, whereas more complex filesystems manage data-integrity and atomic snapshotting themselves, and can do so more easily due to existing at the correct layer.
I wanted it in part for me, but I also wanted it for all the people in my life who were doing a terrible job of taking care of their important data.
“Don’t worry,” I wanted to tell them. “Get one of these, slap some drives in it. When you need some more space, buy a new drive, take the smallest one out and put the new one in.”
Some of the early PR for ZFS was unclear on whether that was possible, but I was excited. This was by far the closest we’d gotten. Later it was a feature we’d have soon. I waited to adopt. Apple was going to support it, here we come. Apple was not going to support it. Okay, NAS could still happen. Sun craters and I lose track, but check in every so often, and still nothing.
In the grand scheme of things, it is not the biggest tech disappointment I’ve suffered. But it feels like it should be in the top ten, even if it’s #10.
ZFS was not philosophically opposed to those ideas. They were just dropped at some point.
Had Sun remained the powerhouse it was, the BPR would probably probably have eventually gotten done, but with the smaller resources devoted to ZFS development now vs in the Sun days, it's gotten kicked way down the priority list.
That is an insanely time consuming operation that also increases the risk of data loss as you have less redundancy during the expansion.
An enterprise ought to just migrate the entire pool/content to somewhere else and redo it completely.
Expanding by replacing with higher capacity drives just feels like a hack and afterthought that only hobbyists would actually make use of.
If the old device is not faulty then the replace operation will keep both old and new in the array until it is complete.
> An enterprise ought to just migrate the entire pool/content to somewhere else and redo it completely.
That's also an option with send/receive, but that involves downtime. Drive replacements can be done on a live system. Server grade hardware generally supports hotplug.
Which you probably should.
That's not quite the issue.
The issue is that PHYSICAL and HASH block addresses are stored together, intermingled, which means that the PHYSICAL addresses get hashed into the copy-on-write tree, which means any block can't be relocated (i.e., have its physical address(es) changed) without rewriting the path to the block containing a pointer to it as well.
I.e., ZFS is not content-addressed storage. In CAS the hash of a block is its pointer, and the physical addresses are stored separately (not intermingled with hashed data) and not hashed into the tree.
IF instead ZFS had used ONLY the hash value as the blkptr_t, and then had stored the physical addresses at the end of metadata blocks and then NOT hashed the physical addresses into the tree, THEN relocating would be a lot easier. Not trivial, mind you, just easier. Not trivial because of snapshots and dedup: you can find a singular pointer to any data block in the absence of snapshots and dedup, and rewrite the physical address stored nearby, but you can't easily find duplicate pointers, so to relocate blocks in CAS w/ snapshots and dedup you have to leave behind forwardings, or have a database of relocations (which is costly).
> (In the grand tradition of computer science we can sort of solve this problem with a layer of indirection, where the top layer stays immutable but the bottom layer mutates. This is awkward and doesn't entirely satisfy either side, and is in fact how ZFS's relatively new pool shrinking works.)
Yes, and proper CAS also has a layer of indirection, and some awkwardness around snapshots and dedup (see above). For mirror drive evacuation there's nothing to do. For shrinking operations you only need to relocate blocks whose physical addresses are beyond the end of the new device (but you'll need a fair bit of swing storage for the interim relocation or to use as forwardings until you've found and fixed all pointers to any one relocated block).
I've said this many times in the past, and always someone complains that not hashing the physical addresses is unsafe, as if the hash of the contents is not safe but hash of contents and physical address could somehow be so much safer (it can't be much safer, if at all). And again, see CAS.
If you want an array of disks with a mutable configuration, I think the right choice is mdadm and LVM. They're reliable, relatively easy to use, let you move disks between machines, don't rely on hardware RAID, and support all file systems.
If you need something faster and more optimized like ZFS or btrfs, you're probably better off spending on SSDs anyway.
Tell me why I'm wrong, HN :)
You have to manually set up dm-integrity on the disks, however - it is not used by default when creating an mdadm/lvm setup.
But you can start hacking together a lot of the functionality the more dm modules you start using.
In other words zfs and btrfs incur less overhead by having the raid, checksumming and journalling layers talk to each other. Implementing them separately in as in device-mapper comes with inefficiencies.
I make backups regularly (manually, because I want them to be offline), but I'd rather not have to use them if one of my drives die. Unless I literally just finished making the backup, I'm going to lose some amount of work.
My mirrored ZFS pool means that (A) a drive can die and I won't lose _any_ data, no matter how recent, and (B) I'm confident none of the data on either drive is corrupted, because ZFS checks it regularly (every time it's accessed, and during weekly scrubs).
I do use SSDs, where I need speed - L2ARC ;), far better use than buying 19Tb of SSDs, not to mention the price.
Anyway, I do use ZFS on laptop for linux root. Reason? Snapshots. Kvm + ZFS, perfect match. /.zfs/snapshots? Fabulous.
Filesystems like BtrFS and ZFS know what is being used and what is not and won't waste time reading or writing to blocks that don't hold any currently used data.
That's taken care of by TRIM/DISCARD, which all modern filesystems support.
It's like you're saying, why do we need ECC when there is fast DDR4 memory?
ZFS is a filesystem not just a layer like LVM. You have to compare it to LVM + some filesystem. Which filesystem would you suggest using with LVM to get ZFS-level integrity?
(Technically you can use ZFS on top of LVM but that would be a weird thing to do.)
With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing? If the concerns about "data integrity" are really about time until a disk is replaced, then it makes sense. Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems.
Nothing in your setup there protects you from bitrot.
>With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing?
Drive rebuild time is primarily a function of a variety of things - speed of the disks, processing power involved, etc. There's nothing magic about any solution here. ZFS does get a leg up in that it knows what blocks are in use, but hardware raid and mdadm are quite fast when they find empty blocks too, so it's not some overwhelmingly massive improvement.
>Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems.
I think you're fundamentally confused about what journaling provides a filesystem. A journal helps you in situations where your system crashes, loses power, etc. and you had not finished committing your changes to the filesystem. It does not protect you from data integrity issues that are unrelated to this sort of situation - a bit flipping in an HDD from electromagnetic radiation or an SSD from electron leakage will not be caught by a journal.
Isn't that what the scrubbing process is for? My understanding was that RAID is keeping parity bits to make sure bitrot hasn't happened and it periodically scrubs and repairs when bits do rot. Is the difference that ZFS scrubs in real-time?
Sorry if I'm being dense here, but I did a fair amount of reading about this, and I'm trying to understand what I'm missing.
Nope. Parity bits aren't used to detect bitrot. If you get a URE when you go to read, the parity bits will be used to get you data. If you read successfully, nothing happens at all. If you scrub, and the data is found to be inconsistent, md assumes that the parity is wrong and overwrites it.
https://linux.die.net/man/4/md
>If a read error is detected during this process, the normal read-error handling causes correct data to be found from other devices and to be written back to the faulty device. In many case this will effectively fix the bad block.
>If all blocks read successfully but are found to not be consistent, then this is regarded as a mismatch.
>If check was used, then no action is taken to handle the mismatch, it is simply recorded. If repair was used, then a mismatch will be repaired in the same way that resync repairs arrays. For RAID5/RAID6 new parity blocks are written. For RAID1/RAID10, all but one block are overwritten with the content of that one block.
Is it correct to say that ZFS parity is used to reconcile on each read even without UREs, but with md parity is only used when there are UREs? If that's true, it makes much more sense why ZFS is considered the right choice.
Also, do you know if adding luks in the mix changes that?
My data goes md->lvm->luks->ext4 and I hadn't considered which part of that is supposed to catch the error if it wasn't a URE.
It makes sense that md would update parity if there was no URE. It also makes sense that LVM wouldn't know or care about data integrity.
I would think luks would require data integrity to work properly, but I'm not sure what happens if there is bitrot in a luks block.
I'm also surprised ext4 doen't have checksumming, but apparently it's only for able to checksum for metadata and it's disabled by default: https://serverfault.com/questions/456319/how-to-get-checksum...
Dm-integrity sounds worth a look, but https://raid.wiki.kernel.org/index.php/Dm-integrity doesn't inspire much confidence.
>Dm-integrity sounds worth a look, but https://raid.wiki.kernel.org/index.php/Dm-integrity doesn't inspire much confidence.
That wiki page is pretty yikes - I was under the impression dm-integrity was farther along.
Some misc other stuff I like about ZFS:
l2arc/zil on ssd/optane devices for fast read/write caching Easy NFS/SMB sharing zdb ( https://manpages.debian.org/testing/zfsutils-linux/zdb.8.en.... )
I don't think you would be disappointed if you checked it out.
(If I'd been scrubbing regularly, these almost certain would have been found before corruption crept into the mirrored copy of the file, and been recoverable.)
Btw, journaling has nothing to do with integrity, though I believe it is checksummed these days.
I thought this was what RAID was designed to prevent. I don't know what a cable failure would look like in my setup in practice, but I would think the disk would start failing parity checks against the rest of the array and be flagged as bad.
Using more spindles you could get more performance out of a bunch of cheap disks, and to help with the fact that these cheap disks failed more often, you made them redundant.
In general, dealing with data corruption that does not generate an error is not within the purview of RAID implementations. ZFS and btrfs handle it because they're more than just RAID implementations. mdadm can do similar things in a RAID setup if dm-integrity is used on all of the drives (but it is not used by default when just setting up mdadm/lvm/etc.)
ZFS is really designed for end-to-end integrity. Once you're stored data on ZFS, you're basically guaranteed that you'll always get the exact same thing back (or an error)... no matter where in the "chain" of I/O the failure occurred.
50MB isn't much compared with 13PB, but silent corruption is a real thing.
Obviously I moved off the RAID6 soon after, but this is a powerful example of why the checksuming of ZFS is so useful.
A RAID system can only reliably recover when an IO error is reported. In this case, it can assume which element of the stored data is no longer trustworthy and use the other parts to reconstruct it. A RAID scrub operation that finds inconsistencies, without a reported IO error, cannot really determine why it is inconsistent. It doesn't know which element is corrupt and which other elements to use to reconstruct it.
In effect, ZFS adds another data integrity layer over the physical storage, so you can have a lower probability of silent failures where the disk controller or IO channels do their own "parity" or other checks and report a successful read when they actually returned corrupt values.
Something like mdadm or LVM would need to add a new layer of checksums over the physical devices to provide this level of protection. Each block/chunk/stripe written to disk would need another integrity checksum written as well. During operation, these checksums would be verified before deciding that a read operation was actually successful. If the verification fails, a virtual IO error is synthesized to flag the read as bad, so that the other RAID recovery techniques can then be applied while knowing which block/chunk/stripe is invalid.
If you don't have checksums on your data, how do you know it's good (even without drive failures)? Your RAID rebuild could be recreating junk.
> Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems.
Journaling helps with speeding up post-crash reboots: instead of doing a full fsck, it just replays the log.
From experience i can tell this can happen. nervous trauma twitch
But what if you have silent data corruption: a single bit? Even if you have parity (raid5) or a duplicate (raid1), how do you know which part is correct, and which part to repair? Without checksums.
I've had to restore 1TB from Crashplan. They apparently had checksums in their software, and something like 10-15 files were damaged.
Without checksums, you simply don't know if your data is damaged, even on a single hard drive. ZFS gives you data integrity with checksums, and redundancy with mirrors or raid-z.
This is why a regular scrub is recommended, you verify the checksums and repair if possible.
He then reads the file, uncorrupted, with a kernel warning about a bad checksum that was fixed (from the mirror) during the read.
This demonstration is why I switched to btrfs (and ECC) for my new home server/NAS.
That, along with compression sealed the deal. I was disappointed about not having dedupe at the time, but it probably really wouldn't have saved me that much space. (I keep meaning to run the dedupe tool to see what it might save me now.)
This sort of gets into it but isn't a fantastic source: https://www.enterprisestorageforum.com/storage-hardware/driv...
See also https://en.wikipedia.org/wiki/Data_corruption#Silent and https://en.wikipedia.org/wiki/Hard_disk_drive#ERRORRATESHAND... .
Notably, Apple's new filesystem introduced in 2017 does not detect data corruption; it only checksums metadata. https://en.wikipedia.org/wiki/Apple_File_System#Data_integri... Apparently Apple Engineers are of the opinion that Apple SSD and HDD media simply don't have silent data corruption. They have some pretty smart engineers, so maybe they're on to something. It does beggar the question, though: why have metadata checksums at all? http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/
Anyway, I'm not exactly supporting threat models that ignore silent data corruption. Just pointing out that they're not exactly as ridiculous as they might seem on their face.
Linux md will do the completely wrong thing with this: it will treat that as a drive error, and if you accumulate a lot in a short time (such as rebuilding an arrary) then it will conclude that this is a failed drive and kick it out of the array.
How many array rebuilds have you done? You only see the read errors when you go to actually read the sector in question, and even in a week I wouldn't want to rely on not having any sectors go bad.
As a related anecdote, I did some stress-testing on a ZFS filesystem with two Kingston NVMe SSD-s in mirrored configuration and at one point I got two random read errors on one drive, which subsequently got corrected by ZFS.
Silent failures might not be the best phrase to use for these situations, but ZFS does let you know that corruption has taken place and you have to acknowledge that message for it to disappear.
It's not clear to me if you're describing silent data corruption or detected data corruption (at the controller level), but either way Apple isn't selling Kingston NVMe SSDs; they have their own SSD controller and vendor the raw flash.
The data integrity problem is you don't have a way of unambiguously determining data integrity. You have to trust the hardware. Obviously that's good enough most of the time, but it can transiently misstep without outright failing: bit flips, torn or misdirected writes.
By the way, make sure you check for timeout mismatches if you're using consumer drives. Affects Btrfs, mdadm and LVM raid, and I assume ZFS on Linux. https://raid.wiki.kernel.org/index.php/Timeout_Mismatch
File systems like zfs and btrfs checksum every block to ensure that if they do become corrupt, they can get repaired. (otherwise, if you had a mirror of disks, and one disk says the value is 0010 and the other says its 0001, wich one is right?)
Additionally, the 'raid' concept is not just about speed. Its also about capacity, and how many disks can fail. I might have 48 drives in a single server. I certainly don't want an LVM stripe across all drives and lose data. I also don't want raid1, losing the capacity of 24 of the drives. I want it so that any 3 drives can fail in that system, and I can still recover the data.
It certainly made me reconsider my ZFS practices.
ZFS isn't really a filesystem. It's more like a storage backend, that happens to support the POSIX interface. The original intent of ZFS was to completely rethink the volume management/filesystem layering nonsense, and to provide for storage what the MMU/VM system provides for memory.
So it can do much more interesting things like memory backed read caching (via ARC), SSD backed read caching (via L2ARC), and SSD backed write buffering (via ZIL/SLOG), as well as the normal RAID stuff.
Granted I was a much nooblier noob when using mdadm.
https://raid.wiki.kernel.org/index.php/Growing
It should be possible to buy disks of any size, add them to the computer and tell the file system to use them. Drive failures shouldn't increase the risk of additional failures.