Except… there's nothing to really learn. All these weird edge cases that can kill your BTRFS simply don't exist in ZFS. The tooling is also much more polished and in the few cases that something does require user attention, it'll tell you in easily understandable terms, so as a new user the only command you really need to learn is `zpool status`, which you now did.
But of course there are peculiarities that have 'bitten' people (softly) in the past:
- dedup has hardly any use case, novices get enticed to try something that doesn't do what they think it does
- ZIL and SLOG sound like "a simple cache" when in reality they are much more complex
- You can add single devices to a (redundant) pool, at which point you've lost redundancy, perhaps without knowing.
- No defrag
- No re-balance
- No growing pools
- ARC still surprises people (also, I've had kernel crashes under heavy load when the ARC didn't free ram fast enough ... edge case maybe, but still)
That said, of course, ZFS is awesome and I encourage everybody to give it a try.
- current versions of zfs (ie: openzfs) error (unless forced) when adding non-redundant devices to a redundant pool, and have device removal to handle the case where someone bypasses the error.
- pools can grow in zfs (ie: the things one allocates storage out of). what is probably being referenced here is that one can't reshape a zraid vdev to add more disks to it (basically the reshape operation in classic raid setups). This does make expanding in some cases less flexible that old-style raid setups.
Also, I really have to agree that ARC behavior is still not great under load, and I don't think that's that much of an edge case.
- Very limited ability to make snapshots writable.
Would you mind explaining to us how to add a disk to an array with ZFS? Then same question but with a disk which is not the same size that the rest of the array.
To take you up on that with a dangerous example that reinforces the point:
To add a disk to an array whose used space is less than the capacity of a single disk, you can (DON'T DO THIS, IT'S DANGEROUS)
(1) remove a disk from the array (it's degraded now) (2) reformat the single disk, put all data on there (3) create a sparse file with the size of one disk (4) create a new array with N disks, in place of the temporary disk use the sparse file as a device (5) remove the sparse file (the new array is now degraded) (6) bring the new array up, copy all data from temp disk on it (7) replace the sparse file with the temp disk (8) copy files around to make them re-balance
Congratulations, you've now grown your array, and put your data seriously at risk in the process! :)
mdadm --grow /dev/md1 --size=max
FWIW I honestly don't think growing an array vs adding to a pool is actually that interesting given the modern state of drives and how capacity growth has lined up with data growth across the spectrum. At the consumer/soho/small business scale growth curves for storage blew past data needs a while ago, so by the time somebody needs more storage the drives are old and it makes more sense to just replace them with much bigger ones (which ZFS supports fine, replace all the drives in an array one by one then it'll expand), or small drives will be so much cheaper that just adding another set to the pool makes sense. Much bigger then that and people aren't going to be doing array expansion ever anyway. Still, cool to see that one checked off the list, and if the functionality ultimately enables generalization to RAID-Z type conversion or better rebalancing that'd be very interesting.
>Then same question but with a disk which is not the same size that the rest of the array.
Don't do that simple as that? There isn't any great way to handle that with full performance and redundancy, and any possible value seems way way too niche to be worth bothering with. Although again if you want to add uneven vdevs (single or multidisks) to a pool you can always do that if you want.
Edit: One other practical consideration I've always had regarding expansion, adding disks and so on, is that the troublesome brute force method to me is an excuse to promote good habits and practices most of us are very lazy about in general. Having to tear down a pool and restore to get best performance and flexibility out of adding a bunch of disks is certainly a pain. But we should be able to do this at any time anyway, we should be semi-regularly testing that it works, and we do all do that testing right? Right? OK probably not, plenty of people aren't in the habit which is why automation is so important. But even if ZFS gained full perfect flexibility, I still think I wouldn't use it in general because I think the exercise of making sure that yes I can do a full restore and everything is working at least once every few years is so valuable.
What you do is buy not one drive, but a set of drives and add an second array to the system. Then RAID-0 between the two arrays.
Really?
Fortunately, most people don't have data they care about. Their photos and documents are on cloud providers like google/dropbox/facebook (and dozens of others I don't even know of). If disaster happens they might lose a couple days work. There is a lot to not like about cloud providers, but they are probably better than what the average person does - no backups at all. Note that the RAID system I've described here is for data integrity, but there are still house burning down cases where you lose it all,
What the hell? Can you please clarify what you mean by "BRTFS-raid2", since that's not a term that has previously existed and BTRFS does not have any functionality remotely similar to the long-obsolete traditional RAID2, and none of the RAID functionality that BTRFS does have has any limitations or safety thresholds for going beyond 8 drives, especially not the RAID1 mode (which is the most plausible thing your raid2 could have been a typo of).
If you have an 8-drive btrfs RAID array, going to 9 drives is as simple as `btrfs device add`, optionally followed by a `btrfs balance start`. Neither of those will reduce the redundancy of any data stored in the array. And I can't see any reason why you're suddenly talking about 3 drives right after talking about 8 or 9 drives.
You're talking in circles. You're assuming the limitations of ZFS, then providing advice based on those limitations, then declaring that there's no advisable use case for a filesystem that's free of those limitations.
On btrfs, you can start with two or three drives, then expand your array one drive at a time up to 8+ without lowering your redundancy at any point, and each time you add a drive the usable capacity of the array really does increase.
What are you basing this claim on? Did you mean to qualify that statement to only apply to certain RAID modes, or are you again just repeating advice specific to ZFS as if it applies to all storage systems?
Note, good raid6 should have x and y stripes with no more than 8 drives in any stripe. I have no idea what they do, so maybe they can restripe somehow on one drive. Hiding the implementation detail is good if they do this right. But I don't know how to tell if any do it right.
These two sentences are in conflict. There are RAID modes where the failure of two drives is recoverable. You probably know this, since you mention one of them in your next sentence. But maybe you don't realize that RAID 6 isn't the only way for an array to survive the failure of two drives, especially if the failures aren't simultaneous. You should read up on some storage systems beyond traditional hardware RAID and ZFS; there are quite a few interesting ideas out there you should open your mind to. Btrfs would be a great place to start your investigation.
The numbers shift a bit depending on how much you expect drive failures to correlate and how long you take to replace them.
So I analyzed two example cases, using pretty good math based on a binomial distribution.
The first case is a pessimistic one, where after a drive fails we assume that any other possible failures from the next year will hit before we finish rebuilding. In this case, 8 drives with double parity have a 1/740 chance of array failure each year. If we switch to triple parity, we can have 16 drives and a slight improvement in safety.
The second case is a more optimistic one, where after a drive fails we assume that any other possible failures from the next week will hit before we finish rebuilding. In this case, 8 drives with double parity have a 1 / 93 million chance of array failure each week. If we switch to triple parity, we can have 40 drives and a slight improvement in safety.
I would have done "what if all the drive failures happened at once" but that gives >10% odds of an 8 drive array failing, which seems unreasonably harsh.
zpool add tank mirror /dev/sdc1 /dev/sdd1
From the perspective of an individual user it's definitely a suboptimal part of ZFS. But standing in the shadow of business usage is the price to pay for it having been developed in the first place.
As an individual user you can play games with sparse files and failed members as a sibling comment says (and dragging some external drives into service can help, as well as restoring from your backups). But really, just try to plan for the space you want and buy at once. Disks are large enough these days that you should aim to not be tinkering with your storage.
Back in the day I had something like an 8 x 250GB disk external USB drive array for my FLAC collection, with each disk partitioned into 10 slices, raid6'd, and then LVM'd back together. This gave me the wiggle room to take down a single slice at a time for reshaping, and then I could even online grow the filesystem thanks to reiserfs. No backups of course. I did the online expanding thing once or twice and it was neat, but ultimately the whole thing was replaced by a simple 2TB mirror after several years.
There's no reason you couldn't do something similar with ZFS besides performance. But do you really want to?