I'm asking because the main advantage I can see for a zettabyte file system (ZFS) is at larger scale, thus I'm interested in how to actually use it e.g. for a server setup of a company that needs to save data on many disks.
AFAIK yes you could, but you wouldn't want to put them in a single RAID-Z(1,2 or 3) VDEV, but rather multiple VDEVs. This is because I/O operations scale with the number of VDEVs rather than number of disks[1]. So there's a tradeoff between space efficiency and IOPS. But AFAIK you absolutely could put 100 disks in a single RAID-Z VDEV.
On the mailing lists there's frequently people posting with that or more disks in a single pool (split across multiple VDEVs).
[1]: https://www.delphix.com/blog/delphix-engineering/zfs-raidz-s...
So just to clarify. A ZFS pool stores data in VDEVs. A single VDEV can be:
- a single disk (no redundancy, but same error checking)
- N disks in a N-way mirror
- N disks in a RAID-Z (single for disk parity, min 3 disks)
- RAID-Z2 with 2 disk parity
- RAID-Z3 with 3 disk parity (much slower than other two RAID-Z's due to complex math)
You can always _add_ VDEVs to a pool, and ZFS will start using it. It will try to be clever about it, so for example if existing VDEVs are quite full, it will redirect most of the writes to the new VDEV.
What you cannot do (yet) is _remove_ a VDEV.
It's also possible to replace _all_ the disks in a single VDEV to larger disks in turn (important!), and once the final disk is replaced the VDEV will suddenly appear to have the added capacity.
What are the drawbacks of this approach and why can't I just add additional drives to an existing VDEV?
However for the RAID-Z variants it's more difficult, because of how the parity calculations are done and how the parity is stored. If you have a 5 drive RAID-Z, you can store up to 4 blocks of data for each parity block. With a 6 drive RAID-Z you can store up to 5 blocks of data for each parity block.
So if you add a drive to a RAID-Z vdev, the system suddenly needs to keep track of the stripe width that was used when the data was written.
In addition it's the issue of _where_ the data is stored. If your RAID-Z VDEV is mostly full and you were to add a new drive to it (not that you currently can), then you might not be able to use all of the capacity without further ado.
This is because you can only store one block per stripe on the new drive for redundancy to work, a RAID-Z1 can only recover from the loss of one block per stripe. The rest of the blocks in the stripe has to be distributed on the other drives.
However, there is work in progress[1] to support RAID-Z expansion.
That said, RAID-Z expansion is for expanding one drive at a time. If you're adding 20 disks in one go, do it as a new VDEV.
The ZFS devs don't like to push unstable features, so I expect we're still quite a way off.
Careful though, depending on how you're setting up the VDEVs this could introduce fragility to the pool. For example, if you have a raidz2 partition and then add a single disk vdev, that single disk going down would make the entire pool unavailable.
Only way to recover from this would be to either add another disk turning the single-disk VDEV to a mirror, or send/receive the entire pool.
If you want to span a pool across multiple regions, you'll want a distributed file system on top of your ZFS to manage that. Something like Lustre or Ceph. It would still be very challenging, though.