Migrating a ZFS Pool from RAIDZ1 to RAIDZ2
mtlynch.io
mtlynch.io
I know RAIDZ1 is ZFS variant of RAID5 and RAIDZ2 is ZFS variant of RAID6; I will use all these terms interchangeably because I'm not too interested in the ZFS special sauce here.
In the early 2000s, a lot of people were pushing RAID5. Having worked in a hosting / colocation data centre for many years I had witnessed many RAID5 failures. What would happen is an array would degrade, and more often than not a secondary drive will fail due to undue load on the array as part of the degraded status. Also a lot of times failure would happen on the rebuild process because a lot of the HW implementations were flakey -- but again also the undue stress on all drives as you rebuild. This is why I would suggest a RAID10 setup at the time because of the double lucky failure, and more importantly because you can trivially use a software implementation which is much more safe. Also a lot of the motherboards at the time were offering RAID but this was really just a binary blob in the kernel doing software RAID with a facade of making it appear like hardware which fooled a lot of people.
Well we've finally done away with hardware / proprietary RAID and we have ZFS, mdadm, etc. I've normally dismissed RAID6/RAIDZ2 because of the parity/rebuild process and concerns of putting undue stress on the drive. But I think maybe this is premature and that I didn't really understand the consequences of a single drive failure versus a double drive failure. So this is kind of what I want to know:
1. When a single drive fails, is there any undue stress on the array, or because the array can pretty much operate unaffected, there's actually no performance degradation until you rebuild the missing drive, and in the case of software it's really just a negligible hit on the CPU if it has to do hashing/erasure coding/etc. I guess the rebuild process is really just the cost of a zfs scrub at this point but at least it is on a healthy array.
2. The good news of RAID6 over RAID10 is you can always survive two drive failure; but I think this is where things are concerning because the rebuild across two drives places a lot of undue stress on the remaining disks, and if any of those disks die then you're shit outta luck. This scenario is much more similar to a single drive failure in a RAID5 failure. But again, I think the rebuild cost is that of a zfs scrub but with the minimal set of disks. So RAIDZ2 would be a much more solid choice over RAID10 right; at least you will always know you can survive a two drive failure?
1. If a drive is missing, and it contained a data block you want to read, then you have to do parity calculations to recalculate that block. this means potentially all drives must use their read capacity for this calculation. I think this would be considered stress and max read throughput is significantly reduced. If your block size is very large, or your files much smaller you might get away with minimal performance hit but you're also wasting a lot of capacity/benefit of z2. (In certain pathological cases z2 can have the storage profile of a double mirror, but with all the complications of z2) The rebuild process will require a recalculation for every missing block type, basically every drive will need to perform a read for each restored block. Writing new data to a degraded z2 pool can force zfs to be quite wasteful. For example, a 5 disk z2 pool with 1 drive missing will mean a maximum of 2 data blocks with 2 parity blocks, instead of the expected 3 and 2. restoring that drive will not automatically restore that capacity unless the files are written again to the restored pool. The drives will be filled unevenly, this has performance and storage efficiency penalties.
2. If you replace both degraded drives at the same time, and you aren't using resilver_defered, it should only need to read all other drives once and write to both new disks. But you might not want this, depending on many complicated factors.
Zfs also prioritizes user reads and writes over recovery (resilvering), which may not be universal practice with RAID.
It's certainly still the case that if your redundancy drives failed, you're at greater risk of data loss, and IMHO, disk failure rate during repair is higher than base rate: maybe from extra use, but also because of the risk of correlated failures --- if your environment contributed to failure of the first N disks, the other disks were also in that environment and may be about to fail.
If something particularly bad happened, or you tried being really “clever”, you can get into a rare situation of not being able to import the pool or have it import in read only mode. There are tools to help repair that kind of metadata damage. Then proceed with the normal online repair if needed.
With your extra drive solution, I still have to recreate all my datasets and shares, whereas in my solution, they migrated intact, and I still had backups in case of pool failure. I could zfs send into a giant file on the 18 TB drive, but I'd be reticent to do that because it's just an opaque file that I can't verify will successfully restore. Whereas with my solution, I had the two pools running side by side and could verify everything restored successfully onto the new pool before blowing away the old pool.
Why did you pull a drive from your RAIDZ1 to purposefully degrade it?
I am not sure why you keep saying 18TB. Your drives are 8TB. I am suggesting that you should have simply bought another 8TB disk so you wouldn't have to degrade your RAIDZ1.
Yes, I agree I could have reduced the risk of pool failure if I bought an extra 8 TB disk and not degraded the pool.
So, it came down to do I for sure spend an extra $120 on an extra drive or do I just take the <1% chance that one of my three other disks fails in a 6-hour window while I'm migrating data. I took the chance, but I also had my data backed up in cloud storage at the file level in case there was a pool failure.
In other words, it wasn't worth $120 to me to avoid a <1% risk that I'd have 8ish hours of hassle of recovering from cloud backup after a pool failure.
I have to point out that the surprise risk of the $300 bill from Wasabi dwarfs the cost of the 8TB extra disk.
In retrospect, I would have paid the money for the lowered risk, but everybody has a different tolerance for that.
Again, great work and very detailed.
Raidz3 (survives 3 lost drives) is pure insanity in terms of computational demands. Better off splitting up pools, using multiple systems, or using multiple copy systems up the stack like with Ceph on multiple, simple RAID1 or RAID10 storage nodes.
OP described going from 4x disks, add 3x, end up with a 7x raidz2 pool. The steps are
4/4 (steal disk, set up new pool) -> 3/4 + 4/5 (copy and destroy old pool)-> 4/5 (add missing disk) -> 5/5 -> 7/7.
Your suggestion is to add 4x disks, so you can do
4/4 (set up new pool) -> 4/4 + 4/5 (copy and destroy old pool) -> 4/5 (add missing disk from old pool) -> 5/5 -> 7/7 + spare (hot or cold?).
is that a suggestion or do you meam you never used it? because lvm basically uses mdadm under the hood.
Example: I have 5 XFS volumes across 45 drives on a 4U NAS box with 1 cold spare. Granted, I probably should've chopped them up with a single-box hypervisor RAIDless Ceph cluster of VMs in retrospect.
(Original question: Why X hasn't been implemented yet?)
https://www.raidz-calculator.com/raidz-types-reference.aspx
The general difference between the two is RAIDZ2's additional blocks from its 2nd parity function, which seems independent from the 1st.
The (ideal) set of operations that would happen would be:
1) Treat existing parity disk as blank without overwriting it
2) Write to chosen parity disks 1 & 2 from new disks (assertion: new disks >= 2)
3) Blank out old parity disk
Step (1) would be near-instant, since it involves no writes. (3) either involves removing the block entries on the old parity disk and treating it as blank (near-instant), or writing zeroes to the disk (lengthy). The bulk of the time would be spent on (2).
The process of doing this migration would require siloing off writes to another place whilst the new parity disks are being built.
Considering that this migration (A) very rarely occurs, and (B) would require a write silo in the interim, devoting time into creating this error-prone task would not be wise.
Even though there are commenters criticising this approach, it’s your data and you’re the one who knows best how to handle it. I think you did a decent job of highlighting the risks.
Yeah, I was surprised this post ended up being so controversial.
That's neat! I didn't know there was a way to do this while maintaining data redundancy.
> Step 1: Borrow one disk to create a RAIDZ2 pool
> To begin, I remove one disk from my original RAIDZ1 pool, leaving it in a degraded state.
Oh, there isn't. :facepalm:
I have backups, so I was prepared for data loss if the pool failed.
Is this any riskier than recovering from a single disk failure on a RAIDZ1 pool?
1. Build new RAIDZ2 pool with all your disks that you plan to use. 2. Restore backup to the new pool.
I keep a backup too and so this is how I move to a new larger ZPOOL with a new layout.
Either you have to do this because you don't have a backup, and so this is risky. Or you don't need to do this because you have a backup and can just build your new pool and restore your data from the backup.
You keep the raidz1 pool healthy, create a 5x8TB raidz2 pool with two fake disks, migrate the data, and then only pull the disks from the raidz1 pool when the data successfully migrates. Even if a disk dies during reslivering on the raidz2 pool, you still have 3 of your 4 disks in the raidz1 pool.
An alternative would have been to build a RAIDZ2 out of three new disks and two tmp files and then copy the data over and finally offline a RAIDZ1 disk and online it on the RAIDZ2 zpool. Two copies of the data in each pool would require an actual disk failure in both zpools to lose data during the resilvering (even though both were degraded), and when breaking the original zpool to add a replacement disk for the 5th device in the RAIDZ2 pool you'd have to lose two existing disks in that pool to lose data.
No criticism; it's cool to be able to do moves with minimal resources and I've also thought about potential ways to upgrade zpools under weird constraints; especially no free SATA ports for example. I had thought about using the four SATA ports to copy data from half the source disks in a RAIDZ2 to half the destination disks of a new RAIDZ2 while the removed disks would still be a functional zpool on their own as a backup. But ultimately I found a cheap extra box and went for a complete upgrade to larger disks with the benefit of keeping the old box as a local mirror.
My upgrade was from a single RAIDZ2 of 4x 4TB disks (8 usable) to a RAIDZ1 of 4x 14TB disks (42 usable) because once both pools existed I was comfortable running the old disks in a 12TB RAIDZ1 to receive snapshots from the primary zpool while I had <12TB used. Still requires 2 disks to fail to lose data, and now I do some offsite backups as well (rsync and s3backer) and have a test system to perform upgrades on before the main system.