I wound up causing a major outage of my services and destroying my home dir

If you want to use ZFS for root, I can heartily recommend NixOS. I've never had a problem with it there. Of course there are other downsides, but there are plenty of upsides as well.

The only thing exporting a pool does is declare "I am not being mounted by someone else". The FAULTED state would be because the search paths for the devices didn't exist. Clearing /etc/zfs/zpool.cache and doing `zpool import -f` would fix it entirely (ZFS identifies the disks by UUIDs in the headers AFAIK).

This is why one should not use sda/sdb or vda equivalent but use /dev/disk/by-uuid/

Turned out at least on the ones I had that it was always the outer edges of the platter that failed, so a (from 20 year old memory, may be wrong) 60Gb Death Star if carefully partitioned out to a maximum of about 58Gb ran absolutely beautifully and you could get them dirt cheap because (understandably if you weren't a crazy person like me) almost nobody wanted to touch them at that point.

Add nightly or weekly offline backups and you should be good to go. And by offline, I mean physically connecting a storage device, making the backup (preferrably via an automated script), and then disconnecting the device and storing it elsewhere in a safe place.

In your scenario, I understand that the original physical host where you're making a backup of a VM has no redundancy or data integrity guarantees? I'd just like to point out that backups are good, but if all you have are backups of the same corrupted data, then they are pretty much useless.

My approach is to have redundancy and data integrity guarantees (ZFS mirror) at the source and have offline backups on external hard drives. The original data is likely to be OK, and well, if I need to recover from a backup, I'll be happy enough with what I have there anyway.

Btrfs raid1, on the other hand, uses the checksums of the data stored in its metadata trees to validate which copy of the data is correct and repair the corrupted copy. 3-disk makes this even more robust, as you get an exponential growth of data-metadata pairs to pull from. If any one pair matches, that data can be assumed to be correct.

And the drive would return bad data instead of just erroring out - this is sometimes a difference between “home use” drives vs enterprise - enterprise assume you’re in a multi redundant raid setup and instead of retrying will just fail out fast and let the raid take care of it. Home drives will continue retrying and perhaps even eventually give you what they got - the idea being you’d rather have one byte bad of a word doc than lose a 4K chunk of it.

ZFS catches these tricks because the checksums will fail. Multi-redundancy RAID can do it - but since most are enterprise they are built with the assumption the drive will return good data - or none.

This can be the worst scenario as corruption can be silent and infect all backups after a time.

I would therefor say it is extremely unlikely any company (let alone Oracle) would ever sign-off on such a thing.

Then you can upgrade by putting the partitions on different drives etc. Of course must then be careful not to put two partitions of the same vdev on the same disk.

The major downside is that ZFS isn't aware that the underlying devices are the same, so IO which looks nicely scheduled as far as ZFS is concerned might cause massive random IO on the underlying disks.

That said, if one could tell ZFS about which zfs-devices share physical hardware, so the IO scheduler would be aware, then that might be a viable strategy. Might be handy in case you have multiple iSCSIs sharing a 10G link as well.