Every generation rediscovers why ZFS exists, and why it was designed the way it is.
Difficulty aside, I'm wondering if it would be possible for ZFS to implement that as an option without giving up other features.
Then you can upgrade by putting the partitions on different drives etc. Of course must then be careful not to put two partitions of the same vdev on the same disk.
The major downside is that ZFS isn't aware that the underlying devices are the same, so IO which looks nicely scheduled as far as ZFS is concerned might cause massive random IO on the underlying disks.
That said, if one could tell ZFS about which zfs-devices share physical hardware, so the IO scheduler would be aware, then that might be a viable strategy. Might be handy in case you have multiple iSCSIs sharing a 10G link as well.
I don't need the feature that badly though. It's just something that's interested me about Unraid (and most consumer NAS devices actually), especially since I'm running 4 2TB drives and 2 3TB drives as a single RAIDZ2 vdev in my home TrueNAS setup.
You can even use files[1]! Mostly available for testing though.
It would be nice with a bit more flexibility with regards to the underlying storage, hence this concept. I tested it in a VM and it works fine, just slow when the underlying disks gets trashed. Mostly noticeable on heavy writes and scrubs.
Should work fine on SSDs though, and might possibly even be a boon to very fast NVMe drives (increasing concurrency).
[1]: https://openzfs.github.io/openzfs-docs/man/7/zpoolconcepts.7...