Hacker News Reader

Comment by kreetx | Hacker News Reader

I've recently tried ZFS on an NVMe drive in a USB enclosure, the USB connection is fragile and when rsyncing data onto it, it dis- and reconnects, after which the zpool is in SUSPENDED state and only a reboot helps :(.

View on HN

This is a very Stack Overflow response but, yes, don’t use USB for any storage you expect to be reliable.

Lol, ZFS is supposed to be this super reliable filesystem that does your laundry. But if you hard drive disconnects for a moment tough luck.

Lungs are usually pretty reliable, but they're still working on a long-term fix for when air becomes unavailable.

ZFS is a very reliable file system. At some point ZFS will give up and say "I cannot reliably save bits."

It is your hardware where the unreliability lies and ZFS detects that. Would you rather prefer silent corruption?

To be clear, I don't mind it getting suspended when there is a hardware failure. What I don't understand is why aren't there sane way (i.e some command sequence) to re-attach it.

Otherwise the feature set seems great: both a volume manager and a file system, seamless compression, deduplication, COW and snapshots - what's not to love :)

> why aren't there sane way (i.e some command sequence) to re-attach it.

But there is. It's `zpool clear <poolname>`.

That said, you probably need to have created or imported the pool using stable device names (e.g. `zpool import -d /dev/disk/by-id` or `by-part-uuid`, for example). Otherwise, when reattaching the USB device it might get assigned a different device name (e.g. `/dev/sdb` instead of `/dev/sda`) and ZFS might think the device is still unavailable.

Yeah, super reliable in the enterprise. It’s not designed around hanging a single ssd of your rpi with a five dollar usb to m.2 adapter. It’s designed around having a disk shelf or at the very least a mirror.

> it's super reliable on high quality reliable hardware.

Okay

There's fuck all that ZFS can do if hardware lies about what has or hasn't been written to disk yet. There have to be points where ZFS blocks until something critical has been absolutely, totally, 100% written to disk before continuing. Unfortunately, USB devices lie all the time.

But the issue isn't really at storage level, it's about the filesystem driver not dealing with it. Most modern journaling file systems don't really care if my drive intermittently disconnects, all that will happen is some failed file operations, and at worst the necessity to run a quick filesystem check. With ZFS's emphasis on storage being fallible (checksums to detect silent data corruption etc) it's not weird to expect it to also handle connection issues at least as well as other file systems.

I have a machine with dual-boot. Kali Linux (think I used ext4 for it), and Windows 11 (NTFS). Due to space constrains I mount /var/cache/apt on the NTFS partition. Whenever I forgot to cleanly close Windows 11 though, I cannot mount that partition because of potential data loss and I first need to go to Windows 11. This used to never happen. But you know what? I'm OK with it. Heck, I should probably run both OSes on ZFS. Although the Windows 11 partition is just to play around with Windows 11; nothing serious on it.

There's fuck all that ZFS can do if the drive lies about what has or hasn't been written yet, journal or no journal. USB devices are known to do this all the time.

If the drive lies and says that part of the journal has been written when it hasn't yet, and ZFS goes ahead and writes the next part of the journal, then when you unplug the drive and the first part of the journal goes away (which the 2nd depended on) you're hosed. There have to be places where ZFS blocks until something critical has definitely, absolutely, been written to disk.

At that point the only thing ZFS can do is try to unwind back to whatever it thinks is a consistent state, but this isn't 100% guaranteed. (it depends on what old data is still hanging around)

I've been using ZFS for over a decade and love it, but it was created as an "enterprise server filesystem" and in certain areas like this it really shows.

That said, I have you tried something like this[1]? Also what device name did you use when creating the pool? Using one of the /dev/disk/by-'s that doesn't change when you reconnect would make it a lot smoother I imagine.

[1]: https://github.com/openzfsonosx/zfs/issues/104#issuecomment-...

Reply on news.ycombinator.com