Hacker News Reader

Hacker News

Comment by nayuki | Hacker News Reader

Parent Full thread

nayuki

Btrfs has experienced some data loss bugs in recent memory. It looks like ZFS is the only remaining option.

https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Da...

https://www.spinics.net/lists/linux-btrfs/msg59190.html

https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg...

View on HN

Yes I have tried to use btrfs several times for work projects and personally because I was very excited about it but every time I have ran into severe bugs even though it was said to be "stable". I have given up for now, maybe I will check back in a couple more years.

The last time I was using btrfs was around 2014, and I was wondering why my hard drive was always showing 100% utilization even after I moved/deleted a ton-o-stuff.

Turns out at the time, re-balancing still had to be run manually. I'm not sure if that still holds true.

It is still a problem, I just had one of my personal servers hit issues where I couldn't even manually re-balance because the metadata was full. Had to apply weird workarounds to be able to write to that filesystem again...

I really like btrfs for my backup machine, but it sometimes manages to hang when it's cleaning up deleted snapshots. This is a problem that's much worse when on a hard drive or fragmented file system, but I've gotten it to happen even on a recently-balanced drive with a small amount of snapshots.

I recently had a server be crippled by running snapper on default settings for a few months. And after a couple days of balancing (which desperately needs a throttle control) it wasn't much better, so I gave up on having it run btrfs.

I think the record I managed was slightly over two minutes of btrfs blocking all disk I/O. Something is deeply wrong with how it organizes transactions.

Funny, my btrfs story is somewhat similar. I don't hit severe bugs, though, just severe performance issues. E.g. btrfs on / always sooner or later means for me that the entire IO to the disk that's on will be _thrashed_ during system updates or package installs / uninstalls.

Any idea if ZFS plays well with Ubuntu's full-disk encryption? I've used the FDE option at install for years and every time I upgrade (I wipe & reinstall every year or so) I try to understand how to first set up ZFS, then FDE, and then I realize it's far too complicated for me. Any good tutorials or setup guides that even a moron could understand?

I've got a pretty good setup now with a fairly complex fstab, multiple SSDs, backup drives, and everything fully encrypted and auto mounting at boot. I'd really love to move this to a file system more resistant to data corruption.

You can always use zvols and just use ext4+luks on that. There is also work on ZFS native encryption that looks pretty promising, not sure if it's ready yet.

I haven't tried to do it with Ubuntu specifically, but I do know that ZFS-on-linux works fine atop LUKS full-disk encryption. My laptop is running NixOS with such a setup, and I'm pretty sure I followed the Ubuntu ZFS documentation while figuring out how to do it.

(summary: I don't know how to make it super easy, but what you want should totally work)

I use it all over with LUKS on Ubuntu. It works fine, but there's one little hitch:

when calling anything that ultimately calls grub-probe (e.g. apt-get upgrade), you have to symlink the decrypted device mapper volume up a layer into /dev because grub-probe can't seem to find the ZFS vdev(s) otherwise. ie: "ln -s /dev/mapper/encrypted-zfs-vdev /dev".

This is in fact the case on every Linux distribution I've run ZFS over dm-crypt on.

EDIT: IOW its a grub bug not an Ubuntu bug.

The other problem I've found is that grub's update-grub scripts do not handle mirrored ZFS volumes well at all - they wind up just spraying doubled up invalid commands everywhere even up to yakkety so far.

I've had it on my backlog to at some point go in and sort out my initramfs's insanity when it comes to handling crypt'd disks in general - it should be a lot less brittle then it is.

We are rolling our own update grub as a bash script, which does some sanity checks:

- is the crypto module still present in GRUB config?

- is grub running in text only mode? Otherwise we cannot see the LUKS prompt to decrypt the devices.

- does initramfs know about all mirror devices (instead of an early return after the first)

Native ZFS encryption may become available soon. It is pending code review: https://github.com/zfsonlinux/zfs/pull/5769

There was a presentation of this by Tom Caputi at the most recent OpenZFS Developer Summit.

Slides: https://drive.google.com/file/d/0B5hUzsxe4cdmU3ZTRXNxa2JIaDQ... Video: https://youtu.be/frnLiXclAMo Conference: http://open-zfs.org/wiki/OpenZFS_Developer_Summit

We run with ZFS over LUKS encrypted volumes in production on AWS ephemeral disks and have done so in over two years on Ubuntu 14.04 and 16.04. The major issue for us has been getting the startup order right, as timing issues does occur once you have many instances. To solve this, we use upstart (14.04) and systemd (16.04) together with Puppet to control the ordering.

Performance wise it does fairly well, our benchmarks shows ~10-15% decrease on random 8kb IO (14.04).

We are definitely looking forward to ZFS native encryption!

What is the right order?

Since ZFS will run on blocklevel devices and you want to get the ZFS benefits of Snapshots/compression/(deduplication), in my opinion it makes sense to do the encryption at the blocklevel, i.e. LUKS has to provide decrypted block level devices before ZFS searches for its zpools. When ZFS native encryption is available on Linux this will be different, since you much finer control on what to encrypt and you can keep all ZFS features.

So:

First decrypt LUKS (we are doing this in GRUB) Then mount zpool(s)

Reply on news.ycombinator.com