OpenZFS bug reports for native encryption | Hacker News Reader

OpenZFS bug reports for native encryption

docs.google.com

38 pointstranspute29 comments

I was pretty active in ZOL when encryption was added. It was handshakes and applause all around... but shortly afterwards the developer started contributing less and less, while several bugs cropped up... some necessitating sending entire hard drive images over ssh to other developers.

Don't get me wrong, I like zfs and use it every day, but that code should have never been merged, and given more scrutiny considering it completely changes how your data is written to disk. Thankfully not many people use encryption.

As much as I lament the status of native encryption in OpenZFS, and think it would have been better to merge it with more testing, I think what I find to be really more problematic is the lack of enough fixing afterward.

Software is buggy, and it's going to have bugs you don't find no matter how much you tested it. The problem was that (AFAIR) one developer at the company that developed this contributed fixes, and then he left, and then almost nobody did.

I think people were more interested in giving Oracle the finger than anything, considering missing feature parity and compatibility were constantly waved in our faces. (Not that ZOL encryption is compatible at all with Oracle's code)

I often wonder if encryption would have been merged if Matt had usurped ZOL sooner.

"usurp" is an interesting word for "decide ZoL was the project of record and replace the illumos reference platform with it", and since Matt was the one who said no because there's no mitigation other than not using it when I wanted to put a warning in the docs about it, I wouldn't anticipate him doing anything positive.

Sorry, I might still be a bit salty about how that all went down years ago. Not to mention, some of my documentation changes got reverted in the process, as well as my name being stripped off other documentation entirely.

I don't think my creation and abandonment of this spreadsheet suggests I'm not frustrated either.

That's unfortunate, stripping anyone's name off anything is always unpleasant and shouldn't happen. I rather wish they hadn't just serialized a snapshot of the wiki out into GH pages and then said "just open PRs to edit this", but here we are, I guess.

What's the best way to catch up on those historical events?

Web search yielded https://papers.freebsd.org/2019/BSDCan/jude-The_Future_of_Op...

I found this, although some of the other information might no longer exist due to the changes at the time. https://zfsonlinux.topicbox.com/groups/zfs-discuss/T13eedc32...

I was really surprised when I found out how late-added and immature encryption in ZFS is. I had presumed it would have been table stakes for "enterprise" deployments. Were people just using it on top of block encryption like GELI or LUKS or was I just wrong to assume that most servers are encrypted?

I want to say yes, but putting zfs on luks could leave you with issues where you couldn't import the pool, I wouldn't recommend it personally.

I believe FreeBSD didn't have any qualms with putting zfs on geli, but I'm not the best person to ask about that. In what little I've used FreeBSD, I never bothered with geli.

I use it on top of geli and it works great. <3 on my daily driver which I sometimes hard reset too.

Not sure what the issues are with luks, as I'm mainly on FreeBSD.

A lot of it boils down to "zfs thinks the data is on disk, but is still being passed through the encryption layer, then something bad happened like a power outage or corruption at the encryption layer"

At least the ones I'm familiar with boil down to "there is a logic bug in the encryption layer where some settings get munged" or "there is a memory correctness bug in some edge cases with the weird double-existence that was plumbed in for having encrypted and unencrypted buffers violating expectations in places".

12 Feb 2024, https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt and https://github.com/openzfs/openzfs-docs/issues/494

> A new ticket has been opened for OpenZFS as well in proposing to add warnings against using ZFS native encryption and the send/receive support in production environments.

  "Among experienced zfs users and developers, it's conventional wisdom that zfs native encryption is not suitable for production usage, particularly when combined with snapshotting and zfs send/recv. There is a long standing data corruption issue with many firsthand user reports...Additionally, if you join #zfs or #zfsonlinux on freenode and mention that you're having an issue with zfs native encryption, you'll be met with advice from developers that zfs native encryption is simply not reliable."

From Phoronix comments, a suggestion on implementation:

> It should be something like that:

  ERROR: RAID 5/6 support has know problems and shouldn't be used on production systems.
  If you really want to use it please type "Enable dangerous RAID5/6 support" or add argument "--enable-dangerous-raid56-support" to this command

Why are they still using freenode? Everyone jumped ship like 2 or so years ago?

Hmm, good question.

I guess someone’s mentioned how great ZFS is and this page is here to prove the opposite. Every file system has bugs.

I mean, the spreadsheet was to illustrate how many bugs this feature had and how unfixed they are.

Source: it's my document, though I didn't submit it here, and haven't updated it in a while after I gave up trying to convince other people about how bad an idea using it is.

Thanks for creating the spreadsheet. Discovered while banging head against docs and builds.

The feature should be emblazoned as experimental and advertised as seeking developers/funding, to protect ZFS reputation.

Were the "stale" / timed-out bugs due to user or developer attrition?

https://github.com/openzfs/zfs/issues/10769

I wasn't a fan of that change, it serves no purpose other than to sweep issues under the carpet (which can be said of any project that uses one).

Which is funny because they have just as many open issues now as they did when it was implemented.

Had they had better user/developer relations and documentation, it's possible that a lot of open issues would have never been opened. (edit: or remained open otherwise)

That's a troubling policy signal, with impact beyond encryption.

To be fair, I believe the test suite has improved over the years, but adding new features requires meticulous caution to make sure you don't miss any edge cases, at least on the functional side. But then you occasionally get bugs like https://github.com/openzfs/zfs/issues/15453 which can't easily be caught by test suites.

ZFS is great and generally stable and mature, but it seems like the support for encryption, a buggy optional feature, was merged in without enough scrutiny and the problems are still unfixed.

All the advice you'll get from anybody experienced (which I am not very with ZFS, but have been setting up some NAS projects with it so am researching) is to not even touch encryption with a 10 foot pole. As long as you don't enable it (and I expect a lot of GUI driven NAS products probably don't even let you through the GUI or warn you if they do), you're good.

Which is a shame, since zero-trust zfs send is a pretty nice feature, if it could be relied upon

Yeah, true, I want to use it for backups but unfortunately I can't given the instability!

Most bugs seem to be around `zfs send`. I don't use this feature and my 2 encrypted zfs pools seem work without any issues...

I've said this before, but while a lot of the bugs are most easily triggered with `zfs send`, you can also trigger them other ways depending on which bug on which platform - illumos, for example, used to be able to reliably trigger a crash with the same trace as one of the "send/recv" bugs on ZoL/OpenZFS with just "zfs clone" (it might still, I haven't tried in a while).

Another one of them got what looks like a duplicate posted that triggered on pool creation in the test suite sometimes.

So I don't think it's accurate to say "it's safe if you don't use zfs send" - while some of the bugs might actually be exclusive to send/recv, it seems like a number of them are just most easily triggered that way.

It clearly works "fine" for some people - e.g. I believe the original authors of the feature still use it in production, last I knew, because their workflow doesn't hit any of these issues. But "using this feature adds a 1% chance of a lion eating your face" seems like something worth warning people about, if there was not a face-eating risk prior.

(Especially because a number of them seem to be races, so how much the problem comes up is really going to be hard to predict a priori...)

FWIW, I believe we've now fixed the clone-related crash issue, as of https://www.illumos.org/issues/14003

Reply on news.ycombinator.com