Ask HN: ZFS native encryption vs LUKS | Hacker News Reader

Ask HN: ZFS native encryption vs LUKS

66 pointsaborsy61 comments

I use full disk encryption with LUKS to encrypt data at rest in Linux systems. Lately, ZFS has offered native encryption in Linux. I am thinking if new Linux systems should be encrypted with file system encryption (such as ZFS native encryption or ext4 with fscrypt) instead of full disk encryption (mostly LUKS).

I know some of the differences. For example, ZFS leaks some metadata, mostly dataset and snapshots names (which are useless in my case). LUKS encrypts everything, has a better KDF and multiple key slots, and is mature at this point. ZFS uses AES-GCM which is more complex but stronger than AES XTS.

Are there other reasons to use one versus the other? What do you choose?

Openzfs encryption is still rather new and I worry there might be bugs or pitfalls breaking confidentiality, or cause pool corruption.

Any feedback on the implementation quality of ZFS native encryption?

View on HN

Caveat emptor: I've run ZFS on a two-digit number of machines for >10 years. But no corporate deployment.

To me, a big benefit of ZFS' rampant layering violation is that it shrinks the surface of weird interactions and edge cases to learn about. You get to know the FS, play with it, and then off you go. Most other solutions require a lot of thought and planning. Perhaps there is a better way to tune the storage setup to exactly what you want, but with ZFS you aren't seduced into getting creative with the architecture.

So I'm in favor of native ZFS encryption due to its usability.

That said, it is not as mature as I'd like, and there is at least one very big issue to know about: I recently sent an encrypted FS with zfs send/receive, and the received copy ended up being unreadable. I'll see if I can find the bug again. So that means you should test your setup precisely.

[edit] Here is the issue: https://github.com/openzfs/zfs/issues/12594

The technical detail from that thread: "This happens because when sending raw encrypted datasets the userspace accounting is present when it's not expected to be. This leads to the subsequent mount failure due a checksum error when verifying the local mac. I tried unsuccessfully to tackle this in #11300. See also: #10523, #11221, #11294.

Edit: If you have critical data lost due to this case I could help you recover them."

And there's this comment [https://github.com/openzfs/zfs/issues/12594#issuecomment-929...] which has a lot more pointers to recent problems.

yup see rincebrain's first reply where nearly every word in the sentence is a url to a different encryption issue

> [I recommend](https://github.com/openzfs/zfs/issues/11679) [not using](https://github.com/openzfs/zfs/issues/12014) [native encryption](https://www.illumos.org/issues/14003) [until it gets](https://github.com/openzfs/zfs/issues/12270) [a fair bit](https://github.com/openzfs/zfs/issues/12439) [more polish](https://github.com/openzfs/zfs/issues/12418) [in the future](https://github.com/openzfs/zfs/issues/11983) ([I'm only so hopeful](https://openzfs.topicbox.com/groups/developer/T86620b1082ed9...)).

Unfortunately this knowledge isn't wide-spread, and as far as I see the zfs devs seem to NOT recommend using encryption right now without accepting the risks of dataloss

Most of the devs seem to be of the opinion "we don't use it so we have no opinion", or there wouldn't be horrifying bugs in it over 3 years old.

That said, very few if any of them have involved actual irrecoverable data loss versus just requiring lots of handholding to get your data back.

edit: I suppose at the point where I'm being quoted on HN about this, I might as well link my list of encryption bugs.

https://docs.google.com/spreadsheets/d/1OfRSXibZ2nIE9DGK6sww...

I still need to add a couple of new bugs, but almost all of them are dupes or related to one of two flaws - a narrow race in the ARC (...probably), where you can sometimes NULL dereference for fun and sadness, and zfs change-key and send -i resulting in the receiver incorrectly resetting the metadata for which key an encryptionroot is encrypted by.

Right, that's the comment I linked to.

> So I'm in favor of native ZFS encryption due to its usability.

Another reason to favour it is because if you do ZFS Send/Recieve to do incremental backups (like with rsync.net), you can backup your encrypted datasets to an off-site location without shipping the encryption keys.

That means you don't have to be afraid of your backups somehow resulting in breaking your confidentiality.

So you can still retain your incremental ZFS based backup-flow, and you don't have to change any tooling to use it. With LUKS you will either have to store your backups unencrypted, or you lose the ability to do incremental ZFS-based backups.

The LUKS headers doesn't contain the encryption keys, they're encrypted from the user's passphrase through a KDF. Additionally, you can make the LUKS header "detached" (external) see https://wiki.archlinux.org/title/Dm-crypt/Specialties#Encryp...

But my point was that if you want to backup such a volume (the encrypted LUKS volume), you will have to backup the whole volume, and can't do incremental backup.

If you want to do incremental ZFS send/recieve backups, you have to do it on the unencrypted ZFS volume, and thus your backup wont be encrypted.

That is you can't have both encrypted and incremental backups at the same time.

With native ZFS encryption you do get that.

I filed #11294, it's a non-issue for me now. Seemed like an API change stability issue, which happens. I think what a crypto-user might care about more in ZFS vs LUKS is precisely what gets encrypted: LUKS can be the entire disk, whereas ZFS is going to need some metadata to know what is there, which keys unlock it, etc on at least a per-dataset basis. It's not a tradeoff that affects my particular threat model, but it's worth understanding as a crypto user.

If I read this right these issues may actually be resolved as of 2.1.3 ? At least there's nothing else open on the tracker.

Yes all the issues should be resolved by now.

But given that the bugs exist, anyone using encryption should test and verify all the functionality before using it in production.

The worst thing about this bug is that it doesn't manifest on the local machine but in a situation where things seem ok.

Nope, not even close.

Care to elaborate?

Sure, have an explanation and a link to my (currently missing one or two newer bugs) spreadsheet.

https://news.ycombinator.com/item?id=32342900

As a seasoned storage administrator and ZFS veteran, reading over these comments is mildly infuriating. The amount of misconceptions and flagrant untruths is appalling. Using a beta version of FUSE ZoL one weekend 10 years ago does not make you an expert on storage and filesytems. People have no appreciation for what a wonderful gift ZFS is to the storage community, how well architected it is and how hard behlendorf and the rest of the team have worked.

That is HN. Anecdotes taken as gospel. Comments written as absolutes. Simple tech blog posts that turn into 500 off-topic-comment threads. It's wonderfully useless at times.

I haven't run ZFS + LUKS but I'm running ZFS Native Encryption.

There is an outstanding issue [0] that I've encountered on two separate machines running in this setup. The issue occurs when using ZFS Native Encryption + (NVMe?) SSDs. The issue appears to be snapshots getting corrupted (or maybe failing to create?) occasionally. It happens roughly 1/1000 snapshots in my case. I take a lot of snapshots so this occurs weekly.

I haven't lost any data yet but this issue is annoying because I get spooky alerts from ZFS warning me about "potential data corruption". To clear them I have to manually intervene by deleting the corrupt snapshots, restart the machine, then scrub.

What's most annoying is that this breaks ZFS send/recv until I intervene. send/recv was the whole reason I went with native encryption instead of LUKS in the first place.

Again, no data corruption so far (if you don't count the snapshots that get lost, which I don't, because I have so many of them). Just very annoying and tedious.

[0]: https://github.com/openzfs/zfs/issues/12014

As others, I'm running ZFS with native encryption on my daily driver Linux box (Arch).

As others have said, I like the benefits, which are easy send/receive, etc (but mind the bugs some other commenter mentioned).

I will also give a few drawbacks / points in favor of using LUKS.

In my case, Arch usually has up-to-date kernels. These are not always compatible with the latest stable OpenZFS release. Issues go from benign: ultra-high CPU usage so my 980 PRO ssd is slower than spinning rust to the module not compiling at all.

Another drawback is that booting with systemd in initramfs doesn't support zfs native encryption. There's no support for using a TPM [0]. So you have to type in your password every time you boot up the computer. If you want hibernation to work, you need a separate swap, encrypted separately, so that's two passwords to type in.

Systemd recently got support for TPM so can automatically decrypt devices. I have this set up on a separate machine (with ext4) and it's nice.

---

[0] I know not everyone likes the idea of automatically decrypting the drive, but that's another debate. If your threat model allows for this, it can be done with LUKS but not with ZFS.

There recently was an issue where the deprecation of a kernel header (with version 5) lead to ZFS losing access to hardware accelerated (vectorized) checksums [1].

As far as I know, the issue is now resolved, but I haven't followed this closely.

[1] https://github.com/openzfs/zfs/issues/8259

Yes it's been resolved. It was around the time kernel 5.16 came out, but it took a while for the fix to land in stable.

I actually followed this closely, since that's when I had got a new Laptop with a new SSD and when I tried to benchmark it, I was going crazy trying to find the culprit. Because an older PC with a slower ssd (and older kernel at the time of benchmarking) was something like 25 times faster.

I've been using ZFS native encryption since it was beta (~2017 I think), and it's nothing short of awesome. It's amazingly convenient and easy to use, and it integrates wonderfully with the rest of the filesystem - in comparison, Btrfs on top of LUKS feels very clunky and cumbersome.

I don't think I'd use it on cold storage though, I like using it on systems I daily drive, such as workstations and laptops, where I feel sacrificing convenience is not worth it, and where I don't want to lose the option to not be able to select what's encrypted and what's not (for instance, I don't want or need for all of my VMs ZVOLs to be encrypted). With ZFS you can pick what gets encrypted, and with what key, without having to fumble around with loopback devices and other strange machinery.

In my experience it is very reliable, I didn't really feel any performance degradation after moving my system on top of encrypted datasets on any of my systems. I also happen to have to use every once in a while a system installed on an external USB-C SSD (with Arch+ZFS), and it works very well without any performance hiccup even with encryption on.

The only time I had to fumble with the dataset was when encryption still wasn't stable and I had to rsync everything to newer format datasets - something around 2018 I guess? Still, it was a pretty straightforward experience. I never had any other issue with encryption after it got moved to stable (3 years ago I think?), daily driving several (personal and work) systems.

On my cold storage, such as drives I backup my stuff onto, I still use LUKS + XFS/ext4/ZFS because it's tested and solid though - I don't really feel that moving disks I often keep offline to ZFS native encryption would help at all. I honestly worry more about them being misplaced or lost, than convenience.

For cold storage, the appeal of ZFS is the usual features of a backup software, primarily, checksums and snapshots (and to some extent ZFS recv, compression and integration with ZFS live systems).

In particular, with ext4+LUKS, there is no way to know if there are silent bit errors.

Yeah, that's why I like to use LUKS + ZFS nowadays for cold storage, at least you get a bit of support against bitrot and it's not that unwieldy.

With XFS and ZFS available on Linux, I haven't used ext4 for maybe a decade.

I wonder what is a compelling case to create a new ext4 volume as of now.

Using Rancher's Longhorn, because XFS causes random volume corruption.

[0] https://github.com/longhorn/longhorn/issues/3597

According to Phoronix [0], at least with kernel 5.14, ext4 seems somewhat faster.

[0] https://www.phoronix.com/news/Linux-5.14-File-Systems

I've been using it on FreeBSD and NixOS (both ZoL based) since the feature's release - no issues at all.

I can't speak to the metadata leakage, but from my grumpy perspective the fewer layers in my IO path the better. Disks should be boring (ish).

ZFS is still a little on the immature side; there are sufficient recently-fixed bugs involving data-loss that it is hard to be confident that there aren't others yet to be discovered.

That being said, I do use it on a laptop where losing the data wouldn't be a big deal, but having it easily compromised would be. It was just so much easier to transition to encrypted datasets; the whole thing was done with maybe 10 minutes of downtime.

ZFS dataloss bugs? I think you’re confusing it with btrfs.

LOL yes, ZFS is better than btrfs by far in that regard, but there have been issues specifically with zfs send/recv of encrypted datasets. I believe all known issues are resolved now.

LUKS has its own use case. You can create portable containers with it, which is nice because not everyone wants their OS drive encrypted, sometimes it’s better to be able to move a container from usb stick to drive to whatever.

Shameless plug but i wrote a nice cli script to manage luks containers.

https://codeberg.org/emanresu/Luksman

> AES-GCM which is more complex but stronger than AES XTS

I'm not sure if complex/stronger is the right phrase to use here. If someone wants to learn more, this is a nice summary of differences https://crypto.stackexchange.com/questions/84188/is-xts-alwa...

Also

https://sockpuppet.org/blog/2014/04/30/you-dont-want-xts/

There are too many open issues with ZFS (on Linux) and encryption that I could recommend it with confidence.

ZFS's encryption is also certainly not as thorough as LUKS is. Depends on your threat model if that's a problem for you.

Zfs has a per dataset data encryption key. When you destroy a filesystem or volume, the key is also destroyed. This makes recovery of data in deleted datasets exceedingly hard even with raw disk access.

LUKS and other FDE schemes tend not to have the ability to have fine grained encryption keys. If you have raw access to the LUKS device on which your filesystem resides, you can recover plaintext of blocks that have not been overwritten or erased via TRIM or similar.

zfs on a self encrypting drive seems like a good combination. Even if the Opal implementation is garbage (see CVEs from a couple years back) the exposure is minimal because zfs is encrypting data. My measurements of performance impact IO with and without encryption suggest there is no performance nor increased cpu consumption with self-encrypting drives.

Can you elaborate on this a bit further?

Suppose that I have encrypted ZFS datasets. Naturally, sometimes the process of working with datasets is interrupted (due to, eg, power loss and so on). Is it known how much plaintext could potentially leak to disk outside encrypted ZFS datasets, possibly accessible by forensic analysis?

That would violate the confidentiality guarantee in the event of an accident.

Related to this, if I recall correctly, if the encryption key is changed, ZFS may not destroy the old wrapped key and may keep it stored on disk outside the dataset. This is apparently due to the copy-on-write system.

> Is it known how much plaintext could potentially leak to disk outside encrypted ZFS datasets, possibly accessible by forensic analysis?

When a write to an encrypted dataset happens, no plaintext data is written to disk. Your data is every bit as secure from disclosure as if the write completed normally.

Think of ZFS encryption like you would TLS encryption. They both hide your data, but leak metadata. With ZFS encryption, someone with access to the disk can see what you name your datasets (don't name your dataset "people-i-killed" or name your snapshot @before-starting-the-fire), the order that each block was written (via birth txg in block pointer), how much data was written to which datasets, when the last write happened (via ub_timestamp in uberblock copies in vdev labels), etc. Likewise, with TLS someone with access to the network can tell who the endpoints of the connection are, when each party sent data, and how much data was sent.

If you have full disk encryption under encrypted ZFS, you get the goodness of encrypted ZFS plus after your drive has been powered off, no one can determine anything about the content of the drive (except maybe that which is leaked via SMART and similar data). This is kinda like having a VPN and using TLS. Someone with access to the network under the VPN can still see that there's activity but has a much harder time figuring out information about the data and metadata with the VPN.

Take care that if dedup is enabled, information is leaked.

More information at: https://blog.heckel.io/2017/01/08/zfs-encryption-openzfs-zfs...

ZFS encryption on a disk can perform better than LUKS since LUKS has integrity enabled by default and TRIM/discard can't be used. If you encrypt the full nvme this can lead to premature drive failures and loss in performance.

With ZFS there are other things to consider as well, but I would recommend looking at an article like here https://arstechnica.com/gadgets/2021/06/a-quick-start-guide-... that goes into much detail.

Like always it mostly depends on your use case/scenarios.

I'm using ZFS encryption for a couple years on my NixOS laptop.

It works well but be aware that SWAP specifically on encrypted ZFS will lead to deadlock: https://github.com/openzfs/zfs/issues/7734

Thus I have planned to move the swap back to LUKS. If I cannot unlock everything by entering only 1 password entering due to that, I might move everything back to LUKS for that reason.

sometimes it's an alternative to just disable swap.

That's what I'm doing

Nixos with natively encrypted zfs mirror.

Next step is impermanence

I would not trust in the quality of any ZFS features in any implementation that were not developed by Sun or at least thought out by Sun before it was acquired.

Not sure where you are getting that. Lots of former Sun people now work on the OpenSource Solaris. Matt Ahrens was one was the Nr.2 guy who did most of the implementation and he is still heavily involved in OpenZFS.

OpenZFS has a very healthy and active community working on it.

And the code has been much improved compared to originally open sourced version by Sun. Technology has moved on since then and so has ZFS.

Care to elaborate?

Sun had certain values as a company it enforced through its corporate culture which made ZFS excellent.

My perception is that the ZoL based fork had a lot more instability and bugs compared to the parallel development of ZFS by in the Open Source Solaris world. That OpenZFS is based on the more unreliable code base should be evidence that the values which made ZFS excellent under Suns steermanship are insufficiently maintained and that features integrated into ZFS after the fork who did come from the ZoL side should not be used if one wants the reliability of ZFS.

The fact that not all metadata is encrypted shows they are doing things wrong from a principled standpoint. Sun didn't halfbake important stuff like that. The ZoL community will in the long run make the reliability of OpenZFS converge against the reliability of BTRFS because that is the level of reliability open source allows for. Initially higher stability be dammed.

Many of the open source zfs developers are the same ones that developed zfs at Sun and then Oracle. I worked on the Solaris team from 2010 until 2017 and have contributed to open source zfs implantations in illumos and openzfs. Your criticisms seem ill-informed based on my actual experience in my contributions to these multiple zfs code bases. I actually found a fair number of improvements in the testing in ZoL.

With Solaris zfs, you saw features after they were exercised by up to a few hundred engineers for months to years. Now you are getting bits fresh from the factory. Like when software is developed behind closed doors, bugs happen and they are progressively squashed leading to a more stable product.

> Sun didn't halfbake important stuff like that

That is disprove by the fact that original Sun encryption did the same thing and the one currently in Oracle ZFS does so too.

> The fact that not all metadata is encrypted

This was one of my concerns when I first encountered this trade off in Solaris zfs. That is, you claim that the Solaris zfs encryption is better in this regard but it’s not.

What metadata is not encrypted? I thought it was only dataset names and their properties, which kind of makes sense?

Everything outside of encrypted datasets’ data is not encrypted. This includes the things you mention as well as vdev labels, block pointers, etc. The clear text bits can tell you a fair amount about the relative age of written data and how much data is present. It will also identify which machine last imported the pool.

ZFS on Linux is very buggy.

And when something breaks (maybe because one of those numerous bugs) there are no proper tools for repair. Your ZRAID and ZLOG probably won't matter. The to-go tool for removing corrupt superblocks is a 10 year old hacky python script, it's ridiculous.

Not been my experience at all.

I have seen folks get in over their heads though. Like -- "OMG I've somehow broken boot for ZFS on root and I don't know how to fix!!" or "I'm using a custom kernel and OMG things don't work!!"

Canonical has even been complicit. When you download an HWE kernel, you don't also download a newer/matching versions of the tools. This should pretty clearly be considered a bug[0], and fixed but Canonical perhaps doesn't have the bandwidth?

I just can't lay any of these problems at the feet of ZFS though. Right now I trust ZFS a hell of a lot more than I trust Linux to Do The Right Thing. When ZFS seems broken (though I have little experience with native encryption so this may not apply to it), I am far more likely to think the issue is Linux, my distribution, or me, than any problem with ZFS.

I'm a pretty serious ZFS lover though.[1]

[0]: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/193... [1]: https://github.com/kimono-koans/httm

The last time I needed to use fsck to repair a filesystem was less than six months ago. After running it I found that my entire directory hierarchy was garbage and lost+found had at least hundreds of files named by their inode numbers. It would have taken me forever to piece this back together if it was a filesystem that I cared about. This is consistent with my experience with fsck in decades past.

In 15 years of using zfs, I’ve never lost a pool when it was in a mirror or raidz layout.

Not my experience. I've been running natively-encrypted ZFS on Linux for over a year now with zero bugs or errors. It may help that my distro NixOS has excellent support for it.

Only a year?

I've used ext4 for more than seven years and ZFS for a few months. ZFS shat itself irrecoverably - ZRAID, ZLOG (PLP), ECC RAM and UPS be damned. People's issues sit open on GitHub for years.

ZFS on Linux is empirically and anecdotally not trustworthy.

To the best of my recollection…

If you stream your ZFS data from one host to another then the encrypted filesystem remains encrypted at the destination.

If your encryption is done outside of ZFS then the data on the destination is in plaintext.

This might seem obvious to those experienced but to a newcomer it caught me off guard and the interaction between recursive send/receive and encryption was more complicated than I had expected.

I should add that I haven’t looked into this for a while and could be completely wrong but it’s worth researching, at least, because my spideysense is definitely tingling to the tune of something about this was odd and worth knowing about.

This post might give a better hint to the actual details: https://www.truenas.com/community/threads/cant-zfs-send-betw...

Sorry to not be more authoritative. Bottom line: if you are going to use send/receive, spend some time to build a run book for your use case sooner rather than later.

In addition to your points, if you intend to back up your ZFS datasets, it's easier with ZFS encryption to use ZFS commandline tools (zfs send) to ship the whole encrypted dataset over the wire. With LUKS you're shipping the unencrypted dataset instead.

Arch has a good wiki page comparing LUKS vs ZFS native vs a few others: https://wiki.archlinux.org/title/Data-at-rest_encryption

And more discussion of differences over at /r/zfs: https://www.reddit.com/r/zfs/search?q=zfs+luks&restrict_sr=o...

It's great to send encrypted raw FS, but beware of the bugs that I mentioned above. Depending on your distro's packages, you may run an affected version for some time.

If the change isn't too large/frequent you could send it over an encrypted transport (ssh/vpn/whatever).

I use LUKS cos I need multi keys and native ZFS's performance is somehow miserable on my small Gemini Lake box.

I for one have always used LUKS and then created the ZFS volume onto the decrypted device in /dev/mapper . This has worked flawlessly for all the time, about 10 years I have been doing this.

I guess it is about to proceed onto zfs native encryption.

Reply on news.ycombinator.com