Native Encryption for ZFS on Linux | Hacker News Reader

Native Encryption for ZFS on Linux

github.com

169 pointsfunkaster92 comments

OK, I'm curious: why are people shoehorning the filesystem to fit the OS rather than carefully picking the OS that makes the most sense for the application it's being used for?

We originally ran OpenSolaris _just for ZFS_. Then we switched to FreeBSD when OpenSolaris/OpenIndiana dead-ended. We had a production server running PHP and we were sick of PHP5, so we switched to Linux because HHVM on FreeBSD was a joke. PHP 7 came out and obviated HHVM (from a performance perspective), and so our latest PHP deployments are back on FreeBSD.

If your data is valuable to you (and presumably it is and that's why you're looking at ZFS), why are you not on FreeBSD in the first place?

Same story with ASP.NET - tried deploying on Mono way back before .NET Core was even an idea and realized the futility of it and switched a server farm to Windows _because it just wasn't worth it._ PHP on Windows? Same story - not production ready, move to Linux.

(This isn't to disparage ZfsOnLinux, which I think is a great effort and laudable if only for home user purposes. Instead, this is a question for sysadmins on HN who are using ZoL.. .why?)

I'm curious in what way ZoL is "shoehorned to fit the OS"? Sure, a port adds potential for error, but is the technical difference between the interfaces so large?

The ZoL code is not the shoehorning, it's the fact that you're doing so via a community effort to go around the official stance of the kernel developers who've explicitly opted to not support this FS at this time; meaning you're missing out on all the normal development and QA processes and the benefits thereof. There's real value in having the the actual OS community designing the filesystem as well, their experience cannot be discounted.

This isn't just applicable to Linux; we have a strict policy on using only in-kernel filesystems on FreeBSD as well after some poor experiences (_in production_) with aufs-esque ports to FreeBSD. They all look shiny and nice on the outside, and even stress test OK, but when you release them to the masses that's when the shit hits the fan and you realize they're just not designed to the same specifications as the rest of the OS components.

The kernel developers would probably accept zfs if they could. The main blocker is a license incompatibility issue, which is worked around by distributing zfs in source code form.

It's not about whether they could if they would. It's that they didn't, and therefore the things they'd have done if they would have didn't come to pass.

I'm saying it didn't matter why. Just that at the end of the day, the kernel team isn't opting their weight behind this, and that should factor in to the decision.

The Linux Kernel project doesn't own the Linux Kernel developers; those that wish to contribute to ZFS on Linux will do so and those that do not wish to will not. Things aren't quite so black and white in the open source world. Just because something isn't in Linus's tree doesn't mean it doesn't get testing and communication doesn't happen. In fact, because it's designed as a module and we have the DKMS infrastructure, being out of mainline really isn't as big an issue as it might have been in the past for distribution. There are disadvantages surely, but keeping up, taking advantage of interested talent and being in the know aren't really huge examples of them -- especially for an established project like ZFS on Linux.

EDIT: I would also like to explicitly point out the false equivocation to which I am responding. The idea that the Linux Kernel project ignoring the code because it has an incompatible license is, in any way, equivalent to the Linux Kernel Project maintainers choosing to not "throw their weight" behind a piece of code by mainlining it -- essentially that they are rejecting it due to lack of technical merit -- is specious at best and deliberately instils fear, uncertainty and doubt at worst.

Do you know what development and QA processes are in place for in kernel filesystems and how that compares to ZoL's development and QA process?

There are no QA processes per se for all file systems. The kernel is pretty much a bundling of many projects and the big file systems are their own projects, managed largely indepently from the Linux masters.

They are probably referring to the fact that in the Linux world, in-tree modules get a lot of maintenance for free when the kernel refactors the pieces beneath.

If you have ever written a kernel module out of tree then tried to upgrade the kernel version you know this pain.

Zfs isn't the original home of zfs so I'm unclear how using the freebsd port is more recommended at this point than the linux port.

(I think you meant "FreeBSD isn't the ...")?

ZFS was shipped on FreeBSD in 7.0 -- about a decade ago. That's 10 years of development, testing, QA, and use in production.

Canonical just recently began providing kernel modules for ZFS. It appears as if Red Hat may never include support for ZFS and -- unless some changes happen WRT licensing -- it will almost certainly never be part of the Linux kernel.

I've had "ZFS on root" setups on my laptops and workstations on Ubuntu (previously) and Arch Linux (currently) and have several servers using ZFS on FreeBSD. Initial installation and setup of ZFS is a major pain in the ass on Linux, compared to FreeBSD -- especially when dealing with anything more complicated than a single ZFS pool on a single disk.

I don't care how you capitalize it.

On funtoo/gentoo/arch basically any distro where the installation is manual zfs is exceptionally simple to set up. This stage took me about 30 seconds to figure out having never used zfs.

On any rolling release you could easily be using the same setup for the next decade or more simply duplicating an existing setup to a new machine periodically.

A small amount of work up front seems like a good trade for never having to do so again.

I tend to make use of lots of datasets on my machines, including in and under /var (e.g., /var, /var/backups, /var/cache, /var/log, /var/mail, /var/spool, /var/tmp, and probably a few others I'm forgetting) and some hosts have multiple pools as well.

I never had any issues with this on FreeBSD. Linux, however, was another story. Installation would go smoothly and without incident but I'd run into issues immediately upon first boot. I spent many hours trying to figure out what/where the problem was and it turned out to be race conditions with systemd and filesystems/partitions being mounted. (At the time, I had a few years experience with ZFS and had installed who-knows-how-many servers using ZFS exclusively, yet I had never had any issues with this. Of course, none of those servers were infected by systemd either.)

Now that I'm aware of the issues I can work around them. If you just want a single pool on a single disk and a minimal number of datasets, you'll probably be fine -- you're right, that's a cinch to set up. Like I said, however, don't be surprised if you have problems when trying to do anything more complicated than that.

So once again systemd is a pos shocking. I think the solution is to mount via /etc/fstab or don't use systemd.

Most small shops don't have the expertise -- or appetite to develop it -- to do an OS and kernel comparison in depth. Larger shops like Netflix do (and Facebook, etc). So if you want a detailed answer by a systems/kernel engineering team who have spent weeks studying it, you need to ask a larger shop.

I'm one of those people. I'm on the Perf and OS team at Netflix. We're very familiar with FreeBSD -- we use it on our CDN. We're also very familiar with Linux -- we use it on our cloud. We're very familiar with a lot of other OSes too (Windows, AIX, Solaris, HP-UX, etc).

So why don't we use BSD everywhere, or Linux everywhere? There's no single reason -- it's many reasons -- and it's serious engineering time to document them all and do the topic justice. I'd like to, but I don't have that time right now.

But I wanted to point this out because I do see it commonly asked, and it sounds like a question that can be answered simply, but isn't. You're asking for engineering work, even to just summarize all the factors. I've written up an explanation of why not Solaris before on HN, since that's easy. But Linux vs FreeBSD is much more work, since they are both compelling options.

No, you misunderstood me. I am not arguing for FreeBSD everywhere or Linux everywhere. To the contrary.

I'm saying use whatever filesystem Linux provides when you need to use Linux. And use whatever operating system provides native support for ZFS when you need to use ZFS.

We're using ZFS on FreeBSD to do our data storage, ASP.NET on Windows to run our business logic, and PHP under FreeBSD where PHP is required (previously HHVM on Linux because HHVM was the requirement). I'm saying pick the right OS for the right job.

I'm saying figure out your priority with stability in mind.

ZFS is a must when managing big data. It's nice to have otherwise. Decide which factor is the "must" in your case, and if it's ZFS, don't use Linux.

Problem is, Linux does not have any decent general purpose filesystem.

That is why people risk their data using an untested hack.

What is not decent about ext4 as general purpose FS?

ext4 is fine for most people but it's not comparable to ZFS. Even ZFS aside, I've been unconvinced by ext4 these days and have starting using XFS on all new Linux server builds.

So it IS a decent general purpose FS. ;)

I think there's two incompatible arguements going on here:

1. Is ext4 a decent general purpose file system?

2. Is ext4 a decent alternative or comparable to ZFS?

My answers to those questions would be:

1. It's an "acceptable" general purpose file system but it's really starting to fall behind the competition. When I see comments like how people don't need features like snapshots et al I'm reminded about just how popular Windows Shadow Copy is. Ok that is a service rather than a file system feature but it shows that there is a demand for desktops to have this feature. Checksuming is another example - it's practically free service these days as CPUs have hardware extensions for popular checksum hashes. So the argument is akin to saying "why do we need file system journaling"; sure you don't need them but you're massively grateful when you do have them and your consumer hardware throws it's inevitable hissyfit.

In my opinion Linux really does need to up its game. Ext4 feels almost stuck in time when compares to the likes of XFS which is years older than ext and yet they're working on snapshotting. Then you have really forward thinking file systems like what DragonflyBSD have been working on. They understand the need for better resilience of our data on desktops due to ever expanding storage capacities and HAMMERFS looks extremely exciting as a result.

So to get back to your question. Yes ext4 is acceptable for most people, but is acceptable really good enough? Maybe we should invest more time and energy into XFS?

2. Is ext4 a decent alternative or comparable to ZFS? Simply put, no.

* stores data - yes

* handles crashes - yes

* detects corruption - no

I'd want all three for a general purpose FS.

Snapshotting is nice but it's complex and expensive, and you can do fine without it. But telling you if your data is safe should be an expected feature for a general purpose FS.

I take you intended "modern" (eg. data checksumming, and maybe volume management)? It's hard not to qualify ext4 as "decent".

"Modern" would have been suitable a decade ago. The current state of filesystems on Linux is inexcusable.

If you’re gonna say something is out of date, at least say what could be better.

That is fairly obvious isn't it? Copy on write and checksumming are two basics. And on top of CoW you can get snapshots (which in turn helps with backups, etc. etc.).

Not everyone needs copy on write and checksumming, it doesn't make sense to waste CPU cycles on checksums when the use case doesn't require it.

ext4 is a perfectly fine filesystem for most desktop users and a wide range of servers simply by not being ZFS.

If you need CoW and checksumming, there is btrfs too, which is natively supported on Linux and IMO fine if you're only on a single disk or use md or LVM for RAID.

It's quite amusing tbh when people think that a filesystem needs to do everything.

Checksumming is free on todays CPUs, you could use your argument agains filesystem journaling as well. But for some reason you are not proposing using ext2? Why not? Why waste clock cycles on a journal when you might not need it? This truly is a non-issue but in very exceptional circumstances, and in such you don't use a general purpose filesystem anyway.

Checksumming and snapshots etc. give substantial game-changing improvements to the whole system. More than a journaling filesystem ever did. They also provide means as to vastly simplify other tasks. Roll-back last system-wide update? Done. Backup consistency? Free and vastly simplified.

I disagree, these are properties that are expected of any decent general purpose filesystem. If you fall into a niche that doesn't, fine, use one of the many alternatives. But this should be the default for all user-facing machines as well as servers.

Checksumming is rather not free on today's CPUs unless you use something like xxhash which operates close to RAM speed.

Journaling is rather useful because especially consumer desktop computers tend to be mistreated and crash a lot. The physical journal in ext4 prevents the worst when stuff gets hairy. Additionally, it does not effectively delete data like ZFS (or rather, make it very very hard to access data) just because a bit was corrupted.

I think a user will value their family pictures more than some random text file suddenly having another character somewhere, they will loose data with a checksumming FS on a normal machine. There is no second drive to pull a good copy of the data from. There is one. And you just killed the last picture of grandma. Congrats.

>Why waste clock cycles on a journal when you might not need it?

Journaling wastes mostly bandwidth and IO, not much CPU and clock cycles.

>Roll-back last system-wide update? Done. Backup consistency? Free and vastly simplified.

That is indeed something you can do with a CoW system but the tools to achieve this are in my experience not fully mature yet.

The same tasks can also be achieved without snapshots at all, NixOS seems to be doing fine.

>Backup consistency? Free and vastly simplified.

Snapshots are not backups. If you pretend they are you will loose all your data.

Snapshots are snapshots. Nothing more and nothing less. Not backups.

>But this should be the default for all user-facing machines as well as servers.

I've not stated they shouldn't be default, rather, that the end user does not necessarily need them.

Though, ZFS is doing a rather poor job on all these tasks, especially disk management on ZFS is a hassle I wouldn't have a typical end user face. I think Bcache FS is a much cleaner and better design than ZFS, which is rather unflexible.

But again, I don't see the immediate need for every end user machine to have a checksumming and CoW filesystem, people have been doing fine for years without them and I doubt there is chaos and fire everywhere as you seem to imply. It could be a little bit better but ext4 is a perfectly fine filesystem for production data.

If you rely on checksumming and snapshots for your data safety, you're in for a bad time. Proper backups, not snapshots, will beat snapshots any day. Checksums make data inaccessible, a rather bad choice for a user machine. Whoops, I guess those family photos are totally unusuable because 1 bit has corrupted.

>>Backup consistency? Free and vastly simplified.

>Snapshots are not backups. If you pretend they are you will loose all your data.

The good thing about snapshots it that you can use them as a consistent base for your backup: If you naively copy files over from a running system, you'll get inconsistent data, but if you copy it over from a snapshot, it'll work without taking the system down.

That is fairly obvious, a lot of systems provide snapshot functionality irrelevant of the file system.

If you still need snapshots, there are several methods without needing CoW Filesystems, notably LVM and dattobd (both of which I have utilized in the past)

Also keep in mind that a pure snapshot from a running system will behave like a system that just crashed, a half written file won't complete the write magically because you have snapshot.

If you don't tell postgresql that you're doing snapshots or backups, both will likely corrupt data, doesn't give you anything in that case.

Same for desktop systems. You make a snapshot of the runnign system and suddenly the browser only finds corrupted settings because a write wasn't complete until after the snapshot.

If postgresql corrupts your data upon sudden halt you have far bigger problems to worry about.

And as far as I know most software acts responsibly with files, either writing out the new version to a temp file or using sqlite. Both of which are atomic-snapshot-safe.

You might still loose partial datasets (especially when the database is not 100% perfectly constrained) and not all software handles such sudden halts perfectly. Making a snapshot of any working dataset of any software without the data corrupting or loosing parts of the working set is a pipedream.

Yeah I somewhat understand the phrase "snapshots are not backups". But what really constitutes a "backup" ? Because I argue snapshots are backups. I snapshot a dataset. And use ZFS send/recv to a remote ZFS server of the dataset snapshot. Is that not a backup? I argue it is. If the original dataset is completely trashed I simply send/recv again the remote snapshot of that dataset back to the primary server. Plus you can update with incremental send/recv as well. I simply don't need tapes. I suppose for archival purposes one might want a tape copy somewhere. But even tapes are problematic. They degrade, get chewed up, etc. There are fewer variables with digital data for me personally.

I've explicitly said that snapshots HELPS with backups, not that they are backups.

Just because a checksum doesn't match and you don't have redundant copies doesn't mean you throw away data (hint: you don't in practice either). I think a user would rather know about corruption rather than, ehm, not.

Depending on FS you can also disable checksumming, happy?

I'm not aware of ZFS letting you access corrupted files that easily, the Oracle manual [1] notes that files can only be recovered from backup or removed, not how to access a file that has been corrupted. So this does seem to imply that ZFS does indeed throw away data if it's corrupted.

Of course a user would rather know about corruption, but on the average user system with family pictures, you don't run into problems even after years to my experience. The photos function perfectly fine with minor corruption, which as noted above, does not work on ZFS since on ZFS it's either perfect or repairable or lost.

>I've explicitly said that snapshots HELPS with backups, not that they are backups.

You can do Snapshots on other FS' too, LVM and dattobd allow Snapshots of Blockdevices fairly easily. So I still don't see why that makes ZFS superior to ext4 when you can simply layer the solution underneath.

Also, btw, LVM has RAID1 and RAID6, both of which allow the user to detect data corruption on any filesystem, which IMO, is better than restricting to one filesystem. That way you can make an image of an old FAT16 device and be assured that it won't suddenly corrupt data.

Same goes for mdraid and snapraid, the later of which is a file-based solution and allows recovery of data beyond loosing all parity of an array plus additional drives. ZFS does to my knowledge not work well once you loose more than it tolerates.

Why choose ZFS when I can layer the solution together? Or rather, layer until a FS with a better underlying design comes around.

[1]: https://docs.oracle.com/cd/E26505_01/html/E37384/gbbbc.html

LVM and dattobd are not equivalent and only have some overlap in usability and often come with severe limitations and implications in practice.

ZFS is targeted towards the enterprise and such is adapted for that use case. I agree that it isn't perfect for home users and I'm not advocating that ZFS be the acceptable solution. I'm just stating that Linux does not have a decent general purpose filesystem.

There is nothing that says that a checksum mismatch results in data loss, that statement is absurd. ZFS has the stance that you should not work on corrupted data as that will result in further corruption when action are based upon it. I tend to agree with that approach, but ZFS will not remove the data it finds corrupted.

You can also layer a more traditional filesystem upon ZFS to leverage redundancy, checksumming and snapshots, something that mdadm does not provide. There are very good reasons for why a layered approach isn't nearly as flexible, this was controversial at the time and debated alot a decade ago.

Most RAID solutions don't even verify the data when read from normal operations. A checksummed approach is a significant improvement, in combination with redundant information you can automatically recover - something that is hard to do in a layered approach. Also, with with a checksum you can verify the correct result. With raid 5 or degraded raid 6 you can at best know that an error has occured but not fix it. Again something that a layered approach has issues with (even if the filesystem had checksumming the RAID controller/software wouldn't know or be able to act upon it).

XFS has metadata checksumming, and CoW works (but still experimental) with recent kernels. And it generally runs circles around other filesystems, too.

Doesn't copy on write ruin the ability to preallocate files?

We use zfsonlinux in production already. Have for years. It's had some issues, some have been fixed, some not yet. (I'm usually the go-to person to deal with the deepest ZFS issues.) Last I load tested FreeBSD ZFS it wasn't much different: a different set of issues. Some problems, some fixes.

We're now using btrfs too.

What's your technical beef with ZFS on Linux anyway?

He's not against ZoL. He's just saying that ZFS is a first class citizen on FreeBSD so if ZFS is amongst your concerns for that server, then picking an OS where ZFS if a first class citizen makes some logical sense.

To be honest I do agree with him. I find FreeBSD to be close enough to Linux that any backend engineer or operator worth their salt should be able to swap between the two platforms. Nearly all the same tools that are available on Linux are available on FreeBSD, plus a few tools of it's own. I'm not advocating FreeBSD everywhere though - Linux has it's strengths too which would make it a better platform in other domains. But sometimes it feels like people turn to ZoL without even considering FreeBSD - which is a real pity as FreeBSD + ZFS is a dream to administrate. It's a lot easier than people expect and by not even entertaining the idea of FreeBSD I think they're missing an opportunity.

This is all ideal world scenario stuff though. I do appreciate there are often other factors that complicate the decision making. Sometimes even political rather than technological factors.

For us in particular, a big deciding factor was actually monitoring when we decided to consolidate our storage servers on ZFS on Linux (migrating a couple that were FreeBSD back to Linux).

ZFS works well enough on either these days, and most of our linux folks are certainly comfortable enough in BSD-land, but our primary server monitoring stack at the time (NewRelic's server/infrastructure agents) doesn't have any BSD support at all and they didn't have interest in adding it, so we went back to the other set of tools we knew (linux) which integrated better with the other tooling we used and didn't have time to replace.

The purists who try to suggest ZFS should be used from BSD whenever possible are probably forgetting "perfect is the enemy of good"

I don't think the grandparent poster was talking from a purist perspective though (I certainly wasn't). I think s/he was more interested why so few people consider FreeBSD. It might be the case that everyone runs into a deal breaker like yourselves but I think sometimes people don't consider FreeBSD simply because it isn't Linux. However I do completely agree with your points as well.

I feel your pain with NewRelic as I've also ran into a few brick walls with them. In fact on Linux as well (not tried NewRelic agents on FreeBSD).

I'm a bit confused by why ZFS shouldn't be used on Linux. It's just software and can be ported?

In terms of using FreeBSD, I generally work in small (~6 person) sysadmin teams running hundreds of servers. There's already a lot of software and OSes to maintain and not a hell of a lot of time to do it, if we can cut out some of the complexity (making sure provisioning tools work on FreeBSD etc.) by just running as much stuff on Linux as possible then our jobs and lives become easier.

> So why don't we use BSD everywhere, or Linux everywhere? There's no single reason -- it's many reasons -- and it's serious engineering time to document them all and do the topic justice. I'd like to, but I don't have that time right now.

If you ever have the time, that would be an awesome read. I find the differences between operating systems fascinating, and rarely done justice as people overlook the subtleties for a "Just use <<insert favourite tribe>>" approach.

Part of the problem is that to do it justice, I need to get input from at least several other engineers and managers here. The choice of an OS is a big topic, and in the cloud it spans multiple teams (the Perf & OS team have the expertise and time to dig deep, but the choice also affects and needs input from the tools team, the platform security team, network engineering team, developers, etc.).

I would be interested in such an in depth write up.

So would I, but I believe as he states it would be a monumental task across multiple disciplines and would that effort be worth it for probably a very small audience of nerds? :)

Would it be possible to recommend it for a future Netflix tech blog?

I think you are spot on Brendan. We use both Linux and FreeBSD, and we see that applications can behave very differently in some scenarios on both platforms.

Because Linux is absolutely awesome.

Zfs can be made to work, shoehorned in your parlance, and then you have the best filesystem on the very best of operating systems.

What are the angles that make Linux so compelling to you? For example, I am a linux user since mid 90s. But recently I have been doing a lot of programming with an emphasis on IO. From what I am seeing both bsd (through kevent) and windows (through completion sockets) offer far better APIs for complex IO handling. The lack of a consistent IO API in Linux is making me wonder why is has so much momentum. (Drivers are pretty important, and Linux does tend to be strong here.)

I am not a low level developer so I cannot discuss that merit with you.

I too started mid 90s with Linux. Walnut creek cdroms anyone?

I believe Linux merits stands on it's own feet and I don't want to have to write an essay about everything that is awesome about Linux.

- kernel dev speed

- the disruptive gplv2 linux kernel license

- kernel breadth of drivers

- great hardware support

- kernel subsystems available

- container technology in particular lxc

- scalability

- openness

- now - zfs - with encryption support!

- strace

- kvm

- size of community

- size of pool of corporate backed devs

- great great wm's

- great userland tools

- bedrock distributions like Debian

- distributions pushing the envelope like Arch

- The Arch Wiki

I think Linux is cursed by its development model. There is no way for a single, complete change set to be landed in Linux. So you get features like btrfs or lxc which are horribly broken at the start, and then asymptotically approach stability and security.

>There is no way for a single, complete change set to be landed in Linux.

What? Manage a git clone, develop until done, submit patches. Btrfs could have done this, but what is the benefit over saying the feature isn't ready? Nobodies forcing you to use or even install it.

Some of these are not particular strengths of Linux. For example, dtrace compares well to strace (this relates to the kernel up design from earlier post). But you make a good point towards strong hypervisor.

You're not describing anything that isn't also applicable to FreeBSD - nor many other open source OSs. I mean take LXC for example, a great many flavours of UNIX had OS containers long before OpenVZ hit Linux - let alone LXC.

The biggest points you could probably win an argument on is:

1) hardware support: but that's less of an issue with servers anyway. Plus ironically I've had more issues installing Debian onto HP Proliants than FreeBSD due Debian's strict policy on non-free drivers.

2) size of community: but then what's the signal to noise ratio in the Linux community of actual helpful, knowledgeable engineers vs that of FreeBSD or any other POSIX-like platform? Linux does have a significantly bigger community but I've found that does result in significantly more misinformation being published too.

A genuine, non-trolling and non-elitist question: have you actually tried any other POSIX-like platform aside Linux? I'm not saying you should need to since you're clearly happy with Linux, but since the points you praise Linux can be applied to a number of platforms outside of Linux as well it does sound like you're making assumptions that those points are exclusive to Linux as they are why you're describing Linux as the best OS out there.

Considering who funds a lot of FS development on Linux, this isn't that surprising — using O_DIRECT is really really complex and also not in the kernel, so a neat-o way to keep others from doing the same.

> OK, I'm curious: why are people shoehorning the filesystem to fit the OS rather than carefully picking the OS that makes the most sense for the application it's being used for?

Because the filesystem isn't the only feature of the OS that matters, and people often have more than one requirement. There are technical reasons to choose linux that have nothing to do with ext or xfs, and there are nontechnical reasons also. Just because someone has chosen linux doesn't mean they don't value their data.

In my car of using it hardware support. Freebsd just chokes on the data controller in using and solaris and it's flavors didn't even detect it. Along with that freebsd doesn't yet support the virtualization tech that I need to give real pcie hardware to a VM.

The bhyve hypervisor does support PCI-e passthrough.

Not VGA passthrough though, which is needed for doing most ML and gaming tasks.

Do you just buy pre made boxes? Im curious because I have never put boxes in service I did not spec and put together myself. For exactly this reason. You want to make sure the HW is supported by your OS. I didn't think anyone just grabbed a box and tried to install multiple OS'es on it to see which worked. That seems genuinely like a poor idea and way to do things.

No, I built this myself and the hardware is supported on my OS of choice. I tried FreeBSD on it recently to deal with an issue I had on ZFS (was related to a bug in ZoL during bad hardware failure).

My naive first reaction would be: because something up the stack is Linux-specific (or X-, or Y-specific).

In every ZFS on Linux discussion, there's a question like this. The short answer is that nobody really wants to be forced to run FreeBSD just for ZFS...

Also consider hardware support, mindshare and the flexibility/need to use more of the OS than just the filesystem.

I began with ZFS on Solaris, then Nexenta... then jumped to ZFS on Linux in 2012. FreeBSD was a non-starter because my hardware platform (HPE ProLiant) was poorly supported under FreeBSD. That killed it as an option.

I run all 3 in various forms these days - Illumos, freebsd, ZoL - honestly ZoL is where I'm most excited about the future.

I am seeing long standing stability/availability bugs finally getting fixed within the ZoL project - often times being fixes to OpenZFS upstream, fixing bugs that certainly "felt" like the same thing on those other platforms.

There are zero platforms I entirely trust to survive every conceivable drive failure, I've had production outages using every combination of hardware and OS possible due to bugs both in the OpenZFS stack itself, as well as underlying OS issues on all platforms.

At this point I think it's choose which OS you're most comfortable with, the ZFS bits are getting really close to on par.

And there are many reasons to want to run a linux kernel with ZFS - not just "preference" related.

Because NixOS doesn't have a FreeBSD equivalent.

Because sometimes you're stuck to that small intersection where only a particular OS is supported by a particular technology. Case in point: CUDA. There's no support for CUDA for OSs other than Windows and Linux. And there's no support for ZFS in whatever form on Windows. If you want to do GPU computing using programs written in CUDA and keep the data on ZFS then you're essentially stuck to Linux + zfsonlinux and you have to deal with it.

Running a mixture of OSes comes with its own costs too. Having every machine running the same OS (and ideally the same version) can make a lot of things much easier, e.g. no more prod/stag-only bugs that aren't reproducible on dev machines.

How does porting ZFS to Linux differ from the "porting"-via-rewrite of the Unix tools?

The title of this HN post should mention that this is ZFSonLinux.

This is good news, but I'll definitely want to wait a good long while before enabling this in production. Yes, officially zfs isn't good enough to use in production anywhere even without shiny new features, but I reckon zfs as-is is better than some other filesystem.

By the ZFS proponents own standard for "mature", I guess I should still wait a decade to use this, right? ;)

Depends on a). how much you value your data and/or b). how much you trust your backups.

The rate of work and breadth of contributors to ZFSOnLinux is pleasantly suprising and impressive: https://github.com/zfsonlinux/zfs/graphs/contributors

Things that happen when you employ folks full-time to work on something.

Why would one choose it over dm-crypt? What are the advantages?

Authenticated encryption, for one.

Personally, I think the best thing is having (finally!) serious cross-platform encrypted drive support in software. (Yes hw encryption has its benefits, but like with software raid - the flexibility of software is great). Potentially this could also be easier to use for external disks than unlock, detect volumes, mount (or unlock zfs mount...).

In theory, having compression and encryption handled by one codebase might open the door to a safe/clearly delineated tradeoffs of enabling both.

This to me, is one of the GREATEST advances since Unix was created. Seriously, how !@#!@# amazing is it to be able to yank a pool from illumos, Linux, FreeBSD etc and swap it into a box of another OS supporting ZFS and simply import it wether its encrypted or not but especially encrypted!

I don't know if most of you are old enough to remember the absolute HELL of dealing with tape drives, and UFS to trying to get data recovered or migrated before ZFS. It usually resulted in tears.

ZFS - native encryption, resizing, bleaching | The FreeBSD Forums <https://forums.freebsd.org/threads/56869/>

> I hope that FreeBSD will be not too far behind. …

Only by looking at issues in github repo you can tell it's far, far away from being stable, even for homelabs.

I'm curious how long it'll take for support to land in grub for this (or how much work it's likely to be).

Does ZFS use the native compression libs in the linux kernel?

s/Merkel/Merkle/

Thank you so much for your wonderful contribution to this discussion.

What's wrong with pointing out a typo?

What purpose did it serve? The typo was in a commit message. Assuming that the committer ever even sees it -- most submissions here are not submitted by the original author -- he is certainly not going to go back and fix the typo in his commit message.

Besides, in 99% of "typo instances" it is quickly obvious that a typo was made, everyone understands what it should be, and continues on. If it's a typo that actually results in a real problem then by all means point it out; otherwise it just adds to the noise (such as in this case, where it was merely a transposition of two characters).

It might not be fair but reading a .gov address listed as a reviewer does not exactly inspire trust. :'(

Why not? It's LLNL, who are heavy ZFS users and have been at the forefront of large datasets since they were a thing in computing.

Yet also advertise to be into counterterrorism, defense, and intelligence operational support. How many dual_ec_drbg do we need before we stop trusting those people ?

I think you're confusing the NSA with LLNL

Reply on news.ycombinator.com