History of ZFS Part 2: Exploding in Popularity | Hacker News Reader

History of ZFS Part 2: Exploding in Popularity

klarasystems.com

134 pointsveronica_avram65 comments

This is actually Part2.

Part 1: https://klarasystems.com/articles/history-of-zfs-part-1-the-...

I'm pretty blown away that there is an active OpenZfs on Windows Project ongoing.

Personally I would not likely use ZFS for a Windows desktop but for Servers, might be handy.

For desktop, I always virtualize my windows on a host that has ZFS (Ubuntu 20.04). By adding SMB shares from host to guest gives me what I need to snapshot my user data. The only drawback is no hardlinking on windohs guest but that's a rare usecase for me.

There are many ways to use ZFS for windows... even without OpenZfs on Windows!

Check https://happyadministrator.com/2020/11/29/windows-users-prob... : it's not just for servers. The performance gains and extra features make it worth doing for a regular desktop.

Personally, I am a Windows 10 fan. I have tried Linux but found the command line experience sub optimal ( explained on https://github.com/csdvrx/cutexterm ) - except for the filesystem!

So for my next Windows 10 installation, I preparing a KVM GPU passthrough, just to keep using msys2 on MinTTY for a wonderful user interface, yet with a great filesystem behind (a raid of NVMe, datasets snapshotting etc...)

That should give the best of both worlds!

I am going to be honest, I can't figure out if the cutexterm README is trolling me or not. You list some features that cutexterm has, but don't make it clear what advantages it has over existing terminal emulators. Sixel support is nice but maybe show examples on why we might want support for it. I don't know what I would use it for.

If it was not intended as a troll, I suggest not to referring to linux enthusiasts as Millenials (sic) with Stockholm syndrome.

/unsolicited-feedback

> You list some features that cutexterm has, but don't make it clear what advantages it has over existing terminal emulators

cutexterm is just a configuration of xterm. I tried to make the best out of my Linux experience by spending a lot of time working on what I perceive as commandline usability issues. Hopefully, it will save some time to other people, if only by showing the limitations of the approach.

In the end, I give up, because it's a dead end. This is why I will do a KVM GPU passthrough: to get a proper terminal experience with proper font support, sixel support, etc. thanks to mintty on Windows 10.

> Sixel support is nice but maybe show examples on why we might want support for it. I don't know what I would use it for.

When I ssh to a server, running gnuplot remotely saves me a lot of time. When I need to check a video file, I run mpv - and so on, so proper sixel support is very important to my workflow.

Maybe it's not important for yours. Or if your terminal emulator did not support it, maybe you never became dependent on this feature?

Personally, I couldn't do without it.

> I suggest not to referring to linux enthusiasts as Millenials (sic) with Stockholm syndrome.

Uh, that's just my personal judgement. I can't explain otherwise why someone would prefer anything besides mintty if they have had the opportunity to use it on Windows 10.

With Edge running on Linux (and even syncing works now!) I have less of a need to run Windows 10, but I still prefer it. So I'm going back while keeping the good things I gained from ZFS :-)

If you are not annoyed by the Linux terminals, I guess we just have different tastes/preferences.

What’s the advantage to running windows virtually in Ubuntu/ZFS vs just natively with backups enabled?

Is there a specific flow you use that makes this more optimal?

How is performance?

I have automated snapshots of the Windows VM which are backed up to a remote ZVS Pool. This protects me from BSOD corruptions. BSOD corruption can be avoided with bare metal windows but it takes a bit more work.

Performance is fine. I don't use this for gaming or GPU heavy windows tasks and am running on a dual CPU Xeon tower. Have also run This setup on my i7 ZBooks and Elitebooks in the past with no issues with the right config.

Challenge is always disk space and memory when virtualizing. Must reserve RAM for VMware workstation or else you will have intermittent swapping issues.

Biggest reason for me to use Linux on bare metal is security. Windows is and always will be the Swiss cheese of security issues. With Linux, I can harden to host much tighter and the windows host never can be pinged externally as its on a private Network behind VMware.

ZFS is perfectly useful on desktops, why not on Windows? I have data that I want to be preserved, and ZFS is great.

Just a personal preference. I will only install windows on physical hardware for gaming or GPU intensive purposes. I don't see much advantage to using ZFS on a VM guest except for bitrot protection. This can be accomplished using SMB share on zfs dataset on the linux host.

For my day to day programming and devops, I virtualize Windows Server for all my desktop work that I have to use windows for which is anything I cannot do on Linux which is not too much actually. Just use it for Remote admin of IPMI's, VCenter, IIS, SQL Server and Visual Studio for rare coding scenario's.

On linux, I even use WinSCP which solves all my change request code syncing to remote servers. With ZFS on root, its awesome. Only time I ever lost a linux server is when windows update on another drive screwed up the boot sector on my linux drive. I cannot count the number of times a windows blue screened permanently for me.

I swore to never depend on windows again and switched to Linux for good. May switch to OSX in the future once hardware becomes more reasonably priced (M1 chip appears to be awesome).

Given the recent topic on the government's forming of a taskforce to manage investigations into ransomware attacks - I wonder how much of a big deal ransomware would be in a world where linux used ZFS by default. I have read reports of users running ZFS who were easily able to shrug off a ransomware attack but I don't know how that would play out in general. Could widespread adoption of ZFS significantly blunt ransomware attacks?

AFAIK, Windows has support for snapshotting as well (Volume Shadow Copy Service¹), and (again AFAIK) certain malware disables it.

The whole point is that if malware gets root access (or anything that can manipulate the disk at will), there's nothing that can be done.

EDIT: briefly searched, and indeed, my suspicions were correct: https://redcanary.com/blog/its-all-fun-and-games-until-ranso....

¹=https://docs.microsoft.com/en-us/windows-server/storage/file...

> The whole point is that if malware gets root access

Strictly true, in that if it does get root access, and is aware of zfs snapshots and deletes them, then yes. Nothing you can do.

More typically, the malware runs as the user id and has no awareness of zfs, so it just overwrites files the user owns. Which can be trivially undone with snapshots.

As with everything related to security there is no absolute, but every layer helps.

That's why you need another system to pull snapshots to for backup.

If that system is online, then it can be hacked.

Even if the backup system isn't hacked, you can hack the original system in such a way that the pulled snapshots look like they work, but in fact did not work (ex: hack the rsync data requests to be encrypted with the ransomware key).

You gotta stop them from having root access to begin with. Hard in this day of zero-days, but a lot of these ransomware attacks don't seem very hard to pull off these days...

Well, if you take snapshots every hour then you can always roll back to the latest snapshot if you need to. It would be a nice defence against the traditional "encrypt all writable files" attacks. As long as the root user isn't compromised, using ZFS (or other filesystems with snapshots) should prevent a lot of problems, and not just ones related to ransom-ware attacks.

Why would ransomware that can encrypt all your files not be able to delete your snapshots?

In the scenario where a client computer with access to files via a network share becomes infected and encrypts files, but the server is snapshotting the file system that those files are stored on.

Also, OpenZFS can export ZVOLs over iSCSI so you can have formatted block storage for other devices that can be rolled back by snapshot on the OpenZFS.

In a virtualized environment all of the VMs' storage could be protected with snapshots.

There's a number of reasons I feel this assumption is valid in most cases:

1) Unless it's a targeted attack it's highly unlikely that a ransomware strain will be designed to target a specific filesystem. Snapshot deletion requires filesystem specific APIs, whereas reading/writing files is just standard libc fare. You would need to maintain different strains of a virus to target an organization running Btrfs vs ZFS vs ReFS, for example. - If the attacker has deep knowledge of your storage stack, and has gained privileged access to it, you've already lost: go to step 4.

2) Separation of privileges. Deleting a snapshot is a privileged command. A ransomware infection that doesn't successfully elevate privileges might be able to encrypt my home directory, which would suck, but it wouldn't have permissions to delete snapshots of my home directory.

3) Virtualization / network storage: snapshots are transparent to virtual guests, a virus inside a guest can't delete snapshots if it's not aware they exist. With network shares and ZFS: you can choose whether or not you want to expose snapshots. If you do expose them: don't grant network clients privileges to delete them.

4) Replication: with any ransomware infection the only real defense is off-site backups. Snapshots makes backups trivial. You can incrementally replicate and restore from them easily (e.g: zfs send/recv.) This alone makes using a CoW filesystem worthwhile in the defense against ransomware. Ransomware will have a hard time deleting snapshots on an air-gapped disk.

"Why would ransomware that can encrypt all your files not be able to delete your snapshots?"

If you keep your backups on a third party cloud storage provider that runs on ZFS then those snapshots are taken by their root, not yours.

You send your backups to this cloud storage provider and they take the snapshots on your behalf.

You have an unprivileged login and have the ability to read the snapshots, but not destroy them.

In this scenario, Mallory can destroy your data and gain access to all of your cloud credentials and destroy the primary dataset ... but cannot destroy the snapshots.

I'm sure there must be some cloud provider, somewhere that offers this ... right ?

Your service does so and does so well but I would go further and say that any service that does not is not suitable as a backup solution. No matter if the service runs tapes the customer can't decide to overwrite, snapshots they can’t remove or whatever solution they use, the customer should not be able to delete old versions prematurely.

Indeed, that's what they [can] do: https://redcanary.com/blog/its-all-fun-and-games-until-ranso....

The windows clients cannot delete the ZFS snapshots that are presented as shadow copies.

If there was ever a use case for certain operations to require a privilege level higher than or separate from root, snapshot deletion is it. I’m generally a bit skeptical of such measures, but the purpose here is obvious.

I don't see how that is anything more than adding an additional hurdle that the attackers would overcome. If they can already jump through the hoops to get root access then what's to stop them from doing the same for this other theoretical access level?

The user may have granted root access to do something relatively normal and unrelated to ZFS. You probably only want one or two programs to ever be able to delete snapshots.

ZFS on Ubuntu requires root to destroy snapshots, I don't know about ZFS on root, but I would guess the same.

This is a permission which can be delegated to users on a per-dataset basis. See "zfs allow". It didn't used to work on Linux, but has been possible since Ubuntu 20.04.

Often ransomware runs with user or local admin access since all they want to do is encrypt your work files. Hopefully deleting snapshots would requite a different level of access.

Zfs allows remote replication, which would prevent this.

Any snapshot-capable storage setup should be a decent remedy against ransomware, just roll back to the latest snapshot (unless of course the ransomware manages to kill your snapshots), unless there's another specific ZFS functionality you're hinting at?

ZFS, btrfs, even LVM does snapshots, and it seems silly to use anything else for critical data/configuration.

I’ve always wondered why Oracle wasted all that money on Btrfs when ZFS could have been made GPL compatible by a mere stroke of a pen. Could someone perhaps shed some light on this?

Well, since the Oracle purchase of Sun was only announced in April 2009 and Btrfs was in a stable release of Linux in March 2009, I suspect it was being worked on well before Oracle had that capability, and because it makes different tradeoffs than ZFS, there are reasons to prefer one over the other in both directions.

As for why they haven't done it now? Who can say.

Do you suppose the reputation for lack of production-readiness of Btrfs at that point was earned? What about now, 12 years later?

But even if it was production ready in 2009, it seems like it would make more business sense to keep the most mature and just cancel the other one, like Adobe did when they bought Macromedia.

I have very limited Btrfs experience, now or then; I've heard a lot of people complaining about it, but A) I hang out in ZFS circles, so the people I hear there talking about Btrfs are a self-selecting group, and B) the plural of anecdote is not data (unless very large numbers are involved).

One notable caveat is that in 2009, ZFS was basically available on FreeBSD and Solaris - zfs-fuse existed (and I even used it sometimes), but I would be astonished if you could boot from it, for example, sometimes it just broke, and FUSE is going to have at least some overhead by dint of being FUSE, which may be a nonstarter for performance applications. If Oracle had wanted to throw all their weight into ZFS on Linux (not the FOSS project that was spawned later, their own project), it'd have taken a bunch of engineering time for the initial port, plus ongoing maintenance on multiple OSes (unless they just decided to throw Solaris out, which, with support contracts, would have taken some years to do anyway), and they _still_ would have run into some of the fundamental limitations which make some of Btrfs's features infeasible on ZFS (though IIRC closed ZFS worked around one of them somehow, and I'm kind of curious how, but you know, closed source).

I use btrfs for my primary or OS disk and ZFS to store all of my data. Until btrfs can fix their raid 5/6 or ZFS on root becomes easier this will remain the case.

FYI: ZFS root on Ubuntu 20.04 is supported in the installer. Encrypted ZFS isn't supported in the installer, but is relatively easy to do: https://linsomniac.gitlab.io/post/2020-04-09-ubuntu-2004-enc...

I've been running with encrypted root ZFS on a couple workstations for over a year now, and it's worked great!

Not Invented Here (NIH).

https://en.wikipedia.org/wiki/Not_invented_here

I wonder what would have happened to ZFS, java and all those technologies if IBM have bought Sun?

This is an interesting question. Despite HN's hatred against Oracle. I still think they are a better steward for Java, JVM and MySQL. If it was IBM it would have fallen under the management of Ginni Rometty.......

Current IBM CEO seems to have a much better vision and strategy. Although time will tell whether he could execute and succeed.

I am thinking about the big problems they had with getting J2EE transferred to the Eclipse Foundation. In the end they had to rename it (to Jakarta EE) and even change all the package names (!). Nobody can know for sure, but I have the suspicion that in an alternate timeline in which IBM bought Sun, the same transfer may have happened, but probably much more smoothly.

The days of Solaris and SPARC were always numbered but I wonder if IBM might have managed their decline with more grace and less abruptness than Oracle has. I was particularly struck by Oracle's decision to suddenly (and without explanation) discontinue UNIX V7 certification for Solaris, after investing so much effort in obtaining it.

Given IBM already has POWER, it is unlikely IBM would have wanted to keep SPARC going long-term. It probably would have absorbed some of the technologies, patents, and engineering talent, and transferred that to POWER. Still, at least all that would have given all that somewhere to live on, Oracle's semiconductor division ended up being obliterated.

Likewise, the fact that IBM already has AIX would mean they probably wouldn't want to keep Solaris going long-term. I wonder if they might have somehow tried to merge AIX and Solaris. That could have been interesting. I also wonder if IBM might have, unlike Oracle, kept OpenSolaris alive. And I also wonder what the ZFS licensing situation would have been with IBM as its owner (would IBM have been more likely to relicense from CDDL to GPLv2?)

(Disclaimer: Former Oracle employee, although I never really had any personal involvement in the topics I'm discussing.)

Java and mysql are treated well, yes, but those are major money makers. Would IBM have done worse? I don't think that's clear.

Solaris languishes, and dtrace/zfs run better on freebsd today. IBM acquired redhat, demonstrating a willingness to develop opensource operating systems.

>IBM acquired redhat

Yes that is my point. Redhat acquisition was entirely the current CEO's making. Before that I am not so sure.

With my impeccable timing, I took my first looks at ZFS right at the very twilight of OpenSolaris, literally weeks before Oracle took over. I ended up running one of its successors (OpenIndiana) for a while, but it wasn't too friendly with white-box hardware; AHCI hot swap was risky, and KVM only worked on Intel CPUs. Getting VirtualBox VMs to start on boot was fun (not). It was still worth the aggravation just to have ZFS, though; it's saved my bacon more than once, and made life a lot easier when I need to migrate data or VM images.

I was glad indeed when Ubuntu started shipping an OpenZFS module. ZFS with virt-manager makes for an excellent VM host.

Someone correct me if I'm wrong, but the raison d'être of the Illumos kernel (upon which OpenIndiana is based) has always been the cloud, with Nexenta and Joyent (SmartOS) doing the heavy lifting. They're basically ZFS/DTrace machines.

I've played with OpenIndiana too but it didn't fill any real use case for me, at least locally. The userspace was just a cut down version of Nth Linux distro and I don't have a file server home that would benefit much from ZFS. Neither do I have to debug the exotic stuff you would want DTrace for.

No mention here for NetApp’s WAFL. It was a checksummed (block, and later sector) file system with instant snapshots many years before ZFS, which took plenty of inspiration from it; NetApp even sued Sun (and lost).

Netapp and Oracle settled out of court a few weeks after Oracle acquired Sun.

https://www.theregister.com/2010/09/09/oracle_netapp_zfs_dis...

https://www.eweek.com/storage/netapp-oracle-settle-old-paten...

You’re right, my memory was slightly off, it was a settlement. Awkward because NetApp sold a lot to Oracle.

Nice try throwing shade, but they're totally different approaches.

WAFL only detects data bit flips with a larger sector size containing parity info, ZFS uses Merkle trees and hashing.

Well ZFS also doesn’t use RAID-4 so of course they’re “different.”

Nonetheless many of ZFS’s internal features are or were directly taken from WAFL, at least at an algorithmic level (nobody copied source code that I’m aware of). The snapshot mechanism in particular.

How can you even know or assume any of this? What evidence do you have?

Snapshots aren't special magic from NetApp. Windows, HP-UX, LVM, and VMware have snapshots.

I think you're stuck in religious reverence for all things NetApp. Good luck with that.

Was? Does NetApp no longer use it?

>Just because Apple decided to drop support for ZFS, that didn’t mean that ZFS was no longer available on Mac OS. In 2008, Dustin Sallings created the MacZFS project with the goal of “continuing where Apple left off with ZFS”. The MacZFS project ceased in 2013. In 2014 the effort was resumed, partly based on the ZFSOnLinux repo, under the name OpenZFS on OSX, which continues to this day.

I'll admit I'm a bit disappointed by the Mac section of this history. In particular this completely skips the very interesting Z-410/ZEVO year or so period mid-2011 to 2012 wherein somebody attempted to actually do a commercial release of ZFS for Mac. It was actually a pretty impressive effort for the brief development period that was far more functional, polished and integrated vs MacZFS at the time and I ran it on a few Mac Pros for years before O3X took over. I was bummed when the effort ran out of runway and got bought by "getgreenbytes" or some company along those lines IIRC, which did a single free community release and then canned it. O3X ultimately moved things much farther on the technical side and has been fantastic, but a project like ZEVO would likely have ultimately added more GUI tools and made it more approachable for typical Mac users.

Really it's one of those real tech historical shames that but for a few turns of events Apple might have thrown their weight behind it rather then APFS. The reasons given in this article are also a bit curious in that they completely skip over the shitty NetApp patent lawsuit going on literally right at the time Apple was deciding (September 2007). Then again maybe then we wouldn't have gotten the wonderful OpenZFS project in the same way either, hard to say how it'd all wash. But if there'd never been any patent concerns, or and CDDL/GPL compatibility concerns, it would have been neat to have an FS like that become the universal standard (sans windows granted).

An interesting article about some of the history of ZFS and the birth of OpenZFS

Site seems to be missing a clear opt out on privacy. Not gonna recommend anyone click through.

Outline: https://outline.com/SwjZtr

When you click the here for more info, there are the cookie requirements/explanation with optout link pointing to: https://s.pubmine.com/opt-out

"Explode" is a pretty strong word for a technology that was somewhat stillborn. Yeah, it showed up on Solaris as Solaris was winding down. I've never seen it used anywhere, and I've been doing this for awhile.

You clearly haven't been doing enough or in various-enough industries "for awhile." I guess you missed massive Solaris shops in academia, enterprise, military, and government. Sun OS (UNIX) before '92 and then to Solaris on-top from about '94 to '13 as gear lifecycled out.

The classic use of ZFS was on the Kealia / Sun Fire X4500 "Thumper" box. 96 TB in 4U in '06. It was a beast and worthy to throw rocks at NetApps.

Anytime someone insecurely touts "I've been doing X for Y years," I know they've reached peak complacency rather than continual learning and mastery.

I used SunOS from 1992 and Solaris up to around 2004.

I'm in the semiconductor industry and we used to have thousands of SPARCstations and SPARC servers for larger jobs. All of the EDA (Electronic Design Automation) tools ran on Unix / X11. Around 2002 the vendors started porting some of the tools to Linux / x86. The machines were faster AND cheaper.

By 2003 the only jobs run on the Suns were processes that required >4GB RAM. When AMD released their 64-bit x86 Athlon CPUs we bought some of the first ones. After that the Suns just faded away.

In the 1990's all of our file servers were from Auspex. We have been using NetApp fileservers for about the last 20 years. Other than a few scratch drives I don't think we ever used Solaris as a file server.

An energy-sector engineering shop had SPARCstation 2's and 4's used for running nuclear reactor Fortran compilations and simulations. The software ran on any UNIX (SCO, HP-UX, BSD, SunOS/Solaris, SGI, AIX) and I helped port it to Windows about 1997. Oddly enough, Windows of that era had extremely inefficient virtual memory swap strategies.

The university I worked at had a large Solaris deployment for internal IT systems for the CS department. They kept it for many years because it worked, it was a known quantity, and was reliable. They could've changed it over to commodity boxes running Linux or FreeBSD sooner, but that would've been a major headache for downtime, costs, and staff processes.

A friend-of-a-friend ran an extremely large deployment of Solaris boxes for batch jobs of a military data processing facility.

The scariest SunOS feature was metadisks: striped drive volumes to expand storage.

I worked in a Solaris shop from 1999-2000.

I'm not insecure, and I'm not mad at ZFS. I'm just pointing out that it showed up when Sun was already circling the drain.

Seamicro managed to pack 2000 cores into 10U. Cool. I've never seen of those, either.

Yeah, 1-2 years, you must've missed it then. Regardless of the sustainability of corporate strategy, the Solaris kernel team pushed out a solid fs and David and Andy capitalized on it. ZFS is technically great, but like the awesome software no one knows about, tying it to Solaris hardware was a mistake. They should've spun ZFS off as freemium open source (against the wishes of the Solaris team./?) almost like what FreeNAS is now.

I think you're reacting to me saying "I've never seen ZFS in the wild" as "I've never seen a Solaris box in the wild." It sounds like we can agree that "explode" doesn't really cover the story of ZFS adoption.

It's become pretty popular in a number of circles, and has been for many years at this point - rsync.net offers ZFS sends as an option for backups, for example, which is not a service you implement unless you think it'll be popular enough.

For being the receiver of rsync backups, and for long term archival storage, ZFS is really best in breed.

Reply on news.ycombinator.com