ZFSin: A Windows kernel-mode ZFS driver | Hacker News Reader

ZFSin: A Windows kernel-mode ZFS driver

github.com

265 pointsFilligree108 comments

Wow.

Of the 26 items in the Milestone section, three (near the bottom) are currently not done. All the rest are. (The last one should read "swim". And I hope that last one definitely happens!)

The first substantial commit (https://github.com/lundman/ZFSin/commit/6ea8093d48da0a0abc3b...) includes a large pile of code. It's clear this was (wisely) built out privately to begin with. I wonder how long it took to get to the above commit point.

While unrelated, thanks very much to the author for the TIL on how to do Windows kernel development! TIL about the free Windows VMs too.

I've opened https://github.com/lundman/ZFSin/issues/6 to hopefully prompt a license file to turn up.

> TIL about the free Windows VMs too.

Not quite free, just an evaluation version that expires after a while.

Ah. I'd middle-clicked that link but not actually looked at it before hitting Submit. :)

That page says the images expire 12/1/17. So that's... two months from now? Huh. That's... 341.33MB per day in bandwidth (20GB over 60 days). Weird way to measure it (the cost of continually reconfiguring would theoretically be higher), but I had a really tiny Internet cap for a few years some time ago.

Windows licenses aren't that expensive. If you need a copy of Windows more often than the trial image makes feasible, you should probably buy a copy.

Fair point.

Glances at nonexistant bank account - and computer from 2006

A Windows 10 Home edition costs 145 euros, pro is 260 euros, it's not exactly impulse purchase territory for me.

I never use Windows because I want to, I use it because I have to. So paying hundreds of euros to have the privilege to use an OS I don't care for just because some developer decided not to make their application portable is very hard for me to swallow.

Especially now with the built-in ads, telemetry, OneDrive and whatnot. You're not going to guilt me into purchasing a copy of that garbage fire of an OS just so that I can boot up the odd windows-only application every other month.

The cheapest version of macOS for anyone that needs to use it, costs 570 € in Germany (Mac mini) for the very first version, excluding the free upgrades.

Any other commercial OS at enterprise level is way beyond that in developer licenses.

And it comes with a very elegant and compact computer totally for free.

Still not worth it.

That's debatable. I kind of like mine. Not as fast as its big brother, but that was never the intention.

And the OS is a beautiful successor for NeXTSTEP.

That's not a very fair comparison. And while I'm sure MacOS isn't exactly cheap I never encountered a situation where I had to use some piece of software that required me to run MacOS.

For what it's worth I was mainly thinking of the Linux/BSDs that I actually want to use when given the choice

It's free forever, easily customizable to fit my various needs, from embedded barebones application to media server to workstation. I actually own and control the OS (my debian doesn't upgrade to a new version behind my back) and I can even dig into the code if I want to.

I'd sooner give 100 euros to the FreeBSD fountation than 10 euros to MS frankly. I gave them enough money indirectly already with all these prebuilt computers I've bought in the past decades which came with Windows preinstalled, only for me to immediately wipe them with something else.

That's just your particular case and not indicative of the situation as a whole. I regularly encounter software that requires me to run macOS. I build virtual and augmented reality software. macOS is garbage for VR, all the major VR systems work better on Windows. But, if you want to build AR apps for iPhones (or any sort of game app, doesn't have to be AR), you need Xcode, and Xcode only runs on macOS.

Well you've convinced me, if I need to buy an OS to run Windows programs, it sure won't be MacOS.

> Any other commercial OS at enterprise level is way beyond that in developer licenses.

Huh? If you need one license, Windows pro is much cheaper? Along with visual studio community edition etc.

It really annoys me that Apple refuses to license OS X for non-appearance hw though. Might even eventually have bought some Apple hw, if I could start developing for ios/OSX without having to start with buying a computer I don't need. (I might have bought an iphone, for example).

For other commercial OSes I mean commercial UNIX flavors, mainframes and support contracts for Linux variants.

That's just wrong. You could (or still can?) buy mac os installation dvds in an apple shop for 30 bucks (iirc my sister bought one a few years ago) also the "upgrade" to sierra is free without any limitation on how many machines you can legally install it on so you can create an installation medium (you can install it in qemu or probably even use coreboot with a modified tianocore to actually install it). Oh and you can download the source for the kernel (I'm not sure which versiin this really mirrors and I'm not a mac guy so I don't care) here https://opensource.apple.com/ Just because you can't easily buy a copy of it at best buy and install it without some hacking on any random computer doesn't mean that you can take the price of the cheapest mac _hardware_ say that's the price of the os and call it a day.

Also your comment did completly miss the point of the parent comment.

> also the "upgrade" to sierra is free without any limitation on how many machines you can legally install it so you can create an installation medium (you can install it in qemu or probably even use coreboot with a modified tianocore to actually install it)

There may not be a limit to how many machines you can legally install it on (I'm actually not sure), but the license is very clear as to what machines you can install it on (i.e. only Apple hardware). Though in reality it's been quite some time since Apple tried to enforce that provision very hard, and never on individual users installing it themselves on usual PC hardware.

> Oh and you can download the source for the kernel (I'm not sure which versiin this really mirrors and I'm not a mac guy so I don't care) here https://opensource.apple.com/

The kernels are usually pretty up to date, but the public code is an even smaller portion of the OS than the Linux kernel is (many key drivers are not open source, and forget anything approaching the level of the UI).

I just went into my study and got my original snow leopard dvd (the one my sister bought) and I can't find the part that states this.

> but the license is very clear as to what machines you can install it on (i.e. only Apple hardware)

Would be nice to get a reference on that. But ignoring this my point still stands that pjmlps comment was more a misplaced snarky remark against apple that missed the point. But hey I neither use Mac OS nor Windows (only in the circumstanced when I'm forced to offer support) so I don't really care :)

But a PS: Most hardware you buy comes with a Windows license so people who don't want to buy Windows can probably still use this.

It's in the EULA:

> 2. Permitted License Uses and Restrictions.

> A. Preinstalled and Single-Copy Apple Software License. Subject to the terms and conditions of this License, unless you obtained the Apple Software from the Mac App Store, through an automatic download or under a volume license, maintenance or other written agreement from Apple, you are granted a limited, non-exclusive license to install, use and run one (1) copy of the Apple Software on a single Apple-branded computer at any one time. For example, these single-copy license terms apply to you if you obtained the Apple Software preinstalled on Apple-branded hardware.

There's also section 2B(iii), that is for copies obtained via Apple Store, that allows you to run virtual machines:

> iii) to install, use and run up to two (2) additional copies or instances of the Apple Software within virtual operating system environments on each Mac Computer you own or control that is already running the Apple Software, for purposes of: (a) software development; (b) testing during software development; (c) using macOS Server; or (d) personal, non-commercial use.

Emphasis mine.

I bought an install disc for Snow Leopard (OS X 10.6) back when that was the current version.

Seems they have it in the offical online store still actually: https://www.apple.com/shop/product/MC573Z/A/mac-os-x-106-sno...

However when I inquired at an authorized store about a later version after it came out they said that it was not available on disc and you had to buy it via the Mac App Store which means you need to have OS X in the first place but I'm not sure if they were talking about their own store or in general.

I'm on mobile right now so not going to bother finding out what other versions can be bought on disc media from the official online store but if someone else would like to chime in with more links that'd be welcome.

Legally I can only use Apple hardware to run macOS.

So anyone that cares about following the law needs to buy an Apple computer to run those DVDs you mention.

Hackintoshes have no place in compliant businesses.

Depending on jurisdiction you may be able to buy a "used" license. In that case, you're looking at maybe 20-30 Euros for Windows 7 Pro.

OEM variants without the media and box are £25 on Amazon. Worked fine for us. Came as a license key from a VAT registered reseller and activated just fine.

be careful, those discounted keys are likely coming from bizspark/msdn accounts. they will work fine, but if you're a business and get audited you will fail the audit. in that sense you might as well use a KMS activator.

The trial images are not for you to use other people's software, they are for you to test your own software, especially for testing website compatibility with Internet Explorer. If your job requires testing more often than the trial image makes convenient, you should make your client pay for the Windows license. And if you're just using the OS, I really think you have a responsibility to pay for it.

And all that other stuff can be turned off.

If you don't need the support and don't need a physical boxed product, you can likely bring this cost down to about £40. Look for OEM.

The VMs for IE/Edge testing purposes seem to be periodically updated, once the current images stop working. I also remember seeing instructions on how to extend the trial period somewhere within the VM, similar to what dexterdog describes in the sibling comment.

https://developer.microsoft.com/en-us/microsoft-edge/tools/v...

If done right, the cost can be close to zero. See :

- https://github.com/GoSecure/malboxes - https://github.com/joefitzgerald/packer-windows - https://www.packer.io/intro/why.html

You can usually extend the evaluations a few times. The Win2k16 server trial is 180 days and can be extended 6 times with a simple script call. If you cron it right you get 3 years.

Oooooh. That's very nice! And legal too. Awesome to know, thanks very much.

Looks like I started 2017/02/08

vdev_disk.c spawns a thread to get around that IoCompletionRoutine is called in a different context, to sleep until signalled. Is there a better way to do async in Windows?

Instead of creating a thread and blocking until being signalled in the continuation of an IO, enqueue a work item to continue work at PASSIVE_LEVEL (i.e. schedule a continuation).

Async means continuations, otherwise you're blocking.

https://docs.microsoft.com/en-us/windows-hardware/drivers/ke...

I'm not ZFSin's creator, and there's no guarantee they're reading this. If you want to inform him, you should probably file an issue.

Looks like the author is here and already responded to his comments.

This would be perfect to dump into a new issue

I tried that way first, but because it called the callback at different level, and ZFS grabs a mutex, it just died. Which why I changed it to spawn a taskq to wait for the callback to come back, then tell ZFS about it. So it is async right now, I'm just not happy it has to create yet another taskq to handle it, for each IO request.

I'm far from expert at NT kernel programming, but I do know that there's not much you can do in the callbacks from interrupts, which is why you use a deferred procedure call (using Io/ExQueueWorkItem) to continue at a level where you can do more things, like potentially blocking calls. The work item execution of the DPC proceeds on a system worker thread, rather than a thread you had to spawn yourself.

But perhaps that's what you did. If you're still stuck, I suggest you email Larry Osterman (Firstname.Lastname@microsoft.com) for a clue; Larry has worked on the NTFS driver and he's responded to my emails in the past. For an improved probability of a useful response, I'd send him a link to a Stack Overflow question, so that other people benefit from the reply also. (This is a general rule for emailing people for developer help, you'd be surprised how many random people email system / compiler developers for coding assistance.)

Ah that looks interesting. I will research that once I leave SF

Another expert to contact: Trent Nelson from http://pyparallel.org/ a few years back.

He pops up here occasionally to defend Windows unique I/O subsystem¹, and monitors Twitter².

¹ https://hn.algolia.com/?query=author:trentnelson&sort=byDate...

² http://twitter.com/trentnelson

Hardly needs 'defending'. It's superb.

Yes, maybe evangelising would be a better word.

Alas, didn't quite work. Turns out DPC's are not allowed to wait, so the first condvar we hit in zio makes us die :)

These are called Deferred Procedure Calls, the purpose is not to delay but to run in a more relaxed context.

Yes, but I'm pretty sure in the past they have also been called Delayed Function Calls in Microsoft documentation. It doesn't change the semantics, of course. Same goes for the old Symbian OS.

I was kinda bummed when Apple announced that they are going with APFS instead of ZFS. I think ZFS is one of the biggest engineering feats in last decade or so of software engineering. It still needs wider adoption.

How do you think they could achieve wider adoption?

The problem is the license, which ZFS on Linux covers a bit here: https://github.com/zfsonlinux/zfs/wiki/FAQ#licensing

You can see some of the fun hoops you have to go through to get ZFS as your root filesystem here: https://github.com/zfsonlinux/zfs/wiki/Ubuntu-16.04-Root-on-...

For now I just have 4 disks in a ZFS-raid for storage, and then I have installed my OS on a separate SSD.

The license is only a problem for gpl. Note that FreeBSD uses zfs and has for quite some time.

Yes - that is true, and it seems like ZFS usage is much higher there because of that.

Ubuntu has it built in and ready to go. But not for the root drive (that way lies madness).

It's quite excellent for the root drive, you just have to install via debootstrap. I personally do this on all my 1/2 dozen boxes (Straight Debian usually, but some Ubuntu). I've also written up how to build an AWS AMI this way. Good stuff, It's only going to get easier.

> I've also written up how to build an AWS AMI this way

If you've written it up in public, any chance of a link?

from their submission history: https://scotte.org/2016/12/ZFS-root-filesystem-on-AWS

It's not difficult really (assuming one can/is willing to RTFM), just a little more involved and time-consuming to get set up. I've used ZFS on / on Ubuntu a handful of times and all of the machines I use on a daily basis (a high-end workstation and a pair of laptops, all running Arch Linux) all use ZFS exclusively.

I have one (centos) machine with zfs on /. Looking back, it was a mistake - I always dread, when I see grub update in the yum update list, what will break this time.

And of course, after each kernel update, link your /dev/by-id/... devices into /dev and rerun grub2-mkconfig by hand. Otherwise, you won't boot the new kernel you just updated.

If this machine weren't a pet, it would be wiped clean and reinstalled with xfs /.

Wow. That sounds like something isn't configured quite right. Or perhaps ZFS just isn't as well "integrated" on CentOS.

All of my Linux machines with ZFS are running Arch Linux (as well as a few servers running FreeBSD) and I don't have to do anything special.

For example, I'll update them every few days (or whenever I don't mind having to "re-open" all my applications, tabs, etc.) and I just run "pacaur -Syyu", have it update everything, and reboot when it's done. It starts back up, I enter in my FDE passphrase, and it boots right up.

It is actually a known issue between ZoL and GRUB:

https://github.com/zfsonlinux/grub/issues/5

zpool status will show only basename component of the device name and grub will look for it in /dev only. As I created the pool with ata-* device names, which are constant per device, it will of course fail, because they are in the /dev/by-id/ and grub is not looking for them there.

Next time there is a kernel update, I' going to try the env variable mentioned in the august 2017 comment. Hopefully it will work right.

ZFS and CentOS are not a good combination, in my experience. The Red Hat/Oracle feud doesn't give RH much incentive to be ZFS-friendly, and the interaction between DKMS and weak-modules is a fiasco.

ZFS woes, and Red Hat's penchant for yanking hardware support (like a PERC/4e RAID controller I encountered this spring), have brought me to the point that the only new CentOS installations I'll make will be KVM guests.

The last straw for me was the 1708 release, when the ZFS modules failed to build, leaving my home NAS stuck on the old kernel. Out went CentOS, in went Ubuntu Server. No regrets.

In my case, Ubuntu makes PITA running stuff I want to run (especially FreeIPA/sssd/gssapi integration). On the other hand, I'm using kmod repo for ZoL, not DKMS, because I wanted to avoid having a compiler and dev packages on the server in the first place. CentOS/RHEL have kernel ABIs, why not use them.

That's odd. I've been using it on Ubuntu as the rootfs for over 18 months now and have been through 16.04/16.10/17.04/17.10 upgrades and numerous kernel upgrades and it's worked flawlessly every time.

And that's the thing. I used SmartOS for a while and while there's nothing wrong with a ZFS root, per se, you're still at the mercy of boot loaders and whatever Amazon will or will not let you do.

So I've taken to having ext4 or whatever as the boot drive and a zpool for anything I care about :)

Not really, I use it on Arch as my root drive https://ramsdenj.com/2016/06/23/arch-linux-on-zfs-part-2-ins... and never have a problem. It's been useful multiple times for informing me that I have a problem disc, and being able to go back in time and look at how things have changed on my system with snapshots is of great benefit. Being able to rollback, clone filesystems and send them around is great too.

With archzfs I get binary upgrades with every kernel, and I've never had a problem at boot after a kernel update.

The problem back when was that Sun wouldn't indemnify Apple against third party patent claims. Now that Solaris is dead it is unclear what Oracle's intentions are, but in any case, surely Apple would have similar desires.

It's happening via NAS-type devices - a lot of those use ZFS behind the scenes. Once people get used to that kind of functionality, they'll ask why their computer doesn't do that on its local drive, maybe.

It would really be nice if there were a way to shrink vdevs on a live filesystem. That's the one feature that ZFS can't do at all that other filesystems can do equivalents of.

if you count XFS as “that other fs”, then no, it can't

Other FSs can shrink partitions without losing data; ZFS can't. The fact that XFS can't doesn't make my statement false.

Other filesystems will lose your data or write corrupted data

APFS and BtrFS at least won't.

APFS doesn't checksum the data. You've been deceived.

BTRFS still cannot be trusted for various reasons. It was deprecated as of RHEL 7.4

What reasons? openSUSE is defaulting to btrfs. snapper is really sweet.

Because RHEL kernels are very old and the maintenance burden of backporting btrfs changes is too great (and fragile).

That sounds like a problem with RHEL, not a problem with btrfs

For the end user it's a problem with the ecosystem. Red Hat needs to keep the kernel patched for a very long time and some pieces move way too fast.

Right, it's not a btrfs problem other than btrfs is moving too quickly and RHEL's kernel stands still (by design).

Being deprecated in RHEL means nothing; Red Hat didn't have any btrfs people anyway. They have all the xfs people, so that's what they focus on.

As long as you don't use Btrfs's RAID 5/6 features. I happened to be more lucky and my filesystem only locked up read-only on me, and I had to transfer everything off.

The moment Oracle bought ZFS and Java Apple ditched all plans for both.

ZFS would be terrible for most of Apple's use cases. It is definitely designed around the assumption of fairly active sysadmins a la Solaris in 2005.

It's very good at what it does within that assumption, of course. But do you think behaviours like "you filled up too much of your disk, performance is crippled unless you wipe and rebuild" is "Just Works"?

Are they ever going to implement built-in encryption in this Windows ZFS driver? I can't see it in the Milestones listed by the Github project. My pools largely make use of native encryption, so this is quite useless to me without it being able to import keys and decrypt the drives.

Just a matter of cherry-picking it in. I'll get around it it soon

Fantastic. If/when encryption stabilizes that give a solid cross-platform fs with both encryption and snapshot support. Hopefully there'll be something for Apple users as well.

Would be ironic if Linux has so-so fatx support because of patents, and OS X lacks zfs because of patents..

I'm sure "they" would appreciate any help.

Does anyone know anything about this author? Is this likely something that's been developed for a business or is it "just" a hobby project (albeit a seriously impressive hobby project!!)?

Jorgen Lundman is also a main coder for the OpenZFS project afaik. I've seen his name many times since the very early betas of the OpenZFS macOS port that I've been using back in 2012/2013. I don't know how it's financed, but I'd think it's not just a hobby.

Oh that's good. Even if this turns out just to be a hobby project then at least the brains behind it is someone with real experience hacking ZFS.

Just a hobby, working on it because I want to :)

Your hobbies make my hobbies look pathetic. Kudos :D

This is exciting to see. I currently have a separate NTFS disk for Windows on home machine - it would be awesome in the future to share a single mirrored zpool between Linux and Windows.

wait what? Can you do that - do you mean with both operating systems live?

I believe he means dual-boot. Thus only one would be live at a time.

It is unlikely that you could install Windows on anything but NTFS. That doesn't mean you can't put your home directory on some other filesystem.

Still, there is a fair amount of fragmentation in the ZFS world. It's possible that it will work fine (provided you got all the feature flags to match), but I am very skeptical that being able to mount in both will happen any time soon.

> It is unlikely that you could install Windows on anything but NTFS. That doesn't mean you can't put your home directory on some other filesystem.

Even Microsoft's own https://docs.microsoft.com/en-us/windows-server/storage/refs... can't be used as a boot drive currently.

Most definitely no.

You'd still need to go for SMB or NFS for that.

If this is possible I would consider it one of the most impressive things I'd ever seen.

ZFS supports connecting a pool to multiple computers at once, but only by way of locking the pool as a whole, and as others have pointed out it's unlikely to work even then. There's too much feature divergence.

NFS is what you're "supposed" to use.

Hah. ZFS really does have every single almost-reasonable feature baked into the design :)

Problem with NFS is that it's really glitchy. For example I cannot disconnect my laptop's (higher-power) USB Wi-Fi dongle when I want to move around, because the PHY disappearing from under my NFS mount produces a kworker that consumes 100% CPU.

The good news: I found a Red Hat bugzilla thread with repro info that perfectly matched what I was seeing, which identified the root cause and generated an upstream kernel patch that completely eliminated the problem. This all happened a couple years ago, for kernel 3.8 or so.

The bad news: I'm using kernel 4.1.6.

(I'll upgrade eventually, and see if the problem has conveniently disappeared. If it hasn't, I'll make the effort to switch to SMB. I'll also see what happens if I trigger this bug on a kernel with NFS compiled as a module at some point... >:D)

There's also interesting technologies that use ZFS as a base layer too such as MarFS which effectively provides network based object storage (or lustre etc) - https://www.usenix.org/system/files/login/articles/login_spr...

> ZFS really does have every single almost-reasonable feature baked into the design :)

Except some of the nice features btrfs has. Like almost arbitrary online restriping; on-demand deduplication, defragmentation and compression of existing files; reflink copy; nocow-flag, ...

Also: advanced/expensive iSCSI options (simplest: read-only)

https://social.technet.microsoft.com/Forums/en-US/21889ca0-2...

I have feeling he just means for the data. ZFS doesn't allow two systems to use it at the same time (unless it is shared through NFS or similar) and booting Windows from ZFS would be another challenge (I bet much more difficult one).

Currently there's no good FS that can work together with Windows and Linux/Unix. I mean the one that works on both would be ExFAT but its age is showing up.

I currently found a workaround to have FreeNAS installed on VirtualBox and then share the volume through Samba, it works, but it is hacky. A native solution would be far better.

He's doing as a hobby but he isn't against any support you can give him https://www.patreon.com/lundman :)

But why? I’m thinking for data recovery. I’m hoping people wouldn’t try to set up a ZFS fileserver within windows.

I would love to have an advanced file system that works on portable drives across multiple OSes.

ZFS is not designed for this use case, but at least it is something. Having compression, encryption, integrity checking, and data duplication on a potentially flaky external drive would be killer features.

That would be awesome. I’d love to have c or root on zfs as well. That could make clients really easy to back up.

But wait! WinFS (or whatever its name) is just around the corner!

Sorry. Couldn't resist. Microsoft promises something better than NTFS - something "object oriented" or "database driven" - since before I moved to Windows NT 3.5.

You joke but ReFS has been out since Windows 2012. Combined with Storage Spaces, there is a lot of overlap with zfs.

I will concede it's close to what we have with ZFS, LVM+XFS or BtrFS, but, still, it's not quite up to what WinFS promised in terms of semantic awareness.

Is this not an IFS driver?

Reply on news.ycombinator.com