Hacker News Reader

Comment by shmerl | Hacker News Reader

Do they explain the reason why they needed a new filesystem instead of using some existing one (same OpenZFS)?

We can speculate on the management decision, but from an engineering point of view Dominic Giampaolo said that he didn't dive into btrfs, ZFS, or HAMMER in order to not get potentially bad influence. At least that's how I read his answer as cited by Adam. It's interesting that Larry (Oracle CEO) being a good friend of the late Steve Jobs didn't result in finding an acceptable ZFS licensing agreement for Apple. I mean, they've incorporated DTrace (into the kernel of all things), so why not ZFS as well?

Another possible interpretation of "to avoid becoming tainted" is that it refers to avoiding "residual knowledge" that in some circumstances can be used to make a copyright claim. E.g. if someone reads the ZFS source code to see how they solved a particular problem and then goes and solves it the same way that residual knowledge might cause legal problems.

That being said IANAL and have no idea whether residual knowledge has been explored w.r.t. the GPL or CDDL.

> from an engineering point of view Dominic Giampaolo said that he didn't dive into btrfs, ZFS, or HAMMER in order to not get potentially bad influence.

I'm not sure I quite understand this approach. It's sounds like a pure NIH kind of method (which is admittedly common for Apple). I.e. of course one can always try to reinvent the wheel, but why is it bad first to analyze what already exists and to evaluate good / bad sides of that? Or his approach is simply always to make everything from scratch and not to look at anything else?

>why is it bad first to analyze what already exists and to evaluate good / bad sides of that?

It's a legal defense strategy.

ZFS is covered by multiple patents.

If someone who have never read about any of ZFS's designs and patents independently reproduces one of ZFS's patented features, then the courts could rule that that was non-infringement.

That's why the author said of Giampaolo: "...but didn't delve too deeply for fear, he said, of tainting himself". Reading too much about a patented product effectively taints yourself from being able to freely create your own designs.

So you think it's Apple's paranoia? They should blame themselves for contributing to this sick patent climate.

Independent reinvention is not in general a defense to a claim of patent infringement. Is there some fairly specific set of circumstances you're referring to here?

No, but it helps you to avoid the potential treble damages bit for wilful infringement.

I understand what you're saying, but I also find it funny that Apple would try to avoid patents. If you're saying that Apple is happy to patent stuff but not so much when it comes to using patents by others, maybe you're right. I wish filesystems were more like media codecs in that Apple had no choice but to support an industry standard.

I don't believe Dominic and the team were in NIH-mode. There are most likely other factors which due to Apple's secrecy, and this not being a public project like Linux or Firefox, we may or may not learn. Though, with the information we have right now, it's easy to come to your conclusion.

A tracing framework is much easier to rip out if it presents legal difficulty than a filesystem.

How so? If anything, ignoring ZFS's cache subsystem, a filesystem is another module in a pluggable system if many fileystems. DTrace on the other hand to be most effective requires adding probes all over the kernel.

You're forgetting the weight of data and user practice for a mainstream project. Consider how those two scenarios would unfold:

Apple makes a mistake in 10.11.6 and ships a kernel without DTrace. Everything runs but a few nerds notice and file bug reports. 10.11.7 quietly ships and all is right with the world.

In contrast, shipping a new filesystem is enormously invasive: billions of hours are spent globally rewriting EVERY storage pool in existence. Some percentage of those will fail due to hardware problems or corruption which happened years ago (or even somewhere else) but was previously unnoticed, flooding support and the news with dire predictions. Every crash or performance issue noticed for the next year will probably be vocally blamed on the new filesystem, even if there are clear signs pointing elsewhere.

You don't want to deal with that any more than you have to.

Apple will _not_ remove the existing filesystems, and neither will they automatically migrate an existing HFS volume to APFS without first asking, especially not during an OS update. There are many filesystems in the kernel, and compared to DTrace probes, they are very minimally invasive. Adding DTrace probes means touching all kinds of places in the kernel, so it's definitely more invasive.

Again, look at it from the perspective of a user or system administrator rather than a kernel developer. Yes, someone has to remove a bunch of easily searched for and removed code and rerun their test suite but that's a handful of people for a very small amount of time compared to the number of people involved with something which will make a major change on hundreds of millions of running systems. More critically, the development work is completely under Apple's control and shipping can be delayed as long as it takes to complete testing but once they ship a filesystem it will very quickly reach the point where it will be in the wild for years even if they immediately ship a new release which takes a different direction.

The question isn't whether Apple will force this on users soon or without notice but rather the observation that they're going to be very careful not to do this more often than actually necessary. That's why despite having implemented and shipped ZFS in the 10.5 beta series they removed it prior to release. Sure, it might have gone okay but if that changed later there would be no easy way to go back without forcing users to migrate existing data and if that hit a licensing/patent case, the other side's lawyers enjoy the extra leverage which that would give them.

As an aside, “There are many filesystems in the kernel” is technically true but misleading: HFS+/HFSX is used constantly on every device, FAT/ExFAT are used regularly by many users, and everything else is a rounding error. A few Mac users use NFS or SMB, even fewer use UFS, etc. and no iOS device uses any of those.

I agree with most of that, but I don't understand why you think DTrace is less invasive than another filesystem. They could have shipped ZFS all along and only now declared it stable, if they wanted to. They also could have integrated it again, now that it's actually more portable than before due to OpenZFS. I don't use Macs, so I don't have a direct interest in ZFSonMac, but using a filesystem which can be accessed from Linux, illumos and FreeBSD has some practical aspects which are definitely missing with APFS. Someone will eventually, unless Apple prevents it with IP and Patents, implement APFS for BSD or Linux, but just as NTFS-3G, it will never be fully stable and reliable as a Mac version of OpenZFS could have been. Apple's own APFS implementation will be stable of course, and for other versions to be good as well, they would have to open source the critical bits.

I think the point is that if Apple had to remove DTrace for legal reasons, the impact to Apple's business and customers would be minimal. If a hypothetical AppleZFS shipped on a billion devices and then Apple lost a patent lawsuit, Apple would be totally screwed. A filesystem can't just be pulled out of the OS without shipping a new one and converting every device. Apple could potentially be in a position where they couldn't sell any new devices at all, and ZFS support in other OSes wouldn't help at all.

That's a fair point, and I'm sure Apple didn't integrate DTrace without checking the legal ramifications.

For one, you're not going to put ZFS on a watch. They want a file system that can scale to all their devices.

What about BTRFS? It's more lightweight. Or it's very Linux specific?

And I'm not sure you need snapshots and many of those industrial features on a watch anyway. Sometimes simplicity is actually a plus. So who said they need "one shoe fits all" to begin with? It's never a good approach.

I don't think BTRFS has all the features they want. One thing you can do with APFS is do an upgrade of an HFS+ partition to APFS. That might be difficult with BTRFS.

You don't necessarily need snapshots on a watch, but I can see it coming in useful on a phone when you're editing pictures and video.

Snapshots on a watch could be useful for system recovery. Say something went wrong during an update you could potentially use the snapshot to revert to a previous version.

Filesysytem snapshots are useful for recovery, but it's a very powerful mechanism for it. I.e. I think for a watch it's an overkill.

On the other hand, computers are gradually moving towards miniaturization, so I suppose all this really is quite transitory, and soon enough all such considerations wills simply be irrelevant.

BTRFS can upgrade from ext4 by slotting itself into free space and marking the old metadata as a special snapshot. The same thing should work on other filesystems with few new challenges.

Support for that has been dropped within Btrfs AFAIK. Also, "upgrading a filesystem in place" is more of a lottery than a feature that people should actually use. So many bad things can happen.

Well I did it a month ago, on a reasonably-updated system, at least.

The main point is that upgrading "in-place" does not require any of the metadata to be in the same place between old and new filesystem. With a mildly flexible destination filesystem, the old filesystem can stay there until you're completely sure the conversion is a success. You end up with two read-only filesystems sharing a partition, and you choose which one to go forward with. Cancelling at any point is trivial, even if the conversion process crashes.

BTRFS is GPL licensed and thus Apple can't touch it with a 10 foot pole.

Apple can touch it, they choose not to. They choose to make exclusively proprietary software.

No they don't. They use and license various pieces of software that's under BSD and other similar licenses. For instance, they released Swift at the end of 2015. It's on GitHub and has an open development process. Plus, they use GPLv2, just not GPLv3.

I was specifically referring to their kernel when talking about "exclusively proprietary software" (which is what matters in discussions about porting btrfs). Sure, they've liberated some software but their entire stack is essentially proprietary.

I believe that https://opensource.apple.com/source/xnu/xnu-3248.20.55/ is the kernel source for the current version of of Mac OS X 'El Capitan,' and I believe that https://opensource.apple.com/source/xnu/xnu-3248.20.55/bsd/h... is the source for the current HFS+ filesystem. Is that not correct?

Much of the operating system is proprietary, but I didn't think the kernel or HFS+ filesystem were included in that.

I was under the impression that the released sources for the kernel are not complete (they are missing critical features), but I'm not sure whether or not this is the case with the repos you linked. To be clear, you still need proprietary software to compile it (so it's not practically free under the FSF definition).

> Plus, they use GPLv2, just not GPLv3.

Also, btrfs is GPLv2. So that agrees with my point -- if they didn't mind using free software they could use that code.

"Or it's very Linux specific" Bingo. One of their design goals was to re-use as much existing Linux code as possible. It seems like if they succeeded it would be very hard to port it to another OS.

Terrifyingly, ReactOS and Windows have a Btrfs driver. [1]

[1] - https://github.com/maharmstone/btrfs

Reply on news.ycombinator.com