Hacker News Reader

Comment by crawshaw | Hacker News Reader

Why did Apple abandon efforts to use ZFS? Licensing?

I worked on Apple’s ZFS port, and it was indeed licensing. Apple’s lawyers took a closer look at the CDDL when we were close to shipping and found some provisions that were unacceptable to them.

But were acceptable for DTrace?

Different risk assessments. DTrace is a non-critical tool, and a totally standalone one. If it suddenly had to be yanked out of macOS because of licensing issues, there would be gnashing of teeth but life would ultimately go on. The filesystem is a different story: if something happened and it had to be immediately removed, Apple would be up shit creek.

It's not totally standalone (it has kernel components, for example), but yes, it's clearly something far easier to suddenly drop.

I think the point is you can rip the kernel components out, and the (non-Dtrace using dev) user doesn't notice. Whereas removing a filesystem driver leaves users without their data.

Mojave drops support for DTrace probes in Instruments, so this process may already have begun.

Really? Is there a replacement?

Instruments in Xcode 10 has a new way of constructing "custom instruments". I haven't looked into the details though, so I don't know how much of the old DTrace stuff can be replicated.

Very few people internally to Apple cared about DTrace, and the person who was primarily responsible for evangelizing it left that particular role years ago. DTrace support has been slowly degrading over time (to the point that Apple has ended up shipping DTrace scripts on the system that do nothing but spew errors if you try and use them, as the probes changed and nobody updated the scripts).

It's a shame, because DTrace is cool and very powerful, but it seems almost nobody bothered to actually learn how to use it or cared about supporting it.

I've been using DTrace on High Sierra a bit for the past few months, and it has been working fine for me. I'm writing my own scripts though. I did notice that the sample scripts often did not work, but did not think that meant much.

DTrace is definitely cool and powerful. I'm surprised to hear that few people at Apple cared about it - I'd have thought it was an essential tool for them to solve certain kinds of problems. It's funny, because the tool itself is relatively simple. It seems to me that most of the knowledge needed to use it effectively is an understanding of the code being instrumented, rather than of DTrace itself. Hopefully the sample scripts degrading does not foreshadow the probes themselves degrading too.

I'm not just talking about "sample scripts". The system ships with a few DTrace scripts in /usr/bin as tools and they're rather broken. For example, running /usr/bin/iosnoop just prints an "invalid probe specifier" error. I haven't done an exhaustive catalog of all the /usr/bin DTrace scripts but last I checked there's at least one other broken one as well.

Trying to use DTrace with SIP also warns that "some features will not be available", though I don't know offhand what DTrace features don't work with SIP.

I think your problem with /usr/bin/iosnoop might be due to SIP. I no longer remember exactly which I did, but I know that I either partially or completely disabled SIP when I started to use DTrace, a number of months ago. I might have used one of the procedures outlined in this post [1] to do so. I do know that /usr/bin/iosnoop seems to work for me - at least, I just this moment invoked it, and got regular DTrace-style output with no errors like you mention. I am running an up-to-date High Sierra (10.13.6).

Edit: SIP is only partially disabled on my system, just enough to allow DTrace to work.

[1] https://web.archive.org/web/20160809042951/https://internals...

Yep, this was more or less exactly how the conversation went.

What about this? Does this change anything?

https://gnu.wildebeest.org/blog/mjw/2018/02/14/dtrace-for-li...

No, for a lot of reasons:

1) AFAICT they didn't relicense any of the ZFS code, just dtrace.

2) Even if they had, GPLv2 code still wouldn't be able to go into xnu.

3) Even if it could, Apple has long since moved on from ZFS. They've taken the time to write a new filesystem (APFS) already, so it's hard to see them having any interest in ZFS at this point (and yes, I know APFS doesn't have all the features of ZFS).

Sorry, I meant the dtrace code, not the zfs. Thanks for the answers.

Doesn't instruments and activity monitor build on top of Dtrace in MacOS? Which if true doesn't seem so easy to rip out.

You're still comparing, at most, a handful of applications that would need to be removed or updated vs. an entire filesystem that everything is built upon.

Sure certainly less effort, I guess I was responding to the previous poster saying Dtrace was a "standalone" program, because other programs build on top of it.

Cool, as someone still working with that code (a little bit still remains) thanks for your work :)

I always assumed it was Oracle buying Sun and Steve Jobs NOPEing far away from the prospect of putting an Oracle filesystem at the heart of their then-largest product. Oracle is really good at using their products as beachheads to vampirize large companies. I hesitate to even imagine what would have happened to Apple had they gone forward with ZFS.

Jobs and Ellison were friends. This is why their companies are so similar in terms of business strategy. https://mashable.com/2013/08/13/larry-ellison-remembers-stev...

In what ways are their business strategies similar?

Heavy investment in sales, marketing, and litigation relative to R&D.

That sounds more like a just-so story than some fundamental similarity in overall business approach. And what would a 'heavy investment in litigation' even look like? There's a chart of (absolute) R&D numbers here.

https://www.recode.net/2017/9/1/16236506/tech-amazon-apple-g...

> That sounds more like a just-so story than some fundamental similarity in overall business approach.

And yet both companies' business strategies are primarily known for those things, and both companies spend more of their money on those things relative to R&D than their competitors.

> what would a 'heavy investment in litigation' even look like

Like this: https://www.theguardian.com/technology/2013/jan/23/steve-job...

The fact that Jobs would even threaten such a thing shows how similar he is to Ellison. The only reason he backed down was that Palm would have taken Apple to the cleaners in the countersuits.

And yet both companies' business strategies are primarily known for those things

I don't think they are and really, it's the thing I'm asking you to defend which I don't think you can do very well by just repeating it.

As to the other thing, ok, Apple was involved in a lawsuit? Lots of big companies are involved in lawsuits. What you are saying is that Apple and Oracle's business strategies are marketing and suing people and that they are known for this. That doesn't sound right at all.

> What you are saying is that Apple and Oracle's business strategies are marketing and suing people and that they are known for this. That doesn't sound right at all.

The reason I repeated what I said is that you keep repeating what I said incorrectly, even after I emphasized the part that you left out. The key phrase is relative to R&D.

> As to the other thing, ok, Apple was involved in a lawsuit?

Is that what you got out of it? The (maybe too subtle) point is that Apple is so quick to litigate that it even threatened to bring lawsuits against another company for not participating in its illegal wage suppression agreement. Jobs would rather spend money on lawyers than on poaching engineers away from competitors or paying his own engineers enough to stay.

The key phrase is relative to R&D

Great. And what's the evidence for this?

Apple is so quick to litigate

Again, lots of big companies are litigious. The salary-fixing thing involved a veritable who-is-who of big SV companies. Good thing? No. But Apple doesn't stand out more or less than the rest with its badness. Your claim isn't that Apple did some lame thing but that Apple and Oracle make that bad thing a central part of their business strategy.

> The salary-fixing thing involved a veritable who-is-who of big SV companies.

Yet not a single one of them threatened to sue other companies to enforce the agreement except Apple.

As far as marketing, find another tech company that does https://www.cultofmac.com/255618/how-apples-blacklist-manipu... and https://www.zdnet.com/article/apple-prs-dirty-little-secret/. It brings to mind Ellison's retaliation against the University of Wisconsin for publishing database benchmarks (https://danluu.com/anon-benchmark/). In Jobs's and Ellison's eyes, it's better to force the press not say anything bad about your product than to actually produce a good product.

It's interesting that you're only asking for evidence about Apple and not about Oracle. My hypothesis as to why is that their press policy works.

Nah: http://usingapple.com/2011/06/funny-organizational-chart-for...

Actually the opposite is true, Jobs and Ellison were notoriously great friends - they were from the same generation, the "pirates" usurped by Gates. They just made sure to swim in different seas, as big sharks do.

Apple simply gave up on Sun, seen as a losing bet, and abandoned Java too.

They were friends, but there was a lot of rumors going around that Jobs didn't want a business relationship with his friend. The strange part is that Ellison was put on the new Apple board when Steve took over[1], and the multiple stories of Ellison wanting to lead a takeover bid of Apple and Jobs stopping that cold.

1) Check the crowd reaction when Ellison is announced as part of the new board https://youtu.be/PEHNrqPkefI?t=11m40s

The Apple/Java story is still interesting. I now personally don't care much about Mac OS and Java anymore, but I remember me being pissed when Apple abandoned shipping Java, Cocoa bindings, native LaF for Swing and other integrations around 10.7 and leave the field for Oracle, after SJ had promised to deliver "first-party, best-of" Java dev experience years earlier. Oracle had also published a dev build for the Oracle RDMS on Power-based Macs, but then denounced the Mac as a serious platform. Maybe a back-deal or bet was in effect here, or Apple figured they'll never get into datacenter and enterprise anyway?

Steve was once asked about Apple's commitment to Java by one of the engineers working on it. SJ called Java "a big, fat pig." It was on its way out of the OS within the year.

Yes, I was a bit annoyed about that, it came just as I was finishing off a commercial Java application.

That's interesting, to me it seemed the other way round - it was good for Oracle to be responsible for their runtime. The integration was always a really terrible fit and without it, Oracle could ship current versions of Java. The Apple/Java thing was just a bad idea with a long lifetime and somewhat sluggish death.

It was the plan B in case Mac OS developers wouldn't be willing to use Objective-C from the new shinny OS X stack.

I also got pissed off, but I do get the reasoning behind it.

Apple only bet on Java, because the hype wave was still going strong and they were quite unsure how well the Mac developer community would be willing to embrace Objective-C.

So for a while they kept both languages on the platform to see which one would get the uptake.

When they saw that the developers had no issues with Objective-C, the decision was clear.

Jobs and Ellison being friends doesn't preclude Jobs NOPEing away from a position of extreme vulnerability wrt Ellison. Quite the opposite, I'd imagine.

Apple wanting to use ZFS, and then abandoning the idea, predated the Oracle acquisition.

By a couple of months, easily a narrow enough margin to be accounted for by insider foreknowledge.

Apple could have negotiated what it needed, if that's what it was. From what I can recall the issue was indemnification.

Lots of theories but it may have been licensing, a possible NetApp patent infringement lawsuit, or maybe they thought they could do better.

https://arstechnica.com/gadgets/2009/10/apple-abandons-zfs-o...

https://storagemojo.com/2009/08/31/why-did-apple-drop-zfs/

There's also the theory that it was Jonathan Schwartz's (Sun's then CEO) "pre-announcement" that led Steve Jobs to throw it all in the bin:

> In fact, this week you'll see that Apple is announcing at their Worldwide Developer Conference that ZFS has become the file system in Mac OS 10

That was the theory I’d always heard, but it seems like such a huge thing to pull last minute out of spite.

Some of the other comments about licensing issues make more sense, although that announcement may have been the final straw.

If it was one thing, I would agree but Apple/Jobs was known to do this, look at nVidia history as well.

The licensing terms wasn't a dealbreaker on its own, both Oracle and Apple could've come up with a different agreement (like Oracle would protect Apple if they were sued) but that announcement was just a massive no-no regardless of how friendly the company is with Apple.

I know they tended to do this, but ATI was a pretty direct replacement to nVidia, so the Switch wasn’t as big a deal. They’ve certainly done it with other things too.

Giving up on ZFS meant they had to stick with the ancient HFS+ and start a VERY long project to make a new FS.

In the long run, they ended up with their own FS that they can scale from all of their platforms (tvOS, iOS, macOS, watchOS) and pulled it off without any serious issues, which is impressive on its own.

It's going to be interesting to see how well they optimize APFS in the following years since the current performance level isn't that much better.

It’s fits their tendency to want to keep everything in house as well.

But I would have preferred to have ZFS while they worked on their own FS. And I still want data integrity to protect against bit rot whichever APFS does t provide.

IIRC Apple wanted Sun to indemnify Apple as to patent lawsuits threatened (FUD) by NetApp/

As others have noted, there were various complications around ZFS that Apple may well have decided introduced too much business risk. It's also the case that, while ZFS was and is a very good server file system, a lot of its capabilities are much more relevant for large servers than they were for a desktop PC or laptop.

This is a perpetuated myth that is patently false: ZFS loves cheap disks was the maxim when ZFS first came out, and with block checksums, explicit flush requests directly to the drive electronics, a setting for keeping redundant file copies on a single disk and self-healing during a resilver, it couldn't be more wrong. ZFS the is ideal filesystem for desktops and laptops running the cheapest, shittiest consumer grade drives. It is the only filesystem / volume manager capable of guaranteeing data integrity on such shitty, unreliable drives like consumer grade drives are.

What myth?

I wrote "a lot of its capabilities are much more relevant for large servers"

Which is true. Very large scale, integrated volume management, tiered storage, etc. There are a lot of features on ZFS that either aren't relevant at all or are relevant to a vanishingly small percentage of MacOS users.

You're right that ZFS has some nice data integrity features. Apple apparently didn't think those were a sufficiently big deal as a practical matter to offset the other issues.

"I wrote "a lot of its capabilities are much more relevant for large servers" Which is true."

That is a myth. It's simply not the case. What you could argue is that if you solve a problem for servers, it is also solved for consumer grade hardware, but writing a volume manager / filesystem for servers is empathically not what Bill Moore and Jeff Bonwick set out to do. Jeff wanted to solve the problem of data integrity in a filesystem (his own words), not explicitly design a solution relevant for servers. ZFS is perfect for laptops and desktops. See also https://wiki.illumos.org/download/attachments/1146951/zfs_la...

...if you still have difficulty with the concept of ZFS being perfect for shitty consumer grade drives in desktops and laptops, do this mental exercise: which features of ZFS would make it not be a good fit for a cheap consumer drive?

Tiered storage appears to be relevant for desktop systems given the number of "SSD-cache for your harddrives" solutions around (although maybe not for Macs, given their hardware limitations, on the other hand nothing strictly prevents using something like that with external disks).

Snapshots are quite powerful for backup/recovery features.

As long as you disable the disk write cache.

I had a home Nas running zfs on a unreliable power supply and I lost the contents of the Nas a few times before I disabled the write cache. Ext4 seems to tolerate this much better.

> As long as you disable the disk write cache.

> I had a home Nas running zfs on a unreliable power supply and I lost the contents of the Nas a few times before I disabled the write cache. Ext4 seems to tolerate this much better.

Maybe it is because Ext4 by default "bypass"[1] write caches by default.

[1]: Look at barriers in Ext4 documentation: http://man7.org/linux/man-pages/man5/ext4.5.html

Thank you.

I wondered why ext4 tolerate sudden power loss much better.

It doesn't tolerate power loss at all; it doesn't have any kind of facilities to do that, and write barriers will only work if the underlying storage supports it, otherwise ext4 will revert to no write barriers, so it's spotty at best, hit-and-miss.

There is no way to detect silent data corruption on any generation of an ext filesystem and if you lose power, you've not only lost data but it has most likely been silently corrupted as well. ext filesystems will be none the wiser to this. Same with XFS, UFS, or any other classic filesystem, but ext family of filesystems is particularly shoddy in that regard.

Write cache should always be disabled, even if one is running multimillion dollar storage arrays, no ifs, buts, or maybes. That goes without saying, it's the holy law of data integrity to any storage architect who understands the intricacies of storage systems.

However, it has nothing to do with ZFS per se.

> ZFS loves cheap disks was the maxim when ZFS first came out

This is the first time I've heard this expression.

So, you're telling me I shouldn't have bought all those WD Red drives for my ZFS machine?

"This is the first time I've heard this expression."

Slide 18, "ZFS loves cheap disks", underlined.

https://wiki.illumos.org/download/attachments/1146951/zfs_la...

I've been running ZFS in production and at home since July of 2006. Just resilvered a bad consumer drive with two bad blocks. All fixed now.

> Just resilvered a bad consumer drive with two bad blocks. All fixed now.

Hm, just makes one put ZFS everywhere!

> The better the drives, the better the performance and data integrity with ZFS, but WD is really consumer grade hardware: Seagate ES or Hitachi enterprise drives have completely different firmware.

Is the firmware the one that really matters here or is it about the quality of the hardware itself?

The mechanical internals between a consumer grade and an enterprise drive from the same manufacturer are usually identical and might only differ in things like read and write cache sizes. Firmware is the only thing that matters, especially since that is the #1 source of the problems.

The better the drives, the better the performance and data integrity with ZFS, but WD is really consumer grade hardware: Seagate ES or Hitachi enterprise drives have completely different firmware.

I imagine they got nervous about the prospect of being dependent on Oracle. They'd be far from the only company to draw away from Sun stuff when Oracle took over.

Did Apple not continue to use Oracle software as the backend to iCloud?

That's not any real dependency as having millions of consumer installations...

The have iCloud abstracted and diversified anyway (also uses Azure, AWS, etc).

They use whatever tool is best for the job. Take a look at their careers site, you'll see they use all sorts of databases internally.

  Sr. Database Architect
  
  REQUIRED TECHNICAL SKILLS :
  RDBMS: Oracle
  NoSQL DB: MongoDB
  Columnar DB: Vertica
  OS: AIX, LINUX, Solaris
  Replication: Streams, Golden Gate

Apple actually no longer uses Azure. They use AWS (S3) and Google Cloud: https://www.apple.com/business/site/docs/iOS_Security_Guide.....

Wrong link?

No. The section about iCloud says:

  The encrypted chunks of the file are stored, without 
  any user-identifying information, using third-party 
  storage services, such as S3 and Google Cloud Platform.

The 2017 version of this document had "Azure" in this list. The 2018 version replaces it with Google Cloud.

Yeah, but it also says "such as", so it might not be exhaustive.

Oh, okay, I only searched by "Amazon", "AWS", and "Azure", and didn't find anything.

Even today if you look at the headers in a message to or from an iCloud email account, it indicates "Oracle Communications Messaging Server" and years ago it was the same, but replace Oracle with Sun. I think they may well have been using the same thing since iTools email started back in January 2000.

Reply on news.ycombinator.com