Hacker News Reader

Comment by chmaynard | Hacker News Reader

The author worked very hard to determine the root cause of the problem but he was stymied because the APFS source code is not available. Why not? Apple has open-sourced the Swift compiler with great success, but there seems to be no movement within Apple to open up other system software components.

View on HN

They do plan to document it fully later:

> Is APFS open source?

> An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format specification.

Source: https://developer.apple.com/library/archive/documentation/Fi...

My theory is that Apple rushed APFS and its implementation isn't in a great shape for the public review. APFS doesn't feel any faster on my SSDs on macOS. I suspect they'll add it in a few years, after a few iterations.

They released the format spec here: https://developer.apple.com/support/apple-file-system/Apple-...

> They do plan to document it fully later:

> > Is APFS open source?

> > An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format specification.

They claim to plan to document it fully later.

The spec has already been released.

It's a partial spec.

What makes you think it’s a rush job?

Perhaps it’s just the lawyer reviews holding it up.

> What makes you think it’s a rush job?

A global lock on readdir().

My personal guess is that they wanted to release the printed spec before they released the code. The spec has now been released, the 10.14 source has not, maybe apfs.kext will be in the 10.14 source drop.

(also, HFS+ and its fsck have always been open-source with OS X)

HFS+ was first released for Mac OS 8.1 in 1998. The earliest source code I see is from Mac OS X 10.0, which was released in March of 2001 (not sure when the source was released). The first iPod was released in October of 2001 which used HFS+. I imagine by the time that code was released it had been battle hardened quite a bit.

Because on two separate occasions, APFS ate all of my free space resulting into me reinstalling macOS as the system was locking up. (Yes, I filed a radar, and yes they fixed it, only to regress 6-8 months later).

> My theory is that Apple rushed APFS and its implementation isn't in a great shape for the public review.

The rollout of APFS was _incredibly_ well done; I don't think there is even a remote comparison to how successful it was. A new file system was rolled out to millions of devices in an automated fashion with very few issues. I don't think anyone else has even considered such a thing. I highly doubt the code is that rough, I can't imagine pulling this off without fairly solid core code in place.

Agreed. For iOS, the rollout was so smooth that most people weren't even aware it was taking place, with the only observable effect being that the iOS upgrade took a bit longer than usual. For the macOS rollout, there were more hiccups, but still, extremely smooth. In fact, I just had to launch Disk Utility to double-check that this computer was in fact migrated to APFS because I couldn't recall actually seeing anything specific about it (as this computer has a fusion drive, as well as a Bootcamp partition).

Mojave update trashed my Fusion Drive to the extent that plugging any other mac into my imac in target disk mode made the host mac instantly kernel panic. And I lost all my files obv.

But you did backup before update, didn’t you?

You mean besides the fact that the volume encryption password was stored in plain text in the disk?

https://thehackernews.com/2018/03/macos-apfs-password.html

The APFS source code isn't available, but you're free to disassemble apfs_vnop_readdir in apfs.kext. I'm seeing a couple of calls to lck_rw_lock_shared, so it's entirely possible that there's a lock here.

This. It's what everyone I know who works in the Windows space does when they have a deep problem to investigate. Lack of source doesn't mean lack of ability to investigate, and sometimes the source doesn't tell the whole story either.

(Yes, I know about the anti-RE clauses in EULAs. If they were actually enforced as strictly as they claim, people like Mark Russinovich and Matt Pietrek would've been sued out of existence long ago, along with just about every Windows security researcher.)

The Wine source is often helpful too.

Apple releases most of the macOS kernel but I guess it doesn't include APFS: https://opensource.apple.com/source/xnu/

That's missing a good chunk of the interesting stuff related to how users interact with the os though. (app and framework source code)

Yeah, apps and frameworks aren't part of the kernel. But APFS is.

Kind of talking out of my ass, but Darwin can act as a mirokernel, so APFS could be implemented as a user mode service (no idea if it's the case, though).

Darwin's XNU is not a microkernel, in spite of there being a microkernel version of XNU. While Darwin does support FUSE, performance would almost certainly be inadequate, given the constant transitions between user and kernel space.

I thought it was a mixture of OSFMK and BSD, which is why I said it can act like a microkernel. Don't know more than what wikipedia says, though.

Yes, it is, but the Mach layer and BSD layer were fused into a monolithic kernel before NeXT Step turned into OS X. In OS X, HFS (and presumably APFS) are in the BSD layer.

It's loaded into the kernel, but it's a kext (kernel extension) rather than being part of the xnu source tree itself. Only a subset of kexts are open source. (By comparison, HFS used to be in the xnu tree, but it was moved into a kext as well a few years back; that one is open source.)

Pretty sure the code would expose just how much of a rush-job apple did with APFS. It may be released one day..

Reply on news.ycombinator.com