Apple's APFS Migration: A Feat of Engineering
registerspill.thorstenball.com
registerspill.thorstenball.com
If I remember right, it was said at the time that they had included the migrator code in one or more of the releases before the one in which they actually did the migration. They ran it during those upgrades in a "do as much of the migration as we can without actually committing the changes" mode so that encountered errors could be reported back. So they had most of it tested on millions of devices.
And why would it use a large portion of that, rather than an additional 1 write per cell?
Hover just stops working on all windows unless you right click on another window, or hide windows to view desktop and come back.
Pretty impressive.
My reply is Apple supports close to 2 billion devices. They have to work and never crash. They also need to get upgraded for security patches and new features. The fact they find them "boring" is a good thing. Means iPhones have become an intrisic part of their lives.
This article describes one of the many things Apple has to do to make them "boring". Indeed a feat of engineering.
Now, when they replicate that success with Siri, that will be a game changer. Hopefully advances in LLMs will bring that day forward.
gherkin0 on June 26, 2016 | next [–]
> Fun fact: Dominic Giampalo (who wrote the BeOS file system) is on the APFS team. His book "Practical File System Design" is an excellent description of a traditional UNIX file system design. May be out of print now but I think used copies turn up on Amazon. It looks like he has a PDF up on his website: http://www.nobius.org/~dbg/practical-file-system-design.pdf
Maybe the limited capabilities that mobile device APIs give apps and user was helpful avoiding issues - can't have multiple partitions or weird time machine issues on something without partitions or time machine.
What was the previous filesystem, and how different was APFS?
If they both operated on the same core principles or underlying code, then the migration might have been trivial. Is it just a new "skin" on the prior FS?
If it's a radically new filesystem based on all new concepts, then that's a much bigger feat of engineering.
What did this new FS accomplish? Was it faster? More reliable? Easier to work with? Added functionality?
The old filesystem was HFS+, released in 1998 and APFS was released in 2017.
APFS added encryption, data integrity checksums, copy on write, snapshots, logical volume support. The migration wasn't just a "search and replace the name and ship it", but an intensive re-writing of the filesystem metadata during the migration.
I believe it has checksums on its own metadata, but no checksums on actual user data. So you won't lose a whole folder or volume due to bitrot, but it doesn't provide protection against a flipped bit in one of your photos.
"The APFS engineers I talked to cited strong ECC protection within Apple storage devices. Both flash SSDs and magnetic media HDDs use redundant data to detect and correct errors. The engineers contend that Apple devices basically don’t return bogus data."
Note this continues for a bit and criticises that position. Also see the HN discussion at the time: https://news.ycombinator.com/item?id=11934457
I don't think it's a good decision either; obviously you can run systems without checksums (we've been doing so for decades) but "netter safe than sorry" seems to be the smart thing. It's pretty cheap to checksum data, which is why all the next-gen filesystems (ZFS, btrfs, bcachefs) do so.
Released in 1998 for Mac OS 8.1. That HFS+ worked at all for Mac OS X was a minor miracle; it was absolutely not designed for use on a modern UNIX, and support for some features like deleting in-use files involved some egregious hacks (like temporarily stashing files in invisible directories).
Early Mac OS X did support UFS (not sure which variant, probably an early BSD?) but never fully and eventually removed it. HFS support for backwards compatibility was necessary, but making it the boot FS for so many years did hold the platform back.
There was talk about ZFS at one point, but it never happened - maybe due to licensing. A large variety of FSes is definitely something Linux has over BSD and permissive licensed software. Even ZFS isn't fully permissive which only leaves HAMMER(2) as the FS with next-gen features and a BSD license.
At the same time, it's impressive that the basic design of HFS held up as well as it did! HFS was initially introduced in 1985, and HFS+ was a fairly conservative update to support larger volumes (and, later, metadata journaling).
> There was talk about ZFS at one point, but it never happened - maybe due to licensing.
That seems very likely. Apple's experiments with ZFS ended around 2009, right about the same time that Oracle finalized their acquisition of Sun.
Since then ZFS has improved and machines have become faster.
e.g. case sensitive vs insensitive, different path separators : vs / , lack of file IDs on UFS, resource forks, hard links and the above mentioned deleting files that are open.
[0] https://www.usenix.org/techsessionssummary/challenges-integr...
It replaces HFS+, which itself predates macOS/OS X and over the course of its life got a lot of Unix-specific modern features bolted on in the form of various clever/scary hacks (e.g. hard links).
HFS+ was the legacy file system, from 1998, itself based on HFS from 1985. APFS was a wholly new, modern filesystem and quite radically different.
Filesystem experts please correct me.
APFS is a Copy on Write filesystem. The entire disk is organized as a tree starting at the root. Every time you change a file, no matter how little the changes is, you do it by writing a new file, creating a new tree of the whole disk with your new file in it, and finally updating the root pointer to point to the tree. Files are only removed later when they are garbage collected after not having been a part of any tree for a while.
This means that if you pull the power, the filesystem either looks exactly like it did before because the root pointer was not changed or has all your written changes exactly as you described them. The caveat is that Apple chose not to protect against certain types of bitrot.
It also means that you can take snapshots by just saving a reference to a tree, and roll back to or inspect that snapshot later, without having to pay for a copy of the files - creating the snapshot is instantaneous, and the consumed storage is just the sum of unique files in all trees.
This can also be used to have multiple roots (e.g., one for the current OS version, one for the new update being prepared) and subvolumes (e.g., one for the OS, one for the user, one for each app, whatever), again only paying for the sum of unique files.
This may sound expensive over allowing small writes to be done directly, but SSDs can basically only be written in whole 4k blocks, and the SSD controller moves your data around whenever you write to avoid wearing out the same block anyway. As such, there is a lower bound for I/O overhead, and there's a bunch of tricks the OS can pull to make the CoW overhead virtually disappear, like write caches/coalescing.
It was a much more perilous migration on Macs.
A lot fewer devices than Apple, but with changing a device's entire operating system a lot more can go wrong. I wasn't on the team at the time, but maybe someone else can chime in with more details.
No idea what it involved under the hood, but I assume some magic
For additional context, APFS is only supported on SSDs. My iMac didn't have an SSD, but it did have an upgraded HDD that I chose while buying the machine from Apple. While performing the OS upgrade that was supposed to determine APFS wasn't applicable, it thrashed the bootloader and wouldn't proceed (something about not finding the unmounted partition? IDK it was several years ago).
I hadn't been keeping proper backups either, but was able to manually backup the most important stuff using Target Disk Mode. Then, to Apple's credit, I eventually did a full reinstallation of the broken macOS, which amazingly kept my files intact.
EDIT: I originally said the migration "bricked my computer", but greedo is correct that it wasn't permanent damage.
iOS 17.4.1, minor update, thousands of reports with failing cellular data. IT world is going better direction every day.
Congratz for those real engineers, not hype following sheeps :)