I really like the hack of a block pointers being a data structure and containing the birth time (tx id I guess) and how that avoids needing managing bitmaps.
The advantage of doing send/receive vs rsync was also a nice explanation.
It is not often that you see a technology and think "Oh, this is great stuff", this is like that with ZFS. Haven't played much with it. But get that idea just from learning about it so far.
Both Jeff and Bill are great at communicating and explaining the technology. Like how they tag team, with minor funny bits here and there.
Regarding the main issue here, checksums -- yeah I don't see how Apple engineers could have watched this and said "Meh, don't need data checksums". Maybe they do have a secret vault with magic new holographic storage, immune to cosmic rays and other vagaries of physics, who knows.
The question I'm sure Apple engineers have were - How often do we see BitRot occurring off media, and is the media that we're deploying sufficiently resistant to bit-rot?
And, with APFS's flexible structure, this is a feature that can be added at a later time. Probably made sense to deliver in 2017 something that was rock solid that they could build on, than to either (A) push out the delivery date, or (B) not fully bake all features of the file system.
https://www.usenix.org/legacy/event/fast08/tech/full_papers/...
That's exactly what you want – a clean failure which prevents other software from silently propagating corrupt data. Even more importantly, with corruption it's usually much easier to recover from another copy if you notice the problem quickly.
Think about what happens with e.g. photos – you copy files from a phone, USB card, etc. and something gets corrupted on local storage. With integrity checks that means that the next time you load your photo app, go to upload photos somewhere, run a backup, etc. the filesystem you immediately get an unavoidable error telling you exactly what happens. If it's recent, you can go back to the source phone/camera and copy the file back.
With all current consumer filesystems, what you instead get are things like a file which looks okay at first – maybe because it's just displaying an embedded JPEG thumbnail, maybe because software like Photoshop is enormously fault-tolerant – but doesn't work when you use it in a different app or, because you didn't get an error, the underlying hardware file affecting your storage becomes worse over time until something fails so badly that it becomes obvious. By the time you notice the problem, the original file is long gone because you reused the device storage and you have to check every backup, online service, etc. to find which copies are corrupt.
The user would have to be informed, since Apples hardware has no storage redundancy. That will be perceived as an admission of failure on Apples part; your $2k device just murdered your data.
There is not really a UX flow apart from telling the user to recover the file from backup.
Technically it is the dominant strategy to have data checksums. - Even if you assume the Hardware is very awesomely perfect.
I agree with all your points, but just want to offer a possible UX solution:
Since the vast majority of data on people's hard drives are video and images, where minor data corruption results in (in most cases) just visual artifacts, we could have a pop-up dialogue that says: "A higher quality version of this file is found on your backup. Do you want to restore it?" when the corrupted file is a video or image and there's a confirmed backup of it.
If there's no confirmed backup then just silently ignore the corruption since the user won't notice anyways.
There absolutely is, since that could be performed automatically. Inform the user about the corruption, rename the corrupted file, then restore the most recent backup in its place.
In Apple's case, they have complete control over iCloud and could offer this easily for any file which is stored there. They could also add some sort of metadata API so services like CrashPlan, Backblaze, etc. could register the presence of other copies in a generic manner. Third party services could also integrate background scrubs into their existing application.
In each case, the first time that dialog appeared you'd likely have a customer for life from anyone who's gone through the hassle of losing a personal memory, important document, etc. or make a panicked search for other/older copies.
Also, I wouldn't be surprised if Apple adds a cloud-based backup for macOS once APFS is the default filesystem, since change sets would be extremely efficient.