macOS Tahoe brings a new disk image format | Hacker News Reader

macOS Tahoe brings a new disk image format

eclecticlight.co

382 pointszdw139 comments

Open articleView on HN

https://developer.apple.com/documentation/virtualization/vzd...

I guess this is the Apple version of qcow2 and friends

“Asif” is a fun name for a disk image format. It’s as-if it was a real disk. ;)

Andor is a funny name for a tv show

Well, TV show and/or long movie depending on how you look at it.

The benchmarks are weird to me - the ASIF tests were done on M3/4, but for everything else it was done on an M1?

Yeah, that jumped out at me, too.

If they can’t re-run the benchmarks on the same hardware, it’s hard to compare the numbers.

I skimmed through the article, but I have a question that I hope someone can answer. I have a sparse disk image created on a NAS (which runs Linux), and I use it to backup some stuff (not a VM) from the Mac in the native format (the macOS APFS file system).

Would this new format, ASIF, make this faster and better whenever I switch to macOS Tahoe? I hope there wouldn’t be any gotchas with respect to storing this disk image on a NAS.

Yeah I’m interested in this as well. There’s something about the way time machine works over SMB that is absolutely unfashionably dog-slow. I suspect SMB performance on mac is just not very good in general tbf

My kingdom for a documented disk image format.

https://docs.kernel.org/filesystems/ext4/index.html

A filesystem is not a disk image.

If it quaks like a disk image, it is a disk image

A filesystem is not a file.

(Yes, you can store a filesystem in a file - and that's a trivial sort of disk image, but one with some serious drawbacks like "you have to allocate all of the space up front". We can do better.)

Some of the most popular disk image formats are basically a sparse file abstraction for non-sparse files and nothing more. You have a bunch of blocks, a table mapping each block to its virtual location, and a couple convenience headers.

If those count as a disk image when you put a filesystem inside, then I say a normal file is also a disk image when you put a filesystem inside.

Especially because the sparse mapping is optional. For example, lots of VHDs are a raw file plus a 512 byte footer.

What?

About the only decently documented disk image format I've found is qcow2. [0] Which is usually not the best tool for the job.

So very many of them are just header details + "only works with our tools".

[0] https://www.qemu.org/docs/master/interop/qcow2.html

I'll take your kingdom!

ISO 9660.

https://www.iso.org/iso-9660-images-for-computer-files.html

VMDK

https://en.wikipedia.org/wiki/VMDK

Amiga

https://en.wikipedia.org/wiki/Amiga_Disk_File

UDF

https://en.wikipedia.org/wiki/Universal_Disk_Format

Apple Disk Image

https://en.wikipedia.org/wiki/Apple_Disk_Image

VMDK isn't documented there, and is a container for multiple partially documented - and some only reverse engineered - subformats.

Not on wikipedia, but VMDK is part of the DTMF OVF[1] format and thus I believe the .vmdk format would be implicitly made available therein

However, words are words but software is better:

- https://github.com/vmware/open-vmdk#specifications is Apache 2

- the link they cited is bitrotten but Internet Archive has you: https://web.archive.org/web/20210411181842/https://www.vmwar...

1: https://www.dmtf.org/standards/ovf

Unfortunately, no. OVA and OVF are defined in the OVF spec, but VMDK is not.

So congrats on finding the spec for it, as its no longer maintained! Its also no longer followed, either, unfortunately.

VHDX https://learn.microsoft.com/en-us/openspecs/windows_protocol...

What blew my mind recently was that I could store an APFS sparsebundle on a NAS drive, then mount it over NFS and use it as a plain old APFS volume. Despite the filesystem layering, it works pretty much like a local volume, albeit with a bit of performance degradation. Seems preferable to something like iSCSI for using APFS with network storage.

Perhaps this new format would work even better?

Not sure why that would blow your mind, but Apple's old Time Capsule basically mounted a sparsebundle over the network too. And yes I would also guess this new format would work better.

My experience with NFS file management has been less than stellar, so running a full, virtualized, and performant APFS volume on top of it feels like a bit of a magic trick.

Interesting. My employer uses NFS extensively and issues are very rare, and such issues can always be traced to underlying network issues not NFS itself. It's a good choice if you have a good network.

My former employer used NFS and it would lock up a bunch of stuff whenever the network dropped for a bit. In fact a lot of their software had detection for this case so that you wouldn't think it was at fault for the issues.

My favourite blog specifically for the painting articles

They are amazing, for sure.

Yes! So glad GP brought that up. Bookmarked, looking fwd to spending more time absorbing the fantastic images and accompanying texts.

I wonder with the recent changes regarding containers and now disk images, if Apple plans to enter the server or cloud market.

Would that potentially speed up Docker for Mac and others? (since it's using a vm underneath). That would address a major pain point

They also announced new APIs for native Linux containerization on MacOS with a specific focus on security and performance. This seems like it may be in support of that as well. Anything they can do to improve the performance of containers is a huge win. https://youtu.be/JvQtvbhtXmo?si=3OphClGvylHggmSW

Any chance this will reduce the space needed for major OS updates? Those have always been desperately inefficient.

No. Disk images aren't involved in the OS update process.

They are - in that OS updates are delivered in and extracted from disk images. Not sure how they could be any more involved…

Not this kind of disk image. ASIF is a read-write disk image; an OS update would be distributed in a read-only image (if it is indeed a disk image).

Oh awesome! So it's more virtualisation-focused as opposed to HFS+ --> APFS migration?

Apples and oranges. This is a new disk image format. You can create an APFS _volume_ in the ASIF _image_.

Has anyone found the specification for the format?

I have a meta question not directly related to the article but more about HN itself. I posted this exact link 9h before this submission was posted[1]. How is it possible that there is a new entry for the submission given the link is the same?

[1]https://news.ycombinator.com/from?site=eclecticlight.co

Dupes are allowed after some number of hours.

It depends. I tried posting a few links I came across over the past few days, and it just showed me dups but they were from days or weeks or sometimes months ago.

Not sure why this comment is downvoted. Yes, I got redirected for a dupe earlier today for something posted 7 months ago. And even though the content substantially changed, it can’t be posted again.

I seem to remember getting that and having the option to post it again. Maybe it’s a karma related thing.

I thought that number was significantly more than 9.

I think it is time * (votes+comments).

In practice, if it gets any real amount of votes or comments, you have to wait a year to repost. If it doesn't get any attention, it can be reposted quickly (though I think it should be a day later).

It seems like, if the link is the same, submitting a dupe should just raise the original.

My guess is that “fastest gun in the west” might be a bit anti-pattern with respect to community.

Because in ye olden times, mild URL shenanigans seemed to have been a common hack to bypass more strict dupe detection.

And the community probably doesn’t really benefit from aggressive karma seeking —back then being first would give you a point for every resubmission. [1]

But that’s all speculation based on a supposition that what is more likely to be submitted by other users is not a better criterion for choosing to submit.

But I could very well be wrong and probably am.

[1] and number of submissions is probably at best a noisy signal for front page placement and might be negatively correlated with curiosity…I mean even here, what Apple is doing doesn’t stray too far from yesterday’s big press release by one of the most valuable corporations in the world.

Might be something weird with this domain. Look at the list of submissions, yours isn't the first dupe that was accepted in a relatively short time window.

Nice. But I would like it better if the effort was to support ext4, BtrFS, NTFS and other popular filesystems from the Linux and Windows world...

ext4 and Btrfs are only well supported on Linux; they are not universal standards.

NTFS was only supported well on Windows until recently; but extensions like NTFS Encryption (BitLocker) are still Windows only. Mac still does not let you write to an NTFS volume.

APFS and HFS+ are obviously Apple file systems.

FreeBSD does not support ext4 or Btrfs well; but instead prefers UFS2 or ZFS despite also being an open-source Unix-inspired OS.

The world runs on proprietary or non-universal file systems with CDFS (ISO 9660), FAT, and exFAT being the sole exceptions.

if Mac also supported Linux file formats they would get closer to universal, though? we can't get to that world without these individual steps.

creating a new non universal one is backing away from it

Even FreeBSD can't be bothered to do a good job supporting Linux file systems. They're basically proprietary siloed file systems like the rest, even if code is available. Linux, meanwhile, can't be bothered to support either of FreeBSD's file systems officially. UFS2 is hardly patented anymore, but Linux doesn't care beyond read-only support.

Being open-source, and even being in a popular open-source project, does not make a standard, or even imply inferiority to those who do not implement it.

https://xkcd.com/927/

There might be some license issues, but the dirty secret is filesystem portability isn't terribly important, and for the users for whom it is - ExFAT and friends are usually good enough.

exFAT is widely used but it not being journaled has led to so many thousands (if not more) people losing tons of data, many of which wouldn't have lost so much data had they used a journaled filesystem (or even one with redundant file tables.)

If you need to connect a portable drive to machines of different OS's, there is no safe filesystem that supports read and write on both Windows and MacOS.

Alternatively, cloud storage works until the files are larger than the space you have left on Drive/Dropbox/OneDrive/etc., and local network sharing (on certain networks at least) is more complicated than what the average user is willing to put up with. In practice, many use USB flash drives or external HDDs/SSDs with exFAT. Yeah, people should have more than one backup, but we know in the real world that's not what many do. That requires them spending more time (e.g. configuring local network sharing or setting up an old machine lying around to be a NAS) or money (more on cloud storage.) In practice, having a cross-platform, journaled filesystem would lead to a lot less data loss.

Aside from exFAT, the only alternative with native cross-platform R/W capability is FAT32, but while it has a redundant file allocation table (unlike exFAT), it has a max file size of 4GB, which limits its usefulness for many workflows.

Does this even matter, if you can copy/mount files over a network?

What are potential use cases where you'd want to support those additional file systems? External drives?

I just ran into a use case yesterday. I wanted to copy some files from either my Mac or my Windows machine onto the MicroSD card for my SteamDeck, which is ext4.

I wanted to just plug in the card and copy files, but couldn’t.

Interesting, ok.

I don't even want to say "Use NTFS, doesn't everything support it?" because I'm not even sure that's the best solution for an SD card.

Maybe with their new FSKit API [1], someone can build a compatibility layer for it.

1: https://developer.apple.com/documentation/fskit?language=obj...

The best solution is to add support for EXT4 to mac and windows.

Is there a single filesystem in the world (besides "simple" ones like FAT) that both has an open standard AND is licensed in a "usable" codebase (MIT or other "non-copyleft" license)?

I think UDF was meant to be this, though I don't think it supports everything you'd want. Does NTFS have licensing issues though?

NTFS has serious performance issues, nobody should use it.

AFAIK that's an incorrect meme that just won't die. The performance issues you're thinking of have nothing to do with the filesystem itself, but with the I/O subsystem in Windows more generally. If you have evidence otherwise please share.

Microsoft itself admits there are performance issues with NTFS, which is one reason they created ReFS.

For example ReFS removed the MFT which caused various problems.

I'm on my phone and this is a longer discussion than I can have here, but the performance problems they're thinking of and the ones people usually rant about on these forums are not the same ones (or same magnitude). Before being so confident it's NTFS itself that's the issue, try ReFS (and FAT32 for that matter) and tell me if you see the performance problems you've encountered actually improve a lot. And then narrow down the cause. You might be surprised how much of it had to do e.g. with filter drivers or other things. And don't forget you're still testing one particular implementation on one OS, which doesn't say anything about a different one for the same filesystem on a different OS.

I have been running ReFS for many years now on all but the system partition. But for reliability/self-heal reasons.

The MFT was annoying because if you create 1 mil files, and the delete them, the MFT will not be shrunk back.

> I have been running ReFS for many years now on all but the system partition. But for reliability/self-heal reasons.

Have you ever benchmarked it against NTFS? With the same exact set of FS filter drivers running on both? How much does the performance differ?

There are tools like voidtools Everything2 and WizTree that directly read NTFS from disk device bypassing windows FS apis and are blazingly fast (faster than find/du on ext4 in Linux).

'Everything' is caching all the files in memory and indexing them. Completely different than find/du on Linux.

zfs?

correct. but it's also a management system so perhaps they only want a files system? no idea why more don't use zfs. especially after the auto expansion update earlier in the year.

IMO the main reason is that ~90% of ZFS benefits only show up with multiple drives which (unfortunately) is fairly uncommon outside of servers.

That's a major benefit of ZFS for sure, but I think being copy on write is another major benefit for single disk systems. ZFS is the only one I know of that has a full feature set and is supported on every major OS.

Cross platform encryption, backup/restore (zfs send/receive), large file support are all great reasons to use zfs.

Apple almost went with ZFS and Sun pissed off Jobs by mentioning it early (IIRC).

Not AFAICT.

https://news.ycombinator.com/item?id=17852019

xfs.

if GPL is a non-starter for you, youre missing the point of the open standard. apple already discloses a litany of various GPL it ships. XFS would be no different.

> NTFS was only supported well on Windows until recently

I remember using ntfs-3g without any issues on my first Linux laptop 15 years ago. So that's not really "recently".

I intentionally excluded unofficial or third party software, as almost anything is supported if you’re brave enough. The quality of said drivers also wildly varies.

Until 4 years ago, nothing was good enough for the upstream Linux kernel.

I mean, yeah, you could say that. Something being in the kernel is a good benchmark of quality. But IMO open-source is different. For instance, Terraform had no stable release from 2014 till 2021, that didn't stop enterprises from using it on scale.

Apple recently made FSKit a supported/documented API, which should allow third-parties to add support for other filesystems: https://developer.apple.com/documentation/fskit?language=obj...

Is it actually supported and usable now? I seem to recall it spending a lot of time in a "half-documented and not actually available" state.

3rd party supporting a file system would be one of the last things on a list of all software I’d ever want a 3rd party writing instead of the OS maker.

Nightmare to evaluate the options, pure stress testing the options, difficult to know if it didn’t mess something up.

Question, who maintains NTFS support for Linux? Microsoft? The kernel? The distros? Genuinely asking.

The current ntfs3 driver is developed by a company called Paragon, and it's been part of the kernel since 5.15 or so.

I was going to add some additional comments about this, but then I found that Paragon's website has an FAQ that covers everything I was going to say (and more): https://www.paragon-software.com/us/home/ntfs3-driver-faq/

The kernel is built with the NTFS3 driver, provided by Paragon.

https://www.paragon-software.com/home/ntfs3-driver-faq/

That being said: FSKit is a userspace API. In that respect, it's a lot better than filesystem code running in the kernel - it can't crash your computer or corrupt data on other filesystems, and it's much more tightly sandboxed if it gets exploited.

Exactly! Third party file systems support in user space is exactly what I want to see. It seems to me that third party kernel code has always caused me problems. By moving the FSKit to user-space, I’m quite happy to try something, knowing that it won’t affect the rest of my system.

>3rd party supporting a file system would be one of the last things on a list of all software I’d ever want a 3rd party writing instead of the OS maker.

Given how many people use FUSE, Paragon NTFS for Mac, and similar tools, you're hardly totally representative.

Third party read-write NTFS drivers took FOREVER to become really robust. I remember hearing horror stories not infrequently up until maybe a decade or less ago.

> Given how many people use FUSE

What percentage of MacOS users even know what that is. I mean I am in the percent but I know it's sub 0.001

Well there’s at least two of us.

They don’t need to know it to use it. I’ve seen a fair number of commercial cloud drive products use it.

MacOS users' awareness of a mostly linux centric piece of tech is pretty damn irrelevant here. The point is that FUSE is a pretty mature piece of technology, and we know that it can be used productively without being that nightmare scenario you described. There is no reason why Apple's FSKit can't be equally successful.

Do many people use macFUSE? I thought ever since the license change it's really dropped off.

There's fuse-t, which uses a local network filesystem in the background, iirc.

Edit: but to be fair, that's mostly only relevant for unsupported network filesystems like sshfs...

They trust kernel extensions apparently, all of them.

Like 99% of the computer using world until less than a decade ago, when almost all drivers were kernel extensions and things like kexts were very much used?

> Given how many people use

How many people use software like this because they have no choice? I used Paragon NTFS, but the entire time, I thought it was ridiculous that MacOS can't read NTFS on its own.

I'd be happy for VeraCrypt not to have to rely on MacFuse which requires I go turn off some very low-level protection to even use. It sounds like this makes that possible.

I don't really understand your objection to be honest. Drivers for storage are common on other platforms

What would Apple's incentive be to support Btrfs, Ext4, XFS or ZFS ?

Btrfs, Ext4 and XFS are all under GPLv3, which may or may not be a problem for Apple, but "just in case".

They tried with ZFS, but couldn't strike a deal with Sun/Oracle, so instead invented APFS.

Apple already delivers a stable filesystem. It may not be "best of breed", but it works, as billions of devices runs on it every day with zero problems.

Thanks for sharing about this! I didn't know, and I like to use (or at least play with) some third-party filesystems on macOS.

This article describes a new disk image format (on which a filesystem can be put, APFS in the article), not a filesystem, or did I misunderstand?

edit: added the word "image", which I apparently forgot to type. Mentioning the edition because otherwise it would make an answer to this comment difficult to understand.

This is a new disk image format. Not even a disk format. For virtualizing a disk.

On which you can put a filesystem, yes.

There seems to be an extraordinary amount of confusion in many of the comments, I don't know why.

> For virtualizing a disk.

We really don't have enough disk formats in this space. Is this bringing something new ? Or is just polluting the namespace.

>Is this bringing something new?

If only there was a whole post available so one could find out...

The post only benchmarks against UDIF. Whether this brings something new is a very good question that is not answered by the post.

How complex is the format? Did they do anything clever? Did they engage in NIH? For all I know they switched the AES mode and made the blocks bigger and that's the entirety of the performance boost.

Just reading the title, I was like, I hope it is an update to encrypted sparse bundles, and it is.

https://developer.apple.com/documentation/virtualization/vzd...

Yes, its going to be similar to a .dmg file on the Mac.

Only Mac users IMHO are well-familiar with working with disk images. They are not as diverse or well-supported on other OS’es, while nearly every Mac app (prior to the App Store) was installed by dragging it out of a mounted disk image.

Yeah and OP was asking, why was effort spent on a disk image format rather than a file system. Seems like a reasonable question to ask.

Not any more reasonable than asking why they added new transparency effects instead of btrfs support

That's of course also a reasonable question to ask! We all hope by asking these questions some Apple employee using a throwaway account will provide an answer on HN. HN is that magical place where such questions have the best chance of getting answered.

I don't think the question would make much sense. Why did Apple do X and not Y (unrelated to X) is not an interesting question. It would seem close to whataboutism. They were willing to spend money on X. They are not willing to spend money on Y. Or, they didn't think about doing Y. Or, they needed X. They haven't needed Y. Apple want fast VMs and also they haven't needed Btrfs. What insight can you get? There's no relation between the two. You'd get the same insight by asking "Why didn't Apple implement Btrfs?", but by linking the two in the same question, you are kinda implying there's a link.

But the question would have been highly relevant had Apple developed a new FS, and disk images and FS do seem related at first. I didn't want to assume whataboutism, so I figured OP was possibly confused because this is likely, and I wanted to give them a hint without bluntly asking whether they confused things. I could have, really there's no harm in being confused, nor in asking whether someone was confused.

Can't abstract away your property that way.

Does anybody else think that it would make sense for Apple and Microsoft to just get in a room and horse trade a few things like this, if they cared about user experience? Cross-license both APFS and NTFS, and share any internal documentation under NDA so that external drives can use modern formats with safety features like journaling without locking users in.

Oh wait, I just answered my question.

I suspect there wouldn't be an agreement on a minimum set of features for a modern filesystem, even just for external disks, even if you limited it to flash storage devices to avoid all the complexities of spinning platter latency.

After all, there's no such agreement on Linux either - we just have all the Linux vendor options available.

Let me clarify, I don't expect two vendors like that to merge filesystem specs. I just think they should have first-class support for reading and writing on each others' default (NTFS and APFS) filesystems because the alternative is that a user who has a hard drive full of their important documents can't switch between Mac and PC without buying 100% more storage with which to do a complicated filesystem-migration exercise -- and let's be honest, only an absolute nerd (like us) would even understand what that is. Others would just plug in the NTFS disk to the Mac, see the popup saying "unreadable," shrug and give up (or worst case, format it not understanding that means erasing).

This is the kind of thing a reasonable government would just tell them they had to do by virtue of being a fully locked-in duopoly, just like they should tell Apple and Google that users should be able to choose to install an alternat app store.

This is a disk image format so why not VHD? It seems like it's open enough and supports what a virtual disk needs what do we gain with yet another file that's a disk type?

It's a shame that every new cool product/dataformat/cable/cpu/whatever researched by Apple has very little (or no) public documentation. Sure, there are lots of hackers who can test and reverse engineer those pretty quickly, but it's just unnecessary work. I don't know why Apple is so revered in hacker circles, to be honest. Not even Microsoft does this shit anymore, they're open sourcing a lot of research this decade, but they're still seen with extreme distrust. Whereas Apple was always secretive and used underhanded tactics, but it is still loved.

Here's the link to the Darwin, the underlying Unix layer of macOS, which is open source [1].

Go for it!

[1]: https://github.com/apple-oss-distributions/distribution-macO...

You're joking, right? You can see at a glance that this mess is only published for compliance reasons. There is no documentation at all, and most of the code consists of numerous versions of open-source libraries that have been subtly modified due to their licensing requirements, which necessitate disclosure of modifications. Good luck building any flavor of that! See: https://news.ycombinator.com/item?id=35197308

Was this not already a solved problem? Why reinvent the wheel other than for vendor lockin?

It's a solved problem in the sense that Apple solved it more than a decade ago. (The sparsebundle format was introduced in Mac OS X Leopard from 2007. The sparseimage format was even older; I saw discussions of using it with Panther.) Apple solved it again but made it faster.

Doesn’t sound particularly useful, if you aren’t setting up containers.

If Apple would just ship bind mounts and FS/pid namespaces... or even just un-break chroot...

Unless they can migrate Time Machine to it. The performance improvements sound like they might be a real boon there.

I use sparse disk images extensively as a tool to make certain directories require specific passwords. I simply create an encrypted sparse disk image and set a password on them. Seems like this will speed up my use case.

I hope they did something to not load my hard drive with all the AI crap and steal the storage.

This ridiculous arms race can't be over soon enough.

Reply on news.ycombinator.com