What should we know about APFS special files? | Hacker News Reader

What should we know about APFS special files?

eclecticlight.co

74 pointsingve23 comments

I don’t quite understand what’s so surprising here. “Space on disk” and “size” haven’t matched in the longest time. If you’ve ever seen a folder with thousands of sub-4KB files when space mattered you certainly know this (and I’m not that old)

Here I just see great size optimizations that I’d expect from a modern OS, which sometimes can be more complex than what one saw in the 90s, and can’t be expected to make sense without being a magician.

Does it make sense to see a file on my computer when it’s really just stored on the cloud? How can my 64gb phone have a 500gb photo library? I see no difference here, but this has been common for the past decade.

I am all in on the Mac, but I hate APFS — or at least the way it's exposed by the Finder. There's no longer any obvious relationship between deleting stuff and getting disk space back, and at any one time you can find a range of different numbers in different places to tell you how much space is used and how much free.

As it says in the post: "Magic works best when the spectator either knows nothing about the sleight of hand involved, or is another skilled magician". APFS in the Finder makes both impossible.

> There's no longer any obvious relationship between deleting stuff and getting disk space back

Special file types have been around in other file systems for a long time now.

For instance, Hard Links have been a thing on NTFS disks since the first version of Windows NT.

Deleting a hard link for a file on on an NTFS disk doesn't result in freeing any space, until you delete all the hard links as well as the original file.

As far as I can tell that's not what the issue is. For instance I can delete a file and watch the free space dip 20x the size of the file.

> There's no longer any obvious relationship between deleting stuff and getting disk space back

This is a feature of APFS that Finder can't do much about it: local backups via the APFS snapshotting. APFS maintains a number of local backups for a period of time in a preparation (or, rather, in a hope) to permanently transfer the backups to an external Time Machine, after which the local backups are purged. If there is no external Time Machine set up, the backups will be purged after N days or under the disk pressure (free space is needed urgently).

btrfs in Linux creates file system snapshots in a similar way.

It is possible to entirely disable the APFS snapshots via «tmutil disable», but it will become impossible to restore deleted files, which may or may not be important for some.

The snapshots can also be manually deleted at any time by running a one-liner:

for i in `sudo tmutil listlocalsnapshotdates | grep '^20[0-9][0-9]'`; do sudo tmutil deletelocalsnapshots $i && echo $i deleted; done

What idiot approved this design though?

I have a 512 GB SSD and handle large files often on the go. I've had snapshots taking up 100+ GB with no explanation on my Macbook, just missing space.

Only figured this out when i googled support forums, used 3 party software or went into the terminal.

Apple and their "Magick", confuses regular users and wastes my time.

I fail to see what is wrong with the design. The volume snapshotting is the only way to solve the unsolvable problem of «undeleting» a file in a UNIX file system, after all, and the simplified backup management is the end game here.

Volume snapshotting via checkpointing along with the copy-on-write has been used in UNIX file systems for a very long time and successfully, starting from Veritas VxFS. It is also used in ZFS, in btrfs, bcachefs et al and in some form in the log-structured file systems.

If the default volume snapshot reclamation policy does not work for you, «tmutil» provides a number of controls to fine-tune the behaviour:

1. Disable the snapshotting in perpetuity via «tmutil disable».

2. Exclude select directories from snaphots via «tmutil addexclusion directory1 directory2 directory3»

3. Adjust the quota on how much of the disk space to allot to the snapshosts/backups via «tmutil setquota»

All of the above are one-off commands that, once run, have a permanent to a long lasting effect.

> Only figured this out when i googled support forums, used 3 party software or went into the terminal.

If you want a first-party GUI you can also use Disk Utility: https://support.apple.com/guide/disk-utility/view-apfs-snaps...

This is fantastic advice, thank you. I’ve had the problem for a bit and searched around but never came across this!

I've found a use for APFS' copy-on-write cloned files.

I like to work with git worktrees, and I work on an app that has a large (2 gb) node_modules directory.

Previously, every time I made a worktree, I had to `npm install` to get a fresh node_modules, or copy the full directory, which was slow.

    $ time /bin/cp -r node_modules /tmp/node_modules
    /bin/cp -r node_modules /tmp/node_modules  1.98s user 58.26s system 63% cpu 1:34.17 total

The -c option to /bin/cp on a mac is "copy files using clonefile(2)", making clones instead of full copies. It's still not super fast, but almost a minute less of waiting for the copy is a nice improvement:

    $ time /bin/cp -rc node_modules /tmp/node_modules_cloned
    /bin/cp -rc node_modules /tmp/node_modules_cloned  0.43s user 29.85s system 79% cpu 38.211 total

The one downside is that GNU has an incompatible option for creating clones (`--reflink`), leading to this very ugly conditional:

https://github.com/llimllib/personal_code/blob/0e1034f13b31f...

Has something changed in the Finder recently? I remember being able to copy my node_modules-containing folders relatively quickly but now it still takes a few seconds. I wonder if they either dropped that feature from Finder or if it has a hard time cloning many files

Moving forward, GNU coreutils 9.0, released September 2021, automatically uses reflinks when available. I still have several Ubuntu 20.04 systems that don't have that though, so your way is necessary for a bit.

Use pnpm, it hard likes the files in node_modules to a content addressable ‘store’. It makes ‘pnpm install’ pretty fast.

> I’ve used a 100 GB VM, whose size is given as 107 GB

Surprised to see this confusing statement in an article about understanding file sizes. The difference is due to base-2 vs. base-10 units: 100 GiB = 107.4 GB

> whose size is given as 107 GB, although it only takes 18.47 GB on disk

In context The 107 GB and 18.47 GB are being contrasted, not 100 GB and 107 GB.

The APFS feature that I'm having the most trouble with is 'purgeable' space. It's extremely not clear when it purges or if it matters whether it's a user-visible application or a unix-y utility like cp consuming space. I seem to be able to run out of disk space and have problems when 100s of GBs are supposedly purgeable. There are CLI commands to purge space, but it's also unclear from the man pages what the parameters are supposed to be to get it to do the thing.

I have found that a reboot has freed up purgeable space when the CLI commands fail to do so. Very frustrating to need to reboot solely because your filesystem seems to be holding disk space hostage.

To add insult to injury, some Apple apps don't seem to handle lack of disk space very well.

I recently couldn't get Photos to import photos from my iPhone anymore. I didn't get any error message or anything. I would click on Import, it would (seemingly) start and then just stop the import without anything actually imported.

Then I noticed that I was very short on disk space, so I deleted some stuff. Which of course didn't immediately free up the space due to TM snapshots. I didn't want to mess with Time Machine, so I just waited 24 hours, and lo and behold, there was more free space and import in Photos was working again.

This is with an older macOS version, so maybe this has been fixed in the meantime.

But it's not really what I would expect from "it just works".

APFS originated on iOS, but iOS cannot enable APFS encryption on an external USB-c drive. However, iOS can unlock a macOS-encrypted external drive.

There are small enclosure for mSATA or NVME 2230 SSD, with the form factor of a flash drive, and the speed and reliability of an SSD.

How sad that the Finder is now in the state its in.

Once it was joked about "Windows file transfer visits some friends" and now we have "Finder copies some files"

I hope Apple rights this course to make the UX consistent as it was in the pre-APFS days.

So what happens if you clone a VM, then delete the original? What if you clone a VM multiple times, then delete the original? Does the VM that was first cloned become the “original” in terms of links, or do you instantly lose the space savings?

I love APFS. It allows me to create snapshots of stuff I backup offsite. About 1TB worth. No way to get a reliable copy without snapshots.

Is APFS clone files same as deduplication on other FSs?

Reply on news.ycombinator.com