ExFAT Driver Boasts Much Faster "Dirsync" Performance with Linux 6.9 | Hacker News Reader

ExFAT Driver Boasts Much Faster "Dirsync" Performance with Linux 6.9

phoronix.com

57 pointsmikece14 comments

After John sent me a 5.8TB .bzip2 file in the mail, I discovered that the default exFAT drivers in macOS simply keel over and die. https://x.com/theshawwn/status/1695563727326642377

I’m still not sure why. But after intensive debugging, the fix was to install a third party driver, which thankfully exists.

It made me question why default drivers are often so bad.

Default drivers are designed to satisfy 95%-99% of customer use-cases, and are tested for those. Whoever was tasked with writing the default one probably didn't have "Test 5.8 TB file on a single exFAT volume" on their spec list.

The first thing that came to mind is that newer versions of macOS have a filesystem throttling feature (yes, really): https://news.ycombinator.com/item?id=31788894

exFAT should be simple if it's a variant of traditional FAT, although I haven't looked at the spec in detail; hence why I doubt the FS driver itself is to blame.

(FYI no need to keep the additional tracking parameters in your link.)

I winced at the tracking parameters when I pasted the link. I gave up trying to keep my privacy private, but I forgot that another reason to delete them is out of respect for whoever is clicking, not just me. Thanks for calling that out; I’ve edited them out.

And thanks for the data point about the mystery. It always bugged me that it wasn’t quite tracked down. Isn’t filesystem throttling a problem with the driver, though? Switching to a third party exFAT driver immediately fixed whatever was happening.

The new Firefox "Copy Link without Site Tracking" feature may help a lot for this :-)

Gonna guess "find me the next block for this file offset" is accidentally O(n²) in their implementation. Exfat is probably still list-based, like FAT32, and not extent-based, like any other file system from the last 40 years?

This is the most plausible explanation I’ve come across. Thanks; the mystery was bugging me.

Using exFAT under macOS over a period of a few years led to several of my drives becoming corrupted. I have not had issues since I switched away from exFAT, and I am now convinced that macOS's exFAT driver is unacceptably terrible.

If you need a multiplatform filesystem for a thumb drive or small portable hard drive, I have had success with UDF. Surprisingly so, given that it's basically a hack. See: https://github.com/JElchison/format-udf. This basically won't work on drives over 2TB, however.

I second that. At a past job we picked UDF for being the only format that would work across all platforms. That was before ExFAT got everywhere, but still seems like a good choice.

Useful comparison: https://www.sami-lehtinen.net/blog/exfat-extended-file-alloc...

The note about the partition table makes me wonder how he was formatting the drive. https://github.com/JElchison/format-udf specifically does some partition table related hacks to make UDF maximally compatible with different platforms.

Largest issue seems to be regarding corruptions, BSD platforms seems the only ones to include updated UDF file system check/repair tools.

Small ExFAT chunk size has significant performance degradation on macOS and Linux's userspace ExFAT. It also took 5 minutes to mount an SSD with 500k photos.

Bumping it to 4MB fixed it but used more storage.

Windows and the Linux kernel module variant are unaffected.

I haven't kept up to date with ExFAT in a while. Is it still impossible to resize ExFAT partitions?

Reply on news.ycombinator.com