Hacker News Reader

Comment by hnarn | Hacker News Reader

I don't mean for this question to sound inflammatory but I can't help but think: why does this matter? You can already use exFAT on Linux via FUSE, and as pointed out by VentureBeat:

> To be clear, Microsoft isn’t open-sourcing exFAT — but it is making sure anyone building with Linux can use it. 'It’s important to us that the Linux community can make use of exFAT included in the Linux kernel with confidence.'

exFAT is still not open source, and still can't be distributed with the kernel. So what changed? It will now be easier to integrate it into a custom kernel instead of using FUSE? Why is this better than the current situation, where you'd install exfat-utils for example? Are there benefits of using a kernel module that I'm missing which makes this important news?

View on HN

It will be in the "real" kernel soon, here is a patch I submitted adding it: https://lore.kernel.org/lkml/20190828160817.6250-1-gregkh@li...

And yes, this is _much_ better than using a FUSE interface to the filesystem.

For the 99% of people reading who don't know the kernel as well as you (seriously) can you explain why this is the case?

Does a userspace filesystem driver lose a lot of performance to context switching, or is there something unique to filesystems that slows them down in user space?

> Does a userspace filesystem loose a lot of performance...

Yes.

Try the two versions out, and see for yourself if you are curious.

I use a FUSE filesystem on linux- not ExFAT, but as an S3 interface. I often see 1+Gigabyte/sec throughput over the wire from simple multithreaded IO operations. FUSE can be very fast.

For one thing, that's throughput, not latency.

For another, that's probably a desktop, not a smartphone.

That's correct, my metric for performance is throughput I don't really think a desktop vs a smartphone matters much at this comparison, it's just my observation that when people say a fuse filesystem is slow it's just not well-engineered, not that the fuse API is inherently slow

Run a benchmark that measures more than throughput and you'll see the difference. IOPs are usually poor.

Latency is a nightmare if you perform certain types of tasks on FUSE.

It matters a lot. You can squeeze impressive performance from a smartphone, too, but then you need to dump all that waste heat somewhere, plus you need to take battery capacity into account. In other words, in a desktop, power efficiency is commendable, but not critical.

It also removes the requirement of distros packaging it up and users installing and configuring it. Is it's in the kernel it just works out of the box everywhere.

Much faster. Also, FUSE has to be installed by the user, which if it's in the kernel, then it Just Works everywhere. With the spec open and the patent license freely granted, the most common USB flash drive format can now Just Work everywhere.

For copying photos off an SD card you're unlikely to notice any difference on any machine made in the past 10 years, except maybe the fans coming on, but try it on a Raspberry Pi class machine, or doing any kind of random IO and oh boy..

The Spectre/Meltdown situation must have made FUSE much, much worse than it was already. I wonder if anyone did any benchmarks for that

better because you can be sued for using the FUSE interface?

According to John Gossman "Microsoft Distinguished Engineer & Linux Foundation Board Member" in a related announcement, Microsoft plans to contribute it to the Linux kernel:

We also support the eventual inclusion of a Linux kernel with exFAT support in a future revision of the Open Invention Network’s Linux System Definition, where, once accepted, the code will benefit from the defensive patent commitments of OIN’s 3040+ members and licensees.

Still some patent squirrelly words (they imply you have to sign a patent cross licensing agreement with the OIN - not sure of the implications) and no timeline.

Ref: https://cloudblogs.microsoft.com/opensource/2019/08/28/exfat...

OIN Ref: https://www.openinventionnetwork.com/joining-oin/

Most of the major Linux distributions are directly members or licensees of the OIN already. It looks like any distribution that isn't can get a license by filling out a form, no money needed and royalty-free.

To my reading, the OIN is a "mutually assured destruction" organization for patents around Linux that owns some of the patents outright and basically tries to insure that if any member sues another member or licensee over any of their patents the OIN itself and all the other members should band together and counter-sue over the remainder of the patents. It seems an interesting nuclear option to protect Linux from patent lawsuits.

The list definitely is long but I don't see Debian or Archlinux there: https://www.openinventionnetwork.com/community-of-licensees/

Well at least they did not deny patent issues. As for the driver, a community one is already on the way so they'll probably just piggyback on it.

Or, if they want to contribute an official driver, they should have contribute it at the same time as the spec, IMO.

The big deal here is that they are placing the patent into OIN - I'm probably correct in guessing that it is now the most powerful patent in the OIN arsenal. More electronics/software shops are going to be pressured to join the OIN. Consider sharing files on physical media and, oh, that UEFI basically uses FAT.

Patent trolls are probably collectively experiencing a migraine after this news.

Also hope that uefi will soon use exfat. Still better than plain old fat.

> why does this matter? You can already use exFAT on Linux via FUSE

It means that exFAT will be available everywhere. For instance, right now if you want exFAT support on Fedora or RHEL, you have to enable rpmfusion and install fuse-exfat and exfat-utils from there; once it's in the upstream kernel, it will be available on a default install.

But they can put their own implementation of exfat inside the kernel now without worrying about patents, which is what previously prevented it from being included. I will also help that they will have an official spec to go off of now, so they won’t have to waste time with reverse engineering (plus the legal concerns that comes with that).

FUSE is very slow compared to kernel module.

Citation needed.

I don't think we need a citation for common knowledge about how OS kernels work

I’ve been building SD card based Pi devices and the limiting factor in IO perf is the FUSE exFat implementation. There’s a leaked Samsung internal implementation that is over 2x as fast in my benchmarks. I can’t attest whether it’s the fact it’s FUSE or just other performance optimizations that is the reason though.

The leaked kernel was apparently later open sourced [1], although I can't actually find it on the official samsung website.

[1]: https://www.phoronix.com/scan.php?page=news_item&px=MTQzODQ

Doing things in-kernel is almost always faster. No dealing with complicated context-switching, moving memory back-and-forth, etc. I can't necessarily provide a formal, data-backed citation, but it's pretty well understood.

Because Fuse is slooow - I've had an external Exfat drive mounted on Raspberry Pi 4 and writing large amount of data would pretty much max out the CPU. Ext3/4 has no such issue, I can write to it for days without taxing the CPU. Apparently it's a known problem with Fuse partitions, NFTS-3G has the same problem.

legality and implementation straight on linux kernel would be the first thing I'd say.

exFAT will also be available on other devices like cameras android phones

Care to explain how?

Microsoft isn't giving away it's exFAT patents. They are merely suggesting, that everyone licensed them at Microsoft's own terms. Those terms include becoming a member of Microsoft-sponsored patent ring and agreeing to it's terms and conditions: https://www.openinventionnetwork.com/joining-oin/oin-license.... The conditions can be changed anytime (!!) and the patent license terminates if you sue Microsoft or any other OIN member for any patent violations whatsoever.

Nikon, ZTE, Xiaomi, Samsung and thousands of others (basically all, who were or still are patent-trolled by Microsoft) haven't magically become members of OIN, and it's unclear if they ever will.

Google and Red Hat are members, but neither produces any devices with external SD-cards...

Reply on news.ycombinator.com