NTFS read-write GPL kernel driver | Hacker News Reader

NTFS read-write GPL kernel driver

lore.kernel.org

287 pointsButSpider112 comments

Open articleView on HN

Discussion on reddit:

https://www.reddit.com/r/linux/comments/i9mdqg/ntfs_readwrit...

When exFAT was announced about 5 months ago Paragon Software posted a rant against that.

Rant is posted here:

https://news.ycombinator.com/item?id=22688058

Paragon, purveyors of fine commercial filesystem drivers that corrupt my hard drives? LOL

It's easy to laugh at them but I've had corruptions from Linux's existing NTFS implementation too.

I've had issues with their Windows HFS+ driver too, shit wiped out a hard drive that I later learned wasn't fully backed up

+1 had the same issue with their HFS+ driver, and it did not take that long to encounter it.

Isn't Linux's existing NTFS implementation read-only? I'd be very surprised you would get corruptions from that..

FUSE one is not just read-only. It had capacity to do write on NTFS partitiona for decade at least.

That’s FUSE which is in userspace. The kernel driver is read-only. I believe it never progressed past that due to licensing issues.

I know what FUSE driver is, but my educated guess would be that almost no one ever used in-tree kernel-space driver which you referring to. So when people talking about issues with old driver that's almost certainly NTFS-3g

But, it is FUSE.

Am I correct in saying that there shouldn’t be any reason that being FUSE would in and of itself cause corruption?

Performance is a different question, since it’s not in kernel.

Off the top of my head... ordering violations. FUSE has a lot less control about the exact order in which things get written to disk, meaning that you are more likely to wind up with the wrong parts having committed down first if there's a crash.

I guess it's not that surprising really[0], but it is quite amusing in combination with the company name.

[0] If you tie your company's worth to a proprietary technology.

I mean the thread is they're open sourcing their work (if by commercial you mean proprietary).

Everyone (on Reddit at least) seems to be yakking away at whether Paragon Software is good or bad and what other things they've done. I just wanna know if this driver is actually good or bad. Far as I know, a Linux NTFS driver capable of writing has been a long time coming.

We have licensed this driver and the fsck-like tool.

The driver was deployed to 65,000 Ubuntu-based servers (as far as I remember), but we had a ton of issues with it, to the point that we removed it entirely and went back to NTFS-3g. We were told by paragon that we used a too-old version but at that point we had already moved on.

However, their NTFS check tool that ships with the driver (I forgot the name, but I can dog it up if anyone is interested) is actually super awesome! It is so so so much better than even the Windows chkdsk tool itself, and corrects many more issues. We use it to fix NTFS errors in millions of backup images. Can highly recommend!

Hope this helps.

ntfsfix?

So far, signs are not super positive: https://lore.kernel.org/lkml/87h7t5454n.fsf@suse.com/

This sounds like "yeah, it has bugs that need to be fixed", not fundamental reasons not to merge it. (Though I think they should've tested with KASAN+KUBSAN before submitting.)

Sure, bugs can be fixed. The bugs are one reason not to merge this iteration of the patchset. I don't dispute that it could eventually be polished into something worth merging.

(The second bug in the email linked above is a NULL ptr deref during an in-order recursive directory copy into the volume. That is real basic stuff; I don't think it required ASAN/UBSAN to hit. It shows a lack of even relatively basic testing, if I understand correctly.)

If a tester can't simply create a volume, mount it, and copy a directory tree into it without encountering a null pointer dereference, this to me raises serious questions about how this driver is tested and the lack of care in building it.

I've been writing to my NTFS volume for around 5-10 years I think, out of the box with Linux Mint. (Or maybe an extra package from apt-get, not 100%). Extra options needed and possible data loss when writing used to be a thing, but a long time ago.

You're probably using ntfs3g which is separately installed.

> You're probably using ntfs3g which is separately installed.

Not sure about Linux Mint, but MX Linux & antiX has NTFS driver bundled into ISO-images.

I've been writing to NTFS for at least 10 years. It was the FUSE driver but it worked fine and I never had any problem other than it seemed to write at half the speed of ext4

NTFS is super slow on windows too. You get better performances with ext4 on WSL2 (Windows Subsystem for Linux).

This is not an NTFS issue. You can verify it by trying a different file system.

Here's more or less official description of this issue.

https://github.com/Microsoft/WSL/issues/873#issuecomment-425...

WSL2 simulates a virtual ext4 file system. If you want to access ext4 natively from WSL2, wait for the next Windows Insiders release!

how is this possible? isn't wsl 2 literally a virtual machine image on top of your ntfs drive?

I think it bypassed windows filesystem filters (antivirus, etc) and thus faster because there is nothing taxing your read/write operations.

It least with WSL1 I know that McShield and cyvera/Traps are fully capable of massively interfering with linux performance under windows 10 just like they do normally. Whether they actually effective is not something I know but it seems like they understand ELF files.

The WSL2 kernel at this time cannot access ext4 partitions natively. Only partitions available in Windows can be mounted in WSL2. So that means installing ext4 drivers...

WSL2 itself uses a virtual file system.

Yes but it's only one big file, the ext4 partition.

And to extend, Windows/NTFS is not particularly slow at accessing a large file (actually, until recently they have been ahead for async read/write into big files), the area where they really suck is anything that has anything to do with metadata.

I had heard the issue was with how Windows/NTFS let a bunch of extensions place hooks into reading and writing files. Antivirus apps would sometimes put a bunch of heavy-ish hooks in that would be terrible for performance with lots of tiny files, which many Unix-y workflows expect to be able to do quickly.

The "sometimes" includes the one Microsoft ships by default with Windows. It is ridiculous how much faster random file access becomes when disabling Defender's real time protection feature.

It's one of the most impactful AVs when it comes to file r/w speed, especially on lots of small files. I have no idea why I still see people calling it lighter and faster than other AV options. It's not even like the performance loss is getting better protection. Products like Bitdefender and Kaspersky have better detection rates and don't murder performance. (Personally I just turn it off and don't install a replacement)

This is probably why JetBrains IDEs suggest to add your projects to antivirus exceptions for you. I rarely ever have performance issues while developing anymore after I let JetBrains do its thing.

I've been told this has to do with NTFS using a complicated metadata scheme and also all permissions in NTFS being backed by a complex ACL-security-descriptor scheme. Is that an accurate representation?

I don't believe it is. Windows metadata performance is not just bad, it's so bad it beggars belief. I don't believe the root cause is any specific technical decision, but rather that no-one cares about it. It has always been so bad that no-one can count on it, so no-one depends on it, so it's something that can safely be ignored.

On Linux, and other unices, there have always been important workloads where it really matters, so that itch got scratched.

Windows filesystem metadata performance is also highly dependent on how applications access metadata. It's a lot slower than it needs to be, especially for software ported from Linux (or POSIX-like environments in general), because metadata access has to be done in a Windows way to be performant.

Windows metadata performance is horrendously bad for code that needs to get metadata for a large number files in the same folder the way it would be done on Linux: by calling the Windows equivalent of `stat` (`GetFileAttributesEx`) for every file.

Metadata performance is much better if the code uses `FindFirstFileEx`, which IIRC has no direct Linux equivalent, to grab metadata en masse for all files in a folder and then filtering the results down the files that the code needs to know about.

Unfortunately, most code ported to Windows from Linux uses `GetFileAttributesEx`, under the assumption that it's as fast as `stat` is on Linux, but results in a poor user experience. Worse, many modern cross-platform languages (including Python) don't expose `FindFirstFileEx` without platform-specific add-ons, which means that devs have no easy way to do Windows metadata access right even if they know there's one specific way to do it.

os.scandir in Python was designed to encapsulate "listdir replacement APIs" and uses FindFirstFile/FindNextFile on Windows, and also allows most stuff to be done without an extra per-file stat() syscall on Linux. On top of that, when accessing many/all files in a folder in Linux, this gives you the inode, and if you sort your accesses by inode, throughput can rise to the ninth power on hard drives.

On the other hand, if you look at e.g. Windows Explorer. Explorer will take somewhere from a few seconds to ~tens of seconds to list the contents of a SSD-backed directory with a couple thousand files. The same operation on Linux is practically instant. Copying a bunch of small files on Windows takes forever, maybe doing around a few hundred files per second. Meanwhile Linux will happily copy thousands of files per second, all on the same hardware.

Explorer might be using incorrect ways to do I/O, but if that is the case, then that would mean that the proprietary, only-ever-developed-for-Windows-by-Microsoft tool that is also the main way users do I/O doesn't do I/O properly. What would that say about Windows?

If I remember correctly Windows Explorer still doesn't support long paths (paths with a total length over 255 characters). It only recently gained the ability to create files starting with a dot. It has no support whatsoever for alternate data streams, let alone more obscure NTFS features.

I wouldn't be surprised if it used the wrong API (in fact the lack of long path support is an example of using an outdated API). The only reason Windows Explorer appears reasonable is because people generally consider creating files you can't access with Windows Explorer a bug or misfeature.

Explorer not only appears to be heavy on the "Windows Classic" APIs, somehow to this day, but it also does a bunch of stuff ower file compared to let's say ls.

In fact, if you use Total Commander, in the settings you will find option regarding things like "getting correct icons for files", with the option for "display icons for all files" explicitly marked as "SLOW".

Explorer always gets the icons, and possibly other metadata.

Os.scandir looks like a very good improvement to Python. I haven't coded anything to Python 3.5 or newer, however, so I have no experience with it.

The last time I needed something along these lines the only option were os.listdir or using FindFirstFile through win32py. Os.listdir was an order of magnitude slower for my use case.

I don't agree that it's because no one cares about Windows metadata performance; both userspace/3rd-party developers and the kernel engineers care. Rather, they are loath to break any published API, and their metadata performance suffers from the limitations imposed by those historical design decisions (filter drivers, etc).

They didn't anticipate a workload where files would be opened and closed very frequently. E.g. running npm. Applications accessing large numbers of small records were expected to do so via a big file containing all the records a la SQLServer.

Yet MSI and Windows Update love their gazillion of tiny files, probably one of the reasons why WU is so slow (aside from the "let's solve P=?NP" thing that Windows 7 does). Game devs have got it right for decades though, practically every game ships their own little take on random-access ZIP archives (they often used profiling to arrange the files inside the archives in the order the game would load them as well, though that is probably less important these days), because that's so much faster than using the host file system.

Inherited ACL's are not cached, so it's super slow. When you turn that off and set every ACL manually, it's fast.

Inherited ACLs are applied at file creation time, they can even get out of sync.

I've seen numbers in some workloads where ntfs-3g is 40% faster than NTFS on Windows, despite crossing between userspace and the kernel three times for every filesystem call (recent work in the kernel will potentially cut this down to two context switches each way for FUSE).

Same story here, I do not remember a time when I couldn't read/write NTFS from Linux since I started using it in with Ubuntu 8.04

I've been able to read/write NTFS for years via NTFS-3G. Am I missing something? Check it out if you haven't already.

This is about a kernel-space driver, as opposed to a FUSE (user space) driver. It should allow for better performance.

The way most people use NTFS on Linux is through ntfs-3g, that's why the situation in the kernel has been tolerated so long.

Beware, it does not seem functional: https://lore.kernel.org/lkml/87h7t5454n.fsf@suse.com/

Any guesses why Paragon decided to GPL the driver now?

They didn't. Samsung wrote this driver and Microsoft proposed its inclusion into the kernel.

Are you thinking about exFAT?

Yep. My bad.

Does anyone really care anymore since ntfs-3g works well enough?

I haven't checked in a really long time, but years ago ntfs-3g had a really suboptimal allocation pattern for spinning drives, and optimizations were only present in the commercial version. I don't know if the block allocation strategy was owned by ntfs-3g or if it was using something in-kernel.

If I'm remembering the details right (and that's a big if), large files written sequentially would have their blocks scattered all over the place, making e.g. large uncompressed videos written from Linux too slow to play due to seek times, when if they'd been written sequentially they would have been fine.

ntfs-3g is entirely implemented in userspace, and is far more trivial to hack on and test if anyone wanted to improve its file allocation strategies than an in-kernel driver would be.

Years ago when working on a commercial virtualization product I had to support initializing sparsely populated ntfs filesystems where only the filesystem metadata was written out, with allocated file extent ranges logged externally and not actually populated in the sparse ntfs filesystem in a file. I ended up deriving a small library for doing this from the libntfs-3g code, I don't think it took more than a weekend to do, the code was quite sane.

What I recall from that experience was that the file extent allocations on an empty filesystem was perfectly efficient, since you'd have to go out of your way to make it bad.

The real trouble was when the filesystem was used enough for its free space to become fragmented. At that point the allocation strategy was quite poor and you would end up with extremely fragmented large files. IIRC it was really just a naive sequential find first free blocks kind of thing. This aspect is consistent with your memory.

I think if we wanted to improve that aspect of linux ntfs support we're better off working from ntfs-3g.

Not to mention there are some significant down sides to in-kernel filesystem drivers, they are rather problematic when it comes to containers/namespaces and the security story is a total shit show.

I think it'd be nice to have a native driver instead of loading fuse and then a userspace driver. Is it going to revolutionize my NTFS access and make me toast in the mornings? No. Do I care to see it included? Yes.

Maybe there's something about your use case where the potential performance difference between a kernel driver and FUSE driver is hugely relevant and outweighs the security problems.

For me, the use cases for NTFS in my Linux-centric life are all fraught with trust issues when it comes to knowing what's in that filesystem isn't malicious and about to exploit a kernel fs driver bug for unfettered ring-0 execution.

It's less of a concern with filesystems you initialized from your host, on devices permanently attached to said host, with a clear chain of custody, accessed purely from the trusted environment. There's a lot of implicit trust when it comes to mounting filesystems with a kernel driver, the kernel fs devs aren't shy about admitting there's significant trust assumptions throughout those drivers - they aren't exactly hardened against malicious input, and they aren't isolated processes like w/FUSE.

Personally when a box has a sensitive enough use case (e.g. secret storage hosts) that attacks via the local filesystem are a design concern I simply don't include either FUSE or additional filesystem modules nor do I plan on mounting additional storage after boot. I configure everything with minimal possible surface regardless of if it could be constructed with more convenience.

I suppose some people may treat every box with that level of security profile but I don't think it should come as much of a surprise that level of hardening is not the normal use case considered when new drivers are staged in the kernel tree.

> Maybe there's something about your use case where the potential performance difference between a kernel driver and FUSE driver is hugely relevant and outweighs the security problems.

The idea of handling I/O errors is pretty recent in most Linux file systems, I don't think security concerns are much of an issue with adding another file system - the kernel is probably full of easily exploitable holes anyway.

ntfs-3g does not support NTFS journal replay/rollback (e.g. after Windows unclean shutdown), also it does not maintain NTFS journal for its own operations

Although I'd be surprised if this or any Linux NTFS driver could do that. It would require reverse engineering the journal format, which may even change with every Windows release.

> It would require reverse engineering the journal format, which may even change with every Windows release.

It does not. Thanks to Microsoft’s obsessive attention to backwards compatibility, you can safely mount a Windows 10 NTFS partition with journal on Windows XP (and I hear even Windows 2000) and get some form of safe read/write. However certain chunks of the fs on disk are not backwards compatible, this is mainly the system volume information used for system restore and shadow copy (and not accessible to typical user land in all cases).

More to the point, the submitted driver does, too:

> and supports journal replaying.

> which may even change with every Windows release

The logfile has changed like once. They don't want to break compatibility when you move removable disks between systems.

And even then, supporting just one version would be a big improvement.

Would that be any more surprising than reverse engineering the filesystem allocation/layout?

ntfs-3g is pretty slow with a max throughput of around 80-140 MB/s when using an SSD.

Not only slow, it also makes CPU usage skyrocket. it's especially painful on underpowered boxes like ARM SBCs or Atom portables etc.

I think that may be the point - it works, but not really well enough. If you need to use it to shift a large amount of data around, the performance is crippling.

I don't know about Paragon's NTFS driver for Linux, but their non-free Ext4 driver for Windows had a nasty bug until recently, where it would change file ownership and permissions in the filesystem mounted in read-only mode, rendering the driver unusable. The bug apparently got fixed after numerous complaints on their support forum, but I still use this driver very cautiously and only when I have to.

So I wouldn't trust Paragon's claim of "decades of expertise in commercial file systems development and huge test coverage."

I haven't used a dual boot Linux/Windows machine in ages. How popular is this setting these days?

On Linux, my most common scenario of deploying Windows is in small VMs and, when needed, mounting host folders from them. It's true these VMs will probably not run some PC software well, but I have a Windows PC for that. The other way around we have WSL and WSL2 on Windows, again kind of erasing the need to dual boot from the Windows user side.

The money for them here is in support. Maybe they have licensees that bind them to support contracts and merging software into mainline Linux counts as a "best effort" to make others chip in at keeping it functional as the kernel evolves.

Dual booting is common for people playing games. While steamplay/proton works well enough for some games a lot need either a lot of hacking or a full blown windows to run. You can do GPU passthrough VMs if you have two GPUs but I have a miniITX board and no iGPU so that is not an option for me.

Personally when dual-booting I don't share anything at all between the OS'es, but my understanding is that most do.

I haven't dual-booted in ages, but sometimes use NTFS for USB drives or inspecting drive images from Windows machines / VM guests.

I used to do this in my college days. Had some work (Unity projects especially) that would run into FAT32 file size limits and NTFS was the one more advanced filesystem that at least somewhat worked on all three OSes, albeit read only on OS X. exFAT support for Linux didn't exist at the time and FAT32, exFAT and NTFS are still the only three OS portable filesystems. Windows cannot handle HFS+/APFS, ext* is not supported on Mac and poorly supported by third party software on Windows, and ZFS/btrfs/XFS/UFS/etc are out of the question for both Windows and Mac.

Amusingly, there was actually a historical Mac ZFS effort that was, I think, sidelined in favor of APFL: https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-ap...

They had a GPT GUID for it: 6a898cc3-1dd2-11b2-99a6-080020736631.

Time Machine looks like it was designed for ZFS and some poor engineer at Apple had to graft it on top of HFS.

I really wonder how common that need is and whether it's worth adding this sort of baggage to Linux.

Honestly, for me, mounting Amiga file systems would be more useful.

ntfs-3g is adequate for me, so I don't care one way or the other if _this_ driver gets merged. And Linux is not a commercial OS, there's not some prioritised backlog controlling everything goes in, a lot of the time its what someone is willing to make the case for and sonmeone is willing to put in the work for inclusion/support. See all the esoteric/obselete cpu architectures that are still supported in the kernel - there are absolutely more users mounting NTFS drives than running Linux on PPC for example.

However, "mounting windows filesystems" is undeniably a more common case for Linux users as a whole than "mounting amiga filesystems". Maybe WSL changed this, but Windows/Linux dualbooting used to be one of the two gateways for Linux users alongside VMs. Perhaps the absence of a apfs-3g is a contributor we see less Mac/Linux dual boots.

Perhaps I'm confused, but we not had a NTFS r/W driver working since many years ago ?

There is a userspace driver with rw capability, and a kernel driver that only supports ro.

Given the initial testing people on the mailing list have done and the issues that have arisen I hope the kernel maintainers so f no and push back until said issues are fixed and code quality rises.

Was there something terrible about NTFS-3G?

The main problems are there have been no released versions since 2017, combined with an annoying "open core" development model where some nice to have features have been held back from the free version (although not ones we care about too much). For our use case (virt-v2v) speed is not an issue, but data integrity when reading, writing and modifying many dozens of files absolutely is, and ntfs-3g has been very reliable.

(Disclaimer: I've made occasional contributions to ntfs-3g upstream)

Uhhhmmm... Speed? Stability? NTFS-3G has been a lifesaver for occasional use, but with anything remotely I/O intensive, it starts to crack.

Why would you use NTFS on linux for anything other than occasional use?

To make backups of data on a Windows partition. Playing games residing on a Windows partition. The main purpose I can think of though is forensics (images might contain NTFS).

It's the only non-FAT filesystem with good cross-OS compatibiity.

This. Now, you've made me wonder if in the near future you could have both Windows and Linux installed on the same -gasp- partition.

I had for years. Usung Wubi.

Can't you use ext2/3 pretty much everywhere too?

To deploy Windows from WIM imeges. Windows PE isn't something one would even consider using.

I didn't know this was even possible but now I'll be looking into it. Thanks.

Why should it matter?

Why not? It's a good filesystem, why not use it?

It's kind of a garbage filesystem though.

HN gets all bent out of shape about minor differences between btrfs and zfs. Like how zfs supports raid5/6 but btrfs raid5/6 support is immature. Meanwhile NTFS doesn't even support checksumming, multiple block devices, copy on write, the performance on SSDs is garbage, etc etc.

The whole of the NT kernel was essentially a futurized implementation of VAX-VMS.

Both VMS and NT had the same manager, in the person of Dave Cutler. Microsoft hired a large team away from DEC to build NT (which became the subject of litigation from DEC).

I have read that NTFS shares common architecture with Files-11/ODS-2&5 (native VMS filesystems). Aspects were also lifted from OS/2.

ZFS was developed a decade after NTFS. Most UNIX was still on FFS when NTFS was born.

https://wikimili.com/en/Files-11

Ext2 was AFAIK a FFS-style design.

At least that 'garbage' hasn't died on me like the majority of my BTRFS partitions have. (Thankfully they all mostly still worked in read-only mode.)

Stability hasn't been an issue for us. Virt-v2v uses ntfs-3g intensively and we have migrated a large fraction of a million VMs, many (the Windows ones) all using ntfs-3g. We've not had problems with ntfs-3g at all.

I'm still interested to see if an in-kernel driver can be better.

(Disclaimer: I have contributed a few patches to ntfs-3g upstream)

Has this been merged already?

It was posted yesterday and the replies indicate issues with the patch. So as these things usually go it'll probably take a long time until it's ready for inclusion.

Assuming that Paragon is willing to actually follow the process to get this included in the mainline kernel, it will likely be at least several months before this is fully available in the kernel (and a bit longer for it to become available in your distro).

Argh, Russian hackers, again!

Bad line-wrapping in the first commitmsg does not leave a first good impression.

Good news because NTFS-3G seems dead - last update[1] was in 2017. Moreover, quality of the code is far from perfect.

[1] https://tuxera.com/opensource/ntfs-3g_ntfsprogs-2017.3.23.tg...

Reply on news.ycombinator.com