Hacker News Reader

Comment by lc64 | Hacker News Reader

That's a very rigorously written article.

Let's also note the 4x speed increase on windows 10, once again underlining just how slow windows filesystem calls are, when compared to direct access, and other (kernel, filesystem) combinations.

View on HN

maybe the malware detection program adds to the performance as well

NTFS is really horrible handling many small files. When compiling/watching node modules (easily 10-100k files), we've seen a 10x size difference internally (same hardware, just different OSes). At some point that meant a compile time difference of 10-30 sec vs 6-10 min. Not fun.

NTFS is perfectly fine at handling small files and performs on-par with other modern file systems.

The issue is Defender in sync mode/other AV/other file system filters.

DevDrive as noted by default uses an async scanning technique as well as ReFS. ReFS will suffer the exact same performance issues with Defender (or other AV/other file system filters) doing its thing when running in sync mode, which it does by default for ReFS-formatted drives in Windows Server.

https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/

https://news.ycombinator.com/item?id=26737521

> Except for CloseHandle(). These calls were often taking 1-10+ milliseconds to complete.

> While I didn't realize it at the time, the cause for this was/is Windows Defender. Windows Defender (and other anti-virus / scanning software) typically work on Windows by installing what's called a filesystem filter driver.

This doesn't take away from your point that _it is slow_, but the reasons are not due to the file system in use.

>The issue is Defender in sync mode/other AV/other file system filters.

I've had folders take a full minute to open on an SSD.

It got to the point where I went to open the folder, it started loading. I needed the file quickly, so I searched for it online, found it, and opened it before windows finished loading that folder for me.

After exempting that folder from Windows Defender the folder loads instantly. For the life of me I cannot understand why Defender blocks Explorer.

Probably because Explorer hosts shell hooks which can potentially execute arbitrary code.

Just one example: File icons or thumbnails can be dynamically generated by shell extensions based on the file contents. A maliciously crafted file could potentially exploit a vulnerability in such a shell extension.

> For the life of me I cannot understand why Defender blocks Explorer.

I suppose if you wanted to find out, you could use dtrace/ETW.

Explorer has other things going on, though, including other apps that hook into it (shell extensions, like Adobe Reader, TortiseGit/SVN, and so on) which can certainly cause performance issues.

Technically, they're because of the filesystem in use: it's providing the APIs these garbage-ware utilize... which causes the performance issues ( ◠ ‿ ・ ) —

File system filter drives apply to all (RW) file systems on Windows. It's not exclusive to NTFS or ReFS.

Windows has an extensible model. It's a different approach from most (all?) other OSes. It offers a different set of features.

Sure, AV could perhaps be done in a different manner that would be more effective/faster, I can't comment on that as I lack the insight required -- only MSFTies that work on kernel code could respond in any authoritative way.

I was of the understanding that these sync APIs are only available on Windows filesystems, so a fat32 formatted filesystem wouldn't suffer the same performance impact, which is why windows provides "virtual drives" for performance on their cloud instances that give you extra performance... Precisely by formatting them with a filesystem that doesn't support these sync/blocking Apis.

But I'm not particularly knowledgeable either on this topic, just a (forced) consumer of the operating system with the occasional reading on the side

Azure Temporary Storage disks are stored on the host hypervisor machine. Non-temp storage is stored in Azure Storage. That's the only difference.

FAT[32] does implement minifilters.

https://learn.microsoft.com/en-us/windows-hardware/drivers/i...

https://www.osr.com/nt-insider/2019-issue1/the-state-of-wind...

Thanks for broadening my horizon, esp. that first link was extremely interesting!

I guess my mental model on this was just lacking/simply wrong.

I find this stuff super fun to learn about. My knowledge is far from complete but the more you know, the more pedantic you can be about things like 'virtual memory' and the misnomer of setting the min/max page file to 1.5*RAM (was used to ensure full memory dumps, not for performance).

If you want to know more, grab the book Windows Internals.

I don't know if async, AV, NTFS, or what, but some years ago we could demonstrate reliably that running a "big" maven compile on a lot of .java files was faster in Linux than Windows.

The kicker... Linux was running inside a VirtualBox VM inside the very same Windows host.

This could also be some variance in the `javac` command between OS's, granted.

Doesn't it (Windows VFS layer) also lack the equivalent of dentry cache, making all metadata lookups slow?

The MFT describes resident (in-MFT) metadata about files, or non-resident (external to the MFT, but the MFT has a pointer) if the metadata does not fit within the MFT. The MFT itself is a database; Windows makes use of file system caching, be it in-memory or another caching mechanism, like on-device disk cache.

So why is it still pretty slow for file operations on DevDrive (no filters, async antivirus)? I found the performance improvement not that significant.

Doesn't match what I found.

> NTFS is really horrible handling many small files.

To pile onto NTFS, it's performance is so notoriously bad that there are developer teams working on Windows projects that configure their build farm to do cross builds from Linux to Windows just to avoid the performance penalty.

As an anecdote, we had a really long build time for our pipeline (going back prob 15 years).

I argued for a Linux laptop, and the boss said, "OK, prove it. Here's two equivalent laptops, time it.".

Turns out there was zero difference, or negligible (Windows won), between compilation times. That has always annoyed me.

> Turns out there was zero difference, or negligible (Windows won), between compilation times.

I think there was something seriously flawed in your test. If you Google for a minute, you find multiple posts on how moving the same builds to Linux led to performance improvements in the range of 40% drops in build times.

Some anecdotes even compare doing the same builds in Ubunto with NTFS to see double-digit gains.

NTFS is notoriously awful in scenarios involving reading/writing many small projects. This is the bottleneck in Windows builds. There is a myriad of benchmarks documenting this problem.

Nowadays there are plenty of cross-platform projects to serve as benchmarks. Checking this can be as easy as checking out a project, start a full rebuild, and check how long it takes.

NTFS can be especially awful if you're used to Linux behaviour, port code over to Windows and expect the same performance.

Here's a talk on porting rustup to Windows: https://www.youtube.com/watch?v=qbKGw8MQ0i8

To begin with, it takes rustup 3m30s to install on Windows. After rejigging the code again and again away from its naive implementation which works fine on Linux, to perform sympathetically towards NTFS, it takes the same rustup 14s to install. That's quite a performance gain! But it needed a lot of changes to rustup, and to Windows itself.

Yeah you can work around the filesystem perf problems on Windows to some extend. Unity builds are not just popular because it reduces duplicated compiler work but also because it reduces filesystem access. There is a reason why Windows games tend to pack their assets into archives.

That's survivor bias. Fewer people blog about unsuccessful initiatives.

> That's survivor bias. Fewer people blog about unsuccessful initiatives.

I don't understand what point you're trying to make. Should we give equal credit to unsuccessful initiatives when their failure is due to screwing up a critical part of the project?

I mean, the successful attempts document what is possible. What do you expect to report when you couldn't even manage to get something working?

Say there are N initiatives to move to cross-compilation. Of those, there are M that deliver the hoped-for benefits, where M <= N. But practically, M < N significantly, say, M < 0.9 N.

Assume bloggers blog mainly about content that contains a positive message. I'm asserting that people blog more readily about their success than their failures.

So when you look at the blog literature, your population is not N, it is M. You don't see the failures because they don't tell the tale.

It seems possible that their pipeline just had large files, or something like that.

Java/dotnet pipelines don't have Windows penalty. Both have dependencies as compiled modules zipped into large jar/dll. While even simple node project have 10s of thousands of files inside node_modules folder.

For the most part the two are pretty equivalent in performance, it's particularly constant blocking on small file IO where Windows falls way behind. Also a bit in the "tons of cores" and "tons of networking" categories but not as nearly as bad as small file IO.

Well yeah, if you avoid the OS (syscalls) as much as possible then all operating systems will have about the same performance.

Love that your boss was open to the idea but also wanted data to back it up. (imagine a slow clap)

A single datapoint from an experiment done by only one person you call it data? What's not to love...

It was the only data point which mattered for the decision that needed to be made.

Yep, was a good experience. I'm sure with fine tuning it (Linux) could've been better, but I ate that humble pie.

You can't have windows laptop without running a ton of Antivirus and other crapware .e.g MS store, HP/Dell diagnostics and MS Teams. If you factor that in, Linux will win everytime.

If your company has windows and Linux but only guards against threats on windows, then you may have a different problem.

I believe you can lock a Linux box down tighter than a windows box, but then you're trading compile times for other costs.

Not the case unfortunately; you can. I'm not defending Windows here (bitter hatred of Microsoft here, since they started charging 2k for the compilation tools back in the 90s).

In your particular corporate environment that might be the case, but not in this case, I had free run of a fresh install and no offensive AV there, and detuned to remove the crap.

Other posters have said certain optimizations (which I'm not sure would help, it was pure compilation, no large files that I'm aware of). Just saying, always good to keep an open mind.

Were all your tests pure unit tests? Then sure—the OS isn’t getting in the way (and if your Linux distribution didn’t set the correct cpu scheduler, windows could be faster!)

If, however, your tests make any filesystem calls or fork a child process, there’s slim chances that Linux doesn’t absolutely trounce Windows.

To throw in a morsel of anecdata: same laptop with a dual boot runs the PhotoStructure core test suite (some 11,000 tests which have a pleasing melange of system and integration tests, not just unit tests) in 8m30s on Windows 11. Almost all the same tests complete in 3m30s on Linux.

Seems weird to base the argument on compiler performance alone. I'd think that developer tools will be a much bigger factor for the time difference but that depends on what you are used to.

Anecdotally though, git is unbearably slow under Windows and compiles make all filesystem operations lag while I have never seen such problems under Linux.

Yeah sorry, I left out a lot of details for a quick anecdote. It was Java compilation, so the different JDK's may've made a difference (almost certainly). But I figured the better file system handling etc etc would've gone in favour of Linux, versus Windows.

In any case it wasn't much of a fun codebase. But I think a good lesson was, always test it, always measure. Not casting shade on either OS.

Do you remember which file system you used on Linux?

Didn't they introduced ReFS ("dev drives") to alleviate this?

Why wouldn't they work on improving NTFS performance instead?

Much of the cost is imposed by semantics and the security model; there's no silver bullet solution that can be imposed without turning it into a different filesystem with different semantics and/or on disk representation.

At one point they planned to replace it with a database filesystem, but that was too complicated and abandoned. That was probably the end of replacement work on NTFS. https://en.wikipedia.org/wiki/WinFS

WinFS was never designed as a replacement as it rode on top of NTFS. It was a fancy SQL Server database that exposed files as .NET Objects for API interaction.

Sans the API piece, think of it like storing blobs in SQL Server, just like SharePoint does.

I was lucky enough to play around with beta 1. Not much you could do with it, though.

> Why wouldn't they work on improving NTFS performance instead?

There are other old-school techniques which are far easier to implement and maintain, such as using RAM drives/partitions. Expensing 32GB of RAM is simpler than maintaining weird NTFS configurations.

Splitting your project into submodules/subpackages also helps amortize the impact of long build times. You can run multiple builds in parallel and then have a last build task to aggregate the all. Everyone can live with builds that take 5 minutes instead of 3.

Even Microsoft doesn't touch NTFS code (if they haven't lost it). All new file system features like new compression methods are implemented on layers above ntfs.

Because likely they aren’t a. allowed to, and/or b. have no file system code experience or understanding.

That may be due to a combination of Malware detection + most unix programs not really written to take advantage of the features NTFS has to offer

This is a great talk on the topic https://youtu.be/qbKGw8MQ0i8?si=rh6WJ3DV0jDZLddn

It's more a case of Linux programs not being written to work around the performance issues of Windows filesystems + layers above them. NTFS doesn't offer magical featurs that fix the performance.

Must be why windows 11 added that dev drive feature

That, and continuous scanning of all file IO corporations love so much can 2x-10x that time still. Dev drive is excluded from Defender scrutiny by default because of that.

Not excluded, but the scanning in a dev drive is asynchronous rather than synchronous.

It’s somewhat more complex than “NTFS is slow”. Here’s a good explanation: https://github.com/Microsoft/WSL/issues/873#issuecomment-425...

I’ve benchmarked deleting files (around ~65,000 small node_modules sort of files) and it takes 40 seconds through Explorer, 20 seconds with rd, and roughly a second inside WSL2 (cloned to the VM’s ext4 virtual hard drive).

Why not use https://github.com/bobranten/Ext4Fsd ??

So we should put node_modules into SQLite database?

Honestly, maybe? Would be a fun project to look at at least.

Their "windows dev drive" setup addresses this. I haven't tested it myself but I saw a couple of inexpertly executed tests showing significant performance gains. I honestly have no idea if my compile times are quicker.

You can get benches from the horses mouth.

https://devblogs.microsoft.com/visualstudio/devdrive/

But the primary improvement comes from async A/V, not the file system.

ReFS offers other improvements such as CoW and instant file initialization which can benefit developers. From this standpoint, it is the correct choice over NTFS.

https://devblogs.microsoft.com/engineering-at-microsoft/dev-...

With the right code, NTFS is not much slower than ext4, for example. Nearly 3% for tar.gz and more than 30% for a heavily multi-threaded use like Pack.

https://forum.lazarus.freepascal.org/index.php/topic,66281.m...

https://news.ycombinator.com/item?id=18783525 has previous discussion on why windows filesystem is slow

But is it faster than accessing the filesystem with io_uring as well? I feel like this article should be updated

io_uring/IOCP won't change how long it takes to access a file.

So what's exactly the performance benefits of using io_uring to access files?

It allows you to do something else while waiting for the request to complete. Linux has historically bad hacks to enable this. Windows NT has had IOCP since it's inception, but not only for storage devices, as io_uring is limited to, but networking, TCP sockets, mail slots, named pipes, etc.

"Are you done yet?" vs "Get back to me when you're ready, gonna go do something else". Aka blocking i/o vs nonblocking i/o.

(I always wonder why Windows NT doesn't get more high performance implementations for networking et. al. with this feature; licensing? trade off in perf elsewhere? can't tweak kernel params to your heart's desire?)

io_uring implemented an additional feature of a ring buffer, but Microsoft has followed this feature; I think it was first introduced in a later build of Windows 10 and should be in 11 & 2022.