Hacker News Reader

Comment by jhallenworld | Hacker News Reader

This is all fine, but Window-NT file access is still slow compared with Linux- this shows up in shell scripts. The reason is supposedly that it insists on syncing during close, or maybe waiting for all closed files to sync before allowing the process to terminate. Shouldn't close finality be an optional async event or something?

View on HN

The reason is due to file system filters, of which Windows Defender is always there. There is a significant delay from Defender when performing CloseFile()[0].

> As I was looking at the raw system calls related to I/O, something immediately popped out: CloseFile() operations were frequently taking 1-5 milliseconds whereas other operations like opening, reading, and writing files only took 1-5 microseconds. That's a 1000x difference!

This is why DevDrive was introduced[1]. You can either have Defender operate in async mode (default) or remove it entirely from the volume at your own risk.

The performance issue isn't related to sync or async I/O.

[0] https://gregoryszorc.com/blog/2015/10/22/append-i/o-performa...

[1] https://devblogs.microsoft.com/visualstudio/devdrive/

Windows FS stack is still _way_ slower than Linux. Filesystem operations have to create IRPs and submit them for execution through a generic mechanism. These IRPs can get filtered and modified in-flight, providing quite a bit of overall flexibility.

In Linux, filesystem paths are super-optimized, with all the filtering (e.g. for SELinux) special-cased if needed.

But even still, Windows also had to cheat to avoid completely cratering the performance, there's a shortcut called "FastIO": https://learn.microsoft.com/en-us/windows-hardware/drivers/i...

I wrote a filesystem for Windows around 25 years ago, and I still remember how I implemented all the required prototypes and everything in Explorer worked. But notepad.exe was just showing me empty data. It took me several days to find a note tucked into MSDN that you need to implement FastIO for memory mapped files to work (which Notepad.exe used).

That's interesting, why would notepad.exe use mmapped files?

Probably to save space when opening large files?

But it simply isn't slower than Linux.

Robert Collins explains that performance is just as good as Linux and the performance loss on Windows is due to file system filters (Defender)[0].

This is what DevDrive intends (and does) fix.

[0] https://youtu.be/qbKGw8MQ0i8?t=1759

The last time I did FS tests was admittedly around 4 years ago, but back then Windows was several times slower on pure FS performance benchmarks (creating/listing/deleting large number of directories and files).

It used to be _much_ slower, like orders of magnitude slower, especially for directories with a large number of files.

You're not seeing "pure" FS performance. You're seeing all of the abstractions between you and the file system.

To get more of the abstractions out of the way, you want DevDrive. And don't use Explorer.exe as a test bed which has shims and hooks and god knows what else.

I was testing the performance using plain C++ code.

These days there is a full native Btrfs implementation for Windows.

https://github.com/maharmstone/btrfs

It's complete enough you can boot and run Windows from Btrfs.

https://lilysthings.org/blog/windows-on-btrfs/

I wonder how many of these limitations affect this, notably performance.

If WSL1 could natively talk to Btrfs without the performance-sapping translation to NTFS, would that resolve the poor Git performance etc?

This probably has a lot to do with the mandatory file locking on Windows - afaik on Windows, the file is the representation of the underlying data of the disk, unlike on Linux, where it's just a reference to the inode, so locking the file on open is necessary. This is why you always get those 'file in use and cannot be deleted' prompts.

This impacts performance particularly when working with a ton of tiny files (like git does).

Reply on news.ycombinator.com