Improve Git monorepo performance with a file system monitor

Git for Windows currently supports Vista and above - the AF_UNIX support was only added in Windows 10 1803.

Named pipes are fine, the semantics are basically identical, and you can guarantee there is a separate namespace versus the filesystem (for AF_UNIX, \x00 prefixes work on Linux but not on macOS).

My assumption was that on Linux it's just been using inotify or something for a while and so hasn't needed a bespoke monitor. I have no idea if that's true or makes sense though.

My guess is that inotify is so slow with large directories that it wasn't worth it. Plus inotify has cumbersome user limits.

inotify has a number of other relevant limitations, like not being able to create recursive notifications or handle "move" operations. Implementation effort is going to be way higher for an inotify-based system, and of course that's made far worse by the numerous file systems in linux - I imagine any implementation would probably start first with ext4.

I suspect an ideal solution would be via ebpf, but I'm not sure.

I've been wondering about why there was no linux support, and found an e-mail from the author of the subcommand (as well as the github.blog post) explaining the situation.

Apparently an older implementation using inotify was dropped because inotify does not work recursively, so you would have to do an inotify call for all directories of the hierarchy which is obviously very inefficient. There are system wide limits in the number of directories you can listen to, and even if you increase the limit you would probably cause a lot of overhead.

Newer linuxes support the fanotify system call, which does allow recursive listening. They haven't implemented something using fanotify yet however.

https://lore.kernel.org/git/e1442a04-7c68-0a7a-6e95-304854ad...

Yeah not supporting Linux in a Git feature? WTF what a free software sellout moment.

I've stopped using anything language specific like Guard or nodemon. Instead I use the inotify commands on Linux and entr/ack on macOS so no matter what I'm doing, I can watch for changes in a directory.

Except it's not cross-platform:

> It is currently available on macOS and Windows.

I’ve got qualms with just about all of the big tech companies in one way or another.

My 2c is that one of the unambiguously positive externalities of the tech mega-corp trend is all the great OSS we get as a by-product of their operations.

I mean, I don’t exactly love how iPhones get made, but I’m pretty stoked that clang kicks ass now.

We recently used Turborepo -- https://turborepo.org/ -- for a project that started as a two Electron-based builds that quickly escalated to a three Electron-based build. Once we had it set up for the two, it was quite easy to add one more. Our shared components were in a central package while custom ones existed in their respective app directories.

The nice thing with this type of separation for us is being able to target CI/CD scripts to specific apps. Previously we were using targeted dev script logic to initiate the different app builds which just wasn't maintainable. This new approach this time around made for Electron deployments to be super simple, consistent, and repeatable.

All this was done by two team members on dev side.

In JS/TS world, until recently yarn was a de facto tool for monorepos. Recently pnpm looks like getting traction. There are a few other new tools recently getting popular as well, it's a hot topic.

My holy grail implementation would be a "partial clone" that downloads desired files like normal, but creates stubs for selected files that are not stored on the device but downloaded on-demand upon opening them, like the OneDrive Files On-Demand [1] or Google Drive File Stream.

[1]: https://support.microsoft.com/en-us/office/save-disk-space-w...

On the bright side, filesystem operations on Windows are already slow so you don't need to worry about turning on a file-watcher there.

This has been the way to get performance on a large Git repo for over a decade now, just not built into Git. It provides very good improvements even in environments that aren't the fastest, like laptops.

Hooking writes at a kernel level shouldn’t have much impact, provided it’s actually hooking there

My main complaint with the Windows implementation is that it does not not play well with the lock-by-default policy of the NTFS filesystem. I deactivated the the filesystem watcher on Windows after the agent repeatedly locking files so that checkouts would fail.

I'm using ReadDirectoryChangesW() to read a filtered stream of events from the USN journal. I've not noticed any reliability problems. Technically, the kernel API can always drop events, but whether that is a kernel can't keep up problem or the daemon application not servicing the event stream fast enough doesn't really matter. The API does know if/when events were dropped. And the FSMonitor daemon guards against that and forces a "resync", so the "git status" client is advised to do a normal scan, so the output is correct.

It's simple, Microsoft who implemented this, only needed support for Windows and MacOS. There is an integration with watchman which runs on Linux.

Enabling this messed up something related to projectile/helm-projectile which I use to navigate to files, and is an integral part of my git/magit setup in Emacs.

the buffer projectile-files-errors says "warning: Empty last update token."

so what is?

As with a lot of developer tools, the most adopted solutions are rarely the best tool for the job. But because everyone knows them, thats what continues to be used.

Moving to a continuous, asynchronous strategy versus a point-in-time synchronous strategy, seems like a perfectly reasonable way to improve performance.

All file operations involving many small files are slow on Windows, that's hardly git's fault. It can just do its best to work around the problem (for instance with this file watcher thingy).