github.com
- simple
- efficient
- dependency free
- runnable anywhere with a filesystem
- header only
Watcher is extremely efficient. In most cases, even when scanning millions of paths, this library uses a near-zero amount of resources.
Watcher is simple. The library exposes a single function and a single object. That is all.
Happy hacking.
Option 1 uses a lot of CPU and memory (the map storing the paths being monitored could easily grow to be tens or even hundreds of megabytes if many files are being monitored, which is often the case in large source projects). I have seen tools that use polling with a 100ms interval continuously burn 50% of cpu monitoring a modest sized directory with tens of thousands of files.
Option 2 theoretically would use less memory and little to no cpu, but in practice, the story is more complicated. If you are using an inotify or kqueue like api, you will have to store handles for all of the paths that are being monitored, which can take a significant amount of memory. On macos, the file system events are not accurate in the sense that you can't trust the type of event. It doesn't reliably distinguish between creation and modification events. So if you want to know specifically what kind of event happened, you end up back in case 1 where you have to store an in memory representation of the file system and diff against the in memory representation and the current file system state when you detect an event. For some use cases, you may not care to distinguish between creations and modifications and can get away with a lower memory, but less accurate, solution.
In my experience, getting all of this right is much more difficult than it appears at first glance. Good luck to you.
There is ongoing work attempting to make it more perfect.
I expect a year or two before this is complete.
For now though, it does do what it says. The tests I’ve run show that it is accurate over large amounts of events and time. For under 1 million files and/or directories, it uses a near-zero amount of resources. Testing on older processors shows similarly positive results.
But this is so far from perfect. This is only the groundwork. Most of the bugs have yet to be discovered. The platform support, more often than not, uses the safe “baseline” watcher in favor of accuracy.
Ned14 of Boost fame has given the project some expert advice which will help it along smoothly.
For what it's worth, in spite of ned14's comments, I have never seen inotify fail in practice (except for if it hits the os file descriptor limits in which case it does fail noisily). The tool I wrote uses inotify for linux. It is used by thousands of developers every day as part of an editor integration and there are no open issues about dropped file events.
Your time frame is probably about right. It took me about a year to work through all the edge cases.
The baseline Watcher’s efficiency has a wide spread. When there are many thousands of nested subdirectories, the CPU approaches the limit of the thread it’s on. Flatter directories, or many files without nested subdirectories, do not have nearly as much of an effect. I’ve seen it run on around 10 million paths on a very flat test directory.
So, near-zero is somewhat misleading. There’s a wide spread in efficiency. It was my judgement that deeply nested directory trees were far less common in practice then, so I wrote “near-zero” in the optimistic case.
It uses polling under the hood (at least, I’m sure it does. It uses whatever std::filesystem uses, which is almost certainly polling).
A “baseline” filesystem watcher which uses only the standard library. It has been made to beat kqueue. And it does.
A platform filesystem watcher for Darwin is used, but certain event properties are handled by the standard library. Namely, the event time and the path type.
A platform filesystem watcher is schedule for Windows. Work hasn’t been started.
A platform filesystem watcher for Linux (> 2.4 or so) was toyed with but ultimately rejected out of accuracy concerns. It was far more efficient than the cross-platform implementation “warthog”, no doubt, but it lacked accuracy. Work is being done to get most of the benefits from both worlds.
There are problems with the “baseline” watcher (which I’ve named “warthog” because it’s sturdy and reliable). But those are potential efficiency losses when watcher more than a few million paths. They are, thankfully, not accuracy or safety problems.
Maybe you can see the solution emerging here?
Here’s where we’re going next:
The most efficient kernel watchers can be used on most platforms, but checked for their accuracy periodically by the “warthog” watcher.
How does the baseline filesystem watcher work? If it doesn't use kqueue, does it poll the filesystem periodically and diff against an in memory representation? If yes, see my other comments. If not, I am genuinely curious what you are doing because you know something that I do not.
I moved to a minimal std::filesystem-based watcher and optimized it from there.
There hasn’t been a formal head-to-head test between the two. That should be about halfway down my todo list. It’s worth revisiting more formally.
My response to this question should help here: https://news.ycombinator.com/item?id=33247155#33251437
In short, there’s no secret sauce. There’s an efficiency spread in (what I consider) edge-cases.
Every potential gain over other naive watchers implemented with kqueue is likely algorithmic. I store events in a historical map, compare differences to the current state of the file tree, prune them, and send events when they change. That’s the whole implementation: scan paths, record their attributes, check for differences in the map, and send events when they happen. I haven’t given much thought to exactly why it beats kqueue, nor are there any good tests showing by how much. (Again, this is worth doing.)
Just as a heads up, one of the strange fsevents issues is that it fails if you register two directories where one directory is a prefix of the other. So say that you want to monitor directories $ROOT/foo and $ROOT/fo and you register an event stream first with $ROOT/foo and then $ROOT/fo, you will only receive events for paths in $ROOT/fo and no events for paths in $ROOT/foo (I just double checked that this is still the case in Monterey at least). I never bothered to report this to apple but worked around it by just registering a stream with $ROOT if I detected that one path name was a substring of another.
I'm really surprised that this sort of functionality isn't built into OS's/filesystems. I recently had to do this for HDFS, and I finally "gave up" and polled the file system like you suggest as your first option. Event notification seems like something that ought to be a fundamental feature and is best owned by the file system itself.
The magic file approach described by kevincox below is probably the best way to get > 95% of the benefit with < 1% of the work.
It appears to be built into macOS [1]?
> Whenever the filesystem is changed, the kernel passes notifications via the special device file /dev/fsevents to a userspace process called fseventsd
Which I assume is what they're referring to here:
> A platform filesystem watcher for Darwin is used, but certain event properties are handled by the standard library. Namely, the event time and the path type.
It is, but it's badly implemented and buggy. But the real problem is that there is no posix like specification for file system events so every platform does it differently. Even if every platform implementation were perfect and bug free, it is a huge pain to write wrappers for each one.
This needs at least some bullet points on HOW it does this so efficiently so that I'll keep looking. A blanket statement like this means "they hope it is efficient" or "They want it to be efficient" or "It's good in some scenarios but not others".
With those additional bits, I have a reason to dig around the source.
If so, there are equivalent options, including systemd path units, incron, and the inotifywait utility, in addition to the C API.
The "man systemd.path" page does list explicit limitations of this kernel system call:
"Internally, path units use the inotify(7) API to monitor file systems. Due to that, it suffers by the same limitations as inotify, and for example cannot be used to monitor files or directories changed by other machines on remote NFS file systems." (Files modified by mmap() also don't trigger events.)
https://www.linuxjournal.com/content/linux-filesystem-events...
Windows busybox also has an inotifyd, which appears to do something similar.
The current solution isn’t ideal, and is being addressed here: https://github.com/e-dant/watcher/issues/10
A proper benchmarking program is in the works, however manual testing does show only minimal resource usage.
For more, see this issue: https://github.com/e-dant/watcher/issues/10
https://www.voidtools.com/forum/viewtopic.php?t=9792
but programmatically and its highly scriptable, pretty cool. Will definitely add it to my arsenal of troubleshooting tools.
Edit: never mind, This tool is manually scanning the filesystem instead of listening to OS events https://github.com/e-dant/watcher/blob/989147b183ee0547d71a1...
still checks out :) But I checked just to be sure and no warning in Version 106.0.5249.91 (Official Build) (32-bit). Maybe its your corporate baby content web filter?
I'm particularly interested in something like this, but which will include information about what process made the change, and which user it was running as at the time.
Looks like it:
https://github.com/e-dant/watcher/blob/989147b183ee0547d71a1...
You can check out issue/10 for a full description of how it works now, why neither inotify nor our current solution is ideal, and where the project will be going next.
Have you looked into using eBPF for tracking file system changes at any point? (I don't mean for this project, as it's clear you're taking a particular approach that will work across platforms).
This one looks interesting: https://github.com/emcrisostomo/fswatch
Looks like it works on quite a few systems including Android and iOS.
Although, to be more efficient, I need to write system API calls for Windows.
That will be the 1.0 release.
That’s a good test case. I’ll make an issue.
sane - for node
watchexec - rust based, static binary
1. No need to worry about which files to watch or ignoring build outputs.
2. Works with every project with no setup.
3. Easy to trigger a re-run without actually changing a file.
4. Always runs after all files are saved instead of starting after the first file is saved and racing the rest.
5. Infinitely scalable.
My current setup is documented here but it's easy to tweak to your prefered workflow. https://kevincox.ca/2022/06/14/small-tools/#w