Hacker News Reader

Comment by mizmar | Hacker News Reader

Similar to make, it does mtime chronological comparison of dependencies with target to determinate if dependencies changed. This is just so flawed and simple to fool by operations on filesystem that do not change mtime (move, rename):

1) pick a source file and make a copy of it for for later 2) edit selected source file and rebuild 3) move the copy to it's original location 4) try to rebuild, nothing happens

View on HN

[ninja author] I did some thinking about this problem and eventually revisited with what I think is a pretty neat solution. I wrote about it here: https://neugierig.org/software/blog/2022/03/n2.html

Imagine if filesystems had exposed the file hash next to its mtime.

I might be missing your sarcasm, but this is a common approach for large scale builds. Virtual filesystems are used to provide a pre-computed tree hash as a xattr. In a more typical case, you can read the git tree hash.

Not sure it was meant as sarcasm really. I just think so many build (and other) problems could have been avoided it a file hash was available on every file by default.

That hash would be expensive to maintain, and the end result would still be racy since the file could be modified after the hash was read .

In the current POSIX paradigm yes, it would be expensive. But if the hash was defined as the hash of fixed blocks, it wouldn't be expensive. The raciness depends, a lot, on the semantics we would define. (In the context of a build system, it's no different than that the file could get a new mtime after we read the mtime.)

By not tracking file metadata through an index file, mtime-only incremental build systems trade a lot of reliability for only slightly more simplicity. https://apenwarr.ca/log/20181113

Copy (1) and edit (2) both bump mtime, usually. It's not obvious that in the workflow you describe ninja is problematic, rather than the workflow itself (which is atypical).

ninja fails to detect that file changed from last build - all it's mtime, ctime, inode and size can change, yet it's not detected as long as mtime is not newer than target.

Again, this is just a weird workflow, and you're assuming copy/edit don't bump mtime. That usually isn't the case. If you're doing this weird thing, you can just run `touch` when you move files over existing files like this to explicitly bump mtime.

>you're assuming copy/edit don't bump mtime

Incorrect, I only assume move/rename of backup to original location doesn't change it's mtime (which it doesn't with default flags or from IDE or file manager). And I don't think this is a weird or obscure workflow, I do it all the time - have two versions of a file or make a backup before some experimental changes, and restore it later.

I used to do that some 20 years ago when I was learning programming, but now it does seem like a weird workflow when git exists and handles this case well.

I run into this issue when building against different environments, each with a

1. A library depends on a system package. To test against the different versions of the system package, the library is compiled within a container.

2. To minimize the incremental rebuild time, the `build` directory is mounted into the build container. Even when using a different version of the system package, this allows re-use of system-independent portions of the build.

3. When switching to a build container with a different version of the system package, the mtime of the system package is that of its compilation, not that of the build container's initialization. Therefore, the library is erroneously considered up-to-date.

Because the mtime is the only field checked to see if the library is up to date, I need to choose between having larger disk footprint (separate `build` directory for each build container), slower builds (touch the system package on entering the container, forcing a rebuild), or less safe incremental builds (shared `build` directory, manually touch files when necessary).

> 3) move the copy to it's original location

Copy and edit do, but move (aka rename) generally does not, and that is the part that is problematic.

I don't think the described sequence of operations is all that unusual. Not the most common case for sure, but hardly unlikely in the grand scheme of things.

My guess is that it's for drop-in compatibility with make.

There is (at least) one open issue about this - the solution/alternatives are not trivial:

https://github.com/ninja-build/ninja/issues/1459

Good point. I think it would be fixable by using the Change Time instead of the Modify Time, because that changes when moving the copy over the original.

Reply on news.ycombinator.com