Mtime comparison considered harmful (2018) | Hacker News Reader

Mtime comparison considered harmful (2018)

apenwarr.ca

67 pointsalex_hirner33 comments

Weird that there is such detailed technical information alongside this statement:

> And anyway, purists might argue that the "content" of a directory doesn't change when the files it points to change; the content is merely a list of filenames and inode numbers, after all, and those stay the same, no matter what happens inside those inodes. Purists make me sad.

Or maybe Unix makes the author sad? You can’t wish the OS into doing something different with directory files when the set of contained files has not changed (but a file was modified) so that these Make troubles could be handled differently.

Am I missing something?

The author mentions that they also wrote a backup program (bup), and for backup programs it would be very convenient if directory mtimes would get updated like this (recursively up to the root), as it would allow to skip scanning the entire filesystem for changed files (which in my experience is where backup programs spend most of their time).

I don't remember if mtime is updated on each write call or just on fopen but I could see this being a huge performance overhead, namely for applications that are FS bound. I wonder if io_uring would help the situation though since it's mainly geared toward filesystem operations.

No reason a hypothetical recursive mtime needs to be atomic. The kernel could just stick it in a buffer somewhere and deal with that sort of think out-of-band and in batches. You'd probably need some filesystem journaling trickery if you want to make sure the recursive mtime always updates eventually when a file is modified.

> Annoyingly, when you update the content of a file, the mtime of its containing directory is not changed. All sorts of very convenient tree traversals would be possible if the directory mtime were updated (recursively to the root) when contained files changed, but no. This is probably because of hardlinks:

He lost me here. As much as that's a would be cool kinda feature it smacks of a special kind of tunnel vision: what's the point of anything but what you happened to be working on?

Consider how many writes happen on your system. Consider what sliver of a fraction of them even come under mtime scrutiny. The idea is to amplify those writes all the way up the directory hierarchy? For every write? And contend on ever write to /home/?

We already needed a noatime, if it wasn't for make I could imagine a nomtime too.

Good news is that there is work on making the i_version field available via statx in Linux: https://lore.kernel.org/linux-nfs/20220826214703.134870-1-jl...

So we might have a reliable mtime alternative in Linux soon in 2022.

https://lwn.net/Articles/905931/ for background information on this.

The title should be: “I don’t like Mtime comparison. Here’s why.”

If your name is Dijkstra and it’s 1968 feel free to use the phrase “considered harmful”, otherwise use plain language.

Nitpick: Dijkstra didn’t use “considered harmful”, Niklaus Wirth did. https://en.wikipedia.org/wiki/Considered_harmful#History:

The original title of the letter, as submitted to CACM, was "A Case Against the Goto Statement", but CACM editor Niklaus Wirth changed the title to "Goto Statement Considered Harmful". Regarding this new title, Donald Knuth quipped that "Dr. Goto cheerfully complained that he was always being eliminated."

Reading that same link, Wirth didn’t invent the cliché, either.

Noted.

> If your name is Dijkstra and it’s 1968 feel free to use the phrase “considered harmful”

The phrase "GOTO considered harmful" was actually used by Niklaus Wirth, not Edsger Dijkstra. Dijkstra had submitted his letter with the title "A Case Against the Goto Statement"; and Wirth, as editor, changed it.

But also, it was already a journalistic cliché outside of computer science before Wirth applied it to Dijkstra's letter.

Anti-considered-harmful consided harmful.

What's wrong with the usage? It is short and concise.

"Short and concise" isn't much praise. If I wanted something short and concise, I could say "is bad", which is probably both more concise and more honest. "Mtime comparison is bad" is more clear and more concise.

"Considered harmful" is misleading because the passive voice suggests some kind of general consensus which usually doesn't exist.

> What's wrong with the usage?

It's such a ridiculously inflationary overused meme to the point that it's getting annoying.

There are other contenders: https://news.ycombinator.com/item?id=31836787

(In hindsight, I completely forgot "X shades of Y" in this list)

Who considers it harmful? Some jerk who uses the royal "we" on themselves?

The issue with the phrase is, that it is hopelessly overused also for things that are purely subjective or way to general.

Now, wouldn't you mean """considered harmful" considered harmful" considered harmful"?

It's presumptuous and hackneyed.

We're currently dealing with crashes across all Qt applications using QML on NixOS [1], since Qt utilizes the binary's mtime to invalidate the cache of embedded QML resources.

Since all builds have an mtime of 0 as timestamps are the biggest source of reproducibility issues [2], QML loads outdated cache objects which will then load invalid bytecode at runtime and therefore causing a crash.

Our initial plan was to utilize a hash of the binary [3] which should be IMHO the most straightforward, unlikely to break and future-proof way to solve the problem. The currently suggested implementation's performance could most likely be improved by generating the hash once on startup instead of every time when a QML resource is loaded, but I believe since it's already cached in memory, hashing is not that expensive, binary sizes are usually relatively small and resource loading (of embedded ones) doesn't happen that often, it should be already reasonably fast (real performance test haven't been done yet).

The big challenge will be upstreaming it, once we've proven it to properly work. The current approach has been apparently rejected [4] and was declared to be a downstream issue, which I personally don't agree with, since sooner or later reproducible builds will become the norm and therefore will affect everyone. Current ideas to work around it require individual solutions per distribution/ISV, as this would mean they'd have to come up with domain specific criteria for cache invalidation (as the store path/derivation hash on NixOS) and to maintain a downstream patch for this solution and furthermore wouldn't work for local build processes (e.g. from within an IDE).

Lesson of the day: never use mtimes, they'll bite you in the ass sooner or later!

[1] https://github.com/NixOS/nixpkgs/issues/177720

[2] https://reproducible-builds.org/docs/timestamps/

[3] https://github.com/qt/qtdeclarative/commit/5106afcd76e377a6b...

[4] https://github.com/NixOS/nixpkgs/issues/177720#issuecomment-...

Can't you copy time from the commit of the corresponding source?

I really do think it's unfortunate that the OS cannot keep something like a "write count" for each file. That would be enough. Possibly it would even be good enough to know the number of times a file had been opened for writing.

There's the i_version field in the inode structure which is pretty close to that. Though I'm not sure it's accessible to user space, or whether it's used only by the kernel NFS server.

> Random side note: on MacOS, the kernel does know all the filenames of a hardlink, because hardlinks are secretly implemented as fancy symlink-like data structures. You normally don't see any symptoms of this except that hardlinks are suspiciously slow on MacOS. But in exchange for the slowness, the kernel actually can look up all filenames of a hardlink if it wants. I think this has something to do with Aliases and finding .app files even if they move around, or something.

This reads weird to me.

As the author points out, a directory entry (hardlink) is just a filename pointing to an inode. When you first create a file, and there is only one filename for that file, that entry is a hardlink. So in that text the first occurrence of "hardlink" should probably read "the kernel does know all the filenames of an inode".

But then it gets weird. When the author says "hardlinks are suspiciously slow on MacOS", it sounds like they're saying "accessing files is suspiciously slow on MacOS" - because accessing any file in the filesystem goes through hardlinks. All links to a file, including the first, are hardlinks.

I suppose it's perfectly allowed for MacOS to simulate Unix hard link semantics (ie. names are just pointers to inodes, with each name being equal) even though the underlying filesystem doesn't have the name+inode split. It seems indeed this is the case on HFS+: https://developer.apple.com/library/archive/technotes/tn/tn1...

Ah, that makes it much clearer, thanks.

It reads weird because it's a roughly equal mix of opinion and wrong, possibly informed by some historical understanding of how HFS+ used to work.

Granted, in 2018 HFS+ was still kind of relevant, but even at that point the listed assertions were pretty dodgy.

It would be neat if filesystems like btrfs or ZFS could expose their internal checksum of the file data to userspace, to quickly see if two files are having identical content (eg. think of rsync). (Assuming the hashes computed internally the filesystem are actually purely based on the file data, and not for example metadata like block pointers. But, dedup-capable filesystems should surely have such checksum internally...).

The problem with that is for good performance reasons that checksum is typically on smaller blocks than the whole file. For example ZFS splits files into a maximum of “recordsize” (128K) by default and checksums each block.

Not so likely whatever remote target you are looking at has the same blocks and checksum algorithm.

On the upside you can often ask such file systems to take two snapshots and what changed between them. Or to export some kind of differential to transform the original snapshot to the newer snapshot. Both ZFS and Btrfs can do those.

For ZFS, we have the pool-wide transaction number, which increments with every disk write. Couldn't you use that as the "time" for the last modification to each file so long as you have a starting transaction number to compare it against?

apropos

> Linux added an O_NOATIME

!!Con 2016 - How I fixed UNIX atime! With 10 lines of code and feminism!!! By Valerie Aurora https://www.youtube.com/watch?v=fHjsdyN4UK0

That talk is about the "relatime" mount option, not the O_NOATIME open() flag. The O_NOATIME open() flag was added following the semantics described in the glibc manual (https://www.gnu.org/software/libc/manual/html_node/Operating... ); I don't know where these semantics came from, my guess (given that it is glibc and O_NOATIME is described as a GNU extension) is that these semantics came from Hurd, though I haven't checked. These semantics aren't ideal (you need extra code to retry without the flag if it fails with a permission error, or you have to open() without the flag and add it later with fcntl(), both cases requiring to an extra system call), if it were added today it probably would have simpler semantics.

(Source: I wrote the patch which added O_NOATIME to the kernel, based on an older patch which didn't get in, see https://lkml.iu.edu/hypermail/linux/kernel/0406.1/0894.html and https://lkml.iu.edu/hypermail/linux/kernel/9811.2/0118.html)

I suspected this post was generated bot spam given the name of the convention and how most discussions about feminism go on the Internet but I have to say that the talk was interesting and I am glad I watched it.

Thank you for posting it.

That was brilliant, Thanks for posting it!

> mtime is one of the inode fields, so whenever mtime changes, ctime also changes

Not in my experience.

Reply on news.ycombinator.com