Tell HN: TIL: GNU Make .DELETE_ON_ERROR | Hacker News Reader

Tell HN: TIL: GNU Make .DELETE_ON_ERROR

51 pointskazinator14 comments

If you have a make recipe which creates the target file, or modifies the timestamp of the existing out-or-date target, and that recipe fails, make will not delete the target. Thus it looks up-to-date, even though the recipe failed.

The manual says: "So generally the right thing to do is to delete the target file if the recipe fails after beginning to change the file. make will do this if .DELETE_ON_ERROR appears as a target. This is almost always what you want make to do, but it is not historical practice; so for compatibility, you must explicitly request it."

So, hidden in the manual is the recommendation that, it's good practice to have

  .DELETE_ON_ERROR:

somewhere in every GNU Makefile.

An example where this is relevant:

  foo: bar
      generate bar > foo  # redirection is used

redirection will create the file or touch the timestamp of the existing one before the generate command is run. What if bar has syntax errors and generate fails? The foo file was touched and so looks updated. If you run "make foo" again, Make reports that foo is up-to-date.

With .DELETE_ON_ERROR, foo will be deleted.

If foo exists and its timestamp is not changed, then it's left alone.

If a recipe creates or touches the target, and fails due to Make being interrupted by a signal, in that case make deletes the target without any option having to be specified.

That's why you do

    foo: bar
        generate bar > tmp && mv tmp foo

instead

On Linux, you can use O_TMPFILE to ensure that the temporary file won't linger around if an error occurs:

    foo: bar
        generate bar | python3 -c "import os,sys;f=os.open('.',os.O_RDWR|os.O_TMPFILE);os.write(f,sys.stdin.buffer.read());os.link(f'/proc/self/fd/{f}','./foo')"

Lol. Files were a mistake.

This is everything that's wrong with modern software. Do not introduce a pointless dependency like python because of a lack of knowledge about the build system of choice. Just use something more appropriate (that isn't python).

You're missing the point. The important part isn't the Python, it's the opening of a file with O_TMPFILE, writing to it, and linking it into the filesystem in order to guarantee an atomic result without risking leaking temporary files. If your requirements forbid the use of Python for whatever reason, you are welcome to substitute it with any other program that can perform the aforementioned operations.

Is this a common practice?

I ask because the approach makes sense, but I don’t recall ever seeing it before.

Yeah, I've seen it done, and I use it when needed. Many/most compilers won't touch the output file until they've done a syntax check so they don't usually need it, but if you're redirecting something from stdout, you'll have a blank file often as you develop, and quickly realize the benefit of temp files.

That leaves behind a tmp file breadcrumb if generate fails, or the recipe is interrupted by a signal. You can address some of that that by cramming more logic into the recipe.

I was going to fix it in exactly that way, but I could just could not believe that such a workaround would is required. Why can't Make take care of such an obvious thing?

So I went digging into the manual, which I had read before more than one time, to see if I overlooked something.

Setting DELETE_ON_ERROR is superior.

> Setting DELETE_ON_ERROR is superior.

Relying on .DELETE_ON_ERROR will fail to delete the file if your Make process is killed abruptly, in such a way that Make can't clean up. Things like killed by OOM killer, power failure (laptop battery), network filesystem suddenly dropped, VM killed with make in progress, kernel froze (it's happened to me).

That said, many of those things result in an inconsistent state using the "mv" or "move-if-changed" method too, because who calls fsync from a shell script in the right places. It's better to detect that Make didn't get the chance to clean up, so you know "make clean" or equivalent is necessary next time.

("make clean" can be a right pain for build processes that take hours or days to rebuild though. In one project I worked on, it took 20 minutes just for GNU Make to do the stat() calls to get the timestamps to decide what to build, before it ran the first command. Imagine how long the actual rebuild from scratch took.)

The recipes that I'm concerned about will show this issue 100% of the time if the generating command touches the output file and then fails.

GNU Make offers us a blanket solution for such cases and so we should take advantage of it.

That solution doesn't nail all the the corner cases like being killed by uncatchable signals or kernels crashing.

That issue affects not only the recipes I'm presently concerned about but potentially all recipes. Even those which ensure that the file is gone when the tool fails.

Fixing that problem likely requires changing every single recipe in every Makefile, except for those which invoke a tool that already does an atomic create or update, or reasonably simulated facsimile thereof.

> That leaves behind a tmp file breadcrumb if generate fails

That sounds useful for debugging the failure. But better then to do:

    foo: bar
        generate bar > foo.tmp && mv foo.tmp foo

So you can at least tell what it was supposed to be.

“That leaves behind a tmp file breadcrumb if generate fails, or the recipe is interrupted by a signal.”

Honest question: Why does that matter?

So, is there any shell that can do the oftentimes right thing, and delete that file for you if it created it and ‘generate’ fails? Maybe even redirect to a temp file, and rename that on success, delete it on failure (that probably makes the number of variations on this theme too high)

That would make this work from other tools that run shell commands (other makes, cron, etc)

(Bonus points if it also can optionally revert >>, but that’s trickier, as that doesn’t have to be to a file)

It's nice to have, but always pen a .PHONY `clean' target to clarify what to kill before trying again after a big failure.

Reply on news.ycombinator.com