Hacker News Reader

Comment by CGamesPlay | Hacker News Reader

A much more challenging task is writing a systemd unit that starts gracefully before shutdown. I wanted to write a unit that could issue an API call to delete the instance rather than doing a normal power off. Putting it at a reasonable place in the sequence took a lot of trial and error! The trick is actually to have the unit be started at some point in normal boot up (e.g. “armed”) and then do the actual task when the unit is stopped.

Here’s the unit I ended up with: https://github.com/CGamesPlay/infra/blob/master/private-serv...

View on HN

I tend to push the metaphor of software being meant to be read and only incidentally to be run by a computer as far as I can. We are telling a story to future us or our successors. Stories have rules and it's jarring when you violate them.

There's an idea in software that's a bit like the corollary of Chekhov's gun. Chekhov's gun is about not presaging jarring story elements that will never come to pass. But it's nearly as jarring to leave important story arcs as complete surprises until the end. Producing the gun moments before the curtain goes down would be quite a WTF. We didn't know that was a possibility. That's a niche that some people occupy, but it is a niche.

Introducing things early fights with Locality of Reference, but when we're talking about things of deep, dramatic importance (like an attempted murder, or a reaper process) it's important to introduce that "character" early in the story so that people know that it exists. Failing to do so is a form of deus ex machina and we only appreciate that in very small doses.

So framed that way, I don't see a problem with having to start a killswitch while you're spinning everything up. It's there, people can see it, and know to ask questions about it.

If software is meant to be read, and only incidentally run, then having it in the config to only run at shutdown is still introducing it early on. As a matter of fact, I think it would be like introducing the the reaper character in the very beginning and keeping them around in every scene sitting silently in the background until it was time for their one line at the very end. Many people might not notice, but to someone who does, it would be very odd indeed.

I have the following unit file saved for that purpose:

    [Unit]
    # https://stackoverflow.com/questions/36729207/trigger-event-on-aws-ec2-instance-stop-terminate
    Description=unlink agent from remote server
    Before=shutdown.target
    [Service]
    Type=oneshot
    EnvironmentFile=-/etc/environment
    KillMode=none
    ExecStart=/bin/true
    ExecStop=/opt/service-name/shutdown-unlink
    RemainAfterExit=yes
    User=root
    [Install]
    WantedBy=multi-user.target

If I recall correctly, the KillMode=none is important as it causes the shutdown-unlink binary to escape systemd process supervision. Without it, you may deal with systemd immediately halting your shutdown unit (and killing the process) when it hits the shutdown target.

You unit races with the network being brought down, since it isn't listed as a "Before" target, FYI.

I could be mistaken, but I don't think even that would be sufficient? Before would make your command execute before the network is brought down, but it wouldn't have the latter wait for your command to actually complete.

The post shows long-running stop scripts / containers and demonstrates them delaying shutdown (not with KillMode none though)

@CGamesPlay network.target's "primary purpose is for ordering things properly at shutdown: since the shutdown ordering of units in systemd is the reverse of the startup ordering, any unit that is order After=network.target can be sure that it is stopped before the network is shut down if the system is powered off."

https://www.freedesktop.org/wiki/Software/systemd/NetworkTar...

They need After=network.target, not Before=network.target.

The shutdown order is the reverse of startup order, and they execute the payload on shutdown (ExecStop).

The "Before=shutdown.target" is superfluous, that is a default dependency (as regular services obviously need to be shut down before system shutdown).

"KillMode=none" shouldn't be necessary, unless your command leaves processes running that need to stay running (and in that case you have other problems, since those processes may not actually finish before the system is shut down). Systemd waits for your service to stop before reaching shutdown.target (per the default Before= and Conflicts= dependencies).

Since your command probably needs networking, you also want "After=network.target" to ensure your command runs before network is torn down, as covered in sibling comments.

Thanks. I ripped it from a colleague, and now I have some bugs to report.

That's what this post builds to solving at the end and in the next post - having a unit deletes the instance from a cluster before shutdown

Sadly, I think this would be child's play with SysV init.

Reply on news.ycombinator.com