Hacker News Reader

Comment by oconnor663 | Hacker News Reader

If anyone can suggest an update to https://github.com/oconnor663/duct.py/blob/master/gotchas.md..., please let me know. As far as I'm aware, there's still no reliable way to kill a tree of processes on Unix that's suitable for library code.

View on HN

Thank you for documenting those gotchas.

I know not being a library has different considerations, but some ideas I used in timeout(1) to kill a process group may be useful. Tricky things like using sigsuspend() to avoid signal handling races.

https://github.com/coreutils/coreutils/blob/master/src/timeo...

cgroups might be another avenue to explore, being more modern and so having less compatibility baggage

I wrote a 'subreaper' program to capture and kill such things some time ago. It's pretty basic, having options to pass on signals, start killing subprocesses if it receives a signal, or killing subprocesses after the initial subprocess dies. There is an undocumented --all that prints logs, kills on stats and doesn't start killing when the main program dies. I use it for killing test subprocesses or capturing things to terminals that would otherwise double-fork out of them, allowing me to kill them and all their subprocesses with a ctrl-c.

https://github.com/knome/pj

>Linux is in the middle of adding new APIs like pidfd_send_signal, but none of them are aimed at improving the situation with grandchildren.

While my subreaper is vulnerable to pid reuse, but I think it could be fixed by having it do this:

    loop
        children = scan-for-children
        for each child
            if no pidfd for given child
                pidfd open child ( dropping if error )
        still-children = scan-for-children
        close-and-drop any pidfd's that are no longer children
        safely-kill-children-via-pidfds

If you kill the direct children, the grandchildren will become direct children since you are a subreaper, and then so on.

In summary, a helper process using subreaper+pidfd should be able to properly and safely contain and kill grandchild processes.

Some additional linux-specifics: https://lwn.net/Articles/855049/ https://lwn.net/Articles/754980/

I can suggest an update. jdebp.eu is replaceable by jdebp.info. 17 million people voted to deprive me of a domain name some years ago. I'd suggest using jdebp.uk as an alternative, but Scottish Independence and the Northern Ireland border problem are real things and once bitten, twice shy and all that. (-:

When killing an entire cgroup, you might avoid the PID reuse race by getting a list of PIDs in a cgroup, then for each PID, first opening a PIDFD to that PID, check via that PIDFD whether it’s a member of the correct cgroup, and then killing the PIDFD using pidfd_send_signal(). This way, no process can be killed inadvertently by PID reuse.

There may be a better way to do this, but one way might be to open() /proc/<PID> and use the resulting FD to both to check /proc/<PID>/cgroup (using openat(FD, "cgroup", O_RDONLY|O_NOCTTY)) and then use the same FD as a PIDFD when calling pidfd_send_signal().

ISTM that systemd should use this method if its current method is just looping over PIDs.

Might be easier to freeze the cgroup, send the signals, then unfreeze. Only danger there is that it needs to not be reentrant.

Edit: Indeed both your and my methods were proposed to systemd: https://github.com/systemd/systemd/issues/13101 It's just waiting for someone to implement it.

This won't solve your problem especially for library code, but prctl(2) with PR_SET_CHILD_SUBREAPER is worth mentioning: by setting the flag your process becomes a subreaper, and will become the parent of any orphaned child processes. Unlike cgroups, PR_SET_CHILD_SUBREAPER can be used without special privileges in commonly deployed Linux versions.

Drop them into cgroup so the new pids can't run away is I think only reliable one

Solid approach for desktop Linux. Not possible on Unix in a general sense, because cgroups are a Linux-specific thing. They also depend on having certain kernel options enabled.

So if you want to write software that can target MacOS or any of the BSDs, then you can't use cgroups.

As explained in the link, although constrained processes can not escape the cgroup, they can fork off another PID indefinitely. Also, the old PID the forking leaves behind might be re-used by some other new process in another cgroup, and now you’ve killed the wrong process. Oops!

I think it's a non-problem TBH. You can use process groups (yes like shells do). Alternatively you can track all child process ids and kill them all one by one explicitly and do that recursively in your process tree. Each sub-process in the tree that starts child processes is expected to do also this reliably.

The problem is, the children programs you launch may also use process groups to implement job control for their children, and process groups are not nested. Which means it's all very brittle, and e.g. sending SIGKILL will actually leave lots of orphans around instead of cleaning everything.

OK yeah I get that. Back in the day you used to we used to processes like containers. The system programmer would write the process executables or at least a process manager. If a process ever called setpgrp itself it did it for a controlled reason and because it was supposed to. You were in control of that because you wrote it so it's a non problem.

In modern times wer'e doing more thing like containerizing a bunch of processes they are weakly related instead of using a VM for that so the use case became more pressing. There is cgroups for managing such a group of processes. I'm suprised to learn there is no way to reliably send a signal to all processes in a cgroup though.

> SIGKILL will actually leave lots of orphans around instead of cleaning everything.

Well actually orphan processes exist for a reason. Your supposed to "wait" / reap them to make them non orphan.

Yeah and similarly, if you're a terminal program, using process groups for your children means that they'll keep running if the user presses Ctrl-C. Process groups really just weren't designed for anything that's not like a system shell.

> Linux is in the middle of adding new APIs like pidfd_send_signal, but none of them are aimed at improving the situation with grandchildren.

Maybe Linux should implement something like pidfd_getpfd() (returning the PID FD of the parent process) and pidfd_fork() (returning the PID FD of the child process).

It's not possible at all from arbitrary userspace, for the simple reason that you aren't any more privileged than the processes you're trying to kill. They can preempt you and race against you while creating new processes to kill.

You need to use a kernel feature intended for this purpose, which is what things like process groups (linked article), jobs (80's BSD feature) and cgroups (modern generalization of the idea) were designed for.

Use the freezer cgroup to freeze everything, then kill it all off however you like.

Reply on news.ycombinator.com