Hacker News Reader

Comment by IshKebab | Hacker News Reader

The standard Linux process management API is a total mess IMO. It doesn't even offer a way to avoid TOCTOU. They've made some improvements recently (https://lwn.net/Articles/794707/) but as far as I know you still can't implement something like pstree without just reading files from `/proc` which is shit.

View on HN

The problem here wasn't a case of TOCTOU; rather, the program was using a magic constant value that was outside the range of getpid() but which wasn't outside of the domain of kill(2)'s first parameter.

Proper enum types would have solved this problem; e.g. in Rust

  enum ProcessHandle {
      Pid(pid_t),
      Invalid,
      Terminated
  }

There is no way to pass a Terminated or Invalid process handle to kill, since there is no numeric identifier for it anymore. And, any code that wants to get a PID from a ProcessHandle has to explicitly handle non-PIDs - making it impossible to forget to deal with these special cases.

While correct, everything is a number at the ABI boundary, so it’s still possible - if not harder.

Depending on the layout and arch, -2 could well represent Terminated

I think this misunderstands how Rust enums work. The discriminator is separate from the payload, unless niches are involved (they probably wouldn’t be here - note that -2 is a valid pid_t). You cannot go and pass a Rust enum to any FFI function directly (without extra annotations, at least); you’d have to specifically extract the pid_t from the enum before passing it to an FFI function like kill(). And you can only obtain a pid_t from the Pid variant; if the enum holds a different variant, the compiler will stop you from trying to pull out a pid_t.

I understand how Rust enums work, but you’re misunderstanding how ABIs work (or my original point, or both).

Rust enums are a Rust concept. Two identical enums will have different memory layouts on different machines, Rust versions, etc. Even if this was solved, you’re still passing the enum across a boundary. Both sides need to agree on the layout of the enum in memory.

To illustrate the problem, imagine we had two variants: `Pid(NonZeroUsize), LaunchTheNukes`

This can be laid out in memory with a single `usize`: the 0 value becomes the discriminate for `LaunchTheNukes`.

Thus, if you pass `0` (I.e null) across the boundary the other side will interpret it as `LaunchTheNukes`.

In short, if you naively change the layout of an enum and don’t recompile everything that uses the enum (i.e the kernel) then your -2 value could change from being interpreted as `Pid(-2)` in the application to `LaunchTheNukes` in the kernel.

It gets more complex with larger enums and more discriminators, but the general point is that the concepts of enums, type safety and such doesn’t really exist across boundaries like syscalls, because it doesn’t exist in hardware / at the machine level.

Yeah I know, that was just another example of how bad the API is.

UNIX/POSIX process management.

Heh, "even". BSD process groups were around for two decades at least before anyone even thought of the idea of a TOCTOU vulnerability, much less coined an acronym for it. You're just asking for too much here. Yes, it's an API you probably shouldn't use. No, it's not going away.

The solution for this particular problem is cgroups. The build system just wasn't using it.

What inherent TOCTOU vulnerability exists with process groups? Process groups are the way to shoot down a collection of processes atomically in Unix. getpgrp returns the current processes' process group, and killpg(pgrp) sends a signal atomically; so long as the process doesn't switch process groups between those two calls (and using normal APIs only the process can do this itself), there's no race. (IOW, there's no race in `killpg(getpgrp(), signo)`) There are other ways, too, like using SIGHUP delivered when a controlling TTY is closed (not strictly POSIX, but reliable semantics across all the major extant Unix-like systems).

cgroups also allows killing a group of processes race-free, but Linux only got this feature (cgroup.kill) in 2021.[1] Before then even systemd would just walk the cgroup PID list and send a signal to each individual process, which was inherently racy. If this was done from PID 1--init--this could maybe be done correctly by ensuring zombie's weren't reaped--preventing PID recycling--while walking the PID list. OTOH, Linux supports a prctl, PR_SET_CHILD_SUBREAPER, to change a reaper, so perhaps even systemd could never guaranteed to not accidentally shoot down an innocent process.

[1] Merged https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux... I'm not sure what the subsequent release history looks like, but I wouldn't be surprised if some popular LTS distro versions lack this capability.

I suspect the upstream concern was dealing with services that themselves like to spawn process groups. There's no hierarchy of process groups, once you call setpgid() you've escaped the sandbox and the API can't help your parents manage you anymore. The use case in question is a build system that by its nature needs to deal with foreign software.

Also FWIW, you could freeze a cgroup atomically in the very earliest versions and deconstruct it at leisure. You just didn't do it by sending signals.

With TOCTOU, the solution is often just to try to use the resource and handle errors, not to assume that since you checked it, it's available.

PID reuse is the classic example here. There is no neat solution other than process handles, which Unix/Linux does not historically have.

Aren't process handles still an integer under the hood in the end? The solution is presumably the same - use enough bits.

What makes process handles different is that they have a lifetime of their own that is distinct from the lifetime of the process - even if the process exits, the handle still refers to that now-dead process, not to some random new process, until you release the handle.

But if it's still an integer (that you can't see) then it's still subject to reuse to the exact same extent that an integer is or isn't. It's entirely possible to wrap your internal integer at 16 bits and experience handle collisions. Alternatively you are free to use full 64 bit integers on Linux (at least IIUC).

If you can architect a system that never experiences handle collision then quite trivially you can architect an equivalent system that never experiences PID reuse.

You're missing the point. No, handles are not subject to such a problem because their lifetimes are explicit and distinct from the underlying resource. E.g. in Win32, if you have a process handle, it will not ever refer to a process different from the one that it was originally created for until you specifically call CloseHandle on it. The OS will retain that handle in its table and will not reuse it for any other process.

And yes, of course it would be trivially easy to do the same in Linux. Except for that whole part where it would break all the existing code relying on POSIX semantics.

That only works if you know the resource in advance. Doesn't work if for example you're trying to delete files matching a glob and you have to enumerate them by name and then delete them by name.

The filesystem API offers a solution for that. The process API doesn't (maybe with procfd; I haven't tried that API yet since it's so new).

I'm not sure that counts as TOCTOU?

What I'm saying is that instead of writing code like:

    for fname in someglob:
        if os.path.exists(fname):
            os.unlink(fname)

You should instead write

    for fname in someglob:
        os.unlink(fname)

and handle the errors.

This isn't a TOCTOU problem. This is a problem of using sentinel values to mean other things, when those sentinel values won't ever get returned by getpid(), but kill() will accept those values and do something you didn't intend.

The fix was to check if the PID variable was set to one of the sentinel values, and not call kill() at all if that's what they are.

> It doesn't even offer a way to avoid TOCTOU.

What do you mean? I admit there are cases where it's hard to avoid TOCTOU, but I can't think of any where it's impossible.

For example to kill all of the children of a process you first have to read all of the files in `/proc` to build a process tree, and then send your signals, by which time the process tree may have changed.

Yeah, as far as I know the only way to be sure is by putting the parent process into a cgroup. Then to kill all of the child processes you have to freeze the cgroup, enumerate the pids, send sigterm/sigkill to all of them, before unfreezing again.

Doesn't the new pidfd API you linked give you a race-free way to do that?

Maybe. Tbh I was trying to do something like that on RHEL8 and I think it's too old for procfd.

As long as you aren't trying to reimplement `systemd` badly, you really shouldn't have to deal with this.

"As long as you aren't trying to use the flawed API you won't run into any of its flaws."

Ok then.

More like "if you empty a safe-deposit box that doesn't belong to you, don't be surprised if random people start complaining about their goods being stolen."

Ownership matters for more than just memory. And the "just use GC" answer means deadlocks and leaks galore. If you limit yourself to killing processes you own, you will never have a problem.

So "you're holding it wrong" then? I don't get why you wouldn't want to have a reliable well-designed API for process management?

We already have one. What we don't have is an API for process pillaging.

Think about it - if you aren't init, what exactly is too limiting about "kill the processes or process groups of the children you created yourself"? As a rule, most grandchildren won't bother to enter a new process group, and if they do their immediate parent should be advanced enough to take responsibility for forwarding fatal signals.

Who needs handles for processes when you can have reusable IDs instead? /s

For a while the limit has been around 4 million or something like that, instead of the traditional 32k. Which of course broke a bunch of stuff that assumed "char pid[5];" is good enough. Fun stuff.

The kernel offers up this (here abbreviated) example which you could use for all accesses to /proc/<pid> when you have an open pidfd to ensure that the two refer to the same process:

    static int pidfd_metadata_fd(pid_t pid, int pidfd) {
            char path[100];
            snprintf(path, sizeof(path), "/proc/%d", pid);
            int procfd = open(path, O_DIRECTORY | O_RDONLY | O_CLOEXEC);
            if (sys_pidfd_send_signal(pidfd, 0, NULL, 0) < 0 && errno != EPERM) {
               close(procfd);
               procfd = -1;
            }
            return procfd;
    }

And these days you even can poll for pidfds (readable = exited) and you can even use P_PIDFD with waitid. So the very basic process management cycle can be done through pidfds nowadays.

> as far as I know you still can't implement something like pstree without just reading files from `/proc` which is shit.

It's also slow! Tools like top/htop/ps -aux etc. use a surprising amount of CPU because they have to do a trillion trips to the kernel and back. Compare this to the win32 equivalent where, iirc, there is just one or two trips to the kernel to acquire a snapshot and you walk that in user-space instead.

  char path[100];
  snprintf(path, sizeof(path), "/proc/%d", pid);

You're assuming the pid will fit in 93 bytes! The exact same mistake as `char pid[5];`

Jokes aside, sometimes userspace deserves to be broken, and `char pid[5]` is IMO one of those cases. The fact that we treat integers as a scarce reusable resource is bad, and the fact that we now have a second-order process identifier to cover that up is worse.

Fun fact that broken code was in the runtime used by a major Fortran compiler

Reply on news.ycombinator.com