Hacker News Reader

Comment by evmar | Hacker News Reader

In Ninja, which needs to spawn a lot of subprocesses but it otherwise not especially large in memory and which doesn't use threads, we moved from fork to posix_spawn (which is the "I want fork+exec immediately, please do the smartest thing you can" wrapper) because it performed better on OS X and Solaris:

https://github.com/ninja-build/ninja/commit/89587196705f54af...

View on HN

posix_spawn also outperforms fork on Linux under more recent glibc and musl, which can use vfork under the hood. https://twitter.com/ridiculous_fish/status/12328893907639336...

The issue with posix_spawn is that you can't close all descriptors before exec. This is especially an issue as most libraries are still unaware they need to open every single handle with the close-on-exec flag set.

Solaris/Illumos has an extension[0] for that.

  [0] http://src.illumos.org/source/search?project=illumos-gate&full=posix_spawn_file_actions_addclosefrom_np&defs=&refs=&path=&hist=&type=&xrd=&nn=1
  [1] https://docs.oracle.com/cd/E36784_01/html/E36874/posix-spawn-file-actions-addclosefrom-np-3c.html

For implementations which don't have it, you can stuff, into the file_actions, say, 4093 close action entries into the file_actions, targeting descriptors 3 to 4095. This big file_actions object can be cached and re-used for multiple calls to posix_spawn.

It won't close descriptor 4096, but that's probably beyond giving a darn in most cases. If you have an application that opens high descriptor numbers, you probably know.

A better approach is to exec an intermediate helper program that will do it and then exec the actual intended program. One can also use this approach to do things like reset signal dispositions to SIG_IGN.

... add another option to /usr/bin/env and you got it!

Closing all descriptors is next to useless; you usually need to inherit at least standard in/out/error.

What you need is an operation like "close all descriptors >= N", as posix_spawn opcode.

Indeed, it's very common to want to close all FDs other than 0, 1, and 2, of course, as well as a few other exceptions (e.g., a pipe a parent might read from, FDs on which flocks are held). The reason one often wants to close all open FDs besides those is simple: too many FDs that should be made O_CLOEXEC often aren't, and even when they are, too often there is a race to use fcntl() to do so on one thread while another one forks. Yes, there are new system calls that allow race-free setting of O_CLOEXEC on new FDs, but they will take a long time to be widely used.

I've implemented closefrom() type APIs more than once. Of course, I happen to know about Illumos', so there's that.

Reply on news.ycombinator.com