Fork() without exec() is dangerous in large programs (2016)
evanjones.ca
evanjones.ca
from A Fork in the Road, <https://www.microsoft.com/en-us/research/uploads/prod/2019/0...>
Windows is a great example of the alternative (following "Windows philosophy"): There are about three different API calls for creating a new process, each with a heap of complicated optional arguments. The API becomes more complex, less composable, less extensible and less powerful. But also easier to reason about and easier for the kernel to provide, and arguably with fewer footguns.
It is a MASSIVE, unwieldy tool that is difficult to use correctly. It happens to have a small interface.
In it is original implementation, fork() was pretty trivial. All it did was create a new process entry in the kernel table, with all the pages and capabilities and such copied from the original process. Then mark all pages as copy-on-write, and return to the caller. Maybe not trivial, but much less complicated than loading an executable file from disk.
My understanding of Linux internals is maybe 20 years out of date, so I am legitimately curious what makes fork() so complicated these days.
Use vfork() or posix_spawn().
I could see it being worthwhile to immediately copy a few pages, like the top of the stack, but copying the whole resident set seems excessive. Especially since some of that data might not even be written to.
As well there's the cost of all those page faults that the two processes are likely to take to do the copying.
And lastly there's all sorts of complexity involving multiple parent threads calling fork(), or the child calling fork() again (or vfork()) before calling exec.
It's just much easier to copy the resident set and mark the address space as being CoW, because now you only have to worry about page faults for pages that are not in core anyways and so were going to fault anyways, and that means you don't have to worry about TLB shootdowns either (if a page is not in core, it's not referenced by any TLB either). You still have the multi-fork issues, but now you can use an atomic reference count on the address space.
With what you describe, you'd share nothing because at the point you've loaded it up, all the data you just loaded is resident. :-(
The posix_spawn() and posix_spawnp() functions provide the
functionality of a combined fork(2) and exec(3), with some
optional housekeeping steps in the child process before the
exec(3). These functions are not meant to replace the fork(2)
and execve(2) system calls. In fact, they provide only a subset
of the functionality that can be achieved by using the system
calls.
Windows defaults to CLOEXEC semantics and you have to opt-in to child process inheriting open file handles, and that has caused problems.
Unix defaults to not-CLOEXEC sematincs, and that too has caused problems.
The Unix default can cause unsolvable problems because of races between threads.
You should use CLOEXEC everywhere. Except you can't because you are using libraries.
A common hack I've had to add is an argument FD that is not to be closed because. e.g., an flock is held on it.
Unix actually copied the memory over initially.
This could also solve the issue with forking from multithreaded programs since we can ensure we own all shared resources when we isolate our thread, to effectively thus become a new process.
So instead of fork we have clone/isolate.
A new thread can also of course immediately suspend itself, allow other threads to work on it's data in some way, who then give it a signal to resume itself and then execute if need be.
https://lwn.net/Articles/826313/
Clone is also used to start threads, IIRC:
https://stackoverflow.com/questions/4856255/the-difference-b...
https://hn.algolia.com/?q=https%3A%2F%2Fwww.microsoft.com%2F...
and also here:
Windows doesn't have fork(). It has a real, fully mature thread and process model. In Windows NT, every process consists of a handle that is a "Process", which in turn points to a structure containing a list of "Threads". A process is done when its main thread exits or all threads exit, whichever is defined by the main process. Fork/Exec is replaced with CreateProcess (or ShellExecute, your choice).
For a very zen-like example of the fork/exec and pipe management that you'd do on a POSIX system done in Windows, the [MSDN Docs](https://docs.microsoft.com/en-us/windows/win32/procthread/cr...) are quite informative.
From my perspective watching the various techniques used by a multitude of operating systems over many decades, the Windows-style process+thread model seems to be winning out over the UNIX fork() model.
For example, PostgreSQL seems to suffer greatly from the "forked process per connection" model, necessitating front-ends that do connection pooling. Database engine after database engine seems to go through this phase and then "upgrade" to either a single-process thread pool model, or start using async IO in some way. (Web servers also.)
For reference, Microsoft SQL Server back in the 1990s could on Windows NT 4 could handle more connections than PostgreSQL in 2020...
It does (or maybe did, not sure if it still works) have the literal equivalent of fork() - NtCreateProcess() with NULL SectionHandle argument creates a new process which is a clone of the caller. However, it never really worked. The Win32 API did not support forking, and so any process which forked itself and then tried to invoke any Win32 API calls would soon crash or fail mysteriously. In theory forking did work for pure NT API applications, but those are rather limited in their abilities.
BTW, article title needs a (2016). It appears that the relevant Python bug has long since been closed, by avoiding linking with the system sqlite on macOS.
Is it? There are more descriptive (as opposed to procedural) APIs which behave in a safer and more well-defined manner to do it these days. Unless you're implementing a shell, fork has never been a great tool.
As one commenter noted 3 months back:
> The dense fog lifts, tree branches part, a ray of light beams down on a pedestal revealing the hidden intentions of the ancients. A plaque states "The operational semantics of the most basic primitives of your operating system are designed to simplify the implementation of shells." You hesitantly lift your eyes to the item presented upon the pedestal, take a pause in respect, then turn away slumped and disappointed but not entirely surprised. As you walk you shake your head trying to evict the after image of a beam of light illuminating a turd.
TFA does allow for off-process operations, but all of the inputs to the operation would need to be passed explicitly. In this sense, I suppose TFA isn't arguing against multiprocessing per-se, but against the specific type that implicitly includes all of the current process state (which has both up- and down-sides).
You don't have to suppose anything, TFA specifically says that you should use posix_spawn or immediately exec() after forking.
It doesn't imply or hint, let alone say, that threads are superior, it only mentions them because they interact badly with fork() and that's the issue they'd hit. It's not like threads are the only thing which interacts badly with fork.
- to daemonize
- to fork multiple worker processes
But I think that's about it for good uses of fork().
For all process spawning uses of fork() I strongly recommend vfork() or posix_spawn() instead.
The Linux man pages have a list of safe operations after fork here: https://man7.org/linux/man-pages/man7/signal-safety.7.html
Note that this includes most of your standard syscalls, like (importantly) write(), read(), close(), chdir(), as well as certain “obviously safe” library functions like strlen(), memcpy(), etc.
Non-multithreaded programs can fork() how they like and do whatever they want after (mostly).
Yes, but the async-signal-safe restriction is pretty severe, so you have to know what you're doing. Yes, that's also true of vfork(), but at least vfork() will be much faster.
> Non-multithreaded programs can fork() how they like and do whatever they want after (mostly).
Only as long as they haven't used libraries that are not fork-safe prior to calling fork(). And you still need to do things like fflush() stdio handles prior to fork()ing.
http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_...
In general, I don't see how one could safely rely on a third-party library spawning or not spawning threads unless they explicitly make guarantees regarding not using them as part of their public contract.
BTW, it's unclear whether the turd is... the specific truth revealed, or the revelation itself (since it could be incorrect). It's still a glorious comment.
Now, if only we had a time machine...
"If in doubt, Meriadoc, always follow your nose." - Gandalf
They called it Minix because it was mini.
And they called it POSIX because...
1. No analog to tcsetpgrp, so it's no good if job control is enabled
2. No analog to fchdir, meaning you have to synchronize with fchdir elsewhere in the progarm
3. Error codes do not convey enough information for good error messages (e.g. if a file doesn't exist, posix_spawn doesn't tell you which file)
4. Inconsistent behavior around dup2 fd redirections and CLO_EXEC.
5. Inconsistent behavior for shebangless scripts
These are basically deal-breakers so fish also supports a fork/exec path. However the performance benefits of posix_spawn are too real to ignore so fish uses posix_spawn when it can, and fork/exec when it must.
Though that does seem like a large number of corner cases, probably learned through painful experience :-/
I tweeted a little about it here, with some perf numbers: https://twitter.com/ridiculous_fish/status/12328893907639336...
I think most of these are deficiencies in the available posix_spawn actions, not anything inherent. Of course getting all the relevant OSes to add new functionality is a huge pain. The error handling seems bad though.
For example, suppose I am using library A and I initialized the random number generator with a fixed seed. Clearly when I fork it's not appropriate for A to reseed, because I wanted fixed behavior. Something is very wrong so probably there should be an exception. But now suppose I was using library B which was using A and B handles getting system entropy to seed A. Now it is clear that when I fork I probably want B to reseed A, but alas A has already raised an exception because it was given a (from its perspective) fixed seed. So now A needs to be redesigned to be given a seed and like some sort of intent on what should happen when forking, and oh my god wow this is creating a lot of work for everyone everywhere this is not actually going to be done consistently and cannot be trusted.
For all other RNG uses, you really do want it to reseed.
A cryptographic PRNG vs. a simulation PRNG are very different things, and should be different libraries.
https://www.metzdowd.com/pipermail/cryptography/2017-Novembe...
What's the purpose for this strategy opposed to deriving every single random value from `/dev/urandom`, simple performance?
Go only exports os.ForkExec() -- there is no os.Fork() or os.Exec(), because the things you can do between the calls could break Go's threaded runtime. (Goroutines are implemented with OS threads.)
Some elaboration on that: https://lobste.rs/s/hj3np3/mvdan_sh_posix_shell_go#c_qszuer
That is, the space between fork and exec is where pipelines are implemented, but also entire subinterpreters/subshells. The shell actually uses copy-on-write usefully. (And yes I'm aware that there's a good argument that the shell is almost the ONLY program that needs fork() !)
----
A lot of people have asked me why not implement Oil in Go and various other languages, so I wrote this page:
https://github.com/oilshell/oil/wiki/FAQ:-Why-Not-Write-Oil-...
So the funny thing is that Python is a lower level language than Go for this particular problem. It doesn't do anything weird with regard to syscalls. I'm still looking for help on this (and donations to pay people other than me):
Oil Is Being Implemented "Middle Out" https://www.oilshell.org/blog/2022/03/middle-out.html
It's been a while since I looked at it, but I believe Android uses fork for it's copy-on-write sementics to optimize app startup. On boot it initializes a single instance of the app runtime environment. Then when you launch apps that initial process is forked. As a result you do not need to reinitialize the runtime for every app launch.
I'm not sure which, if any; I'd like to see an analysis of that... The issue is what kernel state is preserved/shared across the process creation call.
Every process sort of has a "mirror" in kernel memory. The user memory is CoW, and I suppose you also have to choose whether to copy or reference every kernel data structure as well --- open files in FD tables which point to disk/pipes/sockets, locks which seem to be nonsensical, etc.
But probably you can get the "warmup" property without the full semantics of fork(). That is the CoW of user memory is a somewhat separate choice from the kernel data structures.
----
As far as the shell .... In the recent linked thread, Ninja uses posix_spawn because it has a simple use of subprocesses: https://news.ycombinator.com/item?id=30503382
So there are definitely cases where a shell uses fork/exec like Ninja, so you could imagine optimizing it. But the subshell/subinterpreter case is probably the most general -- the language semantics depend on it. And it's actually useful, e.g. this "alternative shell challenge":
https://www.oilshell.org/blog/2020/02/good-parts-sketch.html...
and I use that pattern literally yesterday and this morning, etc.
----
edit: looks like fish already addresses this, i.e. where you can use posix_spawn in a shell, and where you can't! https://news.ycombinator.com/item?id=31743230
You have a parent process which uses dlopen() to load all the libraries you want to avoid re-linking. When you want to spawn a child, rather than exec() you dlopen() an object with your child's main() and call it. For the case where you have enough libraries this is much faster than an exec(), saving tens of seconds on every application launch if you have a really bad case of C++.
There some small surprises which become obvious with a little thought. You are responsible for everything that normally happens in your process before main() is called. ASLR is only done once per session. People rarely think to fix-up argv[] for ps and friends in the first version.
Basics: https://gist.github.com/ec8469273c7808d46c7285cd056d0104
Typical use: `./a.out seq 3 2 9 -- cat -n` is similar to `seq 3 2 9 | cat -n` except that the return value is nonzero if either side's return value is nonzero.
that said, I wouldn't be surprised if there's something important I'm overlooking here.
I saved this here! https://github.com/oilshell/oil/issues/1161
But to be fair, the only times I can recall using fork() without exec() were forking network servers, and that was mostly me learning about doing network stuff, and a forking server was the easiest to implement manually.
Oh yeah, and that one time I accidentally wrote a fork bomb trying to stress test a DNS server. At least I learned something from my mistake. ;-)
EDIT: To me, using fork() without exec() is kind of like operator overloading - there are cases where it absolutely is the right tool, but these aren't very numerous, so one should exercise caution. A lot.
execvpehm(
...,
int *handles, size_t,
void **pages, size_t,
/* etc */
);
"After a fork() in a multithreaded program, the child can safely call only async-signal-safe functions (see signal-safety(7)) until such time as it calls execve(2)."
So just use only async-singal-safe function https://man7.org/linux/man-pages/man7/signal-safety.7.html
I don't know why so many people still hit this issue when it already told you what you can do and not do in the document. I've done this sort of things without any issue.
Fork() without exec() is dangerous in large programs - https://news.ycombinator.com/item?id=12302539 - Aug 2016 (101 comments)
https://news.ycombinator.com/item?id=863871 (13 years? Yikes!)
For example, until BSD came along, not much had to change in kernel land for any shell. Job control meant that the shell would need to put all the processes for a job in the same pgrp, and also there was a need to add `setsid()`.
1. Unix has fork() because it was influenced by Multics. I don't have the citation now, but I think some parts of Unix were from ITSS, perhaps the hierarchical file system, and some were from Multics. It was a drastic simplification of those systems, but with the same ideas.
---
2. The shell developer and the kernel developer were really the same person -- Ken Thompson. I link to his original paper in this post [1]. The original Thompson shell had pipelines and redirects, which are most of what happens between fork() and exec() in a shell.
Also of note is a video by Stephen Bourne who says that Ken Thompson being away at Berkeley was a good time to turn his shell into a programming language (Bourne shell) [2].
Similarly, I read that Bill Joy added chroot() to Unix simply because he needed it for something he was doing one time (building another Unix system, I would imagine).
---
3. Bill Joy also added job control to the kernel and the terminal for his csh shell. It's a very tightly coupled and ugly design.
So the key point is that we shouldn't assume any notion of "shell developers" or "APIs". The kernel developer and the shell developer were really the same person -- Thompson and Joy.
Unix is more of a holistic system than a modular one; it's porous by design!
---
Since job control, it appears that have been almost no system calls added for a shell. Although I have a feeling a few fcntl() operations were added for a shell, i.e. the difference between dup2() and fcntl(F_DUPFD). But I don't remember the argument offhand.
So I think the history is basically what usually happens -- once the same person stops working on 2 sides of an interface, the interface calcifies. It's a little like the relationship between ISAs and C. So we will probably have fork() forever, but that doesn't mean there can't be evolution in a better direction.
[1] Unix Shell: History and Trivia
https://www.oilshell.org/blog/2021/08/history-trivia.html
[2] https://www.oilshell.org/blog/2022/03/middle-out.html#more-w...
the posix_spawn mentioned in the article is effectively the equivalent of CreateProcess
In Solaris/Illumos it uses vfork() or vforkx().
In principle posix_spawn() can be a system call.
vfork() solves the problem of not wasting so much time on fork() when you're just going to call exec() afterwards (fork() does A LOT of work - potentially, anyway).
Mostly wrong.
You can call functions on the child side of vfork(), but you don't want to exec() in them -- you want to exec() in the same function that called vfork().
And you can write to local variables, but you have to be careful about it.
There's a ton of vfork()-using code that does these things.
Now, it's true that a compiler optimizer that knows nothing about vfork() but knows about _exit()'s semantics, could delete code it thinks is unreachable. So there is some issue, but you can just disable the optimizer if you run into this.
> The vfork() function has the same effect as fork(2), except that the behavior is undefined if the process created by vfork() either modifies any data other than a variable of type pid_t used to store the return value from vfork(), or returns from the function in which vfork() was called, or calls any other function before successfully calling _exit(2) or one of the exec(3) family of functions.
So sure - you can do these things, but they have very little defined semantics after vfork().
It is true that Linux describes the semantics more clearly, so perhaps on Linux it is safer to use.
Moreover, most posix_spawn() implementations use vfork(), and they call more functions than _exit() and exec on the child side.
Let's be reasonable about these things.
Threads were bolted onto Unix in a hamfisted way, breaking more than just fork. For instance, threads broke relative paths, requiring "at" functions like openat to be invented, an ugly stop-gap measure. Threads were badly integrated with signal handling too, another example.
Blaming those existing mechanisms is purely an emotional argument, from the perspective of being infatuated with threads.
The design of threads (coming from various efforts that became POSIX threads) came from such an infatuation: the desire to get any kinds of threads working at any cost, while ignoring the global state that exists in a Unix process, and the need to make a lot of it thread-local, or at least optionally so.
A thread-local working directory or signal mask would have caused difficulties in hack thread implementations that used user space scheduling or M:N (M user space threads to N kernel tasks).
The situation we have today largely comes from the initial reluctance to accept the fact that each thread has to be an entity known to the kernel; the belief that user space threads are viable into the long-term future.
How do you create a new process and pipe it data in a fast fashion without using fork, exec or posix_spawn ?
Did you manage to forget what you'd read two paragraphs earlier when you reached this bit? Because the essay's first recommendation is literally:
> Only use fork to immediately call exec (or just use posix_spawn).
It seems difficult to infer "don't use exec or posix_spawn" from this.
Since you probably don't know what all the other threads in your process are up to, your only option is to attach a debugger to all of them, halt them all, and copy all their state into brand new threads in the child process.
Do it all correctly and you end up with a multi-threaded-fork.
You still need to fix up signal handlers, interrupted syscalls, various notification API's that no longer work, memory mapped temp files used for IPC, pipes and sockets, and a bunch of other things.
But a fork of a complex process is possible. It just isn't easy.
I also find that libraries that absolutely need to make their own threads are better off being their own process. Then you can use proper communication methods to pass data.
Some people would probably argue that if you use Perl, you have much bigger problems to worry about, but that's another debate.
There will never be a time at which you can reliably expect any program developed on one system to "just work" on a different system. This person wasted a lot of time tracking down what was essentially a portability bug. Did they need this to be portable? Was this time well spent generating business value?
Pick one system for development through production, stick to it. There will be portability bugs hiding in your code, but you will never have to fix them. You will be upset for a minute that you can't use a different system, but you will get over it.