A future for fork(2)
brrt-to-the-future.blogspot.com
brrt-to-the-future.blogspot.com
Instead of having to deal with the duality of threads and processes they they opted to implement threading using light weight processes that share the parents data and bss segments. This is only turned on if you pass the RFMEM fag to rfork(), otherwise it behaves the same as unix fork and gives the child a copy of those segments.
This simplifies the system design as you only deal with processes which is arguably a more correct and simplified approach (I mean the OS only has 30 something syscalls). The api is implemented via a CSP library simply called thread. You replace main() with threadmain() and the thread api gives you all the concurrency tools. Or you can create your own threading library yourself if you so choose to.
If you are familiar with Go's CSP concurrency then you can thank plan 9's thread as Rob Pike, Ken Thompson and Russ Cox came from Bell Labs and built plan 9. And those ideas were born in Alef and Newsqueak.
You can still have say a "multi-threaded program" with a global mutex (the threads are created with RFMEM), and then when you want to fork (without RFMEM), you can be prone to deadlocks.
I guess the thread library has channels, and that doesn't happen? But as far as I know there are still mutexes in Go, and I would presume plan 9.
Go also has problems forking since it has a multi-threaded runtime. It only exposes os.ForkExec(), not os.Fork() and os.Execv().
It does! Have a look at the thread man page: http://man.9front.org/2/thread
It's pretty much a wrapper/helper library built around rfork.
true, and this is causing lots of headache for those who want to implement concurrent snapshotting using fork in Go.
It would be like wondering about what UNIX used to be like in V6, ignoring everything else that followed up.
The example given by the author:
make it practical to introduce concurrency after development, rather than designing it in from the start
> unlike threading it also gives you isolation.
If you want isolation why not launch processes? With forks, that copy-on-write thing can increase total RAM usage by a huge factor, if the parent process writes to RAM a lot.
> There are plenty of IPC mechanisms available to communicate things back and forth when you need to.
For communicating gigabytes of data back and forth you want shared memory anyway.
And even for small messages, with threads IPC is faster (1) they are in the same address space, TLB stays current, context switches are faster (2) In some cases, shared memory enables more efficient user-mode IPC methods like busy waiting and lock-free stuff.
Not really. You are better off communicating what to do with the data, rather than sharing the data.
Not if you actually want to distribute the work over several cores.
Example: Your data is partitioned into 4 segments A, B, C, and D. In step 1, node X processes A&B and node Y processes C&D; in step 2, node X processes A&D and node Y processes C&B. If A, B, C, and D are large, this is a prime example where shared memory trumps message passing in terms of performance. If you think that's a contrived case: dense matrix multiplication behaves like this.
Another way to look at it is that the hardware has a very efficient word-level message-passing interface built-in, via shared memory. Why not take advantage of it?
Because shared mutable state is the root of all evil, or at least very difficult to reason about. Better to have clear owners for all state; if you need to conduct multiple operations on the same data you should move the code to where the data is rather than sharing the data between different processing units (this has long been a truism of distributed system design; modern multicore processors behave much the same as distributed systems if you zoom in enough, and the same techniques are appropriate).
They aren't unrelated, they are parent and child. They know PIDs of each other. Parent can read/write stuff to child's streams, input output error. Child inherits opened files. Parent can set command-line arguments & environment variables for the child.
Also I've seen a project which persists large amounts of data to disk by forking and using a consistent snapshot of the data structures (which the new process will not mutate, so there's no need for locking).
https://redis.io/topics/faq#background-saving-fails-with-a-f...
I'm pretty skeptical of that approach. If there's heavy churn, the copy-on-write behavior means that the mark phase _increases_ memory use, when presumably you're doing GC because you need to _decrease_ memory use. And I assume there's some cost to these VM manipulations in general. Is this implementation competitive with other D garbage collectors in terms of memory usage, CPU time, latency, etc? are D's garbage collectors in general competitive with those of Go and Java?
It isn't an uncommon strategy. For example, the Factorio dedicated server can (optionally) do that if you're running it on non-Windows. The performance is great, but it's marked as experimental, probably because of issues like this article describes.
Apart from that, the only thing I have used fork for (apart from fork+exec) is doing concurrent blocking I/O while prototyping, before rewriting the code to use non-blocking I/O with no forking.
So we had a process that would start up, initialize the framework, then look for jobs to run. For each job, it would fork, giving the job an initialized framework but also giving jobs some isolation.
Android does something similar:
"Each app process is forked from an existing process called Zygote. The Zygote process starts when the system boots and loads common framework code and resources (such as activity themes). To start a new app process, the system forks the Zygote process then loads and runs the app's code in the new process. This approach allows most of the RAM pages allocated for framework code and resources to be shared across all app processes."
( https://developer.android.com/topic/performance/memory-overv... )
I believe the Apache web server's Prefork MPM module does something similar—fork() allows the server to spin up new child processes in a pre-initialized state, ready to respond to requests. Servers in general tend to be a good fit for this style of execution.
Emacs works this way, because it has a lot of Lisp code to load in and compile and doesn't have a central daemon (and even if it did, it would want to avoid recompiling the code when the computer is rebooted.) The compiled C binary "temacs" is instructed to load and evaluate a bunch of Lisp and then more-or-less coredump itself in an orderly way to a binary called "emacs"; when that binary is run, everything is in memory as if you had forked temacs after initialization. https://www.gnu.org/software/emacs/manual/html_node/elisp/Bu...
For your average server process reading config files, it probably suffices to have a "compiled" form of the config files in Cap'n Proto or some similar format that is directly usable as in-memory structures, and mmap that in. (Or perhaps just actual native data structures, if the format doesn't need to have a stable representation across versions of your software.)
I don't classify these use cases are concurrency-after-development cases. Lots of systems use this to do interesting things.
Ages ago I made a DNS server, and near the top of the program was a simple:
for(i=0;i<ncpus;++i)if(!fork())break;
Most of the use cases of threads are covered by fork+explicit mmap/vm_remap.
A "shared-nothing" will also contain less code, and having no need for lock/unlock keeps a lot of spam invalidations away from the memory controller making it for many tasks faster.
> Even if you're safe from other threads, there is a lot of other unknown state like fd's still inherited.
The problems you get with threads are bigger.
We have a single process on each server that starts up and loads every single game resource. After this is waits for requests from the instance manager to create new instances. As new instances are required, this "spawner" process fork()s to create one.
Each process uses less memory because all the loaded resources the game needs to run are shared between all the copies.
You also get slightly faster performance because commonly used data still in the CPU cache when switching between processes.
If we used a thread per instance, then we wouldn't have crash isolation between the different game instances, which is handy.
It's also possible to explicitly mmap a shared memory segment to load resources in that without using fork()s copy on write, but it's just more tricky, and the fork() method is totally robust no matter what code other programmers might write.
fork/exec was a convenient hack for singlethreaded small programs on the PDP-11. Its semantics interact badly with all kinds of subsequent features, and it interferes with porting to other operating systems.
posix does provide posix_spawn() though, which an OS could implement in a more efficient way than fork()+exec().
Some OSes implement posix_spawn() as a system call. IIRC NetBSD has had that, and Solaris 12 has it too.
Not gonna happen without a radically different CPU, OS, and application software paradigm. In other words: extremely unlikely in the next 2-3 decades (at least).
> It's too hard to use
Agree for most languages. Have you tried Rust? It makes shared memory multi-threading not just much simpler, but also safe.
> too slow
Very wrong for multi-core CPUs, where message-passing is emulated using shared memory.
> too broken
What does that even mean? With the right language-enforced mechanism, it is very much non-broken and functional.
> and has no future either way.
Very wrong, see first paragraph.
It makes it safe by not letting you share; data is owned explicitly and passed explicitly.
That is simply wrong. Rust makes it easy and safe to share immutable data – it's the ability to mutate that can't be shared.
> data is owned explicitly and passed explicitly
Yes, data is owned explicitly, but the permission to read or write the data is somewhat independent of ownership – the owner of an object can grant permission to access the object, and it can pass this permission to other threads without relinquishing ownership.
So, I do think you're more right than your parent, but I also think this formulation misses important things.
Rust only lets you share when you cannot cause a data race. A type like Mutex lets you share something that's mutable, because the mutex is guarding the sharing. Functions like split_at_mut let you mutably share disjoint parts of a slice.
fork(2) is evil. It is no longer needed.
Use posix_spawn().
If you need to spawn child worker processes sharing state with the parent, then pass that state via pipe and tell the child about it via a command-line option. Or something like that.
fork(2) should absolutely be deprecated.
vfork(2) can stay. It's hard to use, but not actually harder to use safely than fork(2), and has much better performance (no copies, no CoW). vfork(2)'s reputation as nasty is undeserved.
Fork is also stupid. 99.9% of the time (if not more), it’ll be followed by exec/execve. Why go through all the pain of either copying everything or take the performance and complexity hit of setting up COW memory space if it’ll all be thrown away in a few microseconds so a different process can be launched?
Then it leaves as “a trivial implementation detail” to the developer the process of setting up the pgrp, inheriting or negotiating terminal access (and dealing with the impossible to avoid race condition - except by using vfork instead and accepting its limitations) of determining if the child set its own pgrp before the parent did so tcsetpgrp can be called adept before the child can call exec so it doesn’t end up SIGTTIN or SIGTTOU when it tries to use the terminal after exec.
No thanks.
I used to think fork(2)+exec(2) was genius and much more versatile than the WIN32 CreateProcess*() functions. The last part is still true, but I no longer think fork(2) is genius. fork(2) was genius back in the 70s when it was easier to do all the spawning work in user-land, but it's not actually a good thing, not now.
The only sane way to use fork(2) is to use it very early on in main() to start worker child processes. Or to immediately exit(2) or exec(2) on one or the other (or both) side(s) of fork(2) without calling any async-signal-UNsafe code on the child side. If you're going to exec() on the child side, then just use vfork(2) -- it's much easier and safer. Even better, if you're going to exec() on the child side then just use posix_spawn(3) and be done -- let the C library / OS figure out how best to spawn the child process. (On some systems posix_spawn() is a system call. On others it's implemented in terms of vfork(2) or clone(2) called in vfork(2)-like manner.)
The irony is that vfork(2) has the worse reputation, but it really is fork(2) that is evil. fork(2) is the root of much evil Unix-land:
- there are fork-safety considerations galore
- so you can only call async-signal-safe functions
on the child-side of the fork (unless you're
quite certain of the parent's state at the time
of the fork, which is why it's OK for the child
to continue if the fork was done early in main())
- it's EXPENSIVE
- copying the RSS, or all the writable data, or
arranging for CoW -- all of these are expensive
nowadays
- it complicates accounting of swap / VM space
(JNI code that calls fork(2) in a JVM with a 32GB
heap... really slows you down.)
- it has terrible semantics as to signals
- every time something is added to what it means to
be a process... one has to think about whether that
should be inherited across fork(2) or not
The kernel would need to suspend any thread that attempts to take a lock or is not currently holding a lock. That immediately makes userland-only synchronization besides lockless impossible; the kernel has to know about the synchronization primitives.
Once all threads besides the forking thread are blocked trying to acquire a lock or suspended without holding any locks the process can fork and all threads resume.
A far better payoff might be making processes cheaper to create, faster to startup, and providing easier-to-use IPC mechanisms. Those things benefit lots of programs, not just esoteric uses of fork().
Well, mostly. One significant exception: Synchronization mechanisms sometimes identify threads internally using a system-global ID. For example, some variants of the Linux futex() syscall require you to write your thread ID, as returned by gettid(), to the lock word to indicate that the lock is owned by you. This obviously won’t work if a thread’s ID can change at any time; you would have to switch to some other ID that’s instanced per-process. Except that would break mutexes shared between processes using shared memory, so you’d have to find a different solution for that. Solvable, but not quite trivial.
By the way, as a sort of existence proof, Linux already has a way to do something like multithreaded fork()! Namely, CRIU (“checkpoint restore in userland”), used with containers, is a program that can dump a process (or set of processes) to disk and restore them later – including multithreaded processes. This is normally used for things like migrating containers across physical machines, but AFAIK, in theory you could restore the same dump multiple times to get multiple copies of the same process; it would just be slow, and massively overkill. :) The catch: it relies on PID namespaces, such that all processes and threads get the same PIDs and TIDs after being restored as they had before – but they’re only unique within the PID namespace, i.e. container. Thus it doesn’t have to deal with the possibility of TIDs changing, but the process has to be isolated from others. For a container, isolation is what you want anyway, but it’s probably not what you want if you’re just a random Unix program trying to fork.
This may be easier said than done.
Here is a recent article on creating a process.
https://depletionmode.com/zircon-process.html
One correction on this article I noticed is that the PCI driver is currently in the kernel and not user space.