Fork is not my favourite syscall
sircmpwn.github.io
sircmpwn.github.io
This and other things are a massive cause of trouble for libraries that need to run subprocesses but need to also link to main programs that might use threads (such as libvirt).
For most more complex programs and libraries, you have no control over when they malloc(). If you spawn a thread and run any library function in it, it may malloc().
Doesn't this mean that any program written in any programming language that supports multiple threads and spawns subprocesses is broken (it is racy for deadlocks, if the spawned process also wants to malloc)?
(EDIT: I misunderstood what the parent said, so it's not so bad. See my comment below.)
Is this problem discussed anywhere? Are there known ways how people handle this problem?
Yes.
> Are there known ways how people handle this problem?
The usual way is to ensure that after forking you only execute async-signal-safe code. Which includes exec(). So if you have more complicated logic which cannot be made async-signal-safe, you exec() an executable which performs that logic.
This is a massive pain though. For example setenv() is not async-signal-safe so if I want to exec() something with specific env vars then I have to go through /usr/bin/env. Sigh.
Reading your comment I realised I misunderstood what the parent said. This misunderstanding is what made it a suprise for me.
Critically, I missed "... between the fork and the exec ..." and only considered the statement "the lock is still held by the new child process which deadlocks in malloc", assuming this was somehow the case even after the exec(), which would be a huge problem if even the exec()ed child wouldn't be allowed to malloc.
Of course on second thought this wrong assumption is nonsensical: When you exec(), you get a new process space, so any lock potentially thought up by libc or anything else, will be gone.
So all is fine:
For the purpose of starting another executable / implementing a popen(), one just must assure to not malloc (and as you say, in general write signal-handler-safe code) between fork and exec. This is feasible, and I suppose most programming languages do ensure that.
Instead what I'd recommend is that you have a pipe() for passing an error message (one byte will do) back to the parent, and then _exit().
What if the exec succeeds? The pipe is never closed.
If you mark the pipe as O_CLOEXEC and rely on the parent treating EOF as success: what if the child crashes prematurely before exec? E.g. killed by sysadmin, or by OOM killer.
What if a write() to the pipe fails?
- Don't thread before you fork.
- Don't lock before you fork.
- Don't open before you fork.
- Don't do anything that is not meant to be a shared resource before you fork.
Locks and threads is the worst. Locks can end up in weird states.
* http://jdebp.eu./FGA/unix-daemon-readiness-protocol-problems...
It is a compatibility mode, and it is broken beyond repair because the technique itself is broken beyond repair.
If you try to implement the forking readiness protocol (you mentioned that doing so would be problematic), you are doing something wrong. It only exists for compatibility with existing legacy services, which implement behavior that has been a bad idea and problematic since the 80's.
I'm not personally a systemd fan, but no "forking" readiness problems can be attributed to systemd.
If you instead of stopping at the title read the actual post you linked, you'd notice that it explains that the issue lies not in the forking protocol, but in that legacy applications cannot signal readiness. See the third and fourth paragraph in the "No-one speaks the forking protocol" chapter. It also explains how daemonizing is a bad idea altogether, but that's separate.
The essence here is that init systems predating upstart/systemd/... did not have a readiness protocol. Instead, they generally assumed that a daemonized service was immediately available after the parent exited, as that was when the script continued. Forking is meant to pug
The "forking" protocol is only meant to plug such an application into systemd, nothing more. It grants you zero readiness information (just like when it was started by a sysvinit script), as such information does not exist. It would be stupid to try to implement this protocol. It is nothing but a compatibility layer.
If you have an actual counter-argument (so far you have only posted a link, and then restated that you posted that link several times), please present it. I don't mind being proven wrong.
Using "forking" as readiness protocol doesn't make sense. Unlike "notify", it doesn't signal readiness. It's only there to try to understand old services that always daemonize, which was only done because the old init systems required it. Daemonizing is pointless under systemd or other init systems that handle this themselves.
if I want to exec() something with specific
env vars then I have to go through
/usr/bin/env. Sigh.
0 - https://www.freebsd.org/cgi/man.cgi?query=execve&apropos=0&s...
In either case, the parent process can free the memory allocated for the execve call once it returns.
0 - https://www.freebsd.org/cgi/man.cgi?query=strerror_r&apropos...
That's a problem with threads, not fork(). It generalizes to all sorts of things needing to know about internals of threads and locks.
But there's lots of other issues. Things like PID number reuse, libraries in single-threaded programs that may need to reinitialize on fork() (via pthread_atfork()? no because fork() might have been called directly via syscall() or by trapping from asm code, bypassing the C library altogether), and so on.
Just use posix_spawn().
EDIT: Also, ultimately, the worst thing about fork() is that it can't possibly be high-performance. Either fork() copies the writable parts of the address-space, or sets up COW, or copies the RSS and sets up COW for the rest. Well, on modern CPUs COW is really expensive. And copying a JVM is no fun either.
For this purpose, the choice to start with the same data structures and the same execution point is quite sensible - it's essentially what happens when you create a thread nowadays. So, the hypothetical discussion where someone decided to copy the address space in order to launch new programs, would never have happened.
The day where someone decided to reuse the existing fork code to launch processes for new programs, that one must have been worth to watch.
I'm not familiar with the guts of threading libs, but AFAIK the syscall that spawns threads is clone(), which takes a function pointer and a stack pointer, so it very much does not start "with the same data structures and the same execution point".
This is all fine, there are two scenarios, neither of which is the imagined "race" described above:
1. "Fork-and-exec" Our new execution context will actually run a different program. We immediately do so, the new program doesn't care about most of the state of the old program, it throws that state away, including locks that were held by the old program's allocator.
2. "New Thread" Our new context will share all the state, if another thread has a process-wide lock for some reason, it can and should apply to us too. If it took a thread-specific lock, we're not that thread, so we already don't care.
Huh? I saw that the manpage for clone() is in section 2, which AFAIK means it's kernel API, whereas C library functions are usually in section 3. Am I mistaken?
Even syscalls(2) refers to clone(2) in its list of syscalls.
You are correct that the manual section 2 is "for" system calls, but in practice today these are just the lowest level of the C library, and not the actual system calls.
My copy of the clone(2) manual page hints at this by commenting /* For the prototype of the raw system call, see NOTES */
and more explicitly in its body text by saying "The main text describes the wrapper function; the differences for the raw system call are described toward the end of this page."
Other modern stuff is the same way, the section 2 documentation is describing what an ordinary C programmer needs to know, but if you're say, implementing the C standard library or writing a compiler for a language that runs on the bare metal you need to read the kernel documentation which is not provided as manual pages.
I thought the fork syscall was the only way to spawn new programs in earlier versions of Unix, and that the spawn syscall only came later. Is that not right? If there was no other syscall, fork must have been designed for this purpose.
Parallelism? In the original unix? You're talking absolute nonsense. These were single socket, single core systems. How on earth do you think it was designed for parallelism?
But we don't even need to debate this - read the 1971 manual.
https://www.bell-labs.com/usr/dmr/www/pdfs/man21.pdf
"fork is the only way new processes are created"
exec is also in there, and refers to fork and describes how to use them to run programs.
The suggestion that it was done this way because the expected main use case was a fork without an exec at least provides a rational explanation (I don't think 'threads' as distinct from processes existed yet in unix? a fork without an exec was the closest thing you had), but I have no idea if it's historically accurate. Would be interesting to find people who were around and get some background.
Ah, this is interesting:
> Then came Unix, in the early 1970s. The Unix notion of a “process” became a sequential thread of control plus a virtual address space (incidentally, the Unix notion of a process derived directly from the Multics process design [Saltzer, 66]). So “processes”, in the Unix sense, are quite heavyweight machines. Since they cannot share memory (each has its own address space), they interact through pipes, signals, etc). Shared memory (also a rather ponderous mechanism) was added much later.
> After some time, Unix users started to miss the old processes that could share memory. This led to the “invention” of threads: old-style processes that shared the address space of a single Unix process. They also were called “lightweight”, by way of contrast with “heavyweight” Unix processes. This distinction dates back to the very late 1970s or early 1980s, i.e. to the first “microkernels” (Thoth (precursor of the V-kernel and QNX), Amoeba, Chorus, the RIG-Accent-Mach family, etc).
http://www.serpentine.com/blog/threads-faq/the-history-of-th...
Not totally sure what light that sheds on the question, if any, heh.
There was also no form of interprocess communication other than the filesystem, nor any synchronization other than waiting for a specific process to exit. So having multiple processes work on the same problem would mostly have been very awkward.
Until Doug McIlroy invented pipes. And the modern Unix shell pipeline style of creating complex programs.
I really doubt this. Is there an instance of a program with 1st edition lineage that use processes to achieve concurrency? There were pipes, sure, but that could be achieved with a spawn-like call.
I don’t see processes being used for concurrency in UNIX history until BSD sockets and forking servers emerged.
On the other hand, shell pipelines were used for concurrency before BSD added networking code. And I have doubts that shell pipelines would have developed without the fork()/exec() model---see the author's Windows NT (and earlier) with really, seriously broken shell models.
I would argue that (a) was easily the most important aspect of fork(). One can always get (b) by spawning more instances of the same program and passing them whatever state is to be shared. But (a) needs a decent API and to be kernel-side if one has no fork(). Yes, posix_spawn() is better, but fork() enabled Unix to kick the can of standardizing a spawn API that doesn't suck until enough experience was had with it to make such a thing.
- set overcommit = 1, OOM now comes only when sum(RSS) exhausts swap + memory
- just increase overcommit_ratio if you want to stay under overcommit =2
* https://www.cloudatomiclab.com/antisyscall/ (https://news.ycombinator.com/item?id=15998828)
* https://www.cloudatomiclab.com/prosyscall/ (https://news.ycombinator.com/item?id=15998344)
Well, technically, no.
Originally (1st Edition?), IIUC, fork() would fail if enough memory to copy the process wasn't free. Then, when fork() was changed (4th Edition?) to read-only share the address space, that wouldn't happen, but the exec() might fail if there wasn't enough space to set up the child. But those were all doing segmented memory so they're completely irrelevant to modern systems.
The OOM-killer comes with (IIRC) BSD's paged virtual memory. Some bright spark (yes, I have hated this design since I first tripped over it in the early- to mid-'90s) had the brilliant idea of supporting sparse matrices by not requiring allocated memory to be backed by valid pages, either RAM or disk.
If you don't require allocated memory to be backed, you can allocate a giant block of memory; if you never touch most of it, everything will be copacetic. It's good, I was told ca. 1992, for scientific programs in, like, FORTRAN. Yay. Unfortunately, lots of programs started using the feature, including the X server.
(Why do I hate with the fiery passion of a thousand suns the perverted genius who came up with this? AIX 3.2.5/4.1. Running the machine out of memory (remember, allocations don't fail, everything chugs along until some proggie touches an address and the OS can't realize the page) would kick off the OOM-killer, which would immediately hit inetd in the head with its little silver hammer, resulting in a machine that would still be working, -ish, sort-of, but unless you were logged in somewhere already to reboot it, you would have to go visit the physical console.)
Turning off the OOM-killer and overallocation policy was possible, but not advised, since many programs, including the X server, would suddenly use a truly atrocious amount of memory.
TL;DR: Memory overcommitment doesn't have anything to do with fork()/exec(), but rather is a deliberate design decision made with the addition of paged virtual memory, to support a limited class of applications, that probably didn't work very well even for that class.
What I really want is to have an always responsive system. You cannot get this easily on linux because it doesn't allow to disable all paging (executable pages are still paged out even with swap disabled). This means you can easily lock up your machine if you use a lot of memory (even without swap -- I don't use swap). The above post contains a 20 line C program example that will break your system that you can only cure by applying the oom killer. I mean what other options do you have you have 0 free memory? Could a kernel hacker reading this add such an option to linux to a poor sysadmin like myself?
This was actually the cause of 1-2 second blocks in our Node event loop. We were spawning a clamd scan asynchronously, from a process with several gigabytes of RSS. I never would have imagined that fork would blindly copy gigabytes of RSS every invocation. I should have known. It took a few months to track down.
The kicker is that Node calls fork from the main event loop thread and not from the thread pool, in order to hack around the GC, if I understand correctly.
See the relevant Node issue here:
So my guess is that Thompson and co basically went with a "worse is better" approach and delivered a model that performed worse but was a lot simpler to implement and reason about.
Also... It's easy to criticize designs 40 years after. That time was a time of proposals an research. Despite this, the original Unix design, no matter how flawed, stood the test of time.
I understand why this behavior is default, if you _have_ to support fork() this perhaps the more pragmatic default.
That is a strong claim.
That said, as long as "too small to fail" (https://lwn.net/Articles/627419/) is around (which is presumably "forever"), the oom-killer won't be going away. Fork obviously compounds this massively due to copy-on-write. So ensuring your malloc'd memory is actually writable may just make you a more tempting target for the oom-killer.
> Doesn't ISO C require that if malloc returns a nonzero pointer, it has to backed physically (need not be RAM, could be swap on disk or some other thing)
I don't see anything like that, though admittedly I don't have a "real" copy of the standard. I'd also be at least a little surprised if it were true (though my being surprised by the C standard would...actually not be surprising). While the "spirit of the law" would probably make sense, the "letter of the law" would be difficult. Consider e.g. an otherwise side-effect free function with a statically computable result assuming a successful malloc (and corresponding free), could the compiler optimize away the malloc call?
> That this nearly 50 year old crappy design choice has come to this astonishes me.
That makes me feel old. For me Unix was always invented 30-something years ago...
The other is to start unrelated programs. Think bash which can start vim.
I think for the first use case, it is stupid to have to reap the child process or else it gets inherited by init. Instead you should be able to mark that process as a worker from the get go. It doesn't make sense for apache2 workers to keep running if the parent process died.
In the latter use case, I agree with the OP. I'd rather set up the structure of what I want to happen, then start that process as a completely independent thing. spawn seems like the way to go for this.
The good news is that we can totally have all those options as long as we keep fork as is at least for the next 50 years :)
https://msdn.microsoft.com/en-us/library/windows/desktop/ms6...
The elegance of fork() is in avoiding the complexity of requiring every attribute of the process to be explicitly stated in order to create a new process. We simply inherit our parent's attributes and if that turns out not be desirable it is fixed in the child with the usual syscalls.
EDIT: s/And/Or set/
It would be interesting to see in what ways Unix (especially shell scripts) would be different if processes would not inherit all the baggage by default.
So really fork is the opposite of elegance. It makes state implicit, not explicit.
Nowadays, I think ‘whitelisting’ what the new process can do is the choice to make, not forking and then (hopefully) ‘blacklisting’ what you don’t want (that’s especially important if you eventually will be running code you didn’t write or, maybe, don’t even have source for)
That also is easier to test for. If you forget to specify a capability before forking, the bugs you see will be better reproducible than when you forget a thing you don’t need.
I can’t find it now, but try googling an article on how to properly fork a process nowadays. It is insanely difficult to do right.
[0] http://pubs.opengroup.org/onlinepubs/009695399/functions/pos...
fork()/clone() obliges you to track which things the child doesn't want from the parent, and have a way to override them
CreateProcess obliges you to track which things the child DOES want from the parent, and have a way to override them.
There's a convergent evolution at work, the same way bats and birds have both got wings without a common ancestor having wings, wings work, not quite the _same_ wings, but the idea works, so it's inevitable.
futex() and WaitOnAddress have the same funhouse mirror effect. They're doing the same exact thing (very, very cheap synchronisation primitive, with just barely enough features to be useful), the names and descriptions are really different, but if you use both you're like "Oh, these are basically the same, huh". Both names are equally useless, futex() is a clever name if you already know what it does, but until you do you've no idea what to look for. On the other hand WaitOnAddress doesn't actually er, wait on an address, that would be expensive, like futex waking it up when something changes is _your_ job as the programmer.