Why is spawning a new process in Node so slow? | Hacker News Reader

Why is spawning a new process in Node so slow?

blog.val.town

95 pointsmaxmcd56 comments

It's important to specify which OS is being used because spawning goes a lot faster on Linux which has vfork(). If 40 spawns per second is the Linux speed, then I don't even want to know what that looks like on Darwin and OpenBSD. With Cosmopolitan Libc, 20 spawns per second is about the speed I get when simulating fork() on Windows. What makes fork() slow for large programs like Node is that fork() needs to lock and hold every single mutex that exists in a process while it happens. See the IEEE POSIX.1 notes surrounding pthread_atfork(). So yeah, that means everything is de facto blocked until the spawn completes, regardless of the thread that's doing it. Using the separate process to spawn is a smart idea. Especially if you can talk to it via a process shared condition variable.

vfork() was introduced in 3.0 BSD

> What makes fork() slow for large programs like Node is that fork() needs to lock and hold every single mutex that exists in a process while it happens

Are there any circumstances where you'd want to fork a multi-threaded process?

forking is useful if you need to call other code when you can't or don't want to link that code into your own process. Pretty useful for things like ffmpeg where you wouldn't want that in your process space (and you may want to put more than just a single fork/exec between your code and theirs in that case), but also handy for "simple" os integrations where it's easier to run an existing utility than find a library or interface more directly yourself.

Downsides in forking from a large multi-threaded process make it pretty common to add something in early init that forks a process (or several) that remain small and can be commanded to fork at runtime. On modern systems, it's a very small cost if you don't actually fork, and the small added cost of intermediating fork/exec pays off the first time you don't have to wait for hundreds or thousands or millions of fds to be closed in the child process during the process. Of course, if your parent process remains small with few opened filehandles, maybe it is a measurable negative, but it's still likely to be very small.

Are there circumstances where one might want to fork a multi-threaded process over spawning a child process via posix_spawn?

posix_spawn is not a syscall, last I checked, on Linux or FreeBSD. It's a library function of libcs, implemented in terms of fork, vfork, or clone and execve and has the same underlying performance issues in a large process as doing it yourself.

It would be a different thing if it was a syscall, because then you could presumably use it in a large, multi threaded process without any of the hazards that arise from using fork, vfork, and clone.

Are you asking if there's a good reason to have a fork without exec in a multithreaded program? I would think probably not, it would be pretty confusing to have all the memory and filedescriptors in the child process but only one thread --- you'd still have the other thread's stacks, but no thread running on them; that's asking for massive confusion and I think you'd need to do some cleanup for locks and whatever else. There might well be some applications where it makes senss to fork, and then spawn threads in the child processes, although I can't think of one off the top of my head. I've definitely had experience with use cases where it made more sense to fork rather than spawn threads and vice versa, but I haven't seen a reason to do both.

Yeah, I meant from a libc user's point of view, using posix_spawn() directly seems safer (presumably they've managed to ensure they don't do anything dumb; I don't trust myself to be as good). A random copy of its man page actually says it's implemented with clone() with CLONE_VFORK plus exec().

IMHO, using posix_spawn is OK if you know what libc your application is going to be using.

If you don't, you may have inconsistent results, and you'll end up having to sample the various libcs, and probably take the best posix_spawn so your application has consistent behavior. Or you'll have to build in detailed specifications of which libcs and which versions of those libcs you'll work with.

There are basically 2 unrelated uses of fork():

* In single-threaded programs (potentially without calling exec). This was widely used to get parallelism within a single program prior to 1995 or so, and is still a valid approach (except on OS X, which forbids single-threaded programs). In this case `pthread_atfork` is a reasonable thing to use, and the child process is not limited to async-signal-safe functions.

* In (potentially-multi-threaded) programs calling exec. In this case, `pthread_atfork` is not meaningful (there's a recent `_Fork` that avoids it entirely), and the child is limited to async-signal-safe functions - basically, you should do all the setup beforehand, then after `fork` just call various syscalls. `posix_spawn` and `vfork` may replace this, but both are bug-prone.

> except on OS X, which forbids single-threaded programs

…uh, what?

In recent years, Apple's libc will aggressively create threads to perform background tasks. There is no way to prevent this.

Thus, calling fork() on OS X means you can only use async-signal-safe functions, unlike other platforms where abstinence is a choice.

There’s a lot of system libraries on macOS that’ll spawn threads, but I’ve definitely never seen libc do so.

No, you can fork, what you can’t do is “fork and then try to keep running after the fork”. That’s never really been a supported model on Darwin, and in recent years with more threading occurring it fails faster than it did in the past.

On Darwin there is very little you can safely do in the child after fork(), essentially the only thing guaranteed to work at that point is exec() and a few other basic operations.

Again, this has always been the case for fork under Darwin, the only difference is that you are more likely to fail sooner now than 20 years ago. This is also why all documentation says to use posix_spawn: fork +exec is necessarily slower, and can’t really do anything that posix spawn doesn’t.

> can’t really do anything that posix spawn doesn’t.

This is blatantly untrue. Probably the most common example is `chdir`, but there's a whole list in signal-safety(7).

Using a shim is always a possibility but that's annoying.

I sounds to me that while fork() was perhaps initially a simple idea, it has now (with multithreading etc) become a monstrosity that is better replaced by a number of syscalls that are less general and have more intent behind them.

Perhaps fork() will go away naturally when other options turn out to be faster or people start porting stuff to Darwin and other OSes where fork() is not a good idea anyway.

How else would you expect a green-threaded runtime with multiple scheduler threads to spawn a managed subprocess (think: Erlang port-program) that has access to the runtime's file descriptors, unix domain sockets, etc?

Ah, I will add it to the post, an accidental omission. This was all on `Linux 6.8.0-35-generic`.

Do you have a sense for why things were so much faster in Go and Rust if they are both using a single process? I have also been experimenting with napi-rs which uses https://nodejs.org/api/n-api.html and allows you to run Rust in threads alongside the Node process. Performance with that approach has been very good, so I have wondered about the within-process limitations.

If you want to spawn a new process on Darwin you really want to reach for posix_spawn, not any variant of fork. Using this API should also do whatever is optimal on Linux, which still doesn't seem to be merely calling vfork, as you instead can use clone--with CLONE_VFORK, sure--to limit the work to CLONE_VM (per the man page for glibc's posix_spawn). (Of course, this API would also be the perfect fit for use on Windows, but I don't think MinGW or Cygwin implement it, for reasons which are beyond me, given how horrible simulating fork is in that OS; but, I believe gnulib provides it, hopefully using CreateProcess on Win32, for all of these platforms than are missing out... and, they really are "missing out", as I feel most developers--including me--end up incorrectly dealing with the issues surrounding multi-threaded fork.)

> If you want to spawn a new process on Darwin you really want to reach for posix_spawn

Assuming you can avoid the bugs and version-sensitivities. There are a lot of them on various platforms, and my gut says they're more prominent on Darwin since alternatives are unrecommended.

The dup2/cloexec mess is the one that sticks in my mind but it's far from the only one.

nodejs (via libuv) transparently switches from fork to pthread_spawn on Darwin when the options passed support it. The dup2/cloexec is a mess, but there are some Darwin-specific enhancements to work around it. There is work done to make it also do the same on Linux, but seems it hasn’t been merged yet.

The reason glibc uses clone is that it can specify an extra flag to avoid some undefined behavior inherent in the way vfork was originally designed, so it can get all of the equivalent speed with none of the pointy edges

To be fair for Windows, you shouldn’t emulate fork() but rather spawn new threads, which Windows can do in less than a millisecond. This should give you much quicker results, but comes with other complexities, especially for UN*X-ported apps. Given you emulated fork(), you probably know this.

But those aren't really the same? I mean yes they are both ways to acquire additional units of execution but if you're reaching for subprocesses it's likely specifically because it's something you can't do in threads*.

* Or can do but would be horrifying, like trying to run a compiled binary by mmaping it into your address space.

> But those aren't really the same?

It's the "Windows way". Windows process creation is slow. No, they're not identical, they're different approaches to the problem.

> subprocesses it's likely specifically because it's something you can't do in threads

I suppose the question would be, "what can't you do, if you did it the Windows way?"

I would agree, if you’re wanting to run your server software on windows, your software should be designed to work well with windows. That goes for any OS though - if you’re running a high volume server then you shouldn’t be throwing away perf by designing your software in a way that works badly on the host OS.

That said multiple threads are not multiple processes:

Using multiple threads means you get a single set of globals, and if any of them are contended you get a pile of sadness.

Similarly, tearing down threads is not necessarily sufficient to release all resources a library may have consumed because again: globals.

Related to globals: I’m not sure how heavily TLS is restricted on windows, but I recall various arbitrary limits being annoying on windows back when I did dev work, and I recall some annoying stuff with TLS — but that was back in the days of needing to support XP, so maybe it’s less of an issue these days?

You also get security advantages by pushing things into multiple processes that you don’t get from multiple threads, though I don’t really think that that’s an issue in this particular context.

For .NET Framework, AppDomains were a solution to isolation. .NET [Core] deprecated this feature, moving towards assembly loading contexts (and the, IMO, unfortunate move towards microservices -- like everyone needs them /s).

Not sure what your reference to TLS is about; any more specifics?

EDIT: Ugh was thinking SSL! Duh.

https://learn.microsoft.com/en-us/windows/win32/procthread/t...

> That goes for any OS though

100%. I was of course speaking in the context of this thread, but yes you probably wouldn't take a Windows-developed app and simply shift it to UN*X without some form of re-thought on the implementation.

...Unless it was a .NET app :-)

Address space separation.

Let’s give Windows some credit here. Windows had usable threads long before Linux.

NT works on threads, so yeah :-) It was _designed_ that way.

> What makes fork() slow for large programs like Node is that fork() needs to lock and hold every single mutex that exists in a process while it happens

It really doesn't; if it does that's a bug to fix. It only requires that the child process does not call any function that is not async-signal-safe if threads exist, and end with a call to exec or _exit. In C, this is usually pretty easy to arrange; at most the difficulty is if you want to support arbitrary file-descriptor mappings. However, it is impossible from languages like Javascript (or Python, etc.).

pthread_atfork() is really only useful to call if you know you're using multiple processes and not using threads at all. _Fork() is supposed to be a thing now (to avoid any historical atfork handlers) but can't be relied on to exist.

posix_spawn() has the unfortunate problem of being buggy and version-dependent when various options are used. It's probably fine if you don't need the options (for example, if you exec a shim first), or if you know how recent a platform you can use.

vfork() on Linux is mostly an optimization (for scheduling or for when overcommit is disabled), but compiler optimizations can make it buggy (though given that /bin/sh etc. use it, compiler probably won't break it too much if you call it in the same way). For best results, it's advisable to either call clone (with a separate stack directly), or use architecture-specific assembly to call `vfork` in a way the compiler can't mess up.

vfork() on many other platforms is basically just an alias for fork() (or maybe _Fork() ?), but I haven't investigated the details much.

fork() is fine if your parent process isn't big enough that you're worried about memory accounting though, and you haven't added braindead atfork handlers.

You absolutely need to lock all the locks, because fork() kills all the other threads, so if any one of them was in a critical section while fork was called, then the new process will deadlock if it tries to acquire that lock afterwards. Yes, I know the POSIX standard has turned against fork() and considers vfork() beyond the pale. I don't know how they can forbid UNIX behavior that's essential to the workings of /bin/sh. No it's not realistic for the OS to define its most important function to be as useless as a signal handler, but here we are. That advice will be never be accepted. People on the best OSes like Linux and FreeBSD will always use vfork() regardless of what POSIX says. The de facto standard is that only vfork() must guarantee a swift execve() call. You learn to take the official standard with a grain of salt when you realize it's bankrolled by regulated enterprises who hire cheap engineers to work on life or death machinery and therefore can't be trusted with powerful footguns.

> then the new process will deadlock if it tries to acquire that lock afterwards

which is not async-signal safe and thus illegal if other threads ever existed (or if we're called from a signal handler). All semi-new programs (which might use threads) are responsible for not introducing such bugs.

And if we're the only thread (and not in a signal handler), which all sufficiently-old programs (including shells) will be, it's the caller's responsibility to not call fork() while holding a mutex.

Thus, one way or another, there's really no need to do this. From an application perspective, it's unnecessary, and from a libc perspective it is at most a QoI feature (one which should be limited to a very small number of locks at most, and frankly we've strongly learned should just detect the error (which is a simple matter of 2 thread-local counters - one for "number of mutexes held" and one for "number of async-signal contexts entered" - as well as knowledge of whether other threads exist) rather than trying to continue in the face of UB).

Illegal it may be, but as jart (who has probably seen some shit, to put it mildly) points out, that doesn’t stop a very wide variety of programs from depending on this behavior, intentionally or inadvertently.

Heck, runtimes like Python (and Node, if jart is correct in GGP) ship with atfork handlers specifically for the purpose of massaging runtime-created mutexes/condvars/etc. I don’t think those are there because the language maintainers like breaking the rules; they’re probably there because users kept depending on this behavior in spite of cautionary statements.

> I don’t think those are there because the language maintainers like breaking the rules

You think wrong.

Say more? Is this just get-gud anti scripting language elitism, or a reference to some specific history that the Python/Node/others(?) maintainers have regarding decisions made in this area?

People who maintain language runtimes are typically very smart people who often do things that are inadvisable when maintaining said runtime.

> Using the separate process to spawn is a smart idea. Especially if you can talk to it via a process shared condition variable.

You can even use prctl(2) to set the parent as a subreaper for the spawner’s children so the process hierarchy continues to behave as expected and you don’t end up with orphans. Or just run such that the parent is PID 1 in a container anyway.

With either approach, watch out if you’re using a thicker runtime/language whose child process accounting can get freaked out if an unexpectedly-present child indicates ready-for-reap. Using pidfds on Linux (optionally combined with fdpass via UNIX sockets to signal that the spawner did its thing) can help with this.

> With Cosmopolitan Libc, 20 spawns per second is about the speed I get when simulating fork() on Windows.

Egads, this reminds me of when I used to muck with Cygwin. fork() is quite a complex bit of kernel magic, and that’s on kernels designed for it. Simulating fork() on Windows well enough to be usable has always struck me as quite impressive.

But it’s also ridiculous. Forking to spawn a process is an almost comically poor design, and this is one of maybe two things IMO that Windows got right! CreateProcess() may be ugly, its arguments may involve nasty data structures, the command line string may be a disaster that continues to cause problems to this day, but it’s a far superior design for spawning processes. I hope that no one who cares even slightly about process spawning performance on Windows tries to do it using fork.

Not that Windows has ever been particularly fast at spawning processes…

One addendum I would add to the section on node:cluster is that Deno hasn't actually implemented the cluster module yet (it's just a stub), so using it with Deno is pure overhead [1]. Also, there's ongoing work to add node:cluster to Bun [2].

[1] https://docs.deno.com/runtime/manual/node/compatibility/

[2] https://github.com/oven-sh/bun/pull/11492

> Turns out, that if you use a Unix socket and the filename starts with a null byte, like \0foo the socket will not exist on the filesystem and it’ll be automatically removed when no longer used. Weird! Cool!

Is that…intentional on Unix’s part? Seems kind of a weird thing to implement.

I believe that's a Linux thing.

It's very useful since "how do I atomically delete a stale socket file" is not actually an easy thing to do.

files on unix are referenced counted. and unix sockets are just a type of file. every link adds to the count, a link can be a filesystem name but open file handles are also links. once there are no more links to a file the file can be deleted.

so this is opening a unix socket with a no name(the initial null terminates the C style string) thus leaving the only link to it as the file discripter.

One trick I used to do was to sneak another link to the file to keep it around before the process ended , The serious real use case for this was if you had accidentally deleted a file but some process was keeping it open somewhere, but I mainly used it back when youtube was flash based it would try and keep the video cache file hidden this way. and you could make a copy of the video by snaging the correct file descriptor.

I have fond memories of modifying the memory of flash-based games when they first launched on Facebook.

I don't remember the FB account credentials, but I received thousands of messages from people either telling me I was a terrible person, or trying to bribe me so they could beat their sibling... all for a simple helicopter game.

Fun times.

That sounds like a bug in the spec rather than an intentional feature, though?

Nice article, definitely wasn't expecting some of its results. Was good to see Node finally beat Deno near the end of the article.

As a heads up, the author confuses its and it's. Makes the article look unprofessional. https://youryoure.com/?its here's how to differentiate

Thanks for noting it. It do indeed mix those two up all the time. Pushed a fix!

Don't we all :)

It's easy to forget what a big deal being semi fluent in multiple languages is.

I'm curious if anyone has any insights into the answer to the titular question. The article, while certainly interesting, mostly discusses workarounds, but doesn't really dive into a root cause analysis.

Yes, apologies for the less than satisfying conclusion. I am planning a part 2 of this post where I hope to actually answer the question.

In the meantime the discussion on lobste.rs includes some lower-level speculation: https://lobste.rs/s/tr8ozm/why_is_spawning_new_process_node_...

Have you looked into profiling with strace? You should be able to see the actual system calls each of your tests are making. This will probably give you some good insights into what accounts for the speed differences.

Thanks for the super interesting read.

Did you happen to look at how the load on the 8 cores looked at any given time?

Both Go and Rust had CPU near 100% on all cores. Most of the Deno and Bun workloads were at around 80%, some of the slower Node benchmarks were around 40-60% and that increased over time, I think most of the fastest benchmarks were seeing very good CPU utilization across all runs.

Sorry, this is all from memory of just having `btop` open for some of the benchmarks. Maybe in a future post I will dig into CPU utlization more.

Nice article! I remember three years ago, I needed to spawn a few processes in Node, and I ended up using child process spawn to spawn an external binary. It was fast and instantaneous (no benchmarks). It was on a Linux OS, ARM-based.

The problem is `fork()`. Use `vfork()` or `posix_spawn()`.

A really nice read, thanks for posting. I'm curious about why it's not as fast as Go and is it possible to speed up to Go's level?

Reply on news.ycombinator.com