io_uring_spawn: Launching New Processes with io_uring

I hope I/O write queues at some point. Being able to inject I/O barriers relevant to the I/O stream instead of a system-wide barrier can let you write some higher efficiency I/O.

> One has to wonder, though, given Josh' work with Rust, if any of this work is going to be done in Rust? One can dream ..

Not yet, since the Rust-in-Linux work hasn't been merged yet. And even once it is, it won't be usable in the core kernel yet. But I'm hopeful that io_uring will be able to make use of Rust in the future.

And the userspace bits will definitely be possible to do in Rust. As soon as this ships in a stable kernel, I plan to add a code path in the Rust standard library to launch processes this way. :)

I don't follow it particularly closely, but I recall that there was a recent LWN article, "Security requirements for new kernel features" [1], which discussed the issue.

[1] https://lwn.net/Articles/902466/

seccomp filters generally are whitelists. If they haven't whitelisted io_uring_setup it's a non-issue. They only suffer the performance consequences of userspace having to use the synchronous syscalls.

also, iouring doesn't allow arbitrary syscalls. so it's not really a complete seccomp bypass even when allowed

If you're writing anything with performance considerations using shell scripts, you're almost certainly doing something wrong. If it includes spawning thousands of processes, you're definitely doing something wrong.

I'm not sure if it's really used anymore but inetd is a perfect usecase

Just something as simple as setting up a threadpool adds up timewise if you do it in serial. It can easily be that making it faster adds up use cases where processes are not possible today because they would be too slow to set up compared to the alternatives.

What I missed from the presentation is some discussion about the asyncronicity of the clone calls. Yes, the average time of cloning once is 30us, but if you queue up two of them at once, how long does that take? Will it just sum or will the total time be shorter? Today there are applications that set up a thread pool by recursively open new threads, both in the original thread and in the forked ones. That is more than a little bit icky and could perhaps be simplified with this.

fork bomb

Making process spawning faster would allow for workloads to spawn "so many" processes/threads significantly more practically, and therefore more simply than userspace async-style runtimes.

(Note that threads on Linux are just processes which share memory and some other properties.)

Merely halving it may not be all that's needed to totally replace those runtimes but it's a good step in the right direction.

Spawning a process consists of basically three steps: create a new process, twiddle attributes of the new process, and load the new process image. And for this, we (presently) have three choices: fork, vfork, and posix_spawn.

The cost of fork is proportional to the complexity of the spawning process (e.g., the size of its page tables), not the spawned process. The talk gives an example where fork takes 7ms, although I'm not entirely clear what it's doing (I only read the slides, not listened to the talk).

vfork and posix_spawn allow you to avoid that complexity. However, this comes at a price: posix_spawn only permits a very limited set of process attribute twiddling, and vfork is so unsafe that it's difficult to sanction its use... well in general.

From the slides, the intention is to provide something like vfork except without the inherent unsafety of vfork. It describes vfork as "Effectively a thread with no synchronization running with the same stack as the parent". Meanwhile, you can look at io_uring as describing a kernel thread that does a limited set of tasks... so why not implement something like vfork on top of io_uring? Of course, the interesting question is what process bits can be twiddled in io_uring_spawn, but unfortunately, there's no examples from what I can see to get a sense of how powerful it is or isn't.

Isn't spawning a POSIX thread the same thing?

This kind of optimisation becomes more important as the parent process's address space grows e.g. a browser wanting to spawn a process per origin or even worse an in memory database spawning a tiny shell script as hook. The existing (v)fork()+exec() syscalls have to change the existing address space to copy on write which can involve shooting down TLB entries on all other CPU cores with IPIs (inter-processor interrupts) and setup new page tables for the child only to tear them down immediately and bring in a new executable. It also doesn't play nice with unified event loops implemented with the common epoll()/kqueue() etc. syscalls which is a problem for languages like go which depend on event loops for their runtimes.

Forking process per request in a request handling server would be one example where process fork time would add to request latency. (this would be happening during worker process pool expansion).

IDEs can go insane with process executions. VisualStudio shells out like crazy when doing builds.

Also, processes are an awesome abstraction - they give security and reliability. Reducing their cost is nice.

> What kind of workloads spawn so many processes that saving microseconds becomes relevant?

With a huge process, you have a timeframe between the child is spawned and it executes exec*() where you typically "do stuff" (such as closing a lot of fd)

During this timeframe the parent process has its universe COW'ed, and each write will trigger a page fault.

The performance impact can be concerning in the _parent_ process.

For example rust doctests consist of compiling each little doc snippet and executing it ensure that the asserts are true. That's two processes (rustc + the compiled piece of code) per test. This is an order of magnitude slower than multi-threaded in-process tests. Process startup and teardown is a significant fraction of that overhead.

Now fork+exec isn't the biggest chunk of that. But it fingers the memory subsystem. On many-core systems this leads to lock contention. So the cheaper starting a new process gets in terms of virtual memory operations the less lock-contention you get overall which also speeds up other parts.

> I think a fair comparison would be against a new sys call that does fork_exec vs io_uring_spawn, and not fork + exec vs io_uring_spawn.

I agree that's a useful comparison, although the status quo is also worth benchmarking. A fork_exec syscall sounds like a posix_spawn syscall. IIRC, glibc's current posix_spawn is a library function that does a vfork followed by setup operations that portable code isn't allowed to call after vfork.

I think there are two dimensions in which io_uring can be used to amortize/eliminate syscalls:

* sequence of operations: in this case, (v)fork, then setup calls, then exec. This sequence is unusually expensive because doing them separately requires extra page table work (particularly for the fork case, but even for vfork vs having a blank process). A posix_spawn syscall could achieve similar efficiency gains as this io_uring_spawn.

* batches of operations: in this case, lots of processes to spawn. It sounds like he hasn't posted numbers on that and intends to: "He's looking forward to ... supporting a pre-spawned process pool". If that turns out well, then the io_uring approach seems worth doing, and also can be used to implement a new in-user-space posix_spawn, for less total syscall surface area than doing both io_uring and a synchronous posix_spawn syscall. Maybe the supported setup operations also can be in common with io_uring operations for other purposes. I'm curious about that but don't see a link to benchmark code / implementation code / manpage.

It looked to me like providing a cleaner interface which still effectively supports everything vfork does, with all the performance gains (and fewer context switches) was in fact the point of the exercise.

That's absolutely where much of the performance currently comes from, and I did in fact benchmark exactly that and it provides comparable performance (for now). I also noted in my talk that you can get much of the way there by doing clone with CLONE_VM and doing exec from there.

But also, the tricks to make it faster without using io_uring are much less safe than using io_uring. vfork is dangerous.

io_uring is the standard kernel mechanism for doing multiple operations in one syscall, so it seems like the most logical fit for this.

Jens Axboe (io_uring maintainer) and I have many more plans to make this faster.

Leaving that aside, there are many additional capabilities this unlocks. For instance, we can maintain a pool of processes set up and ready to exec.

Exactly. I started out aiming for performance, and quickly realized that the benefits to correctness were even higher.

If you're doing something where process launch speed is extremely important you're hopefully already not using shared libraries.

Working on that too. One of the wonderful things about Plumbers is that so many different people working on infrastructure are there. I had a great conversation with Carlos O'Donell (glibc maintainer) about using io_uring in glibc, including in ld.so for that exact reason...

Every large process using vfork/posix_spawn/io_uring_spawn instead of fork would eliminate one reason to keep overcommit enabled.

I think there are at least some niche cases in which folks would still want it on. For example, some processes may rely on anonymous mappings being cheap until first write via Linux's "empty_zero_page". IIRC, they once tried to remove this concept but put it back in. Some context here: https://lwn.net/Articles/340370/