Comment by cryptonector | Hacker News Reader

Comment by cryptonector | Hacker News Reader

If you want to be able to spawn processes that fask then `fork()` is NOT your friend. You want either `vfork()` (or `clone()` equivalent) or `posix_spawn()`.

`fork()` is inherently very slow due to the need to either copy the VM of the parent, or arrange to copy pages on write, or copy the resident set of the parent (then copy any pages paged-in when those page-in events happen -- all three of these options are very expensive.

Also, what I might recommend here is to create a `posix_spawn()`-like API that gives you asynchronous notification of the exec starting or failing, that way you don't block even for that. I'd use a `pipe()` that will be set to close on exec and which will therefore close when the exec starts, but if the exec fails I'd write the `errno` value into the pipe, that way EOF on the pipe implies the exec started while read on the pipe implies the exec failed and you can read the error number out of the pipe.

  https://gist.github.com/nicowilliams/a8a07b0fc75df05f684c23c18d7db234
  https://news.ycombinator.com/item?id=30502392
  https://github.com/famzah/popen-noshell

The stdlib already mostly does all of that :)

Check out https://kobzol.github.io/rust/2024/01/28/process-spawning-pe....

Copying the page table isn't free but it isn't particularly expensive either. At least unless the parent is a real behemoth.

Unless you enjoy footguns posix_spawn is probably a better idea than vfork. (Unless you actually need vfork of course.)

The async pipe idea sounds interesting but I'm not clear how it would work. It seems like you'd have to use vfork to implement it but vfork is blocking until you call exec so doesn't that defeat the purpose?

CoW is extremely expensive for threaded processes on multi-processor systems since you need TLB shootdowns.

> The async pipe idea sounds interesting but I'm not clear how it would work. It seems like you'd have to use vfork to implement it but vfork is blocking until you call exec so doesn't that defeat the purpose?

Yes, so have the `vfork()` happen in worker threads.

> Unless you enjoy footguns posix_spawn is probably a better idea than vfork. (Unless you actually need vfork of course.)

I have proof that `vfork()` can be used safely: the several `posix_spawn()` implementations that use it.

Good point about multiprocessor systems. He did say this is being used on HPC clusters.

You can also use C safely if you're careful. Doesn't mean it isn't full of footguns.

> have the `vfork()` happen in worker threads

Fair enough. Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than creating a pipe?

> Good point about multiprocessor systems. He did say this is being used on HPC clusters.

Plus essentially all systems now are multiprocessor systems.

> You can also use C safely if you're careful. Doesn't mean it isn't full of footguns.

You don't have to use much C here at all.

> Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than creating a pipe?

The point is to signal that `execve(...)` did not fail but did start the new program -- but, how? You can use a condition variable or whatever to signal failure to exec, but you can't run code after a successful exec that would signal a condition variable or whatever because the program that was calling exec is now not running. A close-on-exec pipe serves to asynchronously signal success to the parent: it only closes either when the exec succeeds or after the exec fails and you close it after signaling the failure however you like. This works because the "code that runs after the exec succeeds" here is not code in the new program but code in the kernel after the kernel commits to not returning from `execve()`!

What else has this property? Only file descriptors, or you can insist that the program you exec must signal some how that it started. But the latter is intrusive, while the former is not. If it's file descriptors then it has to be something that supports async I/O, and the simplest thing would be a pipe or a socketpair. Now if you're going to use a pipe for exec success reporting then you might as well also use it to signal failure because why have two mechanisms, one for reporting success and one for reporting failure?

You still have to deal with the process' eventual exit, and reporting that asynchronously along with the exit status. But you get to report on all of three distinct events:

  - exec failure (e.g., ENOENT)
  - exec start (really *exec didn't fail*)
  - child process exit/death

You could choose to not report exec start, I suppose. But you often want to know that exec didn't fail, and the only ways to know that are to either wait for the process to exit (which could be a long time!), assume that if enough time has passed then the exec did not fail (a lame heuristic), or just arrange to signal exec success. Since it is possible and easy to signal exec success via a close-on-exec pipe/socketpair then you might as well just do that.

Note my nomenclature here is parent (doesn't block), worker (calls vfork), and child (calls execve).

If you used vfork then the worker doesn't unblock until execve has executed successfully (AFAIU). In that case if the exec succeeded then nothing will have been written whereas if it failed then the child will have written out the error code to the worker's stack before calling _exit.

The worker can then use an atomic write to communicate the ultimate result back to the parent in an asynchronous manner.

Ah, yes, that's true. I'm so used to using pipes like this that I forgot. You're quite right. Though I'd still use something like a pipe rather than a condition variable, say, because on Unix and Linux systems you can't have an event loop handle events on file descriptors _and_ condition variables.

And those are my favorite signalling mechanisms: async I/O and condition variables.

I thought the optimization goal was latency on the parent thread so I had in mind a bare atomic_store_explicit with memory_order_relaxed. However you wouldn't care about latency in the first place if you weren't scaling this up and obviously scaling up requires additional infrastructure that I'm ignoring here.

That said the linked article is using spawn_blocking from tokio meaning you can just call vfork directly and return the result. No need to roll your own in this case.

> you can't have an event loop handle events on file descriptors _and_ condition variables

Since events crossing the kernel boundary aren't fast to begin with I prefer to use a secondary event loop for the sluggish stuff and feed the output to the lower latency one.

Reply on news.ycombinator.com