Patching until the COWs come home | Hacker News Reader

Patching until the COWs come home

lwn.net

76 pointsalex_hirner18 comments

The second half of the article is here: https://lwn.net/Articles/849876/

This reminds me of the Haskell language, which goes to great lengths to be "lazy", meaning it tries to avoid doing computations until the last moment.

Sometimes it is convenient (e.g. to work with infinite/very large lists), but I've heard that oftentimes it makes the performance very unpredictable, as a CPU spike arrives all of a sudden in the last moment.

Anyway, take this with a grain of salt, as I'm still beginning my journey with the language.

> I've heard that oftentimes it makes the performance very unpredictable, as a CPU spike arrives all of a sudden in the last moment.

In a sense, all of the computation occurs in the "last moment", since that's when we're 'forced' to make a decision (more precisely, computation occurs when (a) it's needed by the value defined as `main`; and (b) a computation is 'needed' when we've hit a branch point that depends on its result).

Even more confusing, once a computation has been 'forced', it runs backwards: it's the result which gets forced, in order to choose a branch; if the definition of that result involves branching, that will force other values so we can figure out which branch to take; if those involve branching then more values may get forced; and so on until we reach some branchless 'input'.

Solution: don't use copy-on-write?

That would kill performance of many machines. Some wouldn't even notice (for example hosts dedicated to a Go webserver), some would get absolutely trashed (PHP pre-fork would likely be an extreme case). Additionally your baseline memory usage for each application is a full in-memory copy of every linked library. Say hello to 10+MB hello-world. (edit: actually just 3MB on my machine) Also all stacks would have to get reserved and cleared ahead of time, so thread spawning time grows ~5x.

Basically unless you want to run single-process unikernels, our systems and software really isn't designed or ready for non-cow world.

It's not that bad. The real problem is fork(), which is just a bad syscall in general (note how the single most common desktop OS doesn't have any equivalent to fork and seems to get along just fine). You can still allow mmap in a CoW-less world (you'll have to prevent concurrent file modification, which--again--a certain common operating system does!). Memory reservation is completely orthogonal to CoW: lazy page table initialization isn't the same thing as copy-on-write.

Windows have PAGE_WRITECOPY and PAGE_EXECUTE_WRITECOPY.

Those are essential for efficient shared library loading.

This paper argues that fork+exec is the wrong primitive for launching new processes: https://www.microsoft.com/en-us/research/publication/a-fork-...

posix_spawn can take care of many common cases today.

Unfortunately, on Linux there is no posix_spawn syscall. It's implemented in glibc as a wrapper around clone + exec.

Yes, but it's a wrapper around clone(CLONE_VFORK | CLONE_VM) (aka vfork) + exec, which makes a big difference.

CLONE_VFORK means that the parent process is suspended until the child either exits or execs, and CLONE_VM means that no copy-on-write happens - the child has full read/write access to the original memory space until it execs (or exits). In the child, posix_spawn is careful to work within temporary memory so it doesn't overwrite anything that the parent process might care about, including global variables etc. Then once exec succeeds (or fails, and the process exits), posix_spawn in the parent cleans up that temporary memory.

This means that posix_spawn doesn't require any copy-on-write support from the kernel. Or in other words, there's no posix_spawn syscall and there shouldn't be because it's a library function, but there is a vfork syscall.

I generally agree with the comments in the research paper about clone + vfork, most applications seem to not use it at all (either directly or indirectly through posix_spawn) for those reasons.

You are correct that there is an awful lot of software that came out of the 90s that was explicitly written for forking/multiple processes (we're talking 25-30 years now) but that only reflects the type of commodity computers we had then - pre SMP, pre commercialized virtualization, pre-cloud, etc. Threads back then weren't in a useable state either.

Fast forward to today and we're finally starting to see the tide turn. I can go spin up a 384 thread instance on gcp right now. We have very popular languages like go and rust. In the case of Go they've made multi-threading easily accessible to many developers to the point that many people don't even think about it.

Yes, there's a lot of cruft out there but we can build for the future.

If you don't actually need shared anything but config (memory / fds / task pooling), pre-fork is still better than threading. 384 threads sitting on one fd table is going to bottleneck on open (and maybe close) and anything else that is a lock per process.

I'd disagree here.

You can create a thread pool that matches your pre-fork env as well and it will still be vastly more performant especially in those environments that require more memory - which is many applications today.

Always look at the hammer first. The solution is not use mmunmap() of course. But there are still security cases left

Reply on news.ycombinator.com