Hacker News Reader

Comment by MichaelGG | Hacker News Reader

Isn't fork the real offender, which requires Linux to overcommit by default? Disabling swap shouldn't affect that, right? Just makes your problem happen later, in a somewhat non-deterministic way.

Without fork, what reason do you not disable swap? I can only think of an anonymous mmap where you want to use the OS VM as a cache system. But that's solved easily enough by providing a backing file, isn't it?

View on HN

> Isn't fork the real offender, which requires Linux to overcommit by default?

Maybe I'm missing something, but how does fork require overcommitment? When you fork, you end up with COW pages, which share underlying memory. They don't guarantee that physical memory would be available if every page were touched and required a copy; they just share underlying physical memory. Strictly speaking, very little allocation has to happen for a process fork to occur.

If there's no overcommit, each of those COW pages needs some way of making sure it can actually be written to. Isn't that literally the point of overcommit? Giving processes more memory than they can actually use on the assumption they probably won't use it? And Windows takes the different approach of never handing out memory unless it can be serviced (via RAM or pagefile).

What am I missing? (I know you know way more about this than I do.)

When you fork a process, your application's contract with the kernel is such: existing pages will be made accessible in both the parent and the child; these pages may or may not be shared between both sides -- if they are, then the first modification to a shared page will cause an attempt to allocate a copy for that process; execution flow will continue from the same point on both sides. That's pretty much the extent of it (ignoring parentage issues for the process tree). The key thing here is the 'attempt' part -- nothing is guaranteed. The kernel has never committed to giving you new pages, just the old ones.

I don't personally see this as an overcommit, since the contract for fork() on Linux doesn't ever guarantee memory in the way that you'd expect it to. But in all honesty, it's probably just a matter of terminology at the end of the day, since the behavior (write to memory -> process gets eaten) is effectively the same.

Edit: Though I should note, all of the overcommit-like behavior only happens if you are using COW pages. If you do an actual copy on fork, you can fail with ENOMEM and handle that just like a non-overcommitting alloc. So even in the non-pedantic case, fork() really doesn't require overcommit, it's just vastly less performant if you don't use COW.

Oh. I was under the impression that if overcommit was disabled then forking a large process won't work if there's not enough RAM/swap available, regardless of usage.

So out of memory failure won't happen when you malloc, it will happen when you assign a variable in a COW page. This somewhat invalidates the idea of a failing malloc.

The problem is once you touch those memory pages, total memory usage increases even if you don't call malloc.

Saying that fork forces overcommit is strange. Fork is just one of the things that allocates memory. If you don't want overcommit fork should simply fail with ENOMEM if there isn't enough memory to back a copy of all the writable memory in the process.

I meant the practical considerations of fork means overcommitment is needed in many cases where it otherwise wouldn't be needed. If you fork a 2GB process but the child only uses 1MB, you don't want to commit another 2GB for no reason.

> Isn't fork the real offender, which requires Linux to overcommit by default?

fork() != Linux.

Each UNIX system does it on its own way.

Reply on news.ycombinator.com