Hacker News Reader

Comment by wmf | Hacker News Reader

This doesn't address the fact that forking large processes requires either overcommit or a lot of swap. That may be the source of the Redis problem.

View on HN

Author is just ignorant to the technicals and laser focusing on some particular cases that he thinks are problem but are not.

The redis engineers KNOW fork-to-save will at most result in few tens of MBs of extra memory used in vast majority of cases and benefit of seamless saving. Like, there is a theoretical where it uses double the memory but it would require all the data in database be replaced during short interval snapshot is saved and that's just unrealistic

Why? Most COWed pages will remain untouched. They only need to allocate when touched.

Because the point of forbidding overcommit is to ensure that the only time you can discover you're out of memory is when you make a syscall that tries (explicitly or implicitly) to allocate more memory. If you don't account the COW pages to both the parent and the child process, you have a situation where you can discover the out of memory condition when the process tries to dirty the RAM and there's no page available to do that with...

The described scenario (and, consequently, a concern) is mostly a philosophical question or a real concern for a very specific workload.

Memory allocation is a highly non-deterministic process which highly depends on the code path, and it is generally impossible to predict how the child will handle its own memory space – it can be little or it can be more (relatively to the parent), and it is usually somewhere in the middle. Most daemons, for example, consuming next to zero extra memory after forking.

The Ruby garbage collector «mark-and-sweep» (old versions of Ruby – 1.8 and 1.9) and Python reference counting (the Instagram case) bugs are prime examples of pathological cases when a child would walk over its data pages, making each dirty and causing a system collapse, but the bugs have been fixed or workarounds have been applied. Honourable mention goes to Redis in a situation when THP (transparent huge pages) are enabled.

No heuristics exist out there that would turn memory allocation into a deterministic process.

If you have overcommit on, that happens. But if you have it off, it has to assume the worst case, otherwise there can be a failure when someone writes to a page.

The point of disabling overcommit, as per the article, is that all pages in virtual memory must be backed by physical memory at all times. Therefore all virtual memory must reserve physical memory at the time of the fork call, even if the contents of the pages only get copied when they are touched.

Surely e.g. shared memory segments that are mapped by multiple processes are not double-counted? So it's only COW memory in particular that gets this treatment? Linux could just not do that.

But forking duplicates the process space and has to assume that a write might happen, so it has to defensively reserve enough for the new process if overcommit is off.

COW is not shared memory, it’s an optimised copy. There is no way to guarantee that the optimisation will hold forever thus it is a form of overcommit (and indeed the reason most unices overcommit in the first place): most children will not touch most of the virtual memory they inherited but any can, so if you require precise memory accounting you have to account for that in the same way you account for large anonymous maps.

Reply on news.ycombinator.com