System CPU Usage and Glibc

Nothing in this article makes any sense. You simply cannot use MAP_SHARED by default because then it will wreak havoc the moment anyone uses fork() (for anything other than to immediately exec). And I fail to see absolutely any reason why MAP_SHARED vs MAP_PRIVATE would alter the performance characteristics of multi-threading within the same process, since altought the multiple threads are technically forks/clones, they will all share the same VM space irregardles of whether you used MAP_PRIVATE or MAP_SHARED. And neither MAP_SHARED nor MAP_PRIVATE will alter the way entries in the TLB are prepopulated or not, which is going to be at the choice of the processor itself (after at all, it is a _cache_).

So can someone please explain? Maybe what he actually wants is madvise or hugetlbfs ?

Several paragraphs don't make sense to me:

>Our biometric datasets are sized at 1 GiB for easier file management as we could turn on transparent huge pages, if required. So it is guaranteed that mmap is being used under the hood by std::vector, so why was there TLB misses in the first place? That’s because the MAP_PRIVATE flag instructs the kernel to turn off VM_SHARED flag when creating the mapping in the kernel. This causes TLB to be cold and populate it only on demand, via page faults. As long as this translation is small, there’s no overhead. However as the memory pressure increases with concurrency, the kernel has to fight hard to populate TLB.

There is MAP_POPULATE flag to populate the mapping eagerly without waiting for page faults. I'm not sure what MAP_SHARED has to do with that.

> That aside, the reasoning behind glibc allocator’s use of mmap with MAP_PRIVATE|MAP_ANONYMOUS as defaults, is privacy and anonymity. Which makes sense, because we do not want other processes to peak into the memory region of our process. That would be a security nightmare. But I’m not sure I agree on this for inter-thread design. Threads inherently share the same process space and thus the heap. MAP_PRIVATE gets a copy-on-write mapping for performance reasons (may be). MAP_ANONYMOUS will ensure the mapping is initialized to zero. So the performance optimization is wasted away. However this doesn’t make sense for user-space data structures std::vector or malloc for that matter (FWIW, malloc, although a function, has it’s own internal data structure under the hood). No one is going to throw away a memory region after allocation without writing something into it, after all, why else will they allocate in the first place? As the size of memory being requested is large, it makes sense to use MAP_SHARED|MAP_ANONYMOUS as the default in glibc.

CoW is for separate processes, not just for any tasks. A different thread in the same process writing to a page won't trigger a copy. You don't need to pass MAP_SHARED to mmap to share memory between threads, the arguments to the clone(2) syscall which created the thread already made sure that you share the memory.

What I really suspect happened is that they didn't call std::vector::reserve at the start, so there was a lot of overhead when resizing (allocating new memory, copying the content, freeing the old memory) of std::vector, and when they moved to calling mmap directly, they effectively reserved memory at the start, as would happen if they called std::vector::reserve. Unfortunately the post doesn't contain enough details to really be sure about that.

> This causes TLB to be cold and populate it only on demand, via page faults.

This article is confused. Page faults populate the page tables, not the TLB.

> No one is going to throw away a memory region after allocation without writing something into it, after all, why else will they allocate in the first place?

Hah! Just try running a server with vm.overcommit_memory=2.

FWIW, `MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE` would've done the same job.