Disable transparent hugepages

Given alc@ was an author on the paper (and the paper's FreeBSD 4.x implementation supported multiple superpage sizes), I'm not really sure why FreeBSD's pmap doesn't have support for 1GB page promotions.

It baffles me that THP became enabled by default (is it? I think it’s only a default on RHEL distros?). It really screws up many expectations that applications might assume about memory behavior (like the page size). In the majority of cases, THP is a bad, bad idea and anyone with perf or devops experience will agree with this I think.

Do you want to impose GC like pause characteristics to all processes on your box? And possibly double, triple, or 10x your memory usage? Enable THP then.

It's also important to measure in your actual use-case, and not just with benchmarks that seem "close enough"; I know it sounds odd, but I've seen others adjust settings and then prove that it worked with a benchmark that they claim is "representative", when in reality they didn't actually improve anything because that "representative benchmark" differed from the real use case in precisely the way that would not respond to the adjustment.

Blindly following "best practices" is bad enough, but "proving" that the changes work with crucially-different benchmarks is worse; and when it's some expensive consultant doing such things, I think it may even approach fraud.

That is exactly the reason I wrote the post! Those advice are based on specific use case, bug or outdated kernel. The jemalloc (Digital Ocean post) case is a good example, it just doesn't (didn't) know about THP https://github.com/jemalloc/jemalloc/issues/243

I can only repeat it: "Measure, measure and measure again!"

At the lower syscall level you move the BRK address, which is the highest memory address you're allowed to use. By default this is just after the statically initialized memory.

malloc() is just a library that manages this memory for you.

Linux has no idea if you will use the memory you just allocated, usually this happens dynamically; when you access a memory region for the first time, it is allocated in memory for real.

For example, if I map a bunch of pages of memory, use some, and then set MADV_DONTNEED on just a few of those pages afterwords (so they can be given back to the system temporarily), this will only work if I know the entire page is unused. If a page magically gets 512x larger under my feet, it's possible it will "coalesce" with another page -- into a huge page -- that can't be given back, because some of the (now much larger) page is still needed. This is the case of what happens with `jemalloc` + `redis`, where what looks like a memory leak is actually a failure to give back pages to the operating system, because small pages coalesce into huge ones automatically, defeating MADV_DONTNEED.

Whether or not malloc(2) uses THP "under the hood" in this case is more an implementation decision (there are many malloc variants), but it just punts the problem down the road. Ultimately it manifests as a violation of the "unspoken protocol" between the kernel and application when managing memory mappings.

All that said, there's some posts elsewhere in this thread pointing out that FreeBSD's Huge Page implementation avoids some of these deficiencies (possibly as a trade for some of its own), so all that said -- there's still room for better implementations, clearly: https://www.cs.rice.edu/~druschel/publications/superpages.pd...

The issue happens on specific workloads (databases, hadoop etc) and (this is often not mentioned) after the system is running uninterrupted for a quite of while. The slow down comes due that the workloads mentioned cause memory to be fragmented and when kernel tries to defragment the memory (unsuccessfully) on each allocation.

Since the workload you mentioned looks like it is for a workstation that won't be running a database 24/7 over months/years, you are very unlikely to run into it.