Manual Memory Management in Go using jemalloc

dgraph.io

38 points·chewxy··6 comments

Just wait until they figure out how to use arena/stack-based memory.

Actually, the implementation of arenas is broken in glibc malloc() when running on multicore systems.

I have been debugging that problem for some time. The charts in the article are quite similar to the ones I got when I initially came across the problem.

Wonder how many threads does their process have? and how many cores on the machine?

There are environment variables that can adjust the behavior of malloc. They should try setting MALLOC_ARENA_MAX=4 and see if the memory usage goes down.

That’s super interesting, I had no idea, but I just meant malloc’ing huge chunks of memory at a time and freeing it all at once.

> Over the years, we have tried all tricks of the trade within Go. Using sync.Pool, maintaining our own freelists, avoiding allocations on heap wherever possible, using buffer arenas and so on

I'm interested to know why pools/freelists didn't work out for nodes

If you’ve brave enough to try managing memory yourself, C++ has a whole library of collections that support custom allocators without the readability penalty for FFI. I wish they would have written why they didn’t use it.

IIRC Zig just requires an allocator as an argument to any stdlib function which can allocate.

Reply on news.ycombinator.com