Hacker News Reader

Python uses its own allocator for small sizes (below 512 bytes if I remember correctly). I've had a similar problem to yours. See https://glandium.org/blog/?p=3698 and the followup https://glandium.org/blog/?p=3723 . I have a glibc bug open about the issue https://sourceware.org/bugzilla/show_bug.cgi?id=23416 .

View on HN

Wow, I've experienced an issue that's very similar to that one if not the same. Using malloc/free directly worked just fine on Windows and older versions of glibc but lead to runaway RSS on at least some newer versions. I ended up writing a small custom allocator on top of mmap to work around it. It's incredibly frustrating when the memory leak is in the system allocator and not your code.

We at Crunch.io run Python using jemalloc because we have found some really bad behaviour in the default glibc malloc when using Python with many threads.

Using jemalloc we were able to reduce our memory pressure over time dramatically.

Reply on news.ycombinator.com