Scalable memory allocation using jemalloc | Hacker News Reader

Scalable memory allocation using jemalloc

facebook.com

125 pointsr11t11 comments

I'm impressed that there is still room to eke performance improvements out by fiddling with the base data structure, the original author has been working on refining it for quite a while*

* http://t-t-travails.blogspot.com/2008/07/treaps-versus-red-b..., http://t-t-travails.blogspot.com/2010/04/red-black-trees-rev...

40 points and no comments?

Just wanted to say it is great submissions like this that make me come to Hacker News...

Thank you for taking the time to write this up and post it here...

It is a long article and requires some time to digest. No comments is a good thing that implies people do take time to read. I am still in the progress of reading ...

It also shows people here are unwilling to comment from a position of ignorance; a smaller percentage of readers will be authoritative on such an advanced topic.

YMMV but for my Python web app executing mostly OO Python 2.5.x code, I got a 10% performance increase by using jemalloc compared to the malloc in RHEL 5. It's as simply as LD_PRELOAD=/path/to/libjemalloc.so -- memory usage is also better (millions of objects allocated than eventually released in a long running process ended up with smaller amount of memory used when using jemalloc).

This is kind of surprising given that CPython uses its own ad-hoc allocator (see http://docs.python.org/release/2.3.5/whatsnew/section-pymall...) which resorts to malloc() only for large chunks of memory and buffers... I wonder what's happening there.

It would be interesting to see a performance comparison with nedmalloc. It uses a trie instead a of redblack-tree for a faster information access.

http://www.nedprod.com/programs/portable/nedmalloc/

Some time ago I evaluated both of them, and they seemed the best available choices. But I never found a comparison study.

It is not a performance evaluation, but there is a comment from Niall (the author of nedmalloc) in Jason Evans' blog that might give you a hint.

http://t-t-travails.blogspot.com/2010/04/red-black-trees-rev...

I tried to include nedmalloc 1.0.5 in the benchmark, but it quite simply isn't polished enough for actual use. To start with I had to remove an invalid assertion in malloc() and implement posix_memalign(), only to find that it crashes within moments of application startup.

Why is this not the default malloc shipped with libc's around the world yet?

Because if there is one library that needs to be conservatively packaged, its libc. A critical flaw in libc pushed out around the world could cause nightmares too awful to contemplate. Our entire modern world literally depends upon it.

Reply on news.ycombinator.com