Why does musl make my Rust code so slow? (2020)
andygrove.io
andygrove.io
"When it comes to musl, there’s a certain performance trap: its allocator does not perform well in multi-core environments."
"A common cause of the above symptom is thread contention: multiple threads are fighting to acquire the same resource instead of doing useful work. Although it’s better than deadlock or livelock, since the application runs to completion, it’s still a significant waste of CPU."
"The real source of thread contention is in the malloc implementation of musl. A malloc implementation must be thread-safe, as multiple threads may allocate memory at once or even free memory allocated in other threads. Thus, the thread synchronization logic could become a bottleneck.
Although I have not deep-dived into the musl malloc codebase to locate the contention, replacing it with a cutting-edge malloc implementation like mimalloc is sufficient to rectify the problem and enhance performance. Mimalloc minimizes contention and enhances multi-threaded performance. In fact, this replacement can make the application run even faster than the default glibc build."
Classic HN.
'classic hn' in the negative sense is a one-line dismissal containing no reasoning or information. do better
do better.
so i don't think a cursory investigation supports your vague innuendo, which seems to be implying that adding thread-local allocation pools to musl would require enormously more than 'a few dozen lines of code', as i said. the entire allocator is 50 dozen lines
the code isn't 'involved'; it's nice and straightforward, except for a clever algorithm or two from hacker's delight. what's involved is debugging it when you break it
In most cases it's not even an optimization! Docker's layered caching means the most relevant size is the diff between the base image and the final image. This is generally independent of which base image you use.
It has a bunch of negative consequences like in this blog post.
It's something that many people new to Docker feel they should be doing, so you see it a lot.
Just an all-round great premature optimization.
During development, it really does matter if my images are 4MB or 80MB.
Just like when compilation times and tests start growing in the minutes, my feedback cycle is broken.
I also compile Rust statically to musl, so I'll seriously consider building the final image on the device to avoid copying the base image during deployment. Or ditch Docker entirely.
So just like when you change one c file you don't have to recompile the whole project, updating the last layer in the image should only require a few mb of transfer, the base layers would only be transfered once.
When you create a new Docker image with a recompiled Rust binary, only the last Docker layer will need to be downloaded to the device. Incremental caching means Docker won't re-download the base layer. The last layer is basically the new binary itself. So I don't see how it would be slower than copying the raw binary. But maybe I'm missing something.
Debian images can end up using a lot of apt-get tricks to install dependencies cleanly. Way easier with alpine. I think there are plenty of reasons people are quick to make this decision without the necessary depth of investigation.
Minimizing docker image size is great. Cold starts with large image downloads can cause all sorts of fun issues. Yes, if you're shipping lots of dependencies in your containers the size of the base image can quickly be irrelevant. And yes, I think it is usually reached for prematurely, but the benefits are there for the few that need them.
I don't think it is a premature optimization anymore, it has become the best practice by default. The network effect at work.
Unlike the early days of Docker, today you see popular distros provides multiple official image flavors depends on how minimal you are comfortable with. Or you can go even further with things like Google distroless, or even just scratch. I rarely see negative consequences when it's done appropriately. In fact, we see a lot of benefits by doing this. for instance, developers now understand their own app's dependencies much better ("oh your python app is having a cert error? does it come with its own cert bundle, use 3rd party package like certifi, ca-certificates from OS package manager, or a combination of some of them?" It's much easier to answer this question now than before)
Package only the stuff your app needs.
In ClickHouse, we use jemalloc as a memory allocator and custom memcpy: https://github.com/ClickHouse/ClickHouse/blob/master/base/gl...
So, the Musl build does not imply performance degradations. But the usage of Musl is not related to Docker, because ClickHouse is a single self-contained binary anyway, and it is easy to use without Docker.
In C++, if someone starts frantically searching for a faster general purpose allocator then it's usually a sign of putting each C++ object into its own heap allocation behind a smart pointer, creating and destroying tiny objects all the time, and then wondering why so much time is spent in the allocator.
I'm not familiar with musl's allocator so it's hard for me to say anything about it. However what is pretty common is that modern allocators use thread local allocators to avoid hammering global locks too much. Since Rust is very concurrent you don't need too many allocations to benefit from these optimizations.
Even if you program better for it, one of the prevailing opinions on “learning about lifetimes” is to “not learn lifetimes, just make copies”.
Are C++ allocators so bad that they can't handle this use pattern? Yes, you can make a bunch of manual effort to use the allocator less, but you shouldn't have to.
Why does musl make my Rust code so slow? - https://news.ycombinator.com/item?id=23080290 - May 2020 (72 comments)
They mention this script in the README on docker hub.
Frustratingly though, it feels like most apps built on Debian and debian-slim end up downloading a bunch of different layers. Often over a dozen. It just makes me so sad.
And these layers do seem to shift and churn. I keep telling Alpine lovers their small images dont matter because the base layers will almost always be resident. But the Debian images just feel like they are ever shifting.
Musl should be able to exist without providing any allocator.
and scudo is probably one of the few allocators that would work for this purpose, it absolutely cannot be applied to any allocator; there are some concerns that get in the way of that, e.g. the handling of ELF TLS in a libc (ELF TLS may not be and in musl's case is not available within the dynamic linker, which means you cannot use __thread and the likes anywhere within libc.so; scudo is designed to be flexible so you can get around it (we shove a pointer to the thread-local state directly in the pthread structure and manually mmap the data for it) but that is not necessarily the case with many other allocators, including mimalloc, hardened_malloc, and others)