Hacker News Reader

Comment by Orphis | Hacker News Reader

I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). But in an application whose purpose is to be FAST, I find it weird they haven't bothered replacing it with another more performant one.

mallocng is bad at dealing with contention during multithreading. I've had applications that usually were I/O bound suddenly become "malloc" bound when building with musl in multithreaded scenarios (and only just 8 threads). Switching to mimalloc improved performance by 20x, very close to what glibc offers by default, and just a bit under a glibc + mimalloc configuration.

I get that there's a real issue there and it's interesting (to some) to address it, but it should have never surfaced this way in the first place.

View on HN

> I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient).

ripgrep actually sets jemalloc as global allocator when built for 64b musl: https://github.com/BurntSushi/ripgrep/blob/435f59fc4b43af3ab...

That's good! But I wonder why it wasn't then enabled for that configuration or why the override wasn't "global" enough and the default allocator was still partly used.

Rust doesn't get to override musl internals. Ripgrep uses opendir, a POSIX library feature implemented in musl to look at er, directories. The stack trace suggests we blew up when Rust's std::sys::fs::unix::readdir internal detail called opendir, and it in turn allocated.

On Linux it would be possible for ripgrep to talk directly to the kernel via documented system calls without libc, but that wouldn't work on any other popular OS.

I assume it already doesn't work on Windows. At some point you have to define your compatibility boundary. And high performance often coincides with mediocre compatibility.

A good example is go. On linux you can use a from scratch image fairly easily because it only uses syscalls. But for windows or mac the moving target wasn't maintainable so they link against shared objects. Linus enforcing the don't break userspace rule is what made that possible. That definitely has tradeoffs. At some point relibc or something similar will allow the same (stably) for rust. But using posix as that compatibility boundary gets you a much larger set of OS and only occasionally has a performance penalty. Often, a posix api tuned to the kernel is more performant. Musl is an exception precisely because it makes an openbsd-esque trade of performance for simple small attack surface.

The Linux kernel defines syscalls as its stable ABI (note: there are also non-kernel ABIs on Linux, such as Wayland) while Windows defines the DLL calls as its stable ABI.

One isn't better than the other. Linux's approach allows binaries to be fully statically linked, which is a more predictable environment for binaries, but Windows's approach composes better, as every process loads DLLs and this allows for things like graphics drivers and COM to work more reliably. As things stand on Linux you can't use the GPU in a portable statically linked app, because the kernel doesn't define the semantics of dynamic linking.

TBF the GPU thing makes a fair amount of sense if you dig into it. It arguably falls entirely outside the kernel's domain of responsibility.

However given how fundamental GPU acceleration is (as well as various other pieces of dedicated hardware in various scenarios) it would be nice if the kernel defined some basic portable semantics for linking with important drivers. These could exist independently of libc and the rest of userspace purely as an optional fallback.

... or we could all just include a glibc compatible dlopen routine in our statically linked binaries instead of worrying about pedantic hypotheticals.

Yeah we could all just use glibc. You see the problem with that though right?

That isn't at all what I said. I'm pointing out that in reality there aren't 20 competing libc implementations each with a unique dynamic linking scheme used by the userspace of 20 corresponding linux distributions. Instead we have ... glibc. So if the kernel actually specified a common portable linking scheme to fall back on as a means to interface with critical drivers it would presumably adopt the glibc way of doing things for that purpose.

Also I did not say "just use glibc" but rather suggested to just include a secondary glibc compatible dlopen routine. Exactly as you would have to do if the kernel bothered to specify.

If the kernel wanted to force you to use glibc it would do so. We could get rid of the stupid hardcoded paths in PT_INTERP for example.

Right. Obviously. What I'm pointing out is that if the kernel decided to include dynamically linking to certain drivers within its scope (which it does not wish to do at present as you rightly point out) then it would need to specify a scheme for doing so. At which point glibc is all but a foregone conclusion.

So just pretend they did that and include a second dlopen routine and get on with life because at that point you're compatible with ~all major linux distros.

Far more pressing is a blessed filesystem path for where to find the things.

FYI, Windows's approach is better in some respects, like letting you avoid syscall overhead in some cases.

Notably, Linux has this too, but it's a special mechanism (vDSO) while on Windows it's just a normal function call with no difference from any other function.

Or in other words, they agreed the Windows approach was better, and tried to retrofit it (;

You can call into the kernel just fine in Windows. The fact you use a function call wrapper instead of a raw syscall is not really relevant.

Most Windows NT kernel functions take a buffer to use rather than allocating their own memory. Even many Win32 functions don't allocate (although there you have to be much more careful).

> Rust doesn't get to override musl internals.

Is malloc not a weak symbol in musl? I would expect it to be overridable like it is with glibc. Or does ripgrep only override Rust's global allocator?

> Or does ripgrep only override Rust's global allocator?

That one, as you can see in the code linked above, it only does rust-level overriding as that was where musl's allocator was found to impact ripgrep (https://github.com/BurntSushi/ripgrep/commit/03bf37ff4a29361...)

Last time I checked (a while ago), it wasn't. You needed to compile musl and exclude a few files from the archive to then provide your own allocator.

Depending on the toolchain, I'm assuming you could use some tricks or hacks to make it work better, but the tooling I used (Bazel) did that automatically and I haven't bothered to look at the internals yet .

If you look at the sigsegv stack, the allocation comes from opendir which is in musl libc as well. The allocation override mechanism used in Rust doesn’t replace the allocator process-wide; it merely replaces the allocator Rust code talks to.

It does seem like ripgrep should probably avoid using opendir from libc if it allocates using an allocator with a global lock though.

It’s a kernel bug. While I agree libc allocators suck for no good reason, it seems like this work of equally likely hit other application code including mimalloc and glibc.

Sure, but the problem isn't that the bug exists, is that it has surfaced in a performance application using musl and exerting code paths in a slow allocator.

mallocng should not be used at all.

it's a tradeoff; mallocng does have benefits as well -- for example it uses less than half the amount of RAM compared to mimalloc and glibc for certain Python workloads. While yes it is a bit slower, I prefer mallocng in many cases.

ripgrep is there to be FAST. Trading any speed to improve memory efficiency over a longer period of time for a process with a short life-time doesn't make sense in this case.

It's also a development tool. If your development machine is having RAM issues because it's doing a grep, you have bigger problems to solve.

As for other workloads that might use less RAM with mallocng compared to other performant ones, I'm curious to know about the magnitudes we're talking about. From 10MB to 20MB or from 100MB to 2GB? How was the speed of the program? Was there any multithreading involved?

For context, the Python workload was a general-purpose fileserver with file indexing and image thumbnailing, largely IO-bound. Switching from mallocng to mimalloc resulted in a slight speedup (150% of baseline), but over twice the memory usage, going from 250 to 670 MiB, primarily for thumbnailing. Some pathological cases (libvips calling imagemagick to decode heif) was a 10x multiplier.

There was multithreading, and yes, mallocng visibly became a bottleneck beyond 5 threads. However already at 3 threads there was diminishing returns for both allocators, so this was not an issue in this case.

The speed gain was hardly noticeable in practice since the program was already plenty fast, but the additional memory usage was a very real inconvenience.

Is that memory increase seen in peak usage or average/idle usage?

For file servers, I could be willing to give up 350 MB of memory temporarily for large-scale indexing and thumbnailing operations if it doesn't happen often and makes the file serving/browsing more responsive.

On the other hand, plenty of NAS/file server systems are going to have paltry CPU and memory. Running slower with a more modest memory foot print might be the only way to keep the system stable.

libvips shouldn't be calling imagemagick for heif decode, it has a nice one built in. Unless you were using a very old libvips!

I've found jemalloc works best for long running libvips processes, fwiw.

I imagine there is no free lunch in the allocator space, but a series of trade-offs. Pick your poison on implementation which is going to be sub-optimal for some subset of use cases.

Without a hugely compelling reason to switch, going with the default is reasonable.

Ah, no, the problem is that the bug exists.

Another, separate, problem, noticed by people investigating this problem, is that ripgrep on musl uses a serializing allocator in the hot path.

It's maybe a kernel bug.

The way most programs achieve being fast is by re-using allocations. You don't need a fast allocator if you don't allocate. Nothing of what ripgrep does inherently requires frequent allocations.

The ISO C definition of opendir() requires an allocation in practice because it returns a DIR* and it's bad practice for the library to arbitrarily limit how many of those an application has at a time.

Maybe a C library could preallocate several DIRs and only use the heap when those are exhausted, but this ripgrep use case (lots of threads running in parallel on a large tree) would still be likely to trigger that.

There is no ISO C definition of opendir(), that's a POSIX thing.

But yeah, those DIR objects which opendir() returns a pointer to are probably some kind of heap allocated. But we're talking about a system call involving the filesystem here. The time taken by the allocator is gonna be dwarfed by the time spent in the syscall even with the slowest allocator.

Ripgrep reads through every byte of most files and matches it against a regex. That is the tight inner loop where you want to avoid allocations.

Not that you even need to call the C functions. I don't think ripgrep would gain anything from it, but the syscall to read directory contents just needs a file descriptor.

I think ripgrep does avoid allocations in that innermost loop. That's why this report says all the crashes are associated with opendir() and its memory allocation: the opendir() call is outside of ripgrep's innermost loop but still runs often enough to trigger the race condition.

And that's why musl's allocator's performance isn't a huge concern for ripgrep

That's to be properly measured. Assuming that the allocator will not cause issues there is to be proven.

The application I recently improved by changing the allocator had a similar profile (a C++ include scanner) and thread parallel I/O functions had terrible performance originally with mallocng. Adding threads almost had negative value because of the contention.

There is a real tradeoff:

- The musl allocator is only slow with multi-threading.

- Almost all other allocators have trouble reclaiming memory when using multi-threading. This often results in multiples more RSS than single threaded or musl's allocator.

Agree with you on mimalloc. It can even be configured to be aggressive in memory reclaim at the cost of performance.

We use musl+mimalloc by default for our entire production operating systems: https://stagex.tools

Interesting. Not sure if you are aware, but your toolchain looks nearly identical to another linux distribution.. You might have luck looking for patches there if you ever need them.

https://chimera-linux.org/about/#alternative-userland

While we are building for a dramatically different threat model, we use quite a few of their llvm patches and are very thankful for their work, and alpine making llvm/musl a viable option for us.

this use of musl's mallocng actually lead to the discovery of a kernel bug, thanks to its hardening. without it, this might have gone unnoticed for months, silently corrupting memory in the meantime.

Reply on news.ycombinator.com