dalias

5kloc is about 10x larger than musl's existing (old) malloc in source lines. I suspect lots of that is low code density, comments, etc.

I have to lookup what exactly mimalloc is/does every time someone mentions it, because the readme/documentation isn't very descriptive except discussing extensions outside the normal API. I didn't have time to dig through this again today. But I did look at it in some depth on several occasions in the past and it really wasn't suitable for or comparable to what we're doing in musl.

We'd be happy to address specific problems on the mailing list. I believe it's a known issue that the Rust compiler is making really heavy use of rapid allocation/freeing cycles, and would benefit from linking a performance-oriented malloc replacement. Doing so is inherently a tradeoff between many factors including performance, memory overhead, safety against erroneous usage by programs, etc.

One statement in your post, which some readers pointed out was apparently added later, "Others have suggested that the performance problems in musl go deeper than that and that there are fundamental issues with threading in musl, potentially making it unsuitable for my use case," seems wrong unless they just meant that the malloc implementation is not thread-caching/thread-local-arena-based. The threads implementation in musl is the only one I'm aware of that doesn't still have significant bugs in some of the synchronization primitives or in cancellation. It's missing a few optional and somewhat obscure features like priority-ceiling mutexes, and Linux doesn't even admit a fully correct implementation in some regards like interaction of thread priorities with some synchronization primitives, but all the basic functionality is there and was written with extreme attention to correctness, and musl aims to be a very good choice in situations where this matters.

The justifications are partly the same as what Daniel Micay has written extensively on in the rational for hardened_malloc (https://github.com/GrapheneOS/hardened_malloc) - unsynchronized per-thread state inherently sacrifices global consistency for performance and makes it impossible to detect a lot of types of memory usage errors (DF/UAF, etc) that could otherwise be caught.

However musl has the additional constraint of being compatible with small/very-low-memory environments. Lack of global consistency inherently means you will end up using memory less efficiently and requesting significantly more from the system. The new malloc about to go upstream in musl is, to my knowledge, the first/only advanced hardened allocator using slab-type design rather than traditional dlmalloc type split/merge, but also designed for extremely low overhead/waste at low to moderate usage rather than extreme performance. And in the vast majority of applications, this is perfectly reasonable. Even Firefox for example does very well with it.

With that said, new malloc is expected to be somewhat faster than old on lots of workloads (and considerably faster than old would be if we fixed the flaws in old that motivated it), but it's not a performance-oriented allocator. If you really want/need that you should probably link jemalloc or similar (and accept all the tradeoffs that come with that). In Rust programs without "unsafe", it may make sense to do that by default.

There is no excuse for shell escaping bugs like this. 100% safe and reliable shell escaping is trivial:

s/'/'\''/g

s/^/'/

s/$/'/

Nice to see firm getting noticed. It's a really promising project.

To clarify, the numbers in my tweets are measured on Linux, comparing musl libc's posix_spawn (with CLONE_VM) to plain fork+exec (which should be independent of libc). I just posted the test program on our mailing list: http://www.openwall.com/lists/musl/2015/06/04/1

The biggest problem with Cygwin is that programs linked with Cygwin inherit global state from from a Cygwin installation on the system they're running on. If you want to produce a Windows program that just runs on any system you install it on using Cygwin, it will work right for most users, but if a power user who has Cygwin installed and has their own custom mounts, options (like different binary/text mode settings), etc. tries to run your program, it might break spectacularly. This makes Cygwin a really poor choice for making binaries you want to distribute as standalone programs.

Aside from that, Cygwin tries to hard to be a complete Unix environment on Windows, whereas midipix just gives you enough to use interfaces that were standardized in POSIX as a reasonable, uniform API for all operating systems to provide. Some functions go beyond that, but you don't have to use them. And even some things that are mandatory in POSIX are optional in midipix; as I understand it, you can choose at build time whether you want the overhead of being able to support tty devices (and the associated semantics like job control, signals from the controlling tty, etc.).

These are some good observations. Note that on modern POSIX, calling anything but async-signal-safe functions after forking in a multi-threaded process results in undefined behavior. I think it's totally reasonable to consider any programs using the WinAPI (not just POSIX functions provided through midipix) as being formally multi-threaded, and to consider the WinAPI non-AS-safe. I'm not 100% sure what midipix is doing in this regard, but my advice on the project (I'm the primary author/maintainer of musl libc, which midipix is using) has been not to worry about making arbitrary functions work after fork, but only supporting the things that POSIX requires to work. My view in general is that fork should be phased out, but posix_spawn is not sufficiently powerful yet to replace all uses of fork+exec -- it can't do advanced uid changes, setsid, resource limits, etc., and posix_spawn can't be used effectively from an async signal context because the API (attributes and file actions) inherently involves allocation.

No, fork is only one path that can lead to overcommit. Allocation of new memory as COW references to a zero page, and COW writable MAP_PRIVATE mappings of files (such as the writable LOAD segments of any executable or library file) also lead to overcommit unless you do proper commit accounting. Any system that does not need to do detailed commit accounting to avoid overcommit is basically wasting the fact that it has virtual memory/MMU.

No, OOM killer is completely orthogonal to this and is a consequence of not doing correct commit accounting. With strict commit accounting turned on (vm.overcommit_memory=2) fork will correctly fail when there is not sufficient physical backing. But you're correct that fork+exec is a bad model.

Yes, this is the really big deal that everyone focused on fork and mmap semantics and other details is overlooking. Having midipix as the means of producing Windows versions of cross-platform software like Firefox should make it possible to remove a lot of ugly #ifdeffery and/or whole "portability layers" that are avoiding the standard functions like fopen because of a lack of Unicode support on Windows.

Interix is a lot different because it requires installing a system component, which requires administrator privileges. Midipix produces applications that (at least as I understand it) run on basically any NT-based Windows with no special privileges.

I have a sequence of tweets here summarizing basic measurements I did on musl's posix_spawn versus fork+exec:

https://twitter.com/RichFelker/status/602313644026761216

There are still plenty of applications that use fork semantically, to keep the same process image in the child, which benefit somewhat from a fast fork. But most places where fork affects performance now are things that are already pessimized by using fork+exec instead of posix_spawn: the shell, make, cgi, etc. (GCC still uses vfork, but GNU make recently switched from vfork to fork because of vfork-related bugs.) Regardless of how fast or slow fork is on midipix (but I expect it to be fairly fast, much faster than cygwin), they would benefit a lot more from just switching to using posix_spawn.

The choice isn't between 30sec or 15sec boot times. It's between 30sec, 15sec, or 1sec boot times, the latter coming from dropping all of the crap and writing a flat linear /etc/rc file.

If your boot time was bad before and decent now, this is not thanks to the goodness of systemd but rather the badness of whatever hideous system your distro was using before, and/or because your distro is starting a bunch of useless junk that shouldn't be running in the first place. I've been using flat /etc/rc (all commands in one file, & at the end of anything non-essential) for years (decades?) now, and have always had a login prompt faster than the display can synchronize to the video mode change.

Both forking and double-forking are completely wrong behavior for supervised daemons run as part of any automated system. Forking makes sense for small systems which lack any automated supervision and where a human admin is responsible for all process starting and stopping, but it's a bug anywhere else. Modern Linux has various methods for working around this bug (like the method of assigning an alternate process for orphaned descendants to be reparented to, which systemd uses, AFAIK), but really any production-quality daemon should provide a non-forking way to start it.

There are plenty of viable alternatives: s6, runit, OpenRC, and so on. I'm not really convinced uselessd is a viable alternative - it keeps way too much of the badness of systemd, but I guess that makes it viable if you were willing to consider systemd to begin with.

A much better solution for the problem of user-facing applications (e.g. "desktop environment" software) depending on systemd's public dbus interfaces is to provide a fake service that gives them fake data - the same way you would sandbox Android apps for privacy by giving them a fake Contacts list, etc.

As for the other main "public interface" of systemd that things are starting to depend on, the systemd service file format, it would be easy to add support for this file format to any other process supervision system.

People actually _want_ the logging behavior of systemd? My impression is that it's the most widely hated part; I've heard endless stories of journald thrashing the filesystem forever, losing logs completely on corruption, etc. And even operating properly, its performance is comparable to grepping a flat text log, since despite having a "more efficient" format, it increased the actual data size by something like 4-10x.

Logs are essentially write-once, write-often, read-rarely data. As such, the optimal format is always going to be a flat, append-only file.

Systems which do not use systemd simply do not have these problems because there is no analogous component. If syslogd goes down, the worst that happens is you don't get logs. Init doesn't go down because it essentially has no inputs. Individual services can go down if they're poorly written, but they won't bring the system down with them. Traditional systems (the hideousness that is "sysvinit") have plenty of other different problems (e.g. race conditions in process supervision), but deadlock or bringing down the whole system is not one of them.

With systemd on the other hand, all of the components under the systemd banner are tightly interconnected and communicating. In particular pid 1 has ongoing communication with multiple other components, and misbehavior from them can, both in theory and in practice, deadlock the whole system. In case you missed it, this is roughly what "monolithic architecture" means: even though the components are modular, they're designed for use in a tightly interwoven manner that's fragile. It's completely the opposite type of "monolithic" from the kernel, which has everything running in one address space, but with architectural modularity, where interdependency between components is kept fairly low.

Whether it's all in pid 1 or not is irrelevant. What matters is that it has a monolithic architecture, whereby breakage in any one part or their communication channels can bring down the whole system. This is not just a theoretical concern; it has REPEATEDLY happened.

Is there any proper information on this issue? The patches do not apply cleanly against the latest stable kernel (3.14.5) and there's no indication I can find as to what version they're intended to be applied to.

At the time the project was started, Ulrich Drepper was the maintainer of glibc and the official response to any bug report was "STFU". So, no.

Aside from that, while a more permissive license was not part of my original goal or vision for musl, it's something that could never have happened by working on improving glibc. Right now we're facing a situation where Linux has been fragmented into a "GNU/Linux" minority made up of hackers' desktop PCs and enterprise servers, and an overwhelming Android majority. The latter is using a grotesquely incompatible, poorly designed, non-standards-conforming libc called Bionic, and if you want to run existing C programs on Android, you have to add heaps of #ifdef hackery to them to make them compatible with Bionic (much like the #ifdef hackery needed to make C programs work on Windows).

musl can provide a real, standards-conforming libc for use on Android that's still light and MIT-licensed (important because Open Handset Alliance members have a contractual obligation to Google not to add copyleft code to the system they distribute). This means, in theory, we can do away with the whole #ifdef mess and just run existing, portable software on Android. In short, it has the potential to free FOSS projects from getting bogged down in the maintenance burden of supporting yet another gratuitously-incompatible system (Android) in order to remain relevant.

LD_PRELOAD alone wouldn't do it because glibc's dynamic linker is closely tied to glibc. In particular thread-local storage requires close cooperation between them. Likewise, musl needs its own dynamic linker.

What could be done in theory is replacing /lib/ld-linux.so.2 with a symlink to musl. One of the long-term goals for musl is for this to actually work, at least for many programs. But right now there are a lot of glibc-specific symbols that get pulled in magically by glibc's headers, even when the program at the source-level is 100% portable code, and we don't have coverage for all of these.

Fairly easily.

A relatively small number of syscalls are used as part of implementing other functions. Most if not all of these are made via macros that expand to inline syscalls; by replacing them with a call into your own function that implements the equivalent of the syscalls you need "on the metal" rather than calling into a kernel, most of the work is done.

Some syscalls are made directly from assembly source files because C is unable to represent the work that needs to be done around them (e.g. clone and vfork) or simply because their usage is arch-specific. These obviously use the trap-to-kernel instructions directly and cannot be re-routed as described above. Still the number of files is so small you could just rewrite the asm.

One future direction (albeit with lower priority than further functionality/quality improvements, unless somebody funds it) is making it easier to port to bare-metal and documenting the process.

musl actually has very little in the way of optimization, premature or otherwise. Most functions are written to be as simple and direct as possible. Often but not always this gives near-optimal size and speed too. The places where more complex approaches are used are _mostly_ situations where the naive solution would have corner cases that fail or corner cases that have pathologically bad performance and could be used as a DoS attack vector. The main exception is things like memcpy, memset, etc.; if they're slow, people are generally very unhappy.

BTW, one difficulty in libc is that it's hard to know what optimizations are "premature" because you can't envision everybody's usage cases without an enormous volume of experience with third-party code.

The original idea goes back to around 2005 and frustration at the ever-growing size of glibc and poor (slow) support for UTF-8. At the time I made a prototype and used it personally. In 2010 I relaunched the project with a goal of doing it a lot better and targeting user bases who might actually be looking for a new libc; later on this turned out to include a lot of people unhappy with GPL/LGPL, so we re-licensed from LGPLv2.1+ to MIT.

Some of these things are covered on the comparison page I did at http://www.etalabs.net/compare_libcs.html

In regards to fts.h, the version in glibc is not even usable because it only works in 32-bit off_t mode, and stat() randomly fails in this mode due to modern inode numbers being 64-bit. See the recent thread on the glibc mailing list: https://sourceware.org/ml/libc-alpha/2014-03/msg00408.html and the bug tracker issue linked.

At this point both musl and glibc only cover a very small portion of C11 additions. glibc has a GSoC project proposal (with 3+ students interested in it) to add C11 threads, which is one of the big missing areas. We've been holding off on doing so in musl in an interest of making sure we do it in a way that's ABI-compatible with glibc, so we'll probably add C11 threads in the near future too.

Hi all. I'm the original author and maintainer of musl, and I'm happy to answer any questions anyone might have about the project. Glad to see so much interest!

Yet, there are quite a few. See the wiki: http://wiki.musl-libc.org/wiki/Projects_using_musl

A few big names are in the process of adding musl-based variants or switching entirely to musl, but most of these are still experimental. Alpine hopes to finish switching over sometime this year. Aboriginal Linux too. Bedrock Linux is already using musl in their latest release, but due to the way their distro works, they're in the unique situation where they don't have to build a lot of packages against the system libc, so they have it much easier.

It started out that way, but these days we have several other developers who are pretty active. You can see from our stats on ohloh: https://www.ohloh.net/p/musl/contributors?query=&sort=commit...

Also, without the work of the entire community testing and reporting issues getting various packages to work with musl, it would have taken forever to get this far. Testers and bug reporters are seriously under-appreciated!