Hacker News Reader

Comment by stefan_ | Hacker News Reader

The libc situation is pretty unfortunate. You can go with glibc for compatibility, but ideally you don't actually use the upstream version but copy the one from Debian that has crucial fixes that most of the world uses that they didn't bother to upstream because for a while there, glibc was maintained by the biggest asshole in all of software engineering.

On the other hand you have musl, and we can all appreciate its design goals, but then it is similarly maintained by people that believe they and their work are utterly infallible. Hence why to this date they have not made it possible to detect musl and would rather cite you a POSIX standards meeting from 1980 than fix an incompatibility everyone else handles differently. Just ctrl+F musl on http://landley.net/toybox/ for some insight.

The last time I felt like smashing the desk because of musl was when I realized you can't stacktrace a thread in a syscall because they refuse to annotate their 10 lines of assembler doing the syscall with the required CFI directives. They would rather wait for someone to write a 200 line AWK script that does it ([1]) than fix debugging on platforms like arm64.

1: https://github.com/bminor/musl/blob/master/tools/add-cfi.x86...

View on HN

The inability to detect musl is indeed frustrating.

One example of this problem is in detecting C11 thread support. This is a feature of the C library, not the compiler, but it's the compiler that sets C11 feature flags. For threads, you can only detect that they aren't supported by __STDC_NO_THREADS__. But of course since this can't be defined retroactively in old versions of musl, you can't use it to detect C11 thread support in musl. And since you can't detect musl at all, you can't even just assume the version of musl is recent enough and turn it on unconditionally.

The musl team seems to advocate guessing at supported features by detecting with test compilation (a.k.a. configure scripts.) They don't believe in depending on specific versions of musl because features can be backported or disabled. Okay fine, but then why not give us explicit means to detect it? Why not just define MUSL_C11_THREADS or something so we can detect it at compile-time instead of during a slow, fuzzy, non-portable configure step?

Does anyone know of a workaround for this? Does musl define any kind of feature macro that can be used to detect C11 thread support?

Funny, GCC/glibc had this same problem (except you could work around it by detecting the version).

I've never had a problem with C++ on musl/Alpine. ZeroTier ports fine to that target. In fact we use Alpine/musl images to build static binaries for ancient Linux distributions that lack new enough distro-native compilers.

Do you have a real system that uses threads if available, but can also work without them? What does the architecture look like?

there are other threading options available than C11.

BEAM, the Erlang virtual machine, could be compiled with or without threads until release 21 (from 2018).

Plenty of embedded libraries of optional thread support, though from what I've seen this option is explicitly enabled by setting a define specific to that library, and also passing in other needed info/function pointers since threading can vary so much on embedded systems.

There are many systems that I've worked on that do this in practice.

So firstly, a common use-case of threads is for improving performance. Using a job system is a good mechanism for this that doesn't strongly couple the architecture to the details of number of threads, or etc. Often this can be done with a fork-join model, where when you have a big chunk of work to do, you can create a bunch of jobs to do some work, then either wait on them immediately, or go off and do some other work while you wait, either approach works since it's possible to have a job system that will do work on the waiting thread.

At this point it's then pretty simple to either build in a fallback either directly into the job system, where you either do the work immediately upon launching it, or upon waiting on it. or it's trivial to detect presence of threads in the work code, and replace the job code with a simple for loop or something similar.

This is far from the only style of system I've seen that can work optionally with threads, but it's one I've seen working to good effect.

Depending on your compiler __has_include("threads.h") combined with a check for thrd_t or similar after should cover everything except the craziest cases.

Thanks, this looks like it will work.

This is just C being generally bad at any feature detection. This is why the "./configure" exists, and you check for the existence of headers and functions there.

Adding alternatives (perhaps not usable now for no one but the interested reader): LLVM decided to start building their own last year: https://llvm.org/docs/Proposals/LLVMLibC.html

Of course they did.

Looks like llvm-libc is being written in C++. Which is... odd.

https://github.com/llvm/llvm-project/tree/master/libc/src/st...

Why ist that odd? It makes lots of sense to manage resources via RAII for example. C++ has really good C interop too.

It forces you to include a C++ compiler in your toolchain, which might not be possible for older systems, etc.

Every system has a C++ compiler nowadays, and if they do not, you have plenty options to compile yourself one.

No they don't.

Do you have an example? Keep in mind that we are not talking about the host platform here, but the target platform.

Not a problem for LLVM since it's linker is always a cross-linker so if you're targetting a new (or only use the older platform as a compile target) then the initial cross-compile will be able to produce the binaries. If the older system can't support the LLVM toolchain memory wise or otherwise then it's not really a problem either.

https://lld.llvm.org/

You only need a C++ compiler to build llvm-libc itself. You do not need a C++ compiler to use it in your C program.

Why? libc is just an interface, you can implement it however you want.

Microsoft did the same thing for the Universal CRT.

https://devblogs.microsoft.com/cppblog/the-great-c-runtime-c...

This great news, thanks a lot!

That musl refuses to implement dlclose() says it all, really. Musl's sin isn't all that different from old glibc's, really: both forget that the system serves programs, not the other way around. The musl author doesn't believe programs should unload code. Therefore, you're not allowed to do it.

Musl also removed access to Linux functionality because the author disagrees with the kernel's design:

"linux's sched_* syscalls actually implement the TPS (thread scheduling) functionality, not the PS (process scheduling) functionality which the sched_* functions are supposed to have. omitting support for the PS option (and having the sched_* interfaces fail with ENOSYS rather than omitting them, since some broken software assumes they exist) seems to be the only conforming way to do this on linux."

LOL. Do you really want to depend on a libc whose author thinks this way? Who removes functionality that the kernel provides because in his opinion the kernel's design is wrong?

> The musl author doesn't believe programs should unload code. Therefore, you're not allowed to do it.

Where did you get this? A rationale is given at [1], and it's basically:

1) dlclose() being a no-op is allowed by POSIX.

2) It's common for unloading and reloading libraries to trigger subtle/latent bugs, so making dlclose() a no-op leads to more reliable operation in the common case.

3) Managing thread-local storage in combination with unloading libraries is a complex problem, and arguably requires libc to either weaken error handling or leak memory.

This is certainly consistent with an opinionated-standards-lawyer-curmudgeon view of the world, but it seems pretty far from just saying that you're not allowed to do it because you shouldn't.

[1] https://wiki.musl-libc.org/functional-differences-from-glibc...

Yet the bumblebee flies: other libc implementations successfully implement code unloading and no amount of pointing at the standard will make me okay with musl not implementing code unloading, especially when part of the rationale is that other people write buggy code. It's not the place of libc to break functionality because the author believes other programmers can't use that functionality correctly.

Which ones? Last time I looked that list couldn't have included glibc, which while nominally supporting it had several, long-standing thread race bugs. The nexus of dynamic library loading and threading is a hot-spot for bugs in glibc.[1]

Excluding an interface is better than providing a buggy one, IMO.

[1] What glibc needs to do is finally merge libpthread into libc proper. The amount of stub functions, conditional logic, mutex reimplementation, and other chicanery glibc implements in an attempt to support runtime loading of libpthread[2] makes the code extremely difficult to navigate and debug. Which IME goes a long way toward explaining why there are years of unfixed races in glibc.

[2] By runtime I mean loading libpthread as a dependency of a shared library where the main binary wasn't linked against libpthread.

> nominally supporting it had several, long-standing thread race bugs

Name one.

And yes, pthreads should just be part of libc. Separating them is artificial and awkward.

I'm lazy so it's not specific to dlclose and unloading, here's one for dlerror I submitted in 2015 and which is still open: https://sourceware.org/bugzilla/show_bug.cgi?id=18192

I also point in that report to three other similar races which I didn't have the time to properly document (because, seriously, they're so numerous and the backlog so long, it's hard not to be cynical).[1]

Look, I like glibc. I applaud their focus on ABI compatibility. glibc is burdened by tremendous technical debt, much of it blameless. And the project doesn't see the investment and attention it should. musl by contrast is a breath of fresh air in terms of correctness and simplicity, but they achieve that in part by taking a different approach than glibc. And that approach isn't necessarily better, it's just different, and they each have their costs.

Pick your poison.

[1] And I don't like drive-by complaints. If I submit a ticket for a project I try to back it with substantial evidence and suggested remedies, such as this Python2 + FIPS OpenSSL bug analysis I submitted last year: https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/183... This is especially important for projects like glibc which are drowning in open bug reports.

> I'm lazy so it's not specific to dlclose and unloading, here's one for dlerror I submitted in 2015 and which is still open: https://sourceware.org/bugzilla/show_bug.cgi?id=18192

Okay, so the only bug you've managed to come up with during a discussion of dlclose has nothing to do with dlclose. I am very much unconvinced that dlclose has fundamental correctness problems.

> musl by contrast is a breath of fresh air in terms of correctness and simplicity, but they achieve that in part by taking a different approach than glibc. And that approach isn't necessarily better, it's just different, and they each have their costs.

Yes, musl has a different approach. And this approach makes musl unsuitable for anything I might want to do. The maintainer is too opinionated and thinks he can dictate software architecture to libc users. The glibc maintainers don't think so, not as much.

Okay, so here are some bugs which bit me in real life and at $WORK, around pthread_key_create + dlopen/dlclose:

1. libgobject (and anything depends on it, i.e. anything GTK) uses pthread_key_create to allocate thread local storage slots, but since threads can't be joined during dlclose(), there are no safe timing when we can call corresponding pthread_key_delete. It leads to bug reports like this: https://gitlab.gnome.org/GNOME/glib/issues/1311, and ultimately, GTK developers said they explicitly does not support dlclose()-ing their library because they can't see how is it possible: https://bugzilla.gnome.org/show_bug.cgi?id=733065. They suggest people dlopen() their library with RTLD_NODELETE, and good luck on those who indirectly depends on libgobject.

2. On Android, libcxx (the C++ runtime lib) does the same thing, it intentionally leaks pthread_key if you unload it. What's worse is the amount of available pthread_key is limited to 100, so after 100 iterations of dlopen()/dlclose() your process would simply crash. And, one of our significant customers insisted our library must pass their "while true dlopen dlclose" test, because hey, this is supported by libc, right? Mind you, Android does not use glibc, it uses bionic.

These two instance make me believe that it is not possible to support true dlclose() because, well, the two most used libc implementation tried and failed.

Pretty much 2) is saying "you're not allowed to do it because you shouldn't"...

Actually, it seems pretty consistent with the original point that "the system serves programs, not the other way around". dlopen() being a no-op means the system is more reliable, so it sounds like the right choice.

Apple has also considered making dlclose() a no-op on non-macOS (and the presentation is 3 years old so it may have happened by now)

https://news.ycombinator.com/item?id=20229723

No, dlclose still unloads the image if the reference count reaches 0: https://github.com/apple-open-source-mirror/dyld/blob/f033f5...

> Who removes functionality that the kernel provides because in his opinion the kernel's design is wrong?

The functionality is still available via pthread sched functions as specified by POSIX, so I'd argue that functionality was not removed, just moved:

https://git.musl-libc.org/cgit/musl/commit/?id=1e21e78bf7a5

IMHO this makes sense for a libc that explicitly emphasizes POSIX-correctness. POSIX defines these functions, and it is against the spec to have process scheduling functions work as thread scheduling functions (which are a separate set of functions in POSIX).

edit: Just realized that this might prevent changing scheduling parameters of another process by PID, though.

The sched_* functions, as implemented by glibc, do not do what POSIX says they should. On linux, it is not possible to implement them in a way that is compliant with the standard, so musl does not implement them. Given the principles specified in this webpage, that seems like a pretty reasonable thing to do.

musl does expose the extent of functionality provided by the kernel through the corresponding pthread scheduling functions.

> Who removes functionality that the kernel provides because in his opinion the kernel's design is wrong?

The kernel doesn't provide the necessary functionality. The functionality that the kernel does provide is still available in musl.

I can understand that this is probably frustrating (I'm sure I would be frustrated if I had to deal with it), but I can respect musl's decision not to have sched_* do the wrong thing.

> I can respect musl's decision not to have sched_* do the wrong thing.

Particularly because this actually ensures that programs linked to musl are 100% portable (modulo bugs), rather than introducing subtle platform incompatibilities due to slightly differing platform semantics.

You seem to either forget or not be aware in the first place that the purpose of libc and posix is to allow portability between different operating systems, so it makes complete sense that linux-specific capabilities are not exposed, it has nothing to do with disagreement about kernel design.

No, that is not the primary purpose of libc. The primary purpose of libc is to run programs that the user wants to run, and removing random things make that harder. Whether the standard allows something is irrelevant: what matters is whether any given change is good for the user.

This says absolutely nothing, for any given change you can find a user who benefits from it and a user who doesn't, hence a coordination device called a standard has been invented.

Except that the whole Linux ecosystem, starting with the kernel and going from there, simply doesn't really care about POSIX. Now, that Musl does is... well, would have been, long ago... a breath of fresh air. But the ship has sailed. The Linux kernel ABI has won.

Solaris, Illumos, the *BSDs, Windows -- all tried to emulate the Linux kernel ABI, and could not do a good enough job, so they all pretty much abandoned the project. The Linux kernel ABI won and that's that.

Well, if we could convince devs that static linking is bad... But even then, the Linux kernel ABI leaks for all sorts of things, starting with /proc.

And yes, static linking is bad, for a variety of reasons, the biggest of which is that static linking is stuck with 1970s semantics that suck because accidental symbol interposition happens all too easily. That doesn't mean that static linking couldn't grow ELF-like semantics, but if it hasn't over the past 40+ years... Still, even assuming that, there would be other reasons not to love static linking. If we could convince people to ship only dynamically-linked executables, then the ABI to emulate would be smaller, but still huge.

If the ecosystem doesn't care about POSIX it can hardly complain about choices made by a library whose purpose is to implement the POSIX standard.

It also has nothing to do with ABIs - the idea of a standard is to provide a common subset of functionality supported by all platforms that it targets.

Nothing about musl or any other libc implementation prevents developers from including the linux headers and using whatever linux-specific functionality they like.

> Solaris, Illumos, the *BSDs, Windows -- all tried to emulate the Linux kernel ABI, and could not do a good enough job, so they all pretty much abandoned the project. The Linux kernel ABI won and that's that.

I think the fact that they tried to emulate the ABI is a much stronger argument for the ABI than the fact that they abandoned doing so…

Well, it means they felt the need to support Linux executables on non-Linux kernels, and that the Linux kernel ABI turned out to be too big and too fast a moving target to manage the feat.

The Linux ABI is actually quite small and quite well-defined. Perhaps the most so out of all common kernel interfaces today.

It is most certainly not small.

Besides the system calls, the ioctls, and the fcntls, including all the various driver ioctls, there's also /proc. /proc is quite large an interface.

It is, but you can get quite far with emulating only parts of it.

Yes, but not enough to make a business of it.

> LOL. Do you really want to depend on a libc whose author thinks this way? Who removes functionality that the kernel provides because in his opinion the kernel's design is wrong?

musl can't remove the functionality provided by the kernel. You're still free to use syscall(3) or otherwise invoke the kernel routine yourself.

May I assume you abstained from using glibc prior to 2.30 on account of glibc not providing a wrapper for the gettid syscall? More apropos: do you dislike how glibc and musl both change the behavior of Linux setuid syscall to make it standards-conformant--process-global instead of thread-local? Should they simply provide the Linux semantics rather than the standard semantics? Has the kernel no culpability here for not providing the necessary interfaces for implementing the standard-compliant semantics?

> the chrt command is broken when built against musl-libc because that project's maintainer decided he didn't like the system calls it depends on, so he removed them¹ from his libc.

This software seems to be using Linux-specific functionality through the C library. Wouldn't it be better to use Linux system calls directly in that case?

I think it's weird how Linux has no user space library like the kernel DLLs found in Windows. Everyone assumes the C library is the interface to the kernel, even the Linux kernel manuals. Then people end up being burned when the C library maintainers quote POSIX standards at them or omit new system calls people want.

I wrote a liblinux² when I first realized this but it's currently unfinished. Eventually the kernel developers made an amazing nolibc.h file³ for their own tools but it doesn't seem to be used outside the kernel itself. Wish they'd publish it as an official Linux system call library!

¹ https://git.musl-libc.org/cgit/musl/commit/?id=1e21e78bf7a5

² https://github.com/matheusmoreira/liblinux

³ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Most operating systems define a libc ABI (or sometimes, merely API) as the interface to their kernel. Linux is the only OS--at least, the only major OS I'm aware of--that defines its interface boundary to be the system call layer itself.

I don't understand why they use libc for functionality that's specific to each kernel. A standard C library only makes sense for the POSIX stuff everyone follows. Even then people run into incompatibilities in practice because of implementation-defined or undefined behavior or bugs that have to be maintained.

In my opinion, Linux is the only operating system that did it right. The system call binary interface is clearly documented¹ at the processor architecture² level so it can be used from any language. Since everything in user space is replaceable, people could actually get rid of C and rewrite everything in Rust (or anything else) if they wanted to.

¹ http://man7.org/linux/man-pages/man2/syscalls.2.html

² http://man7.org/linux/man-pages/man2/syscall.2.html

> ... maintained by people that believe they and their work are utterly infallible. Hence why to this date they have not made it possible to detect musl ...

Yes, and they are every bit as fallible as anyone. Here is an example of a pretty pedestrian bug: https://github.com/build2/build2/issues/50

And that's a pity, really, since otherwise it looks like a reasonable implementation. But this dogmatic "we know better" attitude now triggers an allergic reaction.

I think we should cut the musl developers some slack; they're doing some really thankless work and giving it away.

Hypothetically, polishing turds and giving them away is also thankless work. Not saying musl is, don't know the first thing about it.

If it is, it's a turd people ask for.

Except that you don't have to do it if you don't want to.

This is what has always rubbed me the wrong way: (a) if it's given away for free, you do not have a right to complain about its quality, and (b) if you do complain, then why don't you go ahead and contribute. A gift horse, I know, but still...

If the quality is poor people have the right to say so, regardless of the cost.

If the projects existence is causing issues for some people, they have a right to complain, even if it is creating a free benefit for someone else.

Why would someone contribute to a project they think is inferior?

Basically, I don’t understand your objection.

I should have put (a) and (b) in quotes.

> ) if it's given away for free, you do not have a right to complain about its quality

What ? This does not make sense. Being free does not give any passes.

Considering the few lines that are mentioned as libc devs turning down feature requests, maybe they've tried.

I disagree, sometimes presence of something (even if free) can add more work for someone else.

For example imagine that you maintain a popular application, you start receiving issues that your application doesn't work with musl. You try to fix it, but musl doesn't provide a reliable way to detect itself. You try to submit a patch but it is rejected.

You absolutely have right to complain, lack of musl would prevent you from wasting time on making your application work with it.

> Just ctrl+F musl on http://landley.net/toybox/ for some insight.

Hahah wow that's some class A entertainment. I feel for you and yours!

> crucial fixes that most of the world uses that they didn't bother to upstream

If you are talking about eglibc, it's been merged back for years.

The specific anecdote I was thinking of is glibc not reloading resolv.conf automatically, making people think they have no internet connectivity for no good fucking reason for decades now.

Debian carries a patch since 2006: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=272265

Why is my Pidgin thinking I'm offline, 2007: https://developer.pidgin.im/ticket/2825

Firefox 2008: https://bugzilla.mozilla.org/show_bug.cgi?id=214538

Chef, 2015: https://github.com/chef/chef/issues/2894

Someone stumbled on this brokenness in Rust in 2017: https://github.com/rust-lang/rust/issues/41570

And then drums Wed Aug 2 19:12:20 2017 glibc 2.26 released with a patch. Maybe programs can rely on this behavior starting in 2027.

Are there other example than this one that's been fixed almost three years ago?

glibc's use of Intel TSX was also a bit of a tirefire, IMO.

It got turned on by default for all locks, broke quite a few of applications (which were using locks wrong, but still), then the instructions ended up being broken in every Intel architecture to date that has ever implemented them [1].

Most of the distros put in some patches to disable the use of TSX, but the 'no-lock-elision' flag provided by glibc didn't actually disable all attempts at lock elision. So the end result is that the glibc in Ubuntu 16.04 tries to use TSX for rwlocks, but not mutexes. Which is unfortunate, because the mechanism to opt-out of the use of TSX in glibc was only implemented for mutexes, but not rwlocks.

TSX only ever provided performance benefits in specific use cases, and at some point glibc upstream became sane and turned it all off by default and provided opt-in mechanisms instead.

[1]: skylake errata: "Using Intel TSX Instructions May Lead to Unpredictable System Behavior", https://www.intel.com/content/dam/www/public/us/en/documents...

TSX is supposed to fall back to an alternative implementation anyways if there's a transactional abort, right? What's the issue with keeping it in?

1. TSX has been broken in varying degrees in every architecture since haswell, and not always in a nice way like "always abort." In various steppings of haswell and early broadwell TSX was in various states of disablement, meaning that sometimes the instructions were advertised (in cpuid etc) to work but had subtle bugs, sometimes were advertised to work but were disabled such that any RTM instruction caused SIGILL, and sometimes correctly advertised to not work. The detection of the state of TSX was sufficiently unreliable that glibc ended up with a blacklist of CPU model numbers that were too problematic to even try using TSX.

2. The pthread implementation of mutex using TSX behaved differently from the prior implementation in cases of misuse. This was deemed acceptable because applications that were impacted were using mutexes illegally, but the result was that a ton of software in all the distros started crashing randomly (IIRC the scenario was a double unlock. pre-TSX glibc didn't complain, post-TSX the program crashed or something).

3. TSX was not generally beneficial. The glibc implementation did something like try using a transaction 3 or 5 times before giving up. The worst-case performance of starting the transaction, going as far into the critical section as possible, aborting, repeat 2x, then finally taking the lock normally is much worse than just taking the lock.

The reason you don't want to detect musl is the assumptions will break, say there is some incompatibility of some sort and the issue is brought up to the austin group which then clarifies the behavior which musl changes you now have a never ending number of versions you need to detect.

That might justify why you shouldn't detect musl, but it doesn't explain why you shouldn't detect features, a use case the musl devs are apparently also refusing to support.

> glibc was maintained by the biggest asshole in all of software engineering

You mean Roland? What did he do?

Must refer to Drepper.

>> glibc was maintained by the biggest asshole in all of software engineering > You mean Roland? What did he do?

Ulrich Drepper

Out of curiosity: What did Drepper do? A quick DDG/Google search didn't turn up anything.

EDIT: Actually, searching for "Ulrich Drepper asshole" did turn up some stuff. I'd still be interested, as the stuff I found was rather general.

This seems that it could be upsetting to some people https://udrepper.livejournal.com/7326.html

I'd never seen that one. Wow!!

Did you bother to read? It's about the combinatorial explosion that happens when you need to support lots of configurations. It's apolitical.

Yeah, that's one of the few things I've seen from Drepper that actually seemed reasonable and defensible.

Am I missing the part that makes it upsetting? I was expecting something full of cusses and ad-hominems.

You're going down an end-less rathole...

Ulrich made tons of bad technical decisions, and mistreated everyone who disagreed with him, was the reason Debian forked glibc for a long time, etc. There are some famous bugzillas that are utterly unexplainable.

Here's one notorious example of Drepper's behavior:

https://sourceware.org/bugzilla/show_bug.cgi?id=4980

(2007) "gethostbyname() etc break for /etc/hosts with both ::1 and 127.0.0.1 localhost entries" Initially closed by Drepper as "WONTFIX" or "NOTABUG" (can't find state history and it's been a while!) with dismissive comments, continuing to do so after people patiently explained how this was a real issue. It was eventually resolved (I don't know by whom)

Here's a taste:

https://sourceware.org/bugzilla/show_bug.cgi?id=4980

Reply on news.ycombinator.com