Hacker News Reader

Comment by zX41ZdbW | Hacker News Reader

I think the right way to avoid this problem is to avoid using ABI at runtime or build time.

At runtime, it means - don't use shared libraries. At build time, it means - build every library from the source, don't use pre-built artifacts.

This sounds controversial... But it allows you to change compiler or compiler options at any time, and you don't have to bother. It also enables cross-compilation, reproducible builds, and portable binaries. You no longer have to ask developers to set up a complex build environment on a specific Linux distribution because it works everywhere.

I use this approach for ClickHouse.

View on HN

This can work only if you own the entire codebase and have all external dependencies that you depend on statically link (compiled) within your product.

I also very much prefer this way of handling dependencies but it's not a solution for all ABI problems since it also implies that you will need to statically link (compile) against all the transient dependencies. These are including at very minimum libc++ or libstdc++. And with this requirement in place this already isn't possible for many of the codebases out there.

And it also brings another issue at the table: X version of libc++/libstdc++ depends on Y version of libc.

Since you generally cannot statically link against the libc, and you don't own it since it's part of the OS, this becomes a hairy problem. You really need to make sure that your code works across different versions and thereof combinations of libc++/libstdc++/libc.

And then there's ... a bunch of other different platforms which aren't Linux.

Glibc is obstructive to static linking but musl is not. That gives you a binary that relies on the Linux syscall interface and nothing else. I believe bsd's libc statically links without problems as well.

Libc++ is set up for static linking out of the box (if you manage to find or guess the many cmake flags).

OSX and Windows insist on libc iirc but they're closed systems anyway so controlling your dependency graph is unavailable.

The problem with musl is its malloc implementation. It is subpar when compared to glibc malloc. And we know that glibc malloc is generally subpar when compared to jemalloc. For large-scale applications with many (concurrent) (small) allocations this will be a big performance problem.

I guess (?) it would be possible to fix this by replacing the musl inline malloc implementation by statically linking against the jemalloc.

That's a good point, haven't looked too deep as I avoid allocations but musl+jemalloc is workable in Rust at least.

Changing malloc for a different one you like more would work fine in musl. In particular because it has a sane bootstrap structure which means the dynamic loader can call normal C functions, it'll be swap out the code and carry on as normal. I'd suggest replacing it in libc itself instead of requiring a separate library though.

Even then, you still need ABI consistency between compilers if you want to link together codebases written in different languages (e.g. C and Rust).

In practice this almost always 'just works' because most cross-language calls simply don't use the kinds of complicated types discussed in the blog post. They tend to stick to simple integer and pointer types, where ABI consistency is usually a given.

Though you can still get into trouble when passing function pointers, especially when combined with some modern control-flow integrity systems.

>Even then, you still need ABI consistency between compilers if you want to link together codebases written in different languages (e.g. C and Rust).

Let's talk over http, queue or other IPC-ish way

You can't pass values in registers using this model

You _still_ need a consistent way to talk about values; IPC systems tackle the same problems under the name marshalling & de/serialisation. They just tend to take much more conservative options to deal with exactly this kind of problem (you don't have to care about integer endian-ness if integers are expressed as strings).

Okay, how do you propose to talk to your kernel then?

Over an spsc queue (unfortunately you cannot mmap this way yet, and you cannot set up the spsc queue itself this way)

Io_uring?

Your kernel will likely have well defined interfaces. You don't need libraries to talk to the kernel.

But how can you use those interfaces without an ABI?

Fundamentally, an ABI is the way you define interfaces.

Except for Linux, those well-defined interfaces sit behind a C API.

Not really - Linux syscalls are stable, so you are free to run your binary with a statically compiled libc and never touch the installed one. You can also handcraft your syscalls in assembly.

This will not work on Windows, where the kernel API is a DLL and syscall numbes are routinely changed.

That's what I said - on Linux the syscall API is stable while on all other OSes you have to go through libc to talk to the kernel.

Who wants a kernel? Distribute a bootable unikernel image that can be talked to via gRPC or something.

Obviously there are plenty of things you can't build that way (e.g. drivers), but for a server application that's intended to be accessed over the network anyway, like Clickhouse, I'm increasingly thinking that's the way to go.

Tell me you never worked in a big codebase without telling me you never worked in a big codebase.

Reply on news.ycombinator.com