Hacker News Reader

Comment by saidinesh5 | Hacker News Reader

The biggest impact suckless had on me was via. Their Stali Linux FAQ: https://sta.li/faq/ .

They've built an entirely statically linked user space for Linux . Until then i never questioned the default Linux "shared libraries for everything" approach and assumed that was the best way to deliver software.

Every little cli tool i wrote at work - i used to create distro packages for them or a tarball with a shell script that set LD_LIBRARY_PATH to find the correct version of the xml libraries etc i used.

It didn't have to be this way. Dealing with distro versioning headaches or the finnicky custom packaging of the libraries into that tar ball just to let the users run by 150 kb binary.

Since then I've mostly used static linking where i can. AppImages otherwise. I'm not developing core distro libraries. I'm just developing a tiny "app" my users need to use. I'm glad with newer languages like Go etc... static linking is the default.

Don't get me wrong. Dynamic linking definitely has it's place. But by default our software deployment doesn't need to be this complicated.

View on HN

The thing is, dynamic linking doesn't mean using LD_LIBRARY_PATH or building full blown OS packages as the only way to find the correct libraries. There's a first class facility for locating shared libraries, using the -R flag to provide a RUNPATH/RPATH in the binary. The runtime link editor will use that path to locate shared libraries. You can make your binaries relocatable as well, by using $ORIGIN in the RPATH: this gets expanded at runtime to the path of the executable, so, e.g., $ORIGIN/../lib would go up one from bin/ where the executable is and down alongside into the lib directory for your software.

LD_LIBRARY_PATH is a debugging and software engineering tool, and shouldn't ever be part of shipped software.

Build systems often make it a huge pain to get the right rpath. The chrpath tool makes it easy to fix the rpath after libtool got it wrong.

The suckless tools are recompile-to-reconfigure, so they don’t hit the length limitation of chrpath or fatal bugs in patchelf.

You wouldn't have a length limitation in using rpath just to express "find the libs near the executables, wherever those are".

If you know that the executables will be in a certain directory, such that ../lib relative to that is where the libraries are found, then you just make the rpath '$ORIGIN/../lib'. That's it; no guesswork.

Build systems that make it a huge paint to calculate elaborate rpath, e.g. knowing what the sysroot will translate to on the actual target system or whatever, are just doing counterproductive nonsense.

By the way, I seem to recall that Yocto uses chrpath in order to get its own tools to reference its own copy of glibc that it provides (for itself, in the build environment).

And the main advantage of doing all that work vs statically linking is? Don’t get me wrong - dynamically linking for dev builds makes a lot of sense to cut down on relink times. But I just don’t see it for distribution since doing that RPath work reduces the main argument for dynamic linking (i.e. the OS can patch the vulnerability for all installed packages without waiting for each to release).

They didn't claim it was better than static. That doesn't make it worse than static either. They are two different answers for two different problems.

It's simply that when you do want external but bundled neighboring libs, there is a good way to do it.

What advantage does sibling libs provide over static linking?

This becomes the same question as when and why are dynamic libraries useful over static linking!

Suppose whatever you are shipping consists of five executables, which have a few hundred kilobytes of their own code and a 5 megabyte library. If they statically absorb the library, they all become 5.5 megabyte binaries.

Adjust that to whatever sizes you would consider relevant.

You can make it clear when a bug is fixed in a library or one of the executables. If you have a patch update, users can see that only a shared library changed or only a certain executable.

When hunting a bug, you may have certain possibilities (within limits that depend on your library API maintenance), like trying an older executable with the same library, or older library with the same executable.

Bundle the executables into a single larger mega-executable with symlinks as the “true” entrypoint name.

Meaningless question. Simetimes it's what you need. Sometimes it's not. Sometimes you want to write the arabic digit 3, sometimes you want to write the word three. What is the advantage of writing out three? The advantage is whatever the advantage is in those situations when you want it. The possible details are infinite.

All what work? Just adding -R '$ORIGiN' when linking the program will then cause it to look for shared libs in its own directory.

The program still looks for libc in the system directory.

The argument for this is that if you ship software that features multiple executables that share code, they can have it as shared libraries, and easily find it relative to their own location, just like on Windows.

There's definitely value in the static approach in some cases, but there are some downsides e.g. your utility will need to be recompiled and updated if a security vulnerability is discovered in one of those libraries. You also miss out on free bugfixes without recompiling.

If you require a library, you can specify it as a dependency in your dpkg/pacman/portage/whatever manifest and the system should take care of making it available. You shouldn't need to write custom scripts that trawl around for the library. Another approach could be to give your users a "make install" that sticks the libraries somewhere in /opt and adds it as the lowest priority ld_library_path as a last resort, maybe?

"Free bug fixes without compiling". I think YMMV.

It depends a lot on ABI/API stability and actual modularity of ... components. There's not always a guarantee of that.

Shared libraries add a lot of complexity to a system for the assumption that people can actually build modular code well in any language that can create a shared library. Sometimes you have to recompile because, while a #define might still exist, its value may have changed between versions, and chaos can ensue - including new, unexpected bugs - for free!

My current day job has probably 60 apps the depend on one shared library.

Static linking has its place pace, no doubt, but it should not be the norm.

60 programs y'all maintain as part of an overall larger solution or 60 randomly selected apps maintained fully by 3rd parties that just happen to share the same major library version?

The former often makes a lot of sense in terms of deployment or maintenance - it's all really 1 big system so why rebuild and deploy 60 parts to change 1 when it's all done by you anyways. I'm not sure that's really what every build environment should default to assuming is the use case though but going shared libraries makes a ton of sense for that regardless.

60 programs maintained and built by 3rd parties which just happen to share major version of a library (other than something like stdlib obviously) would seem nuts to manage in a single runtime environment (regardless of static or shared) though!

> 60 programs y'all maintain as part of an overall larger solution

That, 100% is our scenario.

To me, the worse situation is when you just want to install a little tool to do a 2 minute job, and apt decides it needs to pull in 60 dependencies. You spend more time fetching and installing all these crappy little libraries, then uninstalling them all later, than you do running the tool.

> e.g. your utility will need to be recompiled and updated if a security vulnerability is discovered in one of those libraries. You also miss out on free bugfixes without recompiling.

This was the biggest pain point in deploying *application software* on Linux though. Distributions with different release cycles providing different versions of various libraries and expect your program to work with all of those combinations. The Big famous libraries like Qt , gtk might follow proper versioning but the smaller libraries from distro packages - guarantee. Half of them don't even use semantic versioning.

Imagine distros swapping out the libraries you've actually tested out your code with with their libraries for "security fixes" or whatever the reason. That causes more problems than it fixes.

Custom start up script was to find the same xml library I've used in the tar ball i packaged the application in. They could then extract that tar ball wherever they need - including /opt and run the script to start my application and it ran as it should. Iirc we used to even use rpath for this.

> Half of them don't even use semantic versioning.

This is a red herring. Distros existed before semantic versioning was defined and had to deal with those issues for ages. When packaging, you check for the behaviour changes in the package and its dependencies. The version numbers are a tiny indicator, but mostly meaningless.

I think semantic versioning actually predates distributions. It just was not called "semantic versioning." It was called Unix shared library versioning.

Imagine a world where every library and every package has their release date as their version. You'd instantly know which software lacks maintenance or updates (bitrot).

To me it seems more attractive than how Nix does it, but I guess they considered such and saw conflicts, therefore went with hashes.

IIRC GNU Parallel versions by date.

Recently in the python ecosystem the `uv` package manager let's you install a package as it was on a certain date. Additionally, you can "freeze" your dependencies to a certain date in pyproject.toml. So when someone clones it and installs dependencies it will install it at the date you've chosen to freeze it at.

Personally I love this method much more than versioning.

I think versioning is mostly useful to just talk about software and maybe major major additions/changes e.g io-uring shipping with linux mainline.

On the opposite side of the world in Gentoo, we compile updates to libraries and applications together on a rolling basis all the time, and it generally works out while letting us have everything as bleeding edge as we want it.

There's software that is done and doesn't need constant updates.

Yes, but if statically linked, this excludes all software that relies on security-relevant libraries (e.g. cryptography) or receives data from the network. I struggle to think of a lot of software that would qualify beyond coreutils and friends.

The interesting case here is dynamically linked software that doesn't need updates.

Would you also instantly know if 20250110 is a drop-in replacement for 20240930 or will it require changes in your code to make it work?

How do you know if 12345 is a patch for 432 released years ago or a major upgrade from 12344 released a moment ago? Pure time versioning doesn't work with multiple release streams.

Thanks for proving me wrong!

>When packaging, you check for the behaviour changes in the package and its dependencies

Yeah, but the package maintainer for a widely used library doesn't actually have the resources to do this. (heck, a package maintainer for a non-trivial application likely doesn't have the resources to do this). Basically they update and hope to get some bug reports from users.

>Imagine distros swapping out the libraries you've actually tested out your code with with their libraries for "security fixes" or whatever the reason. That causes more problems than it fixes.

I don't believe that it causes more problems than it fixes. It's just that you didn't notice the problems being silently fixed!

There are issues related to different distros packaging different versions of libraries. But that's just an issue with trying to support different distros and/or their updates. There are tradeoffs with everything. Dynamic linking is more appropriate for things that are part of a distro, because it creates less turnover of packages when things update.

I often refer to semantic versioning as "semanticless versioning". Everyone disagrees about what constitutes a change warranting each version number to be increased

Fun part is that it actually is true as for different use cases of the same library change might mean something different.

So it is complicated and there is no solution for every context, therefore we use best approximation.

This is never ever a problem unless a developer insists on always using the most cutting edge version of a library. There's no law that says you have to use the bleeding edge of every library when you make a program. Another issue these days, is that library maintainers often add new features or delete old features without incrementing the major version number. In the olden days it was assumed that minor versions were for bug fixes that don't break compatibility, and when you wanted to change how the library works in a major way, you increment the major number.

Now a lot of stuff is contnuously buggified so there is no concept of stable and in-progress.

Fun fact... Many Windows programs generally do some sort of 'hybrid static linking' (this is my own terminology) where programs are distributed with the `.dll` libraries just next to the binary. There is no concept of RPATH on Windows—the loader looks for dynamically-linked libraries in a fixed set of locations which includes the binary's directory.

Windows programs generally do link dynamically to core Windows libraries—which users are never expected to mess with anyway—and the C and C++ runtimes, but even these can be statically linked against with `cl.exe /MT`. Some programs even distribute newer versions of the C/C++ runtimes; that's where the famous Visual C++ Redistributables come from.

I agree, though—static linkage should be the default for end-user programs. I long for a time when Linux gets 'libc' redistributables and I can compile for an old-hat CentOS 6 distribution on a modern, updated, rolling-release Arch Linux without faffing with a Docker container.

For Rust, cargo zigbuild lets you compile against arbitrary old glibc. Probably can get that working with C as well.

> I long for a time when Linux gets 'libc' redistributables and I can compile for an old-hat CentOS 6 distribution on a modern, updated, rolling-release Arch Linux without faffing with a Docker container.

The Linux linking model is so bad. So extremely very bad. Build systems should never rely on whatever garbage happens to be around locally. The glibc devs should be ashamed.

Noice

There was a time when 200Mb was a big HDD. Can forgive for that reason.

Can you explain more? Curious and not able to google and understand this thread.

Specifically, what is stopping libc redistributables from being possible? And why disk size limitations from back then caused this situation.

Once upon a time hard drive space mattered and so it was arguably better to use shared libraries to reduce duplication. This is no longer relevant.

I can’t. At some point you simply have to move on.

It's true. I keep an Ubuntu 16 LTS VM around for making binary redistributables on Linux. It's the only way to ensure compatibility.

Imo this is a much saner solution for a system that supports precompiled applications.

Every time I tried to get a third-party binary app running on Linux, I discovered what the vendor did is they shipped half their dependencies as blobs, and relying on the system for the other half - which is an incredibly brittle system that breaks constantly.

The entry point usually is a script that sets LD_LIBRARY_PATH and then calls into the executable.

Compiling for CentOS 6 is a linking problem, and any linker lets you link with whatever you want, it's a matter of running the linker with the right arguments.

> There is no concept of RPATH on Windows—the loader looks for dynamically-linked libraries in a fixed set of locations which includes the binary's directory.

This is not true - you can control the DLL path via manifests. There's also a "known DLLs" list in registry which can globally redirect basically any DLL system-wide.

Many programs link various libraries of theirs statically, but then dynamically link to the system C library.

For instance, GNU Make uses some of GNULib.

https://git.savannah.gnu.org/cgit/make.git (gl subdirectory)

The problem is that static libraries are actually more likely to break across time in practice, since "the system" is more than just "syscalls".

For example, in places where the a filesystem-related "standard" has changed, I have old static binaries that fail to start entirely, whereas the same-dated dynamic libraries just need to bother to actually install dependencies.

I am convinced that every argument in favor of static linking is because they don't know how to package-manager.

> I am convinced that every argument in favor of static linking is because they don't know how to package-manager.

Which would be a fair reason. People who like to build things might just not want to also learn how to package stuff.

It's not all-or-nothing, though. If you need a dependency that isn't widely available on distros, then statically link it. It's fine. No big deal. But if you actually care about being a responsible maintainer, make sure you follow new releases of that dependency, and release new versions of your app (with the new version of the dependency) if there's an important bug fix to it.

If you're linking to libX11 or libgtk or something else that's common, rely on the distro providing it.

I really don't get all the anti-shared-library sentiment. I've been using and developing software for Linux for a good 25 years now, and yes, I've certainly had issues with library version mismatches, but a) they were never all that difficult to solve, and b) I think the last time I had an issue was more than a decade ago.

While I think my experience is probably fairly typical, I won't deny that others can have worse experiences than I do. But I'm still not convinced statically linking everything (or even a hybrid approach where static linking is more common) would be an overall improvement.

What also help is pacing. If your core dependencies is updating every week with API breaking changes, something like Debian is a poor choice if you don’t statically link. Some people don’t need the latest updates and are fine with security and bug fixes.

Nobody knows how to use the package manager. What happens in practice, is every single program uses the the package versions the distro happens to ship with.

If you want a newer version, too bad - your OS doesn't ship that so better luck in the next release. OR you can set up a private repo, and either ship a binary that has the dependencies included (shipping half the userland with your audio player), or they package the newer version of library, which will unwittingly break half your system, if not today, then surely at the next distro upgrade.

It speaks volumes of Linux package management woes, that no vendor ships anything analogous to brew or chocolatey.

I thought it spoke to windows and macOS woes that package management was a third-party concern.

What is the gap between e.g. `apt` and Homebrew?

There are the following differences usually:

- Packages are useful units of software in homebrew, and both end-user apps dependency libraries in apt (and weird stuff, like headers, or dependencies)

- Homebrew packages are installed for the user, apt is system-wide

- Homebrew packages are updated and maintained usually by people involved with the software they are shipping, apt packages are usually maintained by the distro

- As a result, Homebrew packages almost always work and are almost always up to date, apt ones are almost never (stuff like go, node etc.)

Apt is like a weird frankenstein monster of npm, the system update, and an app store, and all that with a global scope.

The sane thing here is to maintain a clear notion of what the "OS" is versus the "app", and use dynamic linking on that boundary, but not elsewhere. Which is more or less how Windows and macOS do things.

Unrelated to suckless, there's a project (confusingly) named stal/IX: https://stal-ix.github.io/

It is also a statically linked Linux distribution. But it's core idea is reproducible nix-style builds (including installing as many different versions/build configurations of any package), but with less pl fuff (no fancy funcional language - just some ugly jinja2/shell style build descriptions; which in practice work amazingly well, because underlying package/dependency model is very solid - https://stal-ix.github.io/IX.html).

It is very opionated (just see this - https://stal-ix.github.io/STALIX.html), and a bit rough, but I was able to run it in VMs sucessfully. It would be amazing if it stabilizes one day.

It's a trade-off.

Consider how dynamic linking libc works when a critical security bug is found and fixed. To update your system you update libc.so.

If it were statically linked, you need to update your whole distribution.

The article touches on that. If "your whole distribution" comparable to a Chrome in size? That gets updated.

I would be more accepting of the trade-off if it wasn't so brittle in practice.

Nix is much closer to a "good" dynamic linking solution IMO except that it makes it overly difficult to swap things out at runtime. I appreciate that the default is reproducible and guaranteed to correspond to the hash but sometimes I want to override that at runtime for various reasons. (It's possible this has changed since I last played with that tooling. It's been awhile.)

I generally find with Nix and NixOS that I'm able to just use a dev shell to create little custom environments at runtime as needed. Another option is `mkOutOfStoreSymlink` if you want some dynamic config for some GUI you are running.

Depends on what you are trying to achieve though.

(Apologies for off topic.)

Suppose your dev shell contains app Foo v1 which uses lib Bar v2. It links directly against the full nix store path. So loading a different version of lib Bar (say v3) into your dev shell, which puts it on your PATH, doesn't affect Foo - the Foo executable will still link against Bar v2. That's by design and a very good thing! It assures reproducibility and fixes the version incompatibility issue that dynamic linking typically suffers from.

However, what if I want to swap it out for whatever reason? Right now I have to modify the package definition and rebuild it. That's needlessly slow and resource intensive for some one off monkey patch. It also might not be feasible at all on my hardware - most people's rigs aren't cut out to build chromium or llvm, for example.

The UX is also absolutely terrible. Have you ever tried to patch a nixpkgs provided definition? I actually gave up on using nix as my primary build tool over this.

My expectation going in was a neat and tidy collection of independent, standalone package definitions that imported one another to satisfy dependencies as necessary. The reality was a massive monolith containing all packages at once making extracting just one to patch it rather complicated in some cases.

I believe flakes were supposed to address this exact dependency modification UX issue somewhat, but they caused my code to break in other ways and I decided I was done sinking time into tinkering with tooling rather than building software.

Linus Torvalds agrees with you: https://youtu.be/Pzl1B7nB9Kc?feature=shared&t=65

Also the stali approaches to filesystem hierarchy is aspiring https://sta.li/filesystem/

> AppImages

AppImages require a large amount of (obsolete) dependencies to run, making their portabiltiy practically worthless. Newer immutable distros like Aeon don't ship the necessary packages to run an AppImage.

You might want to take a good look at how Nix and Guix do things..

I know. But Nix doesn't make any of this complexity go away. It just helps you to tame this complexity and give you a reproducible system. Even Gobo Linux for that matter.

At the end of the day the apps were a simple end user applications. They used a handful of library functions from different libraries. My users cared about just using my apps to do whatever the apps did. I just care that my app should work on their machines easily - no matter what version of what distro they're using.

Gobo linux seems really interesting.

https://github.com/oasislinux/oasis is also a a small statically-linked linux system using musl and the like. Curious about your thoughts on that?

Two things:

1. You can bake the LD_LIBRARY_PATH into your executable with the -rpath mechanism.

2. Both rpath and LD_LIBRARY_PATH expand certain tokens such as $ORIGIN. $ORIGIN expands to the location of the program.

So you can (for instance) set a rpath of $ORIGIN in your executable, and then it will look for its libs in its own directory. Just like is the intelligent default on on you-know-what Redmond operating system everyone rags on.

I also like the static approach , there is staticx which I am loving which can take a dynamic binary and make it static. I am wondering if I could create something like .deb /.rpm / appimage and flatpak all into static binaries which can work on any device

I have a non-LFS, Stali-like project for myself, 100% statically-linked Linux. Probably the biggest PITA for me is compiling a statically-linked cmake. Even just compiling a dynamically-linked cmake takes an inordinate amount of time.

statically built helloworld in zig is like 8KB

...But the point of dynamic linking is that it allows your tiny apps to actually be tiny, yes?

The value of that isn’t what it used to be — an app being 0.1MB vs 3MB doesn’t even register in the world of 300MB apps

Reply on news.ycombinator.com