Hacker News Reader

> The day I stop static linking is the day I can compile on one distro and ship to many without worrying about users reporting loader errors. That day is not today.

That's completely missing point of the article.

You're free to ship your program binary however you want as the upstream.

It only becomes a problem if dynamic linking is a second class citizen in the language or build tool you use, you bundle dependencies and don't support un-bundling, or pin specific dependency versions.

> I don't really buy that dynamic linking all the things is such a boon to security.

Agreed. It's really not a panacea. When you upgrade a library, you probably want to restart the running applications that depend upon this. Dynamic linking won't save you.

The solution is having a graph of your dependencies! IIRC, NixOS gets this right. I don't think Debian's apt did?

I would really like to know how NixOS solves this problem for things like yarn, webpack, parcel, esbuild, npm, pip, conda, cargo, stack, go modules, ruby gems, etc.

Usually there are tools to extract the necessary dependency information (url and hash) from a package manager's lockfile and produce a fixed-output derivation so a Nix expression can reproduce the build environment. I know of yarn2nix, cargo2nix, and node2nix, and I wrote and maintain gradle2nix for JVM projects.

It's doesn't really, and that's the problem. There are automated tools and scripts to help, but there are always edge cases so that you end up going back to using pip for example.

Nix and language package managers just don't really play well together.

There are scripts and daemons that help you determine what needs restarting[1,2]. NixOS installs can go in separate directory prefixes when there are conflicts. For Gentoo and other Linux distributions, maintainers usually won't mark something stable without resolving conflicts, and this usually means sticking to older stable version of libraries until newer versions of libraries are fully supported by all installed packages. This can definitely be more work for maintainers, and as the blog posts says, it's a sisyphean task.

[1] https://manpages.debian.org/buster/debian-goodies/checkresta...

[2] https://github.com/liske/needrestart

Thanks, checkrestart looks useful. I wasn't aware of it.

Given that it's in a goodies package though, I assume it's not integrated with apt—at least by default.

I'm thinking of that apt prompt that says "There are services installed on your system which need to be restarted when certain libraries, such as libpam, libc, and libssl, are upgraded." I presume then that uses a hardcoded list of important services.

I can't help with checkrestart or apt questions, but if I can plug 'needrestart -r a': it restarts services, messages outdated shells/user logins and checks for newer kernels/microcode. It will also tell you if any interpreters/containers need restarting, but YMMV here. It's never blown up for me even when restarting boot services, and should work on any distro as far as I can tell. I use it on Gentoo.

> When you upgrade a library, you probably want to restart the running applications that depend upon this. Dynamic linking won't save you.

That's an orthogonal issue. You need to restart affected services nevertheless, whether it was linked statically or bundled in nix.

I wish people stopped repeating this red herring restart argument against dynamic linking every time the topic is brought up.

Apt has a graph of dependencies and anti-dependencies. They’ve been getting this right since the late 90s.

The day I stop static linking is the day I can compile on one distro and ship to many without worrying about users reporting loader errors. That day is not today.

Maybe do the trendy thing and link your whole distro together with your app (a.k.a containers)?

The fact that containers exist and are prevalent is a damning indictment on the Linux model imho.

Producing software that can simply launch is so wildly complex that the solution is... to snapshot an entire operating system install to run the application.

That’s INSANE. But it’s unfortunately required. https://xkcd.com/1987/

> Now, for the worst of all — one that combines all the aforementioned issues, and adds even more. Bundling (often called vendoring in newspeak) means including the dependencies of your program along with it. The exact consequences of bundling vary depending on the method used.

Let’s consider this statement at face value. The WORST thing a program can do is... include the things the program needs to run? So programs should... NOT include the things they REQUIRE to run?

I get it. I understand the argument. But if you take just a half step back it should be clear how utterly broken the whole situation is.

I'm not sure it's fair to characterize a container as a snapshot of an entire OS install - you can have containers that are very lightweight, e.g. just a static Go binary copied into a "scratch" container - but often the required effort is not put in to reduce size.

You are comparing having containers versus having no containers and no kind of bundling at all, which is not a reasonable comparison. If you need an OS snapshot, the only option on other operating systems is often to use a VM. That's what containers are used as an alternative to. (The other alternative there is to never install patches or upgrades on the production machines, which probably creates as many problems as using containers)

I’m not sure I made that comparison. I’m saying the popularity of containers is an existence proof that the distro-oriented shared library system is failing to meet the needs of common use cases.

Programs that bundle dependencies have a radically reduced need for snapshots, VMs, or containers. Such tools do provide a variety of value. But they don’t become virtually requirements to merely launch a program without error.

Ship your damn dependencies says I. Either statically or dynamically, but ship them.

You can't get rid of the distro-oriented shared library system, it exists as long as you are building on top of an operating system. To that end, when you say "we depend on operating system minimum version X.Y.Z" that now becomes another dependency you have to ship at some point if you're running the production machines. Containers are popular because they actually solve that exact problem. If there was some other comparison you were making I'd be interested to hear it, but AFAIK containers can only help here so it's not clear what else you were comparing to.

Parent is saying that the distribution model of modern distros is so legacy and out of alignment with what people actually want that a solution involving bundling the entire damn OS to sidestep that pit of snakes has now became the defacto way to deploy software.

To put it another way, what do you think the relative popularity of distributing your own internal software via a private apt repo is compared to bundling it as a stateless container and putting it on a registry? Some big companies that pre-date containers do it with apt. Most don’t. For good reason.

The company I work at has an internal yum repository that we use for application and certain application dependenices. It's worked reasonably well for us, mainly because we stick with using dependencies provided through the internal or public yum repositories or some other public external yum repositories if a particular dependency is not available otherwise.

I'm sure a similar solution could also work with apt.

For an entire OS, this doesn't make sense. It does make sense for business critical software. We did this 20 years ago, even calling the subdir VENDOR, for some 3rd party components. It's part of design, and even support of your stuff.

What's new is containers and k8s. People will put more into that, for dev and critical sw.

However, for full desktop or OS it's probably cargo-cult or hard to scale and maintain.

I'll keep beating this drum, but Nix and Guix are the best solution to this problem.

I don't think this is a fair characterization of why containers are a killer app. Full-OS virtualization was a killer app for hosting providers because it enabled multi-tenancy, allowing you to oversubscribe physical resources knowing that most applications you host aren't going to be experiencing anywhere near peak load 100% of the time. Containers allow you to take this a step further by allowing multiple applications with different versions of the same dependencies to run together on the same physical or virtual server without needing to worry about symbol clashes and without having to install one application's dependencies into some different than expected location while manipulating LD_LIBRARY_PATH and PATH.

This solved a huge problem for us in the geointelligence ground processing community. The various product generation algorithms tend to all be native code, but they're produced by different contractors, on different delivery schedules, some of whom are no longer on contract at all to provide updates. What do you do when four of them all depend on libxerces-c but all four depend on a different version? Tell them to stop pinning dependencies and update? How, if they're not on contract to do it? Tell the US Congress to get off their butts and award more money to the backing agencies so they can get their developers back on contract? Good luck. In practice, what we ended up doing, without wanting to stripe certain applications to only run on specific servers when some of them are used a ton more than others, was to install dependencies on an NFS mount shared by all the fast compute hosts, into application-specific subdirectories, and then set PATH and LD_LIBRARY_PATH before calling the executable that did all the work.

This system was terrible! But before containers, it was the best we could do. Now we just deploy each individual processing algorithm in its own container, and they can depend on whatever they want, all using standard system path under the illusion they have an entire OS and filesystem all to themselves. and never clash with each other.

> Containers allow you to take this a step further by allowing multiple applications with different versions of the same dependencies to run together on the same physical or virtual server without needing to worry about symbol clashes and without having to install one application's dependencies into some different than expected location while manipulating LD_LIBRARY_PATH and PATH.

We agree 100% on the problem. I enthusiastically agree with all of your complaints.

My point is that Linux model of system-wide shared libraries and PATH/LD_LIBRARY_PATH bullshit is terrible. And the fact that containers are required to resolve that spiderweb nightmare is a damning indictment on the Linux library model.

Containers are one possible solution. An alternative is for those applications to bundle their dependencies. If all applications bundled their dependencies then everything would “just work”. No need to hack bullshit envvars. No need to containerize.

Yes that means it’s harder to deploy security fixes. But if everyone is using containers then also those images need to be updated. At which point what have you even gained?

> But I'll link to this: https://drewdevault.com/dynlib

We already discussed this in HN https://news.ycombinator.com/item?id=23654353 . Both data and conclusions are utterly suspect.

It's why we like to use LGPL for libraries: if you ship statically linked software, ship your source too, so we will be able to patch and recompile.