Hacker News Reader

Comment by e63f67dd-065b | Hacker News Reader

It's just classic dependency issues. I'm not familiar with swift specifics, but probably a combination of ABI instability and just plain version incompatibility from one distro to the next with your target program.

My opinion is the opposite: I think the old paradigm of distros managing a giant set of system libraries is a bad one, and is how we ended up in the land of docker. Go and Rust made the right decisions here: vendor all the dependencies, and distros can't mess with them. Makes it easier for upstream and the distro.

Modern languages are not like C/C++: a single non-trivial rust program can easily depend on 100+ crates, and re-creating crates.io in your package manager is just a bad idea, even putting aside that there's probably major version incompatibilities the moment you go beyond a handful of programs. Look at the disaster that's python package management, that's not where you want to end up.

View on HN

> Makes it easier for upstream and the distro.

Until there's a vulnerability in one of the dependencies and now you have to rebuild all of the packages which use it. Specifically for Rust, there's also the fact that most projects use a lock file and if your build process respects it, you now have to wait for the upstream to update it and release a new version (or update it yourself). And if your build process doesn't respect the lock file (or the project doesn't use it) and you just fetch the latest compatible dependencies at the time of build, you now have no idea if you're affected or not by the vulnerability because you don't have the exact resolved versions stored anywhere (https://github.com/rust-lang/rfcs/pull/2801).

I totally agree. Static linking is essentially easier for people who don't want to care. But ignoring security does not mean it solves it, on the contrary.

Static linking comes with a lot of issues, just like dynamic linking is not perfect. I really think it depends on the use-case, and that's why I want to have the choice.

> Until there's a vulnerability in one of the dependencies and now you have to rebuild all of the packages which use it.

Packages get rebuilt all the time. This is fine.

As for the rest, it would be cool if binaries shipped with a manifest of some sort naming all the versions of their statically included dependencies. A SBoM of sorts. It would make this sort of vulnerability scanning much easier to do.

> It would make this sort of vulnerability scanning much easier to do.

Scanning, sure. Fixing... surely much harder than with shared libraries.

Only because the build tooling is less mature.

It should be pretty easy to programmatically update a lock file, run the tests, and rebuild a package. For rust crates that are compiled and packaged straight from git, you could probably automate that today.

> It should be pretty easy to programmatically update a lock file, run the tests, and rebuild a package.

You still fundamentally need to rebuild (or at least relink) and re-distribute all the packages, whereas with shared libraries... well you just update that one package.

I hear the refrain "you'll have to rebuild your packages" a lot in these discussions, and I confess I don't see how this is a problem. Maybe it's a holdover from C and C++ veterans for whom figuring out how to build any given project is a Herculean effort, but for every other language with a first-class build system it's trivial.

Debian has about 59000 packages. Even rebuilding 1k of that is a significant pain on build infrastructure, specially in one go, if say a vulnerable library is found.

You also now get a huge set of users who will now download these 1k of updated packages. Not to mention the utter fun of trying to understand why your system just requires maybe 100 packages to be updated…

Compare to upgrading a single shared library.

The distro repositories are being continuously rebuilt, because packages are receiving continuous updates. In the meantime, as far as I'm concerned dynamic linking is itself a security failure, because if I run a binary I want to know exactly what code is being run, without having to worry about dynamic linking and loading nonsense inserting itself into my process.

> The distro repositories are being continuously rebuilt, because packages are receiving continuous updates.

At large timescales, yes. But not at timescales relevant for rapid security updates.

They aren’t being rebuilt with that kind of frequency as you might think.

They get rebuilt when a change in the package is made, not when a change in a dependency is made — which is a huge difference.

There are plenty of packages that might not get an update for a long time (month, half year, … whatever their release cadence might be).

Dynamic linking makes it _easier_ to handle security, including checking if your program is linking to a broken library — static linking does not have the same means.

Statically linked programs don’t mean you know what is being run either, you can always dlopen() and all kind of shenanigans like Go and Rust do.

What is a security nightmare is statically linked binaries, you have no clue what they are linked against.

I'm against dlopen as well. If something makes use of LD_LIBRARY_PATH, rest assured that I want it excised from my system. Reproducibly-built, fully-static binaries are what I want, ideally with an accessible SBOM embedded.

So all your binaries are statically linked on your system? 100%?

That's the dream. I want a system that's 100% source-available, 100% reproducible, and 100% statically-defined. Nix looks like the closest thing to that right now.

I don’t see how Nix is close to your dream, Nix is just like any other system and links things like any other system.. I.e. mostly avoids static linking.

Debian did the strongest push for reproducible builds, and is 100% source available.

Do you know whether someone has made an estimate based on the history of actual vulnerabilities of how much rebuilding would need to be done in a non-shared library approach?

Not to my knowledge. But consider something like crypto, libssl ... which is linked to almost everything, or libxz, zlib, etc. If there is a bug in a shared library, one can replace it with a interm one very easily, and the whole operating system will use it. One can even replace it with a different implementation assuming ABI compatibility, without having to do "much".

Enclaves like Go, Rust, Swift (things that insist on using direct sys calls, and eschewing system libraries) would need to implement their own versions, or use FFI, and then you're SOL anyway if you really insist to use static linking... And who knows what kind of bugs might crop up there that don't match whatever anything anyone else uses.

Shrug :-)

For what it's worth, Rust uses libc by default in Linux.

On the other side, there are quite a few self-updating applications out there (browsers), and source control systems like Github will push notifications, and even pull requests for out of date dependencies. So pushing out a new version of a given application is pretty easy. If the application comes through Flathub or self-updates then it doesn't have to wait for the distro maintainers to get the update out.

That is a very, very tiny subset of things that are installed on any GNU or BSD system, and that need to get security fixes. Just recall the xz/sshd/system debacle, to patch your system all you needed was to replace libxz / libzma -- a DSO! This obviously ignores lots of other work -- but that work would have ballooned if standard practise was to statically link everything.

The amount of work spent finding usages of static linkage, and figuring out which version of libxz (or whatever it was called) just to be very sure that one wasn't using the compromised version was atrocious, no thanks.

> then it doesn't have to wait for the distro maintainers

No, but it has to wait for each developer of each binary individually. Instead of relying on a team of maintainer, you now rely on all the developers.

"Self-update" isn't something magical: someone has to do build the new version and ship it. With static linking you still have to build and ship for every. single. binary. With dynamic linking you just build and ship the one library that everybody depends on.

Of course... and I'm not suggesting that every app in a linux distro should be as such. You can dynamically link with Rust, Go, etc. That said, for one of the 3-4 GUI apps that most people use, the distro process tends to leave people with relatively old versions missing critical bug fixes or features.

I tend to keep my host OS pretty bare, and most of my runtime apps in Flatpak and the stuff I'm working on in Docker.

> You can dynamically link with Rust

Do you have experience with that, or do you say it because you believe it's possible? Genuinely interested, because I have been looking into it and my conclusion was that it does not really do what I expect from "dynamic linking".

See e.g. https://news.ycombinator.com/item?id=40665343 and https://news.ycombinator.com/item?id=40661874

> But consider something like crypto, libssl ... which is linked to almost everything, or libxz, zlib, etc.

AFAIK, the reason most Linux distributions have an allergic reaction to static linking is because of zlib. There was a vulnerability in zlib some time ago, and it required the distributions to find every single program which had its own copy of zlib (vendoring zlib used to be common back then), update these copies with the security fix (not necessarily trivial, since the vendored copy might be a customized variant of an older zlib release), and rebuild all these packages (which is a lot of "fun" when some of the packages are in a "failed to build from source" aka FTBFS state). The distributions learned their lesson back then.

The aversion to static linkage predates that, back many years static linkage also had other issues that have been solved today (ALSR ..). Many of the very basic points that Ulrich Drepper made some odd 20 years back (https://akkadia.org/drepper/no_static_linking.html) still hold.

Go is the only one of the three that directly uses syscalls. Rust and Swift both go through libc. Swift never even had the option of directly using syscalls, since it was originally designed for MacOS, and syscalls are so unstable there that they've changed in minor version updates.

Most USERS only run a handful of applications any given month. I think upgrading 1-2 applications is easier to reason with over even a hundred packages.

For distributions, it costs a lot of money they don't have to just be rebuilding stuff all the time.

For randomly downloaded binaries from the internet it means you will most likely keep using it with the vulnerabilities for years to come.

It really sounds like you see it from the point of view of a developer who has never thought about how a Linux distribution works.

Yes, for you as a user it's simpler to link statically and not learn about anything else. But distros are a bit more elaborate than that.

It sounds like you see it from the point of view of a disto maintainer and never thought about how awful the experience is for software developers. I want to just depend on some specific, latest version of a library. I don’t want to have to go look at which version of that library exists in 18 different Linux distributions. I don’t want to deal with bug reports because Debian patched one of my dependencies or replaced the version I depend on with something newer or older.

I’m happy for someone to update the version of my dependency if it has security patches. But the rest? A total nightmare. I’ll take cargo / npm / swift / etc style versioning every day.

> I don’t want to have to go look at which version of that library exists in 18 different Linux distributions.

If you make open source software, the solution is simply that you should not distribute your software [1]. Let distro maintainers do it, and suddenly it's a lot easier for you.

If you make proprietary software, then it's not exactly the problem of the Linux distros anymore so sure, feel free to link statically. I personally think you should do a mix of both: libraries like openssl you want to link dynamically from the system. Now some obscure dependency you may have that breaks compatibility randomly and is generally not distributed by major distros, probably you want to link them statically. But you should know that you are then responsible for them and the security issues they may bring (i.e. you should monitor them and update them regularly).

[1]: https://drewdevault.com/2019/12/09/Developers-shouldnt-distr...

> Packages get rebuilt all the time. This is fine.

You forgot to link the image with the dog sitting in the fire.

> As for the rest, it would be cool if binaries shipped with a manifest of some sort naming all the versions of their statically included dependencies. A SBoM of sorts. It would make this sort of vulnerability scanning much easier to do.

Go binaries have that. You can apply "go version -m" to any recent Go binary.

Also "govulncheck".

Examples:

    go version -m $(which go)
    go install golang.org/x/vuln/cmd/govulncheck@latest
    go version -m $(which govulncheck)
    govulncheck -mode=binary $(which govulncheck)
    for b in $(go env GOPATH)/bin/*; do govulncheck -mode=binary "$b"; done

As you say, a typical Rust program can easily depend on hundreds of crates that nobody really checks. That's a security issue.

The whole point of a distro is that someone distributes them, so you can choose which distro you want to trust.

What I don't like about Rust and Go is that they enforce their preference. I am fine if you want to link everything statically. I just don't want you to force me.

You're not forced to with Go. CGO exists and requires dynamic linking.

More software should “enforce” their preferences, many decisions are objectively superior and this “anything goes” attitude has done nothing but hurt OSS/Linux adoption.

Everything about the shared lib model is stupid, and has contributed to poor Linux market share to date.

I couldn't disagree more.

> Everything about the shared lib model is stupid

I don't think we can discuss if that's you're stance. But also I don't think you understand the shared lib model if you think like that.

Dynamic linking seems to work fine for Windows or OSX. Maybe it's open source + dynamic linking that's the problem.

Honestly a problem is that most developers just don't understand how it works. Docker / static linking solves a lack of understanding, mostly. That's why they are popular.

> I just don't want you to force me.

Rust supports dynamic linking, and has since well before 1.0.

In practice, it feels like everybody assumes that Rust is installed through Rustup and dependencies are statically linked from cargo. I have tried going the dynamic linking way, it's painful when it's not impossible.

Regardless of what "everybody assumes", you can in fact build rust binaries without using rustup or cargo. Distros do this. Yes, it's more painful for end users than just typing `cargo build`, but that's irrelevant to distro maintainers.

Some software (e.g. IDEs) will call `rustup` for some reason (yep, it happened to me and I therefore couldn't use that software).

> Yes, it's more painful for end users than just typing `cargo build`

Cargo is fine, I don't mind installing cargo. I just don't want my packages to be handled by my package manager. I don't want every project to ship their own custom package manager like rustup.

That's kind of a disingenuous take. Rust supports producing dynamically loadable libraries. It doesn't support dynamically linking arbitrary dependencies, which is what most people would understand from "Rust supports dynamic linking" in a discussion about static linking.

> It doesn't support dynamically linking arbitrary dependencies

I don't understand why one would think this. You make it sound as though Rust only supports dynamic linking for glibc, but that's certainly not true.

The keyword is "arbitrary". Say you have dependencies on reqwest and clap, want to dynamically link against them? Not gonna happen. That's what people talk about most of the time for static vs. dynamic linking, at least in my circles.

I feel the same. A huge part of the pleasure of Go and Rust are that I never run into dependency problems. I'm happy to pay the "Hello World is 4MB" tax.

And the "I now have to evaluate thousands of libraries instead of 10 well established framework" tax?

I've seen plenty of bad go libraries. For example their authors will be unaware of isatty() so they will output control codes when piped.

If you factor in the time to find a competently written library (and quite possibly write one yourself) it starts to be less convenient.

Some distros do actually break out rust dependencies into separate packages (e.g. Guix does this). It's just that a lot of rust software isn't distributed primarily by distros.

Fedora and Debian do this too. That's why it sometimes takes longer for a project to be packaged - you literally have to recursively package all of the dependencies first.

Which is a feature, and not a bug: some of us want our distro maintainers to actually have a look at what they distribute.

Yes and it doesn't prevent anyone to build software separately, being written in rust or any other language.

> Fedora and Debian do this too.

IIRC Arch does as well (or at least tries to).

Arch does for quite a few ecosystems (Python, Ruby, Haskell, ...) but currently not for Rust.

"vendor all the dependencies"

What does that mean?

It means "add the source for your dependencies to your own codebase and build them with your build system".

Thanks. Not an appropriate use of the noun "vendor," so I would never have guessed that.

It's the verb vendor: https://en.wiktionary.org/wiki/vendor#Verb

"Vendor" isn't the verb. The verb is "vend."

That's a different verb, with a different meaning.

Did you read the link?

> Modern languages are not like C/C++

I think there's a bit of a misconception here.

Ardour is written in C++ and depends on 80+ other libraries.

The dependency situation may be exacerbated by packaging culture that encourages the use of lots of relatively small dependencies, but it is by no means determined by language or even context.

I started a small project both in C++ and Rust.

When in C++ I had 2 direct dependencies it resulted in 6 total dependencies (direct + transitive).

In Rust I had 5 direct dependencies (because some standard stuff is not in the standard lib) and it resulted in... 300 total dependencies.

> but it is by no means determined by language or even context.

Not sure what you mean there. My feeling is that making it super easy to pull 50 transitive dependencies without realizing it does not help. If you have to manually handle every single dependency, first it forces you to look at them (that's a good thing), and second it encourages you to minimize them.

> My feeling is that making it super easy to pull 50 transitive dependencies without realizing it does not help

What is it that makes you think that C/C++ does not do this also?

[ EDIT: this also has something to do with the typical size and scope of C/C++ libraries ]

> What is it that makes you think that C/C++ does not do this also?

My experience. I haven't worked in a single C++ project that had hundreds of dependencies, but I encounter those regularly in Rust/npm.

I would suggest that this caused more by ideas about the appropriate scope and scale of libraries in the 2020s than anything specifically connected to the language in use.

I don't see that in Kotlin for instance.

> the disaster that's python package management, that's not where you want to end up

The one where the only sane option is using distribution packages or conda and ignoring anything that the python community comes up with?

I had my Python and pip installed from distribution packages. One day I needed to build something from source, but it needed Meson...

Ok installed Meson also from my distro packages! But no, the project required a newer Meson version.

Ok let's install Meson with pip! But no, it turns out pip packages themselves can require a minimum pip version!! Go figure.

So I couldn't build that program without first pip-installing Meson, and I couldn't pip-install Meson without first pip-installing a more modern version of pip itself.

Guess how well it worked when I upgraded pip. Spoiler: not a smooth way to discover that Python packaging is a joke.

Were you using a virtualenv? Or were you pip installing into your distro / system python, like a true savage?

I just knew the basics. So I was just following commands from stack overflow or similar. Silly me, thinking that it would work logically like other languages do! Later I ended up learning about all that stuff with venvs and whatnot, not to mention it's not even a unified solution but there are multiple alternatives, adding to the confusion... All this, no doubt, is a consequence of how terrible the Python packaging story is. Rust, Go, Ruby, Node, and more, I had used without remotely similar hassles.

You are right. It's not a good experience for new developers, especially given how long Python has been around.

They are so close, though! I think one lesson Python took is that it's better, even mandatory, to use venv and install stuff in isolated containment.

This is exactly what Node/NPM does: "npm instal" creates a new dir, "./node_modules" and installs stuff there, and next "npm" commands will transparently refer to that subdir. This is, in effect, the same thing as a Python's venv! to the point that in my scripts I always name them "python_modules", to give other unfamiliar devs a clue of what it is.

If "pip install" just did this transparently, without introducing the whole concept of venvs to the user, it would be a huge step forward. I cannot imagine how it would suck if NPM didn't make its node_module dir, and instead it forced you to first learn all the idiosyncrasies of module isolation.

Yeah, they should just define a new command line option (and perhaps a new command that defaults to this behavior, say vpip or vpython or something.) The virtualenvs take some getting use to. I've been using python for over 15 years so it's second nature.

Have you tried "poetry"?

Yes. Having tried it led me to write that comment.

I'm curious what problems you ran into with it?

Reply on news.ycombinator.com