Hacker News Reader

Comment by ris | Hacker News Reader

I eagerly await some concrete implementations of these new ideas.

In the meantime I'm waiting for the first big vulnerability in a widely used golang or rust library to see if downstream projects pinning it also release CVEs as they should (which would probably DDoS the CVE system from the resulting avalanche) or they quietly bump it and move on (in which case their users won't get the nudge to upgrade). This is where someone says "well you should just always be running the latest version of everything", which is of course infeasible on a real life system with thousands of installed packages.

And it's not that I don't completely sympathize or understand the advantages of static linking from a developer's point of view - you don't have to sell it to me there.

View on HN

The difference between dynamically and statically linking doesn’t change the patching story so long as applications ship with all their libs and share nothing.

The only difference is that a statically linked app is a large binary blob in a directory while a dynamically linked app is multiple smaller blobs in a directory.

Whether or not statically or dynamically linked apps are the future, the idea of system wide shared libraries seems like it’s going away.

shared ram too. it adds up.

It does, but we're already moving to a world where deployed apps are one of the very few things running in a container, which is itself running somewhere in a VM, so there's less sharing to achieve.

Yeah. In that scenario the distinction between dynamic and static linking is moot. In both cases you need to update the container image and when you do you only fixed that container image and you still need to update all the other images

There's absolutely no need for just one-ring-to-rule-them-all copy of libz.so. Shared libraries can be versioned and garbage collected like how habitat (hab) does it: in separate directories.

Exactly. The issue is not static vs dynamic; it's bundled vs unbundled.

You could probably even do an "unbundled" statically linked system, e.g. imagine Debian but everything is statically linked. Doesn't matter that everything is statically linked - if there's a vulnerability in libpng you can still easily update dependencies. Just uses more network / disk space.

I agree, but if you're bundling the dependencies why not just statically link them too? I've never understood the way macOS does it (dynamic linking, but app bundles ship with the libraries they need included).

In some cases it's required such as if you're using a closed-source library that doesn't provide a static version or if you're calling LGPL code from non-GPL-compatible code.

This is also how windows apps always worked. It’s good for deployment because an update might not touch more than a few files.

Also not all language ecosystems support static linking at all, e.g .NET

> Also not all language ecosystems support static linking at all, e.g .NET

The “language ecosystem” of .NET supports static linking via .NET Native, doesn't it?

.NET native is a dead end technology that only supported old UWP apps.

There's the new single file builds but those just concatenate the dynamic libs together into one file.

There's also the new .NET linker for trimmed builds, but that requires special annotations from many libs to work.

Not a mac user, but one upside I can see is that this forces apps to at least declare which libraries they are using. If everything is statically linked, it might be hard to figure out where on the system a vulnerable library is being used at all.

> Doesn't matter that everything is statically linked - if there's a vulnerability in libpng you can still easily update dependencies. Just uses more network / disk space.

It's not that simple

* Someone, somewhere - probably distro volunteers - have to manage this process of rebuilding dozens, hundreds, or thousands of packages, and knowing which packages need to be updated to begin with. The amount of work required from distros (again, mostly volunteers) would be an order of magnitude greater even if the process was automated.

* Likewise, the amount of resources needed to rebuild all of those packages, and distribute them, would be orders of magnitude greater.

* The amount of extra work (both for humans and computers) required would delay the rollout of patches.

* And in the real world, a lot of users don't have limitless network bandwidth and disk space.

* And auditing systems for security vulnerabilities would be geometrically more difficult. Now you need to track a recursive bill of materials for every package on the system.

If there's a security bug in libc or something along those lines - which are fairly common- distros would need to rebuild practically every package in the OS. It's all well and good to talk about how it would be no big deal but I think if it was ever actually implemented no user would accept that trade-off.

> Someone, somewhere - probably distro volunteers - have to manage this process of rebuilding dozens, hundreds, or thousands of packages, and knowing which packages need to be updated to begin with.

Packages list their dependencies. You don't have to figure it out manually; that would be insane.

Distros have automated build farms for all this stuff.

As I said, even with automation, there's a surprising amount of manual effort required.

> The difference between dynamically and statically linking doesn’t change the patching story

I think there are some edge cases where the mainteners will always be caught between a rock and a hard place and the only way to move forward is to involve the application developers - at least as long as library compatibility can't be defined more rigidly.

As a dev, you'd like to pin the versions of your dependencies as tightly as possible, so you know the library will behave in production exactly as it did in your tests.

As a maintener, you'd like applications to specify their dependency versions as loosely as possible, so you can transparently update a dependency in case of critical patches.

This means, there can always occur a case where version x of a library has a critical vulnerability but version x+1 breaks the assumptions that the software is making.

No matter if the library is linked statically or dynamically, this situation can't be solved by the maintener alone.

Of course the GP is also correct in a sense: If applications bundle libraries or set tight version restrictions, it's now the responsibility of the developers to track critical vulnerabilities in all of their dependencies and integrate the patches as quickly as possible.

The problem with bundled libraries is that now we have to trust every single developer to get this right.

> As a dev, you'd like to pin the versions of your dependencies as tightly as possible, so you know the library will behave in production exactly as it did in your tests.

Indeed, and this is why the shared nothing models are so attractive. Unless I was developing anything security critical such as server apps, I'd probably never want to deploy anything in classical unix style where any part of the app could be replaced behind the back of my application.

On windows and for desktop software, the risk of a failed install or the user sabotaging their own install without realizing it is more common than the risk of having an outdated version of a library. If the application is mostly or completely offline (e.g. only online for updates or documentation), then the surface area for attack is pretty small. So I'd usually go so far as validating the integrity of the app with a checksum on startup too, and refuse to start if the app isn't byte for byte exactly as expected.

This also obivously means my app has almost no risk of ending up in a package in a Linux distribution too.

> This is where someone says "well you should just always be running the latest version of everything", which is of course infeasible on a real life system with thousands of installed packages.

Is updating a library with thousands of dependents, without individually testing those dependents, any more feasible?

You would have to do that anyway. With static linking, you also have to rebuild those dependents and make sure their users update them as well.

I would rather the developers rebuild the dependents, test them, make any necessary fixes, and send me the updated (presumably static) binaries. As opposed to me, as an individual user with no familiarity with the code base, just switching out the pieces and preying nothing goes wrong.

And if the upstream developers are unresponsive, or only provide the bump with the latest bleeding-edge version which you can't upgrade to yet?

On a system where security is critical, I would probably need to re-evaluate my use of that software.

It doesn't necessarily need to be the upstream developer, it could be some other organization analogous to a distro maintainer. What matters is that someone who is familiar with the software and code base has actually tested the update in a purposeful way.

As a small developer who handles my own support, I agree with this sentiment completely!!

You'll have the opposite problem though, that you have to trust every single developer to properly manage their dependencies and publish timely patches if there is a vulnerability.

> As opposed to me, as an individual user with no familiarity with the code base, just switching out the pieces and preying nothing goes wrong.

Wouldn't this be the task of your distro's mainteners, not you? (Who will have familiarity with all codebases as well as options to contact the original developers if there is an unsolvable issue)

> Wouldn't this be the task of your distro's mainteners, not you?

Sure, and when they’re done, they can recompile the static binary and send me an updated copy.

I don’t want to update until they’ve finished testing the dependents. At least with static binaries, I can update some of the dependents sooner as the maintainers work through their lists.

Well, it's what traditional distributions do every day.

In NixOS, we try to work the test suites into the build process and rebuilds happen when any dependencies change, so I'd like to think we get (almost) the best of both worlds.

> Well, it's what traditional distributions do every day.

Which coincidentally is why I'm not using Linux right now. :) A move towards static linking in the Linux world would go a long way towards changing that.

(Everything I've read about NixOS seems great, thanks for helping with that—it's just that it also has a large barrier to entry.)

(I'm merely a lowly maintainer in NixOS, shoulders of giants etc.)

That's the choice you're making everytime you add another dependency--you are now bound to that repo and all of it's upstreams. You need to follow along send stay on top of Caves, API deprecations, etc. I don't think people weigh this cost properly when doing a build vs integrate decision. Turns out dependency management is a lot like the "when you sleep with someone you're sleeping with everyone they've ever slept with" ideology.

Are you sure about the "CVE explosion"? From the CNA counting rules https://cve.mitre.org/cve/cna/rules.html#section_7_assignmen... :

  7.2.4 If multiple products are affected by the same independently fixable vulnerability, then the CNA:
    a. MUST NOT assign more than one CVE ID if the products are affected, because they share the vulnerable code. The assigned CVE ID will be shared by the affected products.

I'm not sure what would actually happen in reality, whether a single CVE would get endless addenda listing the packages affected by an upstream vulnerability. I certainly see plenty of new CVEs go past which are of the form "xyz had a vendored version of abc, which was vulnerable to ..."

Author here.

> I eagerly await some concrete implementations of these new ideas.

I'm working on them now! They are not public because of a licensing issue where I need to consult a lawyer first (thanks, GitHub Copilot), but they will be public as soon as that's sorted out.

> They are not public because of a licensing issue where I need to consult a lawyer first (thanks, GitHub Copilot), but they will be public as soon as that's sorted out.

Do you mean that you built them using Copilot, or that you don't want Copilot to use them? Or something else entirely?

I don't want Copilot to use them. Sorry; should have said that.

No worries, it's nice to know that someone is working on that with the help of actual lawyers!

The design of the Go build and packages system kinda renders that point moot because the expectation is that provided you follow Go's guidelines then a Go application should always be easily recompilable, and all external dependencies should always be distributed in source-form, not as binaries (e.g. https://github.com/golang/go/issues/2775 ).

Right, but if every application did distribute as source and could just be recompiled, there would be fewer complaints about the difficulties distributing Linux binaries

> provided you follow Go's guidelines then a Go application should always be easily recompilable, and all external dependencies should always be distributed in source-form, not as binaries

So it's a human solution, and not a technical one, which means that it can and will fail.

That's like saying condoms are ineffective because you included people too lazy to put them on in the first place in your calculations.

Provided you (and your org) use Go as Google intends then that isn't an issue.

> That's like saying condoms are ineffective because you included people too lazy to put them on in the first place in your calculations.

I think it's closer to say that the existance of condoms won't make AIDS and unwanted pregnancies disappear, even if they were 100% effective. Which, if you're trying to eliminate completly AIDS, is a fair claim to make.

My point is that you shouldn't assume that everyone is going to follow the Go guidelines, because some people won't, and you don't want your security to rely on that.

It would certainly make the embedded industry more secure if it was always running the latest version of everything. Or at least the ISPs, because imagine the bandwidth they're be able to charge for as everything from your washing machine to your watch to your lightbulbs to your car downloads new images hourly.

Just imagining the joy of broken firmware updates being pushed out to all my lightbulbs...

> This is where someone says "well you should just always be running the latest version of everything", which is of course infeasible on a real life system with thousands of installed packages.

Maybe the problem is have a system with thousands of installed packages? If every application is in its own "space", then a security failure won't affect the rest. If you want the best security, you're going to have to do pretty big tradeoffs. If you're not ready to do them, you're going to have to live with an insecure system. I don't think there is a way around this.

Part of the problem is not just getting all the different applications and libraries up to date, but knowing which of them need to be updated.

For a system that widely uses dynamic libs, that's fairly easy to do. Check the version of the installed library, along with any patches it might have, and you have a good idea of whether it needs to be updated.

Now, imagine there's a zlib exploit. That's a lot of things in the system that need to be updated. It's so ubiquitous that even in a system that's almost entirely built with dynamic libs, there are a few that use it as a static lib, and you'll have to make sure those are updated too (go look at the last zlib CVE and the major distro updates to it and you'll see all those packages, sometimes a couple days later as they discover them too).

A world where these are all statically compiled (or bundled separately in containers) is much more complex. It's not impossible, but you don't naturally get some of the work done for you. To some degree we're already going this way with containers though.

> Part of the problem is not just getting all the different applications and libraries up to date, but knowing which of them need to be updated.

In which was? Is it because it's hard to know which version of zlib was used in X application, or is it because there is no centralized information about this? Maybe that's an opportunity right there: build a graph of dependencies of a system, and alert when something has been compromized and which are the consequences.

To some degree most distro's have this graph and it's somewhat queryable (rpm has to agilities for this, if not easily consumed). The difference is in how accurate and queryable it is, and how easy it is to be fairly sure everything is up to date.

It's not a show stopping problem, but it is something that should be considered and thought closely about. This is a security adjacent problem, and changes in it can have consequences that are larger than "I'm running a library that doesn't have the feature I want".

> build a graph of dependencies of a system

You mean in the sense that every application that comes with statically linked libraries should also come with some metadata indicating which libraries/versions are in there?

This would work, but then you have the problem metadata always has: It can get out of sync with the actual thing, developers can forget to properly maintain it (or even to provide it at all) etc etc.

You're right, I didn't answer the question of "who should maintain that stuff?". I was imagining something like libraries in JS or Rust, where people pin the dependencies of their libraries, so with this, you can build a graph of dependencies. I guess I have a very "github" centric vision of things and that most software isn't built this way, which would complicate things.

I believe when heartbleed happened quite a bunch of go progams were yoinking in openssl because the go ssl tooling was still relatively young.

I have no idea what actually happened there, I just recall some moderately annoyed sysadmins figuring out how to rebuild go applications by developers who'd since left the company.

(this is not a shot at go, this is merely a "there might already be -some- data out there" wrt your question)

Reply on news.ycombinator.com