“Static Linking Considered Harmful” Considered Harmful
gavinhoward.com
gavinhoward.com
ldd is not the dynamic linker, it's only a tool to debug the process of dynamic linking. The dynamic linker is ld-linux.so (exact name depends on glibc version, architecture, etc.)
Also, I think the linked article [1] about the security of ldd is somewhat useless. The ldd(1) manpage [2] is very explicit about the security of ldd and tells you not to run ldd on untrusted executables:
> [...] Thus, you should never employ ldd on an untrusted executable, since this may result in the execution of arbitrary code.
It's a little amusing how the linked blog post explains how to create a malicious executable that runs arbitrary code when inspected with ldd, noting that "I researched this subject thoroughly and found that it's almost completely undocumented. I have no idea how this could have gone unnoticed for such a long time." and concluding with "Never run ldd on unknown executables!" - all while the manpage literally mentions that precise known limitation in the third paragraph!
To be fair, you could argue that this limitation is not widely known and people should be made aware of the risks of ldd, but on the other hand you can hardly criticize ldd when its manpage is that explicit about the issue.
Part of "pax-utils" or similar package on some distros. It's actually a shell script wrapper around scanelf (also pax utils), which reads the appropriate bytes from ELF files without just executing the ELF file.
Thank you for pointing out my mistake. I'll fix it.
/lib64/ld-linux-x86-64.so.2
The argument around ldd having a vulnerability is that it appears to be an introspective tool. It is not immediately obvious that it just executes "LD_TRACE_LOADED_OBJECTS=1 ld-linux.so /bin/ls" to print what the linker does during an actual, live linking operation.
ldd documenting this fact to remind people seems reasonable to me. There are other tools like readelf which can inspect objects without executing their contents. Dynamic linking is, well, dynamic and it can depend on code executed at runtime -- so it is necessary to do so to get an accurate report of what will happen.
In the meantime I'm waiting for the first big vulnerability in a widely used golang or rust library to see if downstream projects pinning it also release CVEs as they should (which would probably DDoS the CVE system from the resulting avalanche) or they quietly bump it and move on (in which case their users won't get the nudge to upgrade). This is where someone says "well you should just always be running the latest version of everything", which is of course infeasible on a real life system with thousands of installed packages.
And it's not that I don't completely sympathize or understand the advantages of static linking from a developer's point of view - you don't have to sell it to me there.
The only difference is that a statically linked app is a large binary blob in a directory while a dynamically linked app is multiple smaller blobs in a directory.
Whether or not statically or dynamically linked apps are the future, the idea of system wide shared libraries seems like it’s going away.
You could probably even do an "unbundled" statically linked system, e.g. imagine Debian but everything is statically linked. Doesn't matter that everything is statically linked - if there's a vulnerability in libpng you can still easily update dependencies. Just uses more network / disk space.
It's not that simple
* Someone, somewhere - probably distro volunteers - have to manage this process of rebuilding dozens, hundreds, or thousands of packages, and knowing which packages need to be updated to begin with. The amount of work required from distros (again, mostly volunteers) would be an order of magnitude greater even if the process was automated.
* Likewise, the amount of resources needed to rebuild all of those packages, and distribute them, would be orders of magnitude greater.
* The amount of extra work (both for humans and computers) required would delay the rollout of patches.
* And in the real world, a lot of users don't have limitless network bandwidth and disk space.
* And auditing systems for security vulnerabilities would be geometrically more difficult. Now you need to track a recursive bill of materials for every package on the system.
If there's a security bug in libc or something along those lines - which are fairly common- distros would need to rebuild practically every package in the OS. It's all well and good to talk about how it would be no big deal but I think if it was ever actually implemented no user would accept that trade-off.
Packages list their dependencies. You don't have to figure it out manually; that would be insane.
Distros have automated build farms for all this stuff.
Also not all language ecosystems support static linking at all, e.g .NET
The “language ecosystem” of .NET supports static linking via .NET Native, doesn't it?
There's the new single file builds but those just concatenate the dynamic libs together into one file.
There's also the new .NET linker for trimmed builds, but that requires special annotations from many libs to work.
I think there are some edge cases where the mainteners will always be caught between a rock and a hard place and the only way to move forward is to involve the application developers - at least as long as library compatibility can't be defined more rigidly.
As a dev, you'd like to pin the versions of your dependencies as tightly as possible, so you know the library will behave in production exactly as it did in your tests.
As a maintener, you'd like applications to specify their dependency versions as loosely as possible, so you can transparently update a dependency in case of critical patches.
This means, there can always occur a case where version x of a library has a critical vulnerability but version x+1 breaks the assumptions that the software is making.
No matter if the library is linked statically or dynamically, this situation can't be solved by the maintener alone.
Of course the GP is also correct in a sense: If applications bundle libraries or set tight version restrictions, it's now the responsibility of the developers to track critical vulnerabilities in all of their dependencies and integrate the patches as quickly as possible.
The problem with bundled libraries is that now we have to trust every single developer to get this right.
Indeed, and this is why the shared nothing models are so attractive. Unless I was developing anything security critical such as server apps, I'd probably never want to deploy anything in classical unix style where any part of the app could be replaced behind the back of my application.
On windows and for desktop software, the risk of a failed install or the user sabotaging their own install without realizing it is more common than the risk of having an outdated version of a library. If the application is mostly or completely offline (e.g. only online for updates or documentation), then the surface area for attack is pretty small. So I'd usually go so far as validating the integrity of the app with a checksum on startup too, and refuse to start if the app isn't byte for byte exactly as expected.
This also obivously means my app has almost no risk of ending up in a package in a Linux distribution too.
Is updating a library with thousands of dependents, without individually testing those dependents, any more feasible?
In NixOS, we try to work the test suites into the build process and rebuilds happen when any dependencies change, so I'd like to think we get (almost) the best of both worlds.
Which coincidentally is why I'm not using Linux right now. :) A move towards static linking in the Linux world would go a long way towards changing that.
(Everything I've read about NixOS seems great, thanks for helping with that—it's just that it also has a large barrier to entry.)
> As opposed to me, as an individual user with no familiarity with the code base, just switching out the pieces and preying nothing goes wrong.
Wouldn't this be the task of your distro's mainteners, not you? (Who will have familiarity with all codebases as well as options to contact the original developers if there is an unsolvable issue)
Sure, and when they’re done, they can recompile the static binary and send me an updated copy.
I don’t want to update until they’ve finished testing the dependents. At least with static binaries, I can update some of the dependents sooner as the maintainers work through their lists.
It doesn't necessarily need to be the upstream developer, it could be some other organization analogous to a distro maintainer. What matters is that someone who is familiar with the software and code base has actually tested the update in a purposeful way.
7.2.4 If multiple products are affected by the same independently fixable vulnerability, then the CNA:
a. MUST NOT assign more than one CVE ID if the products are affected, because they share the vulnerable code. The assigned CVE ID will be shared by the affected products.> I eagerly await some concrete implementations of these new ideas.
I'm working on them now! They are not public because of a licensing issue where I need to consult a lawyer first (thanks, GitHub Copilot), but they will be public as soon as that's sorted out.
Do you mean that you built them using Copilot, or that you don't want Copilot to use them? Or something else entirely?
So it's a human solution, and not a technical one, which means that it can and will fail.
Provided you (and your org) use Go as Google intends then that isn't an issue.
I think it's closer to say that the existance of condoms won't make AIDS and unwanted pregnancies disappear, even if they were 100% effective. Which, if you're trying to eliminate completly AIDS, is a fair claim to make.
My point is that you shouldn't assume that everyone is going to follow the Go guidelines, because some people won't, and you don't want your security to rely on that.
Maybe the problem is have a system with thousands of installed packages? If every application is in its own "space", then a security failure won't affect the rest. If you want the best security, you're going to have to do pretty big tradeoffs. If you're not ready to do them, you're going to have to live with an insecure system. I don't think there is a way around this.
For a system that widely uses dynamic libs, that's fairly easy to do. Check the version of the installed library, along with any patches it might have, and you have a good idea of whether it needs to be updated.
Now, imagine there's a zlib exploit. That's a lot of things in the system that need to be updated. It's so ubiquitous that even in a system that's almost entirely built with dynamic libs, there are a few that use it as a static lib, and you'll have to make sure those are updated too (go look at the last zlib CVE and the major distro updates to it and you'll see all those packages, sometimes a couple days later as they discover them too).
A world where these are all statically compiled (or bundled separately in containers) is much more complex. It's not impossible, but you don't naturally get some of the work done for you. To some degree we're already going this way with containers though.
In which was? Is it because it's hard to know which version of zlib was used in X application, or is it because there is no centralized information about this? Maybe that's an opportunity right there: build a graph of dependencies of a system, and alert when something has been compromized and which are the consequences.
It's not a show stopping problem, but it is something that should be considered and thought closely about. This is a security adjacent problem, and changes in it can have consequences that are larger than "I'm running a library that doesn't have the feature I want".
You mean in the sense that every application that comes with statically linked libraries should also come with some metadata indicating which libraries/versions are in there?
This would work, but then you have the problem metadata always has: It can get out of sync with the actual thing, developers can forget to properly maintain it (or even to provide it at all) etc etc.
I have no idea what actually happened there, I just recall some moderately annoyed sysadmins figuring out how to rebuild go applications by developers who'd since left the company.
(this is not a shot at go, this is merely a "there might already be -some- data out there" wrt your question)
(If you're troubleshooting dylib errors on things you're developing, that's not an end user perspective?)
I don't think I've ever run into DSO problems with any reasonable distro packaging (which, in this case, excludes old Gentoo — current Gentoo doesn't have those problems anymore either.)
In saner OS's where they don't rely on unpaid third party middlemen between the developer and the user this is rarely a problem.
Windows seems to fit your criteria for an OS without unpaid third parties, and there still exists a thing known as DLL Hell.
Sadly ran into an issue with this today. A program I wanted to use I guess forgot to ship the dependencies in the provided binaries of most recent version. I could maybe track them down, but many of not all of them would have to be built from source, which I’d rather not bother with.
All packages that leave the confines of my own systems are built the "correct" way, in a clean minimal chroot with sbuild/schroot/cowbuilder. The docs on how to set that up are pretty good, and after you've done it once it's pretty easy to repeat for extra target distros (e.g. older Debian, Ubuntu.)
What would you point out to me as part of your mission?
[Just to be clear, a package is a package for a particular target distro. There's no such thing as "a .deb for software XYZ". Only "a buster .deb for software XYZ", or "a bullseye .deb for software XYZ", etc.]
I'd argue that if you're not packaging your software or testing your software's dependencies, either you're doing something extremely exotic that lies far outside anyone's happy path or "dylib error" should not even be a keyword in your vocabulary.
https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-...
DLL Hell is not a linking problem, it's a packaging problem.
Right you are. I was also perplexed when I read that non sequitur. The author's reference to DLL Hell also suggests there's some confusion in his analysis of the underlying problem, given that DLL Hell is very specific to windows and at best is only orthogonally related to ABI. The author's references to API changes make even less sense, and definitely cast doubt over his insight into the issue.
It really isn't. Unlike linking problems, where the problem is focused on how you need to fight your dependencies to be able to resolve symbols, DLL Hell has been for over a decade a dependency resolution problem that is solved at the packaging level.
More importantly, on Windows, where sharing DLLs is not a thing, you can simply drop a DLL in the app dir and be done with it. In fact, it's customary for windows apps to just bundle all their dependencies.
Sure, most security fixes do not break the ABI or API. But it only takes one, and then you have a security issue.
So I can understand your sentiment that it was sleight-of-hand, but I argue that it's not.
Also, we only think that most security updates don't break API or ABI because we have no automatic way of checking. It's all thoughts and prayers. I'm a religious person, but thoughts and prayers are not a good way to do computer security.
We think security updates don't break API/ABI because they're backported to all previous major versions that are still actively supported. This isn't "thoughts and prayers", it's work that the packagers and maintainers do. I can't tell you how useful being able to fix a vulnerability in every program is, just think of updating every application that uses libpng on your desktop, or every server application that uses OpenSSL.
I just can't imagine wanting to give that up, because of name mangling ABI issues!
As someone who has been a Unix sysadmin for almost twenty years now, I don't recall every installing a patch or package update that broke the ABI/API.
If one should happen to occur next week, I'll still go with leaning towards linked libraries, as their convenience out-weights (has out-weighted) the once-in-twenty-years occurrence of breakage.
> It's all thoughts and prayers. I'm a religious person, but thoughts and prayers are not a good way to do computer security.
Ditto. But "faith" and "trust" are (almost) synonyms. And I trust the updaters of libraries to be able to get out a security update in short order more than I trust the possibility of all the package maintainers that link to it to be able to coördinate a quick update.
When OpenSSL's Heartbleed occurred it was one package update and restarting of a bunch of processes (helped by the checkrestart utility in the debian-goodies package). That's a lot quicker than updating a bazillion dependents.
I’m looking at you, MySQL, docker, Mongodb, or harbor.
I’m not complaining, those are provided free of charge and we are happy when it works, but semver is really all that matters in this: either you adhere and it’s smooth sailing, or you don’t and then please don’t use semver-looking numbering.
Also there is mature tooling around ABI compliance checking: https://lvc.github.io/abi-compliance-checker/
This is used to document the ABI stability of mature projects like so: https://abi-laboratory.pro/index.php?view=tracker
E: re thoughts and prayers, the mature Linux distro all run extensive revalidation for their security updates. Breaking customer binary applications with a security update is the absolute worst case scenario. For yocto or similar cost free projects, or Linux distros that are source, not binary, that may be a different story. But for sles, rhel, ubuntu it's not thoughts and prayers but a maintenance verification and release process on their side and a monthly bill on yours.
I do for Fedora packages I maintain. There can be false positives, like on symbols that shouldn't be exported, but you can look at the API to start with. (The library author should do that, of course.)
> The ironic thing is that operating systems only supported a stable syscall ABI, rather than requiring programs to dynamically link system libraries, they would have less problems with ABI breaks.
> With a statically-linked binary, you can copy it to another machine with a Linux that supports all of the needed syscalls, and it will just run.
I think this is a misunderstanding. With Linux there is a stable interface at the kernel boundary. Other OSes have their stable interfaces in user space. The difference is mostly irrelevant in this context. In one case you dynamically call into the kernel, in another case you dynamically link a user space library. Either way the code you're calling isn't static but the interface is.
Static linking for future-proofing/compatibility has never been a valid argument in my experience. Not only it is much easier to fix (and debug) a broken dynamically linked binary than a statically linked one, I would even dare to say that statically linked binaries break much more frequently than dynamically linked ones when you change the rest of the system. Even when you include the breakage from libraries changing under the feet of the dynamic binaries.
> assuming the hardware and kernel do support it
Assuming kernel support is doing a lot of heavy lifting there. To put it another way, why is a syscall better than any other stable ABI? If both kernel and stable library are distributed together then why should it be considered different?
A "library OS" like FreeRTOS does exactly that: the kernel is just another library, with functions that you call like any other static dependency. You can only run one process (the OS & your userspace code are that process).
Really the "fully static" side is only seen in practice in RTOSes and similar embedded systems work. Being able to dynamically load more than a single process is just too handy to give up entirely. I don't think the opposite extreme has even been tried (every single symbol in its own .so, even within the kernel), the overhead would be ridiculous.
A syscall may not be inherently too different from any other interface, but a single interface is certainly different from two interfaces.
Kernel support is actually easy: since Linux hardly ever removes syscalls, just build your fully-static executable on the oldest Linux you have, and then deploy it on all of your relevant machines.
The syscalls used by all of your libraries, if they work on that oldest Linux, would work on the newer ones.
In fact, this is why AppImage "Best Practices" includes building on the oldest system. [1]
[1]: https://docs.appimage.org/reference/best-practices.html?high...
This is exactly what I was getting at. Thank you.
A "static interface" isn't a thing. There's stable and unstable interfaces, and static and dynamic linkage. Not strongly related.
[1] https://github.com/ThrowTheSwitch/Ceedling/tree/master/plugi...
For users, it would be better if the application was statically linked, at least in terms of startup cost. But because developers do the edit/compile/debug cycle much, much more often than users start the application, we opt for dynamic linkage.
Drepper's article is from 2006. "DLL Hell" was beyond fully understood at that point, certainly by Drepper and large numbers of other well informed programmers.
In everything in the post, I attempted to give Drepper the benefit of the doubt, though there are cases where I don't want to since he sounds (to me) paternalistic in the way Apple and Microsoft do.
So perhaps he understood, but I didn't want to assume that, especially since it might inflame the discussion, which is exactly what I was trying to avoid doing (again).
Also, there are some additional facets of DLL Hell which happened in the Windows of the mid-90s which are not as relevant to Linux. What made DLL Hell so bad there was that installing any random program could replace globally shared libraries, sometimes even with an older version. That is, installing a game could make an unrelated productivity app stop working, because the game helpfully installed newer (or older!) versions of the shared libraries it needed, into the same globally shared directory in which nearly all DLLs lived. It got so bad (even DLLs which came with the operating system were being overwritten) that Microsoft IIRC initially introduced a system which detected when this happened, and replaced the DLLs again with a clean copy it had stashed somewhere else (and later, made these files more directly protected).
That's before considering the disaster that is in-process COM servers; presenting a standard "open file" dialog, or doing some printing, is enough to make Windows load arbitrary DLLs into your process (shell extensions and/or printer drivers), and these often don't have the highest code quality. And then there are some things which inject arbitrary DLLs into every process...
Compared to that, the dynamic linking issues in the Linux world are much more bearable. You don't see arbitrary programs overwriting the global copy of something like libgtk or openssl or zlib or libc, the only arbitrary dynamic libraries being loaded into a process are things like the NSS ones (usually from a small well behaved set) or plugins from the graphics libraries (also usually from a small well behaved set), and the only dynamic library being injected into every process is the vDSO from the kernel.
> Ulrich Drepper claims, rightly, that you only need to update a shared library to apply security fixes to all executables that rely on that library.
> This is a good thing! It is also a good vision.
> Unfortunately, it is only a vision because it’s not the whole story.
> As of yet, there is no way to automatically determine if the ABI or API of a shared library changed between updates. This is called DLL Hell.
The final two words are a link to the wikipedia page on "DLL Hell". The final sentence of the intro section there states:
> DLL Hell is the Windows ecosystem-specific form of the general concept dependency hell.
That is, TFA defines "DLL Hell" using a wikipedia page that explicitly states that it is a Windows-specific version of a more general problem.
Drepper's article from 2006 fully tackles the general problem, albeit without (as the TFA puts it): "a way to automatically determine if the ABI or API of a shared library changed between updates." Drepper's primary suggestion there is a naming/versioning scheme which describes specifically whether the ABI/API has changed in ways that matter for shared linkage. It's not automatic - that part is true. But is is a solution, widely used in Linux and *nix more broadly.
Did Drepper address the Windows specific parts of "DLL Hell". He did not, but that's because "DLL Hell" is Windows specific (as other comments here fully clarify), and he was not writing about how to fix the full scope of that particular nightmare. He likely understood "DLL Hell" as well as anyone, nevertheless.
The reasoning behind your differing assumption is beside the point, even though your reasoning is more likely to be true.
The TFA's claim is that Drepper didn't get all the issues with DLL Hell. My claim is that Drepper did get all of the issues, and addressed those that were not Windows-specific (e.g. per-process COM servers etc.)
Specifically, your post said:
> DLL Hell was well understood in the mid-90s. I think you need to give him more benefit of the doubt on this point.
The substantive question about whether he adequately addressed the issues with dynamic linking is beside the point. Feel free to agree or disagree with the above poster. Not relevant.
The post I am responding to is about, given ghoward's position that X did not adequately address topic Y, whether it is "giving X the benefit of the doubt" to further assume (a) X did know about topic Y; or (b) X did not know about topic Y.
Because (a) implies dishonesty but (b) merely implies ignorance, "giving X the benefit of the doubt" means you should assume (b) until proven otherwise.
If you want to respond to other topics, feel free to hit the reply button below someone else's post.
> I don’t think Ulrich Drepper could have foreseen all of the problems with DLL Hell (though there were signs on Windows),
I replied to their comment, and substantively responded to something that was actually in their comment.
How is that inconsistent?
Apologies.
I read that essay a while ago and reread it before publishing my post.
I don't think the author makes a good case. They make bare assertions without examples and without evidence, so it's unconvincing.
For example, they say that a "Considered Harmful" essay:
> often serves to inflame whatever debate is in progress, and thus makes it that much harder for a solution to be found through any means.
But this could be true of any essay.
More to the point, I think that the tone of the essay or post matters more. Case in point, my "Dynamic Linking Needs to Die" post probably inflames the debate more than this new "Considered Harmful" post.
Maybe "Considered Harmful" essays have a reputation for inflaming the debate, and there's an argument to be made for that. But I personally don't think my post is one that follows that reputation.
Also, because I was writing a post about a pre-existing "Considered Harmful" essay, it was natural to use it, especially since I needed to make it clear that my previous post is harmful too.
Neither your post title nor your content bothered me but just as fyi... some people are very irritated by "snowclones". My previous comment with various examples: https://news.ycombinator.com/item?id=19071532
I agree with most of the article but the title sucks!
Also I'm not sure how you wrote so much about using LLVM IR as a portable executable format without mentioning WebAssembly or PNaCl once!
> I agree with most of the article but the title sucks!
You are correct. I don't claim to be creative, especially with post titles. To be honest, I'm surprised this post got as much traction as it did.
> Also I'm not sure how you wrote so much about using LLVM IR as a portable executable format without mentioning WebAssembly or PNaCl once!
Because, in my opinion, they will not solve the problem.
WebAssembly is too constrained.
PNaCl uses LLVM, which would need to be redesigned to do all that I want.
Dynamic linking with core system libraries which are guaranteed to exist(!) is fine though, for everything else dynamic linking mostly has downsides.
Also: it's only Linux where this is an issue, because it's nearly impossible to build small self-contained executables which run on another Linux machine, thanks to glibc's versioning mess (at least there's MUSL as an alternative to create portable command line tools).
But to be fair, dynamically linked GL wouldn't be a problem, if this wouldn't also pull in a dynamically linked glibc (it always comes back to glibc unfortunately).
The problem is that the "graphics driver layer" (except a small amount of security-sensitive pieces) runs in user space, in the same process which is doing the OpenGL/Vulkan/Metal/etc calls. The way modern graphics hardware works is that the application writes a very complex and hardware-dependent set of command buffers, which are then submitted to the hardware, and only this final submission goes through the kernel (for security reasons). If the application had to call the kernel for each and every step of the creation of the command buffers, it would kill the performance.
Reading https://golang.org/doc/go1.11#runtime, that’s fairly recent (since go 1.11)
Unfortunately, building on the oldest common ancestor will still not save you from ABI breaks. An example is [1]. If the size of `intmax_t` changes between that oldest ancestor and the machines you are running on, get ready for pain.
https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packag...
I read that article when it came out, and it felt disingenuous.
mgorny helps maintain Gentoo, which of all distributions, does rebuild software as necessary.
It would be a simple change to portage to rebuild all dependencies of a library when that library is updated, regardless of if the user uses static linking or not. In fact, there is a project to have a statically-linked musl-based Gentoo.
So it's possible and (I would argue) would be an easy change. mgorny would not have to worry about much because he could, in essence, treat all libraries as dynamic libraries. The only pain to be felt would be by users who would see their build times increase (to rebuild dependents).
As a user, I would happily take that tradeoff. I already build my browser, which takes 5-8 hours; I'll happily rebuild dependents of updated libraries.
This is ridiculous. I most definitely do NOT want to rebuild literally EVERY SINGLE PACKAGE on my system the moment they fix a typo in libz.
With static linking literally every other Gentoo upgrade becomes a world rebuild. See Nix. It is just insane and the main reason I don't use NixOS (without a 3rd party binary cache/host) for anything other than very small systems.
I understand. You don't. I do.
What my post is arguing for is not one or the other, but giving the user the choice.
I think you would be okay with making the choice of not rebuilding everything, and I would make the opposite choice.
I want to live in a world where we can both choose.
I believe e.g. freebsd's portmaster takes a -r switch to mean "rebuild this and all its dependents" (I presume you meant revdeps here), which gets you quite a long way.
No change needed. It can already be done with the special ":=" dependency slot: https://devmanual.gentoo.org/general-concepts/dependencies/#...
If I installed software through my OS package manager, where else would I get security updates from?!
libabigail is one tool for checking that a library's ELF versioning hasn't lied about ABI compatibility when you're building packages, for instance.
Unfortunately people typically don't follow Drepper's advice (as far I remember the article) on building shared libraries, particularly using symbol versioning a la glibc.
Yeah, there are reasons to use dynamic linking, but I'm still not sure why dynamic linking gives you better profiling and tracing.
> Unfortunately people typically don't follow Drepper's advice (as far I remember the article) on building shared libraries, particularly using symbol versioning a la glibc.
I mention this in the post, but I quoted another person, Ian Lance Taylor, who wrote the gold linker, as saying that symbol combining, a portion of symbol versioning, is an unsolved problem.
Even if developers don't follow Drepper's advice, I wouldn't put the blame on them because of that fact; I would argue that it's more accurate to say they can't really follow his advice.
It's not so much that the tracing becomes better, but that it becomes feasible at all. Two specific situations come to mind, both MPI-adjacent. (1) Running a PMPI-based tool on code you can't recompile yourself (e.g., you need a Q clearance to see the source, but not to actually execute it--weird I know, but not that uncommon in the DOE labs.); and (2) running multiple PMPI-based tools simultaneously which are composed at runtime via PnMPI.
That includes not having access to the source because of security clearances. As far as I am concerned, your superiors in the DOE are adversaries to you and your machine.
Other people will have different viewpoints on that, and that's fine.
As I understood it, there actually has been a push for free software solutions on CORAL systems, but I don't remember where that came from.
If you don't have symbol versioning, and you want to statically link something with parts that depend on incompatible versions of a library, you're obviously stuffed.
That seems to be the Go and Rust default approach, or close to it. Link libc and such, but build in more obscure things.
The idea of dynamically linking everything doesn’t scale. It asks too much of distribution maintainers and makes them the choke point for every little upgrade.
I think Go tried to go fully static but ran into problems as most mainstream OSes except for Linux do not provide a stable system call interface, so applications have to link at least one dynamic library. The Linux system call interface is stable, but expects a C style stack layout and that also bit the Go devs. multiple times. So you can probably go fully static if you are only targeting Linux.
There's yet another problem Go ran into on Linux: resolving hostnames. The official way to resolve hostnames in a glibc-based system is to read /etc/nsswitch.conf, dynamically load the libraries referenced there, and call each of them in the correct order. Go tried to emulate that mechanism, going through glibc only when it saw a NSS module it didn't know, but that caused issues on musl-based systems which used a different mechanism (https://github.com/golang/go/issues/35305). And it cannot ever be fully static on glibc-based systems, since there's always the chance of someone adding a new module to /etc/nsswitch.conf (for instance, some Linux distributions have now added the newer "resolve" module to that line, so Go would have to implement yet another one to keep being fully static).
The Rust approach of dynamically linking to the shared C libraries (statically linking only the Rust libraries) avoided all these issues.
Do you want the GUI toolkit linkage to automatically include every object module for every possible image file format in your static image, or do you want the GUI toolkit to use runtime dynamic linkage (explicit or implicit) to import only the modules required for the image formats actually encountered?
This is the blog version of Re: Re: Fwd: Re: in email and has to stop here.
Ok fine, explaination: "Re:" are not art. "Re:" are not added deliberately and for intentional artistic effect, and are not references (despite the literal) or homages.
See my reply to https://meyerweb.com/eric/comment/chech.html at https://news.ycombinator.com/item?id=28737279 .
https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_W...
I quote that email twice in my post. I decided not to quote that part of the email just in case it might inflame the debate.
But yes, he did say that.
Encryption libraries.
If OpenSSL, or LibreSSL, or Go's encryption modules, or any gigantic encryption library has a vulnerability, you basically have to recompile, distribute, and then have your users download, every single god damn networking program.
But it's worse than that. Because everyone wants to ship a statically compiled Go app, nobody packages for the distros. So now every user needs to go track down where they downloaded each of their static Go apps, safely download the update, verify its checksum or signature via a 3rd party, and upgrade the app.
That is a logistical nightmare. Not only are apps going to remain vulnerable for way, way longer, more users will be compromised by phishing and other attacks because there's no package manager to safely automate all these updates (distros sign and verify packages for you).
I've actually written a Shell quasi-package manager/installer just for statically compiled apps (cliv) so in theory that problem could be somewhat solved... But nobody even knows about that program, so the manual update problem still stands.
Good, then you'd know that these 50 of your networking apps have each been tested by their developers' test suites and have no bugs that have been introduced as a part of this dependency. Essentially, there would be a server farm out there that'd run all of the tests so you wouldn't have to deal with prod breaking because of some new dependency version not taking every possible configuration into account.
Of course, that implies:
- that there are automated builds in place
- that there are automated tests in place
- that both of the above are run regularly, on every dependency update
- that both of the above are actually meaningful (think close to 100% functionality coverage in tests)
> But it's worse than that. Because everyone wants to ship a statically compiled Go app, nobody packages for the distros.
In my eyes, that just adds to the above - there are so many deployment targets out there, all of these different packaging systems that are used by different distros and all have numerous idiosyncrasies as opposed to a single good package manager, that people have been fed up with it and now just either ship Docker containers or expect you to download binaries.
Something compiled against 1.1.0a (Sep 2016) has a fair chance of running on a system that has 1.1.0l (Aug 2021). 5 years of binary compatibility is nothing to be sniffed at for a C library.
https://abi-laboratory.pro/index.php?view=timeline&l=openssl
The last one I remember. It was moving the API that did version negotiation from recommended to deprecated and then removed status. But yeah, the API itself didn't change. All that changed is the recommended way to interact with the library, and the old one removed.
I'm not saying it's great, but it's how it works.
Well this is the explanation for why people are static linking! They don’t want to run someone else’s entirely untested code and bear all responsibility for any problems.
If I'm relying on it, it's getting tested and I'm not going to dork with it without good reason once it is.
I think you are strawmaning my arguments.
Personally, I would include encryption libraries in the "system libraries" (I used that term on purpose) that could be dynamically linked.
However, even if they are, you still want to recompile every user whenever that library is updated because of possible ABI and API breaks. And some of those encryption libraries have a history of API breaks.
Also, I am building the ideas I laid out in the post, which should make it easier to handle dynamically-linked and statically-linked encryption libraries.
More of the choices that get made during autoconf needs to be made lazily at link time. That would make executables and dynamic libraries more like containers or lego blocks that could be rearranged without compilation. Which would fix a lot of issues that people hate about package managers and bloated distros along the way.
I'm open to answering questions, and I'll be hanging in the thread.
> On average, dynamically linked executables use only 4.6% of the symbols on offer from their dependencies. A good linker will remove unused symbols.
Percent of exported symbols used is a terrible proxy for percent of library functionality used.
As an example, an SDL hello world program [1] uses 3 functions out of 750 exported by libSDL2. Does that mean it only uses 3/750 = 0.4% of libSDL2's functionality? Of course not; when I tried compiling it against a static libSDL2 instead of dynamic, the executable's binary size increased by a full 70% of the size of libSDL2.so. [2]
Now, on one hand I admit that's an extreme example, as a real SDL program would use a higher number of functions. And SDL itself is not very amenable to dead code stripping; a good chunk of the hello-world program is for things like sound or input that the it doesn't actually use.
Still, even the hello-world program legitimately needs a bunch of functionality, including support for both X11 and Wayland, and setting up hardware accelerated surfaces using OpenGL. And a real program would need more. Also, I'm not even counting the long list of shared libraries linked by SDL (ranging from libX11 to libFLAC); a SDL-based program will use 0% of their symbols (since it doesn't them directly), but may need a substantial portion of their functionality.
More importantly, this general pattern, where exported APIs are just the tip of the iceberg, applies to many libraries.
I tried a similar experiment with zpipe, the official example program for zlib – a much smaller and shallower library. It ended up using 23% of symbols but 53% of the implementation.
On the other end of the spectrum, the problem would be far worse with a real GUI library, where even, say, a simple text input box truly requires massive amounts of functionality including display, layout, Unicode font rendering, menus, internationalization, and keyboard and mouse input. Plus, of course, a variety of input method editors if you want to support languages like Japanese.
In a statically linked world you would probably want to implement most of that as IPC to shared daemons. Such a design could work; there isn't much that really needs the performance benefits of being in-process. But no major GUI libraries are written that way, because in a world with shared libraries there's no need.
[1] https://gist.github.com/comex/c64b5a7e409d5d48ad6ebf37ec068b...
[2] Test harness also at [1]. This is with LTO; without LTO the percentage is much higher. I also had to disable SDL's "dynapi" feature that makes static linking secretly actually dynamically link. Which they put in for good reasons, albeit mostly applicable only to proprietary software where the user can't just recompile/relink.
On Darwin derived systems (macOS, iOS, tvOS, watchOS) we have invested heavily in dynamic linking over the last two decades. That includes features we use to improve binary compatibility (things like two level namespaces (aka Direct Binding in ELF) and umbrella frameworks), middleware distribution (through techniques like bundling resources with their dylibs into frameworks), and mitigate the security issues (through technologies like __DATA_CONST).
Meanwhile, we also substantially reduced the cost of dynamic linking through things like the dyld shared cache (which makes dynamic linking of most system frameworks almost free, and in practice it often reduces their startup cost to below the startup cost of statically linking them once you include the cost of rebasing for PIE), and mitigate much of the rest of the cost of dynamic linking through things like pre-calculated launch closures. It does not hurt that my team owns both the static and dynamic linkers. We have a tight design loop so that when we come up with ideas for how to change libraries to make the dynamic linker load them faster we have the ability to rapidly deploy those changes through the ecosystem.
As I explained in here[1] dynamic linking on macOS is way faster than on Linux because we do it completely differently, and because of that it is used way more pervasively on our systems (a typical command line binary loads 80-150 dylibs, a GUI app around 300-800 depending on which of our OS you are talking about). IOW, by mitigating the costs we drove up the adoption which amplifies the benefits.
And it is not like we have squeezed all the performance out of the dynamic linker that we can, not by a long shot. We have more ideas than we have time to pursue, as well as additional improvements to static linking. If you have any interest in either static or dynamic linking we're looking for people to work on both[2].
[1]: https://www.realworldtech.com/forum/?threadid=197081&curpost...
[2]: https://jobs.apple.com/en-us/details/200235669/systems-engin...
(edit: fixed link formatting)
I'm curious: did you read to the end of the post where I lay out ideas about how to get the benefits of both in one?
* Distributing IR applications: We do a form of this by supporting bitcode for App Store submissions on iOS, tvOS, and watchOS. For watchOS it was a great success in the sense that it allowed us to transparently migrate all the binaries from armv7k (a 32 bit ABI on a 32 bit instruction set) to arm64_32 (a 32 bit on a 64 bit instruction set), but we had to carefully design both ABIs in parallel in order to allow that to be efficient. It also introduces serious burdens on developer workflows like crash reporting. Those probably are not significant issues for people deploying binaries to servers, but it can be pretty difficult for developers trying to aggregate crash statistics from apps deployed to consumer devices. It also causes security issues with code provenance since you have to accept locally signed code, have the transforms performed by a centrally trusted entity who signs it, or you have to limit to your optimizations to things you can verify through a provable chain back to the IR.
* Split stacks: We don't do this per se, but we do something semantically equivalent. When we designed the ABI from arm64e we used PAC (pointer authentication codes) to sign the return addresses on the stack, which means that while you can smash the stack and overwrite the return pointer, the authentication code won't match any more and you will crash.
* Pre-mapped libraries: We have done this since then Mac OS X Developer Releases back in the later 90s, though the mechanism has changed a number of times over the years. Originally we manually pre-assigned the address ranges of every dylib in the system (and in fact we carefully placed large gaps between the __TEXT and __DATA of the library and have the libraries in overlapping ranges so that we could place all the system __TEXT in single adjacent region and all the __DATA in a second adjacent region that way we could exploit the batch address translation registers on PPC processors to avoid polluting the TLBs). When the static linker built a dylib we look in a file with the mappings and build the dylib with segment base addresses that we looked up from the that manually maintained list, and when the OS booted we would pre-map all the libraries into their slots so every process just had them mapped.
That was a huge pain in the neck to maintain, since it meant whenever a library grew too large it was would break the optimizations and someone would need update the file and rebuild the OS. It also only dealt with rebasing, to deal with binding we locally ran update_prebinding which users hated because it was slow, sysadmins hated because it means we routinely rewrote every binary on the system. That was also before anyone had deployed ASLR or codesigning.
Nowadays we use the dyld shared cache to achieve similar ends, but it is a far more flexible mechanism. We essentially merge almost all the system dynamic libraries at OS build time into one mega-dylib that we can pre-bind together and sign. We also use VM tricks to allow the system to rebase it on page in rather than the dynamic linker doing any work, and we pre-map it into every address space by default.
We even perform a number of additional optimizations when we build it, such as analyzing the dependencies of every executable in the base system and pre-calculating a lot of the work dyld and even the dynamic runtimes like ObjC would have to do in order to avoid doing them during app launch.
So in short, I think your ideas there have more merit than you perhaps suspect, and from experience I can say if/when you implement them you implement them you might find your views on the trade offs of dynamic vs static linking change.
And after reading your response, I might give up on at least one of those ideas.
Maybe a stupid idea but has there ever been thought of distributing applications with source code included?
That way you could have a system that automatically rebuilds the application if a static library changes.
Meaning when a user sees “updating” it’s actually rebuilding the app underneath.
In a case like that static linking is not only unstable, as in Windows or macOS, but also impossible.
And since that has security benefits we will probably see it being migrated to other platforms in the near future.
Another resources is https://sta.li/ I thought there was a Archlinux-based attempt, but can't find references to that.
http://sabo.xyz/ More of a complete distro, but the dev himself says he still uses a couple of programs dynamically linked.
As I see it, the entire world of Docker containers exists primarily to solve the problem that static linking already solved.
Something like Nix would work as well, and I think I would love Nix if I knew how to use it. But I don't, and I don't really have the time to learn.
At least a container provides more function than mere bundling, but a lot of containers actually get used merely for bundling.
IMO, when stuff is self-contained, everything becomes easier! This isn't to say some stuff like OpenGL can't be dynamically linked, but some day I'd love to use a distro with a static-first approach, because right now it simply isn't an option.
I am passionately in-favor of bundled. Unbundled distribution is what has led to software being so outrageously impossible to execute that the only sane path to distribution is to create a bundled docker image. https://xkcd.com/1987/
I have "fond" memories of running perebaseall on cygwin that this sentence reminded me of.
Why not just import external dependencies into your project at a function/class level rather than at a package one: if you only use FooPackage.BarClass and FooPackage.bazMethod(), you should be able to choose to make your project depend on just those two things, ideally in a completely transparent way by your IDE.
Then having to manage the full scope of packages and their versions becomes less relevant, because the IDE can check whether a new version has been released in the package manager of choice, check whether those two things have changed and if they have, then demand that the developer have a look at the code changes to refactor code if necessary, otherwise not requiring any action on their part. Furthermore, if you ever need to migrate to a different package or even rewrite the functionality itself, you could just look at the few signatures that you use, rather than having to remove the package entirely and see what breaks in your IDE. Why can't we have itemized lists of everything that's called and everything that we depend on, as opposed to the abstraction of entire packages?
Better yet, why even depend on binary blobs or hidden code (that's ignored by your IDE) in the first place? Why not just download the source for every package that you use and upon updates be able to review the code changes to the package much like a regular Git diff?
Of course, this would probably require getting rid of reflection and other forms of dynamic code, which i fully support, since those have never been good for much in the first place and only destroy any hopes of determinism and full control flow analysis.
As for the possible counterargument of this being hard to do: it wouldn't really be with more granular packages. Instead of trying to shove an ecosystem inside of a single package, why not split it into 10-20 bits of reusable code instead? Smaller packages, which would be easier to review and manage.
Context: i dislike how Spring and Spring Boot in Java force a huge ecosystem of fragile dependencies upon you, with their reflection which ensures that your apps will break at runtime, for example, when moving from Spring Boot 1.5 to 2.0. Furthermore, in the JS world, the node_modules folders are probably 10-1000x too large for what's actually necessary to display a webpage with some interactive behaviour.
Disclaimer: i have no delusions about any of the above being technically feasible right now. Perhaps "Why?" would be a good question to ask. In my eyes, perhaps the industry is too set in its current ways and as long as we don't have languages and entire ecosystems that approach established practices from a wildly different angle, no one can actually contest the decades of already built packages and tools.
On the bright side, WireGuard kind of displaced OpenVPN somewhat and there are numerous benefits to it being smaller, which is a good example of getting rid of bloated software. Furthermore, Nix may or may not do that to alternative approaches to package management, but its usage still remains really low.
In my eyes, the only long term solution is to be able to tell exactly what's necessary for your code to build and work, and to test every dependency update that comes out against this automated process. Static dependencies but at the speed of updates of dynamic dependencies. Then you'd just have to wait for a day until the devs fix the newest regressions of a new dependency release before getting a new statically linked app, as opposed to using dynamic linking and finding that some dependency breaks your app as it happens in prod.
That's how it works already with static linking (just under the hood), the linker will discard unused code and data (with LTO this happens automatically for all dead code and data, otherwise some special compiler/linker/librarian-options may be needed).
A DLL on the other hand cannot predict what API functions will actually be called on it, so everything must be included.
TFA is about programs that require a linker at either compile and/or run time. The references in the rest of your comment to Java and JS suggest that your experience with programming in languages that require linking against compiled libraries might be limited.
What you describe is precisely what the linker is for! It looks at the code you wrote (and compiled), sees that it calls FooPackage.bazMethod(), and then goes and looks for that call along the linker search path using a combination of implicit and explicit library names. If it's the static linker, it imports the code for that call, and then recurses to check the dependencies of FooPackage.bazMethod(). If it's the dynamic linker, it does the same, except differently (runtime pointer swizzling etc. etc)
When you link your program against libfootastic.a, the linker is doing exactly what you suggest, and in most IDEs that step is going to happen automatically. When you link your program against libfootastic.so, the linker will do exactly what you suggest, but at runtime.
> Better yet, why even depend on binary blobs or hidden code (that's ignored by your IDE) in the first place? Why not just download the source for every package that you use and upon updates be able to review the code changes to the package much like a regular Git diff?
Many large end-user desktop applications do precisely this. For Ardour (I'm the lead dev), our build system downloads 80+ source code packages and builds them locally to create a 2GB installed dependency tree of compiled libraries and ancillary files. There are a lot of arguments against doing this, and some arguments in favor.
However, we can't compile each file from each dependency as part of the Ardour build because the build systems for those dependencies are complex and independent. There's no way to force the developers of libfftw (Fastest Fourier Transform in the West) to somehow prepare their code structures so that we can trivially replicate their entire build system as part of ours. If there was only a single build system for EVERYTHING, sure, maybe this would be feasible. But there isn't, and I'm completely certain that there never will be.
But this is pretty much what a static binary already does, did, 40 years ago. The developer has access to the full universe, but the only thing that ends up in the binary are the bits they actually used. Problem solved, 40 years ago.
https://erlang.org/pipermail/erlang-questions/2011-May/05876...
Honestly, that reminds me of the PHP standard library a bit, which feels oriented towards procedural programming at times. Now, the language and the actual implementation of the functions aside, i found that pattern really pleasant to work with, especially since it let me work more in line with functional programming principles as well - pure functions and no global state (even if passing data around was more difficult).
Now, i won't say that the approach fits every project or domain out there, but at the very least having a clear idea of what your code depends on and what's going on inside of your code base is an idea that i still stand by. Sadly, the only tool that i found that at least tries to make this easier was Sourcetrail, which is essentially dead now: https://www.sourcetrail.com/