Dynamic linking
drewdevault.com
drewdevault.com
That's correct, but also very misleading and leads to the wrong conclusion.
The dynamically linked library has references to itself, externally visible or not. It would be wrong to claim that Application.run(); only uses a single symbol of a library.
> A good linker will remove unused symbols.
With LTO or -f{function,data}-secions + --gc-sections any linker will do. Without those options no linker is allowed to. I believe that this the reason why static libraries are usually shipped as separate object files (.o) within ar archives (.a), as those were only linked in on demand.
Yep. One function per C/obj file for smallest static binary possible.
Worse, I think it would have been easier to measure the size in bytes of binaries, or, with a bit more effort, resident memory size.
For more on this, I highly recommend these two posts, which show how modern symbol tables optimize for symbols NOT being found within a given shared object via the use of bloom filters.
Additionally, a linker not be able to remove symbols that are unused at runtime but are still (transtively) referenced statically. For memory mapped dynamic libaries those code pages will not be loaded unless needed.
For example, statically linking libstdc++ for any C++ program which uses iostreams (or anything else infected with std::locale) will mean including code to format and parse e.g. monetary amounts even if your program never uses this functionality.
That's a very misleading reference and graph. First of all, what did they expect to find? As you add more executables, of course the % usage of a library will decrease.
e.g. say I have a networking library on my computer, and, in a perfect world, all my installed network tools link against it. But now I install Gnome, and my machine has hundreds more binaries. Not all of the binaries will do networking stuff, so the % usage of the networking library goes down. But that doesn't mean that the networking library is not being shared as well as it could be.
A much better metric would be to count, for each shared library on a machine, the number of programs that link against it. If only one program uses a shared library, then that means the 'shared-ness' is not being used. If more than one program use it, then the library is being effectively shared. But the actual count, whether it is 200 users or 20, doesn't mean anything more. That's why comparing all libraries against libc's usage shows nothing useful.
It's understandable that games, especially proprietary ones, distribute statically-linked binaries ensuring any third-party dependencies will be present and be a compatible version. But the value of that decision tends to diminish with time, as those external dependencies are frequently the pieces interfacing with the system/outside world, which keeps changing, leaving such dependencies behind to atrophy at best or become vulnerable/incompatible/broken at worst.
I don't personally think it makes sense to approach this so dogmatically. Static linking makes sense in the right circumstances, so does dynamic linking. For general-purpose operating systems, it seems obvious to me that you'd want most higher-order userspace programs dynamically linked. I want my openssl updates to touch a single library and affect all installed ssl-using programs, for example.
Having said that, I do wish the average linux distro still statically linked everything in /bin and /sbin. It was nice to still be able to administrate the system even when the dynamic libraries were hosed. At some point it was changed to just a single static binary; sln for static ln IIRC, assuming you'd be able to fix your dynamic libraries with some symlinks if they were broken, if you happened to have a shell running and could navigate using just builtins. It was already an impossible situation, but even that seems to be gone nowadays.
It's a more nuanced issue, taking an "everything dynamically linked!" or "everything statically linked!" approach strikes me as just another form of ignorant extremism.
[0] http://hg.libsdl.org/SDL/file/2fabbbee604c/src/dynapi/SDL_dy...
This argument came up back when Solaris 10 was in development and the project to get rid of static link archives for system libraries came up (search for Solaris "unified process model"). The disposition of this argument was that if your libraries are damaged (e.g., someone unlinked them or renamed them out of the way, or maybe ld.so.1 itself), well, the dependent utilities in /bin and /sbin themselves could have been damaged too, so you can't know the extent of the damage, and it's not safe to continue -- you have to use boot media to repair the damage, or reinstall. And, of course, the packaging system has to be safe, but that's not a lot to expect of a packaging system (is it??).
To my knowledge there were no subsequent customer calls about this.
SCO Group was renamed as such from Caldera in the early 00's and was the people suing Linux users for copyright infringement.
For something like SDL that has a stable ABI while providing user-visible improvements in newer versions, dynamic linking makes sense. SDL's solution isn't only for static linking but also for bundled dynamically loaded SDL copies. I am not really convinced that SDL's approach is perfect though as anyone statically linking SDL can just disable this functionality and this feature has already caused problems requiring you to manually substitute the loaded libSDL.
For libs that don't interact with the system, dynamic linking does not really provide an advantage for binaries that are distributed on their own.
The upgrade problem has almost nothing to do with download size. The real problem is that you have > 100 binaries which depend on those libraries, and instead of having the library authors go and update the library, you need each team responsible for one or more binaries to go and take the new library and release a new version of their binary.
And then, when you want to check if your system is safe from Heartbleed, instead of checking if you have libopenssl > 1.0.1g, you need to check if bin1 > 1.2.56 or > 0.6.89h, bin2 > 5.76.1, or > 4.6.215,... bin100 > 1.67.89.
And of course, if one of them does NOT have a newer version compiled with the patched library, you need to fix it yourself, and maintain a patched version of the binary. Assuming that you even know that binary had been linked to the vulnerable library.
This is a solvable problem. Package managers such as Nix and Guix rebuild packages if any of their transitive dependencies have been changed.
The difficult part are now language-specific ecosystems that use lock files to lock all their dependencies. Traditional C/C++ programs, either statically linked or dynamically linked, have a dependency graph such that e.g. each program that uses OpenSSL has the same package definition in their transitive dependencies. However, e.g. Rust programs may have different versions of the same crate locked.
(There are solutions to that, but there is still work to be done.)
You don't have to, most NixOS/Guix systems use binary caches. So, their build clusters do the work for the packages included the nixpkgs/guix package sets. If your organization builds their own packages in top of that, you can use a CI plus your private cache (or something like Cachix).
Yes, Nix/Guix and Spack are up to my knowledge the only systems that got that right. The centralised recipe repository (and their functional nature) make scratch recompilation reliable and easy.
Now good luck to get that with most lock-file based package managers like npm, cargo or pip with ~1000 packages in your dependency tree that hardcode their dependency number...
The distributed approach of some package manager often come with a security cost unfortunately. And that's a problem for static linking.
I don't know about spack or Guix, but nixpkgs has buildRustCrate, which builds each Rust crate dependency as a Nix derivation (it does not use Cargo). In this kind of setup it is possible to override specific crate versions across all packages that use a specific crate.
Unfortunately, currently most Rust-based packages in nixpkgs use buildRustPackage, which does not follow this approach [1].
Wait, so you're telling me dynamic libraries are a way for distros to ship their org chart?
Maybe it's time in 2020 for automated and deterministic builds to allow anyone to ship a library update and have dependent packages being rebuilt automatically if needed.
I'm not sure what you mean by this. The libraries are not maintained by the distro, they are maintained by the library maintainers; the distro just distributes them when new versions are available, after testing.
> Maybe it's time in 2020 for automated and deterministic builds to allow anyone to ship a library update and have dependent packages being rebuilt automatically if needed.
This would be nice, but it is not there yet for most languages/pacakges; and you still need to solve the problem of who will actually run those automatic builds.
- dynamic responsive remove bug: Positive/neutral. Team X would have done it anyway.
- dynamic unresponsive remove bug: Positive.
- dynamic responsive add bug: Negative. Team X will see the bug but only be able to passively warn users not to use Y version whatever.
- dynamic unresponsive add bug: Negative. Users will be impacted and have to get Y to fix the error.
- static responsive remove bug: Positive/neutral: Team X will incorporate the change from Y, although possibly somewhat slower (but safer).
- static unresponsive remove bug: Negative. Users will have to fork X or goad them into incorportating the fix.
- static responsive add bug: Positive. Users will not get the bad version of Y.
- static unresponsive add bug: Positive. Users will not get the bad version of Y.
Overall, dynamic is positive 1, neutral 1, negative 2, and static is positive 2, neutral 1, negative 1. Unless you can rule out Y adding bugs, static makes more sense. Dynamic is best if "unresponsive remove bug" is likely, but if X is unresponsive, maybe you should just leave X anyway.
You are of course right, though, and writing it down like this can be useful.
If you are not using an apt-style package manager then the program must include all of its dependencies (except ones that are guaranteed to be present on the platform, which is none on Linux and a few on Mac/Windows), and you will receive bug fixes when that program is updated, whether or not it uses static linking.
Static/dynamic linking does not affect how likely you are to get bug fixes in any way as far as I can tell.
Ironic considering that the package manager/repo model is largely implemented as a solution to issues with dynamic linking and yet makes getting up to date software from the developer much harder by introducing a third-party maintainer into the process.
So the difference in effort between changing the static library and changing the dynamic library is... maybe not nothing, but not nearly as high as I was assuming.
- static responsive add bug: Positive. Users will not get the bad version of Y.
On the contrary, it's negative because the maintainers of X will update Y and introduce the bug to their users (I'm assuming they don't thoroughly audit the source of Y each time they do a version upgrade).
Also, "unresponsive remove bug" should be given more weight since it's more likely to happen IMO.
It's a side point, but I don't entirely understand the argument here since it's basically guaranteed you were vulnerable to a variety of other bugs due to not updating. It depends how old your OpenSSL version was I suppose, but still, it kinda feels like gloating you survived a hurricane by doing no preparation - I'm happy for you, but that doesn't necessarily make it a great idea ;)
> - dynamic responsive remove bug: Positive/neutral. Team X would have done it anyway. > - static responsive remove bug: Positive/neutral: Team X will incorporate the change from Y, although possibly somewhat slower (but safer).
You sure about that? That's basically your whole argument, and I really don't buy it. In my experience, any dependency (static or not) shipped with a program is rarely actively updated, and also never in any regular fashion - on the list of things to do it's typically near the bottom. You called this a neutral, but it's easily a positive for dynamic (and negative for static), and IMO is the biggest argument for dynamic linking and shipping libraries separately. And once you apply that change, it's just two Positives and two Negatives each.
But with that, there's another situation that you're ignoring which throws a big wrench into your table - you're assuming you already know what the dependencies of a program are, when in practice you usually don't. In a dynamically linked world, to address a bug in OpenSSL I just drop a fixed version of OpenSSL in `/lib` and reboot. If it's statically linked to some or all of my programs instead, I need to figure out which if any programs make use of a statically linked version of `OpenSSL` and either wait for a new version or attempt to recompile them with a new one. And if I miss one, then the bug is still there in some form.
Edit: It's a small point, but you're also assuming that the maintainers shipping the static library will catch every bugged version before shipping their program with it - and library maintainers typically try not to release versions with known unusable bugs, it's an accident. An issue like Heartbleed is always going to be found after it's already in the wild, making the `static responsive add bug` category a bit suspect. If a program maintainer was on top of their OpenSSL version and always shipped the latest version when they released, their users would have been vulnerable - they may have made a release in a few days with a new version (to go with the other category), but their users still got the bad OpenSSL version.
It's a side point, but I don't entirely understand the argument here since it's basically guaranteed you were vulnerable to a variety of other bugs due to not updating. It depends how old your OpenSSL version was I suppose, but still, it kinda feels like gloating you survived a hurricane by doing no preparation - I'm happy for you, but that doesn't necessarily make it a great idea ;)
At the time, I think OpenSSL had several trees that were supported, Heartbleed was in the 1.0.1 tree, but not earlier trees, 1.0.0 and 0.9.8 trees were both still supported. OpenSSL versioning isn't very conventional, so one might not realize that 1.0.1 is a major version difference from 1.0.0, and 1.0.0 is a major version different than 0.9.8, but that's how they roll, minor versions within a series get letters after the name, and sometimes break binary compatibility (ugh). There's a reason a lot of people were slow to update to 1.0.1; unfortunately for me, my company had just updated to 1.0.1 about a month before the bug was reported, so we could get TLS 1.2 support. Would have saved a lot of headache if we had waited :(
Plus the whole reasoning tends to imply old versions are overall better (you don't have just one bug added or removed from time to time, you have to consider the overall bugginess over time, and if the project is well maintained, it will hopefully eventually decrease, especially if your usage scope remains constant)
But the reality is simply: it depends. On the libraries. On the application. On the platform. On the languages. On the tooling. On the maturity of all of that.
And even so, you may want (or be forced) to take an hybrid path (classic example: a program for Windows -- static is reasonable, but not for platform libs, and actually just not even available in this case)
OpenBSD has supported static PIE since 2015; not just supported, but all system static binaries (e.g. /bin and /sbin) are built as static PIEs, and I believe PIE is the default behavior when using `cc -static`. The reasons for the switch and the required toolchain changes are summarized in this presentation: https://www.openbsd.org/papers/asiabsdcon2015-pie-slides.pdf
Also, simply checking the "static PIE" box isn't the end of the story. There are different ways to accomplish it, and some are better than others in terms of ASLR, W^X, and other exploit mitigations. It's been a couple of years since I last looked into and had a hold on all the issues simultaneously[1], but the basic takeaway is that dynamic linking in system toolchains and system runtimes is far more mature than static linking.
[1] static PIE issues are a nexus of exploit mitigation techniques, so if you want to deep dive into exploit mitigation or even just linking issues then chasing the static PIE rabbit is a good approach.
When OpenBSD implemented static PIE, -static and -pie, the literal options and the code generation aspects more generally, were still mutually exclusive in GCC and clang. GCC required patching to enable both static linking and PIE generation. In fact, my local GCC 9.3 man page still says that -static overrides -pie; to build static PIE seems to require the special option -static-pie, in addition to -fPIE/-fPIC when building the object code. But it's not enough for the compiler and compile time linker to support it. libc and libstdc++/libc++ also need support, both internally as well as in the built libraries. And I think libdl might need support if you want dlopen to work from a static PIE binary. Likewise for libpthread. Supporting static PIE requires the cooperation of many moving parts. And that's just to support it. Making it easy, let alone the default behavior, so that it doesn't require many carefully coordinated, obscure flags without the risk of any misstep silently disabling some exploit mitigation, is yet another story.
It can be done. It should be done. But to what extent has it been done? I'm not sure, though some quick Googling suggests GCC and clang are mostly there, at least in terms of nominal support (which, again, is distinct from fully supporting all the same mitigation measures). glibc seems to have gotten some support (e.g. --enable-static-pie in glibc's build), though I'm not sure whether it's available in distros. And I would guess that musl libc support is pretty far along, and presumably more mature than glibc's given that Rich Felker started experimenting with static PIE several years ago, shortly after OpenBSD did their work. See https://www.openwall.com/lists/musl/2015/06/01/12
On vanilla gcc (since version 8 when static PIE was upstreamed), `-static` means non-PIE static executable and there is a separate flag for `-static-pie` for static PIE. Alpine patches gcc so that `-static` and `-pie` are independent flags, so both `cc -static` and `cc -static-pie` will produce a static PIE.
It's difficult to achieve a design that provides all the desirable exploit mitigations without sacrificing startup latency and other features. OpenBSD added a new syscall, kbind (https://man.openbsd.org/kbind), so that they can have lazy binding without being susceptible to the RELRO exploits mentioned in that Usenix paper. (Unfortunately, the 2018 leviathansecurity.com article fails to mention kbind, even though kbind was added to OpenBSD in 2015.) There are other approaches. I think the PaX Team has written quite alot about their preferred techniques. But the point is that static PIE, which is desirable because of ASLR and other reasons (e.g. unification of code generation techniques), touches upon varying and distant components of toolchains and runtimes.
(Unrelated, but since you brought it up: kbind is IMO not a very good mitigation. It seems that all you have to do to bypass it is leak a cookie and ROP to that one place in ld.so that is "blessed" and then you not only have the ability to scribble all over your read-only GOT but as far as I can tell you can overwrite any read-only memory, which means it opens up an extremely valuable exploit primitive…)
[1] National Novel Generation Month
[2] https://github.com/spc476/NaNoGenMo-2015, specifically, https://github.com/spc476/NaNoGenMo-2015/blob/master/C/msdos...
[3] lines 337 to 343 of `msdos.c` [2]
But that's a chicken-or-egg problem, right? The only way static linking tools will reach parity with dynamic ones in terms of maturity is if static linking replaces dynamic linking as the de facto method of choice.
So this widespread lack of knowledge that static is actually more mature than dynamic is kind of ironic.
Pure static linking makes sense when you are deploying code you control onto an environment you control.
While I like ease of deploying Go, where binaries are one big blob that just works, it makes it closer to the Java style than the UNIX modular-tool-does-one-thing-well tradition.
About the only place it kind of matters is Linux distros where you are deploying closed source binaries linked against system libraries. I'm not sure that's a significant use-case.
The only performance benefit is memory. If you have a core infrastructure library that's widely shared (e.g. libopenssl) then it can have some benefit. In practice I much prefer the microservice model with a formal IPC API. Then the SW update is trivial to fix the exploit - just kill the 1 process providing the service.
I get that only a handful of libraries are actually used, but that's a very fat tail and the performance will be very bad if that handful of libraries are always statically linked. This may be less important if the program is in a container (isn't sharing anyway) or launched as an AWS Lambda throwaway. I dunno, maybe dynamic linking isn't relevant anymore now that most Linux programs are meant to run in containers on an expensive cloud with tons of memory.
> In practice I much prefer the microservice model with a formal IPC API.
This converts a simple and solved problem, sharing memory, to a client-server distributed systems problem. There is no need. It's all on the same box.
That's also ignoring that measuring the distributed cost of a component is fundamentally simpler than measuring it for a library mapped into 50 million processes.
Shared libraries can be useful but their value is often vastly overstated.
Because from my telecommunications and mobile OS development knowledge I have hard time remembering at least one.
And binary deployments? They are done all the time.
Apple, Google, Microsoft, Amazon, Facebook all build everything from source. Source: I worked at 3 of those & have friends coworkers at the rest. For cloud users I don't have as good a knowledge of that space. I imagine the majority of them use off-the-shelf prebuilt libraries that come with the OS they run on.
Dynamic linking is not a performance feature, it's a decoupling feature.
As for "high multi-tenancy" there's nobody out there with server occupancy as high as Google's (see the recent Borg scheduler traces for concrete data) and they statically link everything.
Yes, really.
Dynlinking solves a memory resource problem we had in the 80s. These days the only people with the problem are the very smallest embedded systems.
Definitely not true. Every bit of memory that's not available that could be shared memory instead is a reduction in memory available for filesystem caches, etc.
As for "high multi-tenancy" there's nobody out there with server occupancy as high as Google's (see the recent Borg scheduler traces for concrete data) and they statically link everything.
Cloud vendor computing models are not generally the computing model of the rest of the world. Comparison to their environment is not relevant to the general populace.
I worked for a "big iron" OEM vendor until late 2017 and the savings were definitely still significant then both for their customers and the vendor themselves.
There are numerous benefits to shared linking, that doesn't mean it's always the appropriate solution, but is not correct to claim that there are no performance benefits.
Especially on more memory-constrained consumer devices, the shared memory benefits of dynamic linking are still significantly beneficial.
The way that Google shares server compute and memory resources is by having a service oriented architecture. A single high scale service serves many different applications, usually colocated in the same cluster or data center. Each service is based on multiple instances of a statically linked binary.
At that scale, there is no point in try to use shared dynamically linked libraries to reduce consumption because you save more by either reducing increasing your own app's efficiency or relying on one of the major services rather than linking more functionality into your own app.
The vast majority of the contents are r-x text (code) and r-- read-only data. I don't know if the kernel is capable of sharing read-only paged memory references or not.
Any pages mapped from disk should be shared, as a consequence of the FreeBSD unified buffer cache; this is why if you write to a binary in-place with cp instead of unlinking first (like install), currently executing copies are changed, and usually crash :). Read-write program sections will be mapped so their changes are private though, so you get copy on write semantics. Trailing parts that aren't page sized will be copied into a full page, I think.
This is not true, or no longer true: opening a running program's backing file with O_RDWR or O_WRONLY is prevented via the "writecount" mechanism (and vice versa: files being written cannot be executed).
Here you go: Stali
https://dl.suckless.org/htmlout/sta.li/
"Stali distribution smashes assumptions about Linux"
https://www.infoworld.com/article/3048737/stali-distribution...
That was an apples-to-apples comparison of pre- and post-process model unification performance, and it was a win.
Now, this was back in... I want to say 2003 or 2004 -- before S10 shipped. And it's possible that the same experiment today would not have the same result.
I'm not sure how easy it would be to construct a distro with only dynamically-linked executables (at least for core libraries, like the C library) and then the same distro with only statically-linked executables. The S10 work was done by Roger Faulkner (RIP) and it was a huge change.
>Findings: not really
>Over half of your libraries are used by fewer than 0.1% of your executables.
Findings: Yes, lots, but mostly the most common ones. Dynamically linking against something in the long tail is pretty pointless though.
I disagree. Dynamic linking, in the context of an OS which offers a curated list of packages in the form of an official package repository, means that a specialized third party is able to maintain a subcomponent of your system.
This means you and me and countless others are able to reap the benefit of bugfixes and security fixes provided by a third-party without being actively engaged in the process.
In the context of an OS where the DLL hell problem hasn't been addressed and all software packages are forced to ship all their libraries that are shared with no one at all, indeed its pretty pointless.
Getting off the treadmill of integrating interface-breaking upstream changes is one of the biggest practical reasons people prefer static linking and directly adding upstream source repositories into their build. It's at least as important, IME, as being able to use newer versions of libraries unavailable in an LTS distro. It can work well for large organizations, such as Google with their monolithic build, because they can and often do substitute the army of open source packers with their own army of people to curate and backport upstream changes. For everybody else it's quite risky, and if containerization provides any measure we're definitely worse off given the staleness problems with even the most popular containers.[1]
[1] I wouldn't be surprised if an open source project emerged to provide regularly rebuilt and possibly patched upstream containers, recapitulating the organizational evolution of the traditional FOSS distribution ecosystem.
You nailed it, in my opinion. The biggest reason I favor dynamic linking is not because of any inherent advantage that I'm determined to believe in (in the face of purported evidence to the contrary, like this article), but because I fear the ecosystem changes that a major shift towards static linking would allow.
If everything is a static blob, you have an environment that is much more friendly to every dev shipping their app as a binary download on their website, like Windows "freeware". Or worse still, they only support AppImage or some other "modern" method of distribution. This cuts maintainers out of the loop. I want to continue using a distribution with maintainers. I think they solve some important problems: when a maintainer is the gatekeeper, it means that a dev has to convince a third party that their app is (a) worth including, (b) not malware, (c) open source - at least heavily preferred since the dev will be building it themselves. And plus having a maintainer means someone besides the original dev is responsible for making sure you get security updates.
Now of course you can distribute statically built apps that way, but there's a reason it's less common. There are so many Go apps out there where the only supported method to build them is using the Go toolchain and pulling in 150 Github repos.
See also: this defense of the maintainer-based ecosystem, from an Arch Linux dev. http://kmkeen.com/maintainers-matter/
So because that's what you want, let's make what other people want harder.
Personally, I can't stand the repo/maintainer model. I want to cut maintainers out of the loop! I like having a direct relationship with the developer of the software I use. If they update their software with a feature or bug fix I need, I want the update right now, not when some unpaid third party gets around to integrating it with the f'ing distro. When I report bugs to the developer, I don't want to have to involve the third party.
Shit like this is why I stick to Windows.
No? I haven't said it should be harder to statically link software. I just said I want to discourage it. I want to advocate that my Linux community resist these kind of ecosystem changes that would, in my opinion, harm the free software community.
> Shit like this is why I stick to Windows.
Seems like telling on yourself. Compare the results: the software available on Linux distributions vs. the Windows freeware market.
> I like having a direct relationship with the developer of the software I use.
And what if that developer isn't trustworthy in some way? (See the Arch Linux dev's post that I linked.)
> If they update their software with a feature or bug fix I need, I want the update right now
Just one example, but I've sometimes gotten updates for Firefox on Arch Linux before the official binaries got released. The maintainers seem to be on top of their game. And never mind the fact that if you're counting on automatic updates, you're assuming that they roll out to everyone simultaneously (often not true) and that automatic updating is always desirable anyway. (Plus a lot of Windows software doesn't update itself at all, so...)
Obvious choice? WINE wasn't invented because Linux had a great software library.
> And what if that developer isn't trustworthy in some way?
Then you probably shouldn't use their software at all. But that isn't necessarily realistic, which is why I also advocate for sandboxed applications by default, something mobile got right.
> Just one example, but I've sometimes gotten updates for Firefox on Arch Linux before the official binaries got released.
Sure, maybe that happens sometimes for big and popular projects, but there are a lot of more niche projects where what is in the repo is years out of date. Hell, I have to get qbittorrent-nox from a PPA to be up to date and that isn't even very niche.
So you can only use software in the Microsoft Store?
It makes ops so liquid and convenient. I love it.
To really push this idea (and microkernel) to its logical extreme, it would be cool to see a null-kernel. You have socket drivers, that's it. Everything is either a network call or some sort of IPC. You might still need something to handle paging, though, and obviously your "peripherals" like storage would need to be more conventional in nature.
Imagine something like InteliJ or Eclipse, where every single IDE plugin is its own process doing IPC.
For Linux though, where the very concept of a standardized base system is loathed, static the sweet spot.
[0] libraries that are likely to be used by a large number of applications like the GUI libraries, kernel interfaces, networking, encryption, etc.
Imagine statically linking UIKit or Android's UI library!
For example, on Android even the switch to hw accelerated rendering was a mode app devs had to opt in to, that didn't even change the look!
I have little memory of the details of the story, and I'm not 100% sure it's true, but it's a much more satisfying and reasonable argument for dynamic linking than performance/space.
Of course, the more modern solution would probably be a good package manager -- if its trivial to recompile things, and track what needs to be recompiled, then dynamic linking seems to gain little, but bring in a lot of its own headaches (as we know today)
The original reasoning for ELF was that static link semantics suck. ELF's semantics are far superior (https://news.ycombinator.com/item?id=23656173).
I read it on a mailing list a long time ago so I don't have a source.
Moreover, in a recent discussion about this topic on HN, someone said the introduction of shared libs into the Linux user space was mainly in support of porting X Windows to Linux (supposedly because of binary video drivers or to accomodate MIT-licensed code?), but I haven't found any supporting reference for that.
glibc's maintainer Ulrich Drepper also has pretty strong opinions on static linking [1]; no matter what you think about this technically, or Ulrich personally, his paper "How to write shared libraries" [2] is considered reference material on the subject.
Personally, I think that the over-use of shared libs in the Linux userland clearly serves no purpose if users flock to entire new layers of abstractions (eg Docker-like containers) to isolate their app delivery from the IMHO overengineered mechanisms in ELF and ld.so with their multiple RUNPATHs, configs, loader scripts, and versioned glibc symbols (on top of build-time libtool/autootols) that still doesn't seem to get to the point. While the LSB effort for more uniform Linux distros isn't dead, it doesn't seem to be taken seriously. Idk, but maybe the GNU folks also see the lack of binary compat for Linux apps as a desideratum, to frustrate any and all attempts to ship binary apps?
[1]: https://web.archive.org/web/20100527213559/http://people.red...
Sure it is more secure and probably preferable in modern times, but it also slower and requires more hardware resources, specially when one scales it with desktop software running hundreds of processes, each for their own plugin sets.
There is no free lunch as they say.
That's primitive superstition:
- dlopen() works in statically linked programs too.
- Even without dlopen(), you can load code dynamically. It's not magic, especially if the plugin is linked statically.
- Besides, how is IPC bad? I take fcgi over Apache's modules any day.
The idea of using plugins is exactly that various parts are able to ship them at various times during the lifetime of the applications.
Patching files compiled statically is obviously not what one wants from plugins.
Incidently plugins support is exactly why Go added support for dynamic linking into their toolchain.
Where is the problem in this scenario? My application (statically) linked provides a plugin interface. It works by calling dlopen() on the plugin and then passes a record of functions to the entry points. The plugin can be provided by a third party, and top make that easier, I provide an SDK (a header file).
No file is patched, and I have no idea what this has to do with Go. Enlighten me?
We have been there before with solutions like graphics drivers for Borland's BGI library for their MS-DOS compilers. Where a driver interface is provided, and then each "plugin" registers themselves on application startup.
Adding new plugins to an existing application in this scenario requires recompilation and updating the uses/#include being used for plugins, or patch the executable from an existing .obj file.
If you are using dlopen on a third party binary instead of your own, then you are already using dynamic linking by definition, by loading third party code dynamically and revolving the proper address locations for all symbols.
Go only had static compilation on the beginning as the only true way, but it failed short exactly in this scenario, to the point that eventually plugin package came to be and dynamic compilation support is now a feature of Go's toolchain as well, although many seem to still not be aware that Go compilers can also produced dynamic libraries.
With dlopen the application can crash in very interesting ways, if it doesn't handle loading the errors in a proper way.
And there is no way to use something like dumpbin or ldd to actually find out what it is looking for, even something like strings might not help, because the path to the file being dynamically loaded in an explicit way might itself be dynamically constructed at runtime.
"Well, technically..."
I proposed using the dlopen() mechanism (or a custom linker) from a program that is itself statically linked. Because that's what the argument is about: statically linking the stuff you always need. Now, technically, that's dynamic linking. And because I'm linking the plugins dynamically, I might as well link everything dynamically, right? No, absolutely not. Because the latter gives me DLL hell, and rpaths, and library maintainers who think that changing LD_LIBRARY_PATH is perfectly sensible. The former doesn't. Not equivalent. Not at all.
We're not talking about BGI or overlays or Go. We're talking about statically linked binaries calling the dynamic linker. You didn't actually know that was possible, did you?
So my knowledge how these things work goes quite back in time.
This is maybe not an issue for open source packages which are managed by your distribution package manager, assuming they update all the dependent packages once some library gets updated (which would lead to a lot more updates all the time).
However, the maybe more critical issue is about other independently installed software, or maybe closed source software, where you will not automatically get an update once some library gets updated.
If you care about security, you shouldn't be running closed source software anyway, at least not outside of a container.
Statically linked binaries for open source software, containers for everything else, and you're good to go.
Or am I the only one who has occasional problems when replacing all the binaries on my system?
On the other hand, I have a three statically linked binaries for CloudFoundry in my home directory. One for Linux, one for MacOS and one for Windows. They are each between 24MiB and 27MiB each.
As far as I know, the only completely statically linked Linux distribution that is actively developed is my own project (inspired by stali), oasis: https://github.com/oasislinux/oasis
1) when making a static link archive (a .a file), include a .o with a static symbol(s) whose values(s) records the metadata that ELF would have recorded, which here is: a) dependencies, b) how to find them, c) the mapfile / version-script (at least which symbols are symbolic, protected, or demoted to local, and which, if any, are interposers.
2) on final link-edits only the direct dependency -lfoo arguments should be needed, and then link-editor should find the dependencies' dependencies... by looking at the metadata recorded in the .a files.
Really, it's very simple. libstool is a very bad attempt at layering this on top of the linker (recording metadata in .la files), but it doesn't work well. This functionality has to be in the linkers.
In particular, static linking for C has two serious problems:
1. symbol collisions -> accidental interposition (and crashes);
2. you have to flatten the dependency tree into a topological sort at the final link-edit.
Both of these are related, and they are disastrous. They are also related to the lack of namespaces in C.
Besides fixing these issues, the C dynamic linking universe also enables things like:
- run-time code injection via LD_PRELOAD and intended interposition
- run-time code loading/injection via dlopen(3)
- audit (sotruss)
- reflection
- filters (which allow one to move parts of libraries contents to other libraries without forcing re-links and without forcing built systems to change to add new -lfoo arguments to link-edits)
- use of dladdr(3) to find an object's install location, and then that to find related assets' install locations relative to the first, which then yields code that can be relocated at deploy time (sure, "don't do that" is a great answer, but if you statically-link then you think you can, and now you just can't have assets to load at run-time)
- use of weak symbols to detect whether a process has some library loaded
and others.
C with those features is a far superior language -- a different language, really -- to C without them.
(EDIT: A lot of the semantics of ELF could be brought to static linking. Static link archives could have a .o that has metadata like depedencies, "rpaths", exported/protected symbols, interposer symbols, etc. The link-editor would write and consume that metadata. However, it's 2020, and the static link ecosystem is stuck in 1980 because no one has bothered, and no one has bothered because dynamic linking is pretty awesome. Still, it could be done, and once in a while I think I ought to do it to help save people from themselves who want static linking.)
> Do your installed programs share dynamic libraries?
> Findings: not really
> Over half of your libraries are used by fewer than 0.1% of your executables.
The C library most certainly gets shared, as well as libm and such. The rest, it's true, not so much, but it does depend on what you're measuring. Are you measuring C++ apps? Yeah, C++ monomorphization leads to essentially static linking. Are you measuring Java apps with no significant JNI usage? You won't find much outside the libraries the JVM uses.
> Is loading dynamically linked programs faster?
> Findings: definitely not
Dynamically-linked programs will load faster when their dependencies are already loaded in memory, and slower otherwise. The biggest win here is the C library.
> Will security vulnerabilities in libraries that have been statically linked cause large or unmanagable updates?
> Findings: not really
Correct. But, being able to update libc or some such and not have to worry about updating consumers you might not even know about is a very nice feature.
> 1. symbol collisions -> accidental interposition (and crashes);
I've encountered symbol collisions only twice, but both resulted in linker errors due to multiple function definitions. I'm not sure how this could happen accidentally. Maybe you are referring to variables in the common section getting merged into a single symbol? Recent gcc enables -fno-common by default, so those will be caught by the linker as well.
> 2. you have to flatten the dependency tree into a topological sort at the final link-edit.
Yes, this is pretty annoying. pkg-config can solve this to some degree with its --static option, but that only works if your libraries supply a .pc file (this is often the case, though).
I think libtool also can handle transitive dependencies of static libraries, but it tries hard to intercept the -static option before it reaches the compiler so it links everything but libc statically. You can trick it by passing `-static --static`.
For oasis, I use a separate approach to linking involving RSP files (i.e. linking with @libfoo.rsp), which really are just lists of other libraries they depend on.
> Besides fixing these issues, the C dynamic linking universe also enables things like: > - run-time code injection via LD_PRELOAD and intended interposition
Yes, this can be a problem. I wanted to do this recently to test out the new malloc being developed for musl libc, but ended up having to manually integrate it into the musl sources instead of just using LD_PRELOAD.
> - run-time code loading/injection via dlopen(3)
In particular, this is a big problem for scripting languages that want to use modules written in compiled languages, as well as OpenGL which uses dlopen to load a vendor-specific driver.
> Dynamically-linked programs will load faster when their dependencies are already loaded in memory, and slower otherwise. The biggest win here is the C library.
But doesn't the dynamic linker still have to do extra work to resolve the relocations in the executable, even when the dependency libraries are already loaded?
> I've encountered symbol collisions only twice, but both resulted in linker errors due to multiple function definitions. I'm not sure how this could happen accidentally. Maybe you are referring to variables in the common section getting merged into a single symbol? Recent gcc enables -fno-common by default, so those will be caught by the linker as well.
No, this comes up all the time. Try building an all-in-one busybox-style program, and you'll quickly run into conflicts.
If static link archives had all the metadata that ELF files have, then the link-editor could resolve conflicts correctly. That is the correct fix, but no one is putting effort into it. The static linkers haven't changed much since symbol length limits were raised from 14 bytes!
> > 2. you have to flatten the dependency tree into a topological sort at the final link-edit.
> Yes, this is pretty annoying. pkg-config can solve this to some degree with its --static option, but that only works if your libraries supply a .pc file (this is often the case, though).
pkg-config alleviates the problem, but it's not enough. Among other things building a build system that can build with both, static and dynamic linking is a real pain. But more importantly, this flattening of dependency trees loses information and makes it difficult for link-editors to resolve symbol conflicts correctly (see above).
> > Dynamically-linked programs will load faster when their dependencies are already loaded in memory, and slower otherwise. The biggest win here is the C library.
> But doesn't the dynamic linker still have to do extra work to resolve the relocations in the executable, even when the dependency libraries are already loaded?
It's still faster than I/O. (Or at least it was back in the days of hard drives. But I think it's still true even in the days of SSDs.)
For dynamic linking there's usually only a single reference that needs to be fixed up in the PLT or GOT for each referenced symbol and the fix-ups are all localized, so that part is typically a very small cost. But this means every call to a dynamically linked library goes through an extra stub function, which adds I$ and BTB pressure compared to static linking.
(If you do things manually with dlsym there's also an extra indirection but it's a little different mechanically since you do an indirect call of a function pointer rather than doing a direct call to a forwarding stub. The advantage of the forwarding stub is that even if there's a BTB miss for either of the two direct calls, you don't suffer a branch mispredict but just a few front-end cycles waiting for the instruction decode, which VTune labels as a "branch resteer". The direct call option also leads to smaller code size for each call site which helps with I$ pressure. Basically it's the difference between CALL rel32 and MOV tmp, [RIP + disp32]; CALL [tmp].)
RTLD_NOLOAD
> Dynamically-linked programs will load faster when their dependencies are already loaded in memory, and slower otherwise. The biggest win here is the C library.
RTLD_LAZY, though it's one less attack vector to have RTLD_NOW and mprotect your GOT and PLT as readonly.
Using more RAM means having lower cache hit ratios. If dynamic linking means using less RAM, you win. But it's not a clear-cut thing -- it will depend a lot on the surrounding ecosystem.
In any case, for C, the problem with static linking is about semantics. Until those are fixed I'll be resolutely against static linking for C, and for everything else, well, do whatever performs best.
Here's a good example. Firefox does even cross-language LTO for stuff that matters — all the DOM/CSS/etc components that are tightly bound together. But it happily allows you to dynamically link to a system libjpeg, and you won't lose any performance because the libxul<->libjpeg boundary is crossed.. about a couple times per downloaded jpeg. Heck, what would you even inline between the jpeg decoder and the browser? Absolutely nothing!
The world at large is already using static linking where it matters.. e.g. with C++, header-only libraries and anything that uses templates results in a lot of static linking. Even if you link to libboost_something.so, you probably already have most of that something inlined into your code because templates :)
LTO requires source, or at least IR.
Static linking involves object files; there's been a loss of fidelity at that point that prevents inlining.
./test.sh | awk 'BEGIN { sum = 0 } { sum += $2-$1 } END { print sum / NR }'
-698915My guess is that apart from eg libc, the average is pretty low (ie 1 for the pages that aren’t free).
I think the author's using a wrong method to make a point. Dynamic linking feels out of place for most long-running server-side apps (typical SaaS workload). One can argue that in a mostly CLI-environment there's also not much benefit.
But even an empty Ubuntu desktop runs ~400 processes and dynamic linking makes perfect sense. libc alone would have to exist in hundreds reincarnations consuming hundred+ megabytes of RAM and I'm not even talking about much, much heavier GTK+ / cairo / freetype / etc libraries needed for GUI applications.
Put the cold hard numbers right in front of someone's face and still the cargo cult wins out.
Come on.
You did not actually measure the figure GP mentioned and which you are disputing. Your methodology and assumption — that 4% external symbol use translates into 4% size used — is a plausible guess, but you haven't supported it with data.
Even if you had measured the figure you're accusing GP of ignoring, the tone of your remark is just aggressively condescending and inappropriate. Tone it down.
To address your other claims:
> Shared memory is page-aligned entire libraries dropped into RAM.
There is a good reason to page- or superpage-align code generally; it burns some virtual memory but reduces TLB overhead and therefore misses / invalidations, which are very costly. You would want to do the same with executable code in a static-linked binary.
> the majority of [the small fraction of static linked library used] would not end up in RAM with your statically linked binary.
Huh? Why do you claim that?
> And if you used a more selective approach, dynamically linking to no more than perhaps a dozen high-impact libraries and statically linking the rest, you'd get a lot of the benefits and few of the drawbacks.
I think that claim is plausible! But it wasn't an option presented on your blog post, nor was it discussed by GP. Prior to that comment, discussion was only around 100% vs 0% dynamic linking.
But most code isn't performance critical. Thus trying to align functions to page boundaries is just wasting memory. Even in performance critical code, aligning to cache line sizes is enough and aligning to page boundaries doesn't provide any advantage.
Like, sure, mapping your executable's code section in at a page boundary is probably fine, but I think trying to align individual functions to page boundaries would be a counterproductive mistake as a general strategy.
Yes — which is why no one does that. I don't know where bjourne came up with the idea.
Wasting TLB slots on your unimportant code still pessimizes your hot code.
> Thus trying to align functions to page boundaries is just wasting memory.
No one aligns individual functions to page boundaries; you align the entire loadable code segment to a page (or preferably, superpage) boundary.
> Even in performance critical code, aligning to cache line sizes is enough and aligning to page boundaries doesn't provide any advantage.
This is a different kind of optimization (avoiding cache line contention) than I was talking about (optimizing TLB slot use).
Yes, forced page-alignment doesn't help cache line contention anymore than forced cacheline-alignment. But that's irrelevant for code, generally: (outside of self-modifying code, which is extremely uncommon) code doesn't share a cacheline with memory that will be mutated and thus doesn't contend in that way.
First of all, I seriously doubt it (haven't looked closely, but if library-loading is similar to mmap, it should only count actually used segments).
But that shouldn't even matter, as most of these libraries are fully utilized. All of libc is used collectively by 400+ processes, that is also true for the complex multi-layer GUI machinery. The output of ldd $(which gnome-calculator) is terrifying, run it under a profiler and see how many functions get hit, you'll be amazed.
Put the cold hard numbers right in front of someone's face and still the cargo cult wins out.
The coldness of your numbers did not impress. And calling a reasonable engineering trade-off "cargo cult" doesn't get you any points either.
Static linking is better than dynamic linking. Sometimes. And vice versa. That's how engineering is different from science, there are no absolute truths, only trade-offs.
I think you overestimate how much saving you get from dynamically linking libc. Each executable uses only a small portion of libc, so the average savings is going to be in the handful of kilobytes per executable.
test.c:
int main(int argc, char **argv) {
printf("hello world\n");
return 0;
}
$ gcc -O2 -Wl,--strip-all test.c
$ ls -sh a.out
8.0K a.out
$ gcc -O2 --static -Wl,--strip-all test.c
$ ls -l a.out
760K a.out
$ musl-gcc --static -O2 -Wl,--strip-all test.c
$ ls -sh a.out
8.0K a.out jart@debian:~/cosmo$ make -j12 CPPFLAGS+=-DIM_FEELING_NAUGHTY MODE=tiny o/tiny/examples/hello.com
jart@debian:~/cosmo$ ls -sh o/tiny/examples/hello.com
20K o/tiny/examples/hello.com
But what are we comparing to what then ?
A few custom Go applications, compared to whole classic Linux distros, on which deployment is both not really the same thing, but still a breeze ?
So yeah, to different needs, different tools.
The top two entries are 80 and 36 kB respectively. RES is 2.3 giga-bytes (over four orders of magnitude larger) between them. Even multiplying the top SHR by your ~400 processes gives 32 MB (still two orders of magnitude off). That is not a counter argument; that is a agreement that dynamic linking is useless.
Edit: RES, not VIRT.
This is strictly a C problem.
I suspect you would need to compute signatures for static libraries for binaries located /bin/, /lib, ... to make load times faster.
First, this analysis was done on Arch Linux, a source-based distribution. Since you know at compile time what your environment is, I would expect the benefits to be smaller. And of course, this means you're willing to do a lot of recompiles. I'd like to see analysis done on more traditional (& common) binary distros.
Second, the arguments seem a little cherry-picked. "Over half of your libraries are used by fewer than 0.1% of your executables." is cute. But modern systems have a lot of executables, so 0.1% > 1, so that still matters, and what about the other half.
Finally, we're already having serious problems getting containers to upgrade when a security vulnerability is found. Requiring recompilation of all transitive users is not likely to win any update speed contests. If it's completely automated then it would work, but any rocks in the process will leave people endlessly vulnerable. See the Android ecosystem, etc.
Maybe it's doing it behind the scenes or something, but I'm doubtful, because otherwise I think my upgrades would take a lot longer.
Arch Linux is not Gentoo. And AUR is only a secondary method of installing software. So I'm not sure what you mean.
I suspect many users will need to dip into AUR once or twice for unsual packages, but I'm mildly skeptical that most users need to dip into AUR often enough that compile times would be a serious argument against static linking.
Fwiw, I've been using arch as my daily driver continuously for 14 years and haven't ever really felt the strong need for an AUR helper.
I run Gentoo as my Linux box. It takes a long time to compile stuff as it is (and I have a 22 core machine). Being forced to compile a lot more because everything is linked statically would be a nightmare.
This is definitely not a plus for source based distributions.
I fail to see the relevance of this comparison.
It's more like wearing fitted clothing and (rationally) objecting to requiring a visit to your tailor every time you change your socks.
I'm pointing out that saying this will benefit source based distributions more is nonsense.
See http://www.real-linux.org.uk/recursivemake.pdf for a classic explanation.
But in https://doc.rust-lang.org/cargo/reference/build-scripts.html I find comments like, It is recommended to carefully consider each dependency you add, weighing against the impact on compile time, licensing, maintenance, etc. Cargo will attempt to reuse a dependency if it is shared between build dependencies and normal dependencies. However, this is not always possible, for example when cross-compiling, so keep that in consideration of the impact on compile time.
So it looks like cargo can have the same issue that make does with recursive dependencies.
Do you have an explanation for why compile times would be an issue for Rust?
"First, false statement. Since you know false assumption, I would false conclusion. And of course, this means false conclusion. I'd like to see analysis done on what you did them on."
"Second, the arguments seem a little cherry-picked. "Quote from article about W and Z" is cute. But modern systems have a lot of Z, and Obviously You Didn't Consider This."
"Finally, we're already having serious problems getting Thing the Author Almost Always Rags on for Sucking to upgrade when a security vulnerability is found. Requiring recompilation of all transitive users who the author doesn't care about and who the author has already told are wrong is not likely to win any update speed contests for a use-case the author thinks is invalid. If it's completely automated then the perceived invalid use case would still be viable, but any rocks in the process will leave people with perceived invalid use case endlessly vulnerable. See Notoriously Bad Ecosystem That Isn't Relevant to the Article, etc."
Commenting that a comment is exemplary of all that's wrong with HN is... popular, I'll grant you that, but it grates, and it's not really right. It's a "shut up" argument. Just downvote what you don't like, upvote what you do like, and reply with detailed commentary where you have something important to say. No need to disparage commenters.
This would be correct, if the objection were right. But the commentator was objectively incorrect, in a way that could have been avoided with something as small as a web search, and the comment had the audacity to do so in a condescending way. I didn't knock anyone who had a genuine objection to the post, I replied to an incredibly low-effort and low-quality comment pointing out that it was low-effort and low-quality.
Commenting that a comment is exemplary of all that's wrong with HN is... popular, I'll grant you that, but it grates, and it's not really right.
This is the first one I've ever made, and I post pretty frequently. I don't think it's wrong whatsoever, and I'm standing by it.
It's a "shut up" argument.
It absolutely is a "shut up" argument. Comments should not be written when they can't get the simplest of facts straight, and authors of them should think before they post.
No need to disparage commenters.
I didn't, I disparaged their comment. I don't think doing so was wrong, either, because the comment was practically equivalent to replying to a well-considered post with a PNG of a meme.
That comment did make a major mistake. Arch Linux is not a source distribution. But the mistake notwithstanding, it was otherwise well-reasoned. Your response was not.
You can mock the point that there is still a benefit in sharing a library a few times, and the article ignored the few libraries that get shared a lot. But said point remains true.
You may think that Android isn't relevant to the article's point. But the tradeoffs between static and dynamic linking are true across operating systems. The challenges that Android has had because of static linking are therefore worth paying attention to. Refusing to look for parallels to inform your intuition from is simply refusal to learn.
The author didn't ignore this, because it's pretty much false. Every package people insisted was relevant, he shot down before he published the article. Example:
https://cmpwn.com/@sir/104406644780241359
Hitchen's razor.
This must not be counting packages which depend on openssl by a chain of dependencies, some number of which contain binaries or libraries which link libssl. On my desktop Arch system of 1448 packages, 42 depend directly on openssl and 672 depend on it directly or indirectly. (pactree -ur openssl | wc -l, then subtract one for openssl itself)
As a side note, you are almost always better off (also on this case) by plotting power laws on a log scale.
Your time is now.