Static Linking Considered Harmful (2004)
akkadia.org
akkadia.org
I particularly disagree with the idea that doing any dynamic linking in a program at all means that instantly it's just as non-portable as a fully dynamically linked program. These things are a continuum, there are degrees of fragility -- you can make a program that breaks less often when ported to new systems. See some of Ethan Lee's writing on this topic.[0]
On a meta-level, you should be instantly suspicious of any programming advice that includes quotes like:
> Conclusion: Never use [thing]!
> This has never been the case and never will be the case.
In reality, most software techniques have benefits and downsides. If a debate has survived for over a decade, that means that the answer probably isn't so completely universal that you can completely ignore one side.
I'm not going to say that there are no black-and-white correct answers in software development, but people who regularly claim to have found them should be regarded with suspicion. On average, they'll often just be oversimplifying things.
[0]: https://gist.github.com/flibitijibibo/b67910842ab95bb3decdf8...
I'm going to go a step further and say that the whole reason that software exists as a large profession is specifically because these trade-offs and domain and situation specific.
If we really had one-size-fits-all software as a whole would largely be a solved domain.
For example, it is fairly reasonable to require LibSDL2 to be installed. However, if you bundle LibSDL2 with your app, you’re now directly dependent on things like X11 and ALSA.
Herein lies the problem, because Linux has absolutely no distinction between "system" and "application". Everything is part of the system.
This is especially true once the game gets older and less supported. The native Linux libraries might move on over time and the version you require might become unavailable or broken in some way but once a game works on Wine it will likely continue working forever unless you make some change to break it.
These days when it comes to my Steam library I am less worried about the Proton[1] games than the native ones and there have been several cases where I opted to run the Proton version because it worked better than native.
[1] Proton is the version of Wine Steam uses
You mean, "the game's package has libsdl2 listed as a requirement"? Or "if you install the game without libsdl2 present, it will install, but will crash on launch"?
> However, if you bundle LibSDL2 with your app, you’re now directly dependent on things like X11 and ALSA.
Well, either those are present and the game will work, or you've tried to install the game on a headless server and surely you don't expect to miraculously just work?
Are you some kind of malevolent genie? Trying to figure out the worst possible user experience that corresponds to the words I wrote?
> Well, either those are present and the game will work, or you've tried to install the game on a headless server and surely you don't expect to miraculously just work?
The assumption here is that the only two options are either “X11+ALSA” or “headless”, and I don’t agree with that assumption. SDL2 on Linux has two video drivers and five audio drivers. Fact is, I can run horribly old Linux software that uses SDL for audio and it works fine, but old software that uses OSS or /dev/dsp is not going to work. With Wayland, I think it's not unreasonable to prefer a dependency on SDL over a direct dependency on X11.
Yes, that's a valid point actually. But ultimately it's a bet on what API will end up more durable: X11 or SDL2, isn't it?
One of the other factors here is that the ability to play old games is in some sense about the preservation of culture. If you have an x64 Linux binary that uses SDL + OpenGL, I have a strong feeling that you’ll be able to play it 20 years from now, although it might inconvenient to get working.
How? I'm honestly confused what you mean here, can you explain how to build a nontrivial game on Linux while statically linking GL/Mesa/X11/Wayland/ALSA/Pulseaudio/libpng/etc, and guaranteeing that it will work on all systems and won't have any issues or security problems? Because as far as I can tell this is close to impossible. Or do you consider use of dlopen to be a different case than what the article is discussing?
If you go into shipping a game using only dynamic linking (as this article advocates), then you're going to run into a bunch of portability problems. I'm hard pressed to think of a reason you would ever want to dynamically link to something like vorbis or libpng for a game.
The goal is not ideological purity in either direction, it's to minimize compatibility issues. It's to look at each library and decide whether the upsides of using the system's version (assuming it has that library) override the downsides of needing to deal with all of the different systems and library versions people might be running.
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-5146
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-7317
If your nontrivial game has an online component with voice chat or custom avatars or something like that, then you now have an attack vector. If your game is closed source, I hope you are following updates from these libraries and either updating your vendored copy or backporting patches and then immediately pushing out a new release whenever an upstream update is published. And you will likely want to be committed to doing this for however many years you plan to keep your game online.
And if you're going through all the trouble of doing that, I don't see what benefit you're getting from static linking these libraries anymore. (Unless the argument is that you're going to patch it faster than the distros)
If you do have an online component or mod-loader that's affected by a vulnerability, you'll still find that the amount of work required to recompile a game and ship an update when a security release is issued for a dependency is less than the amount of work required to deal with accidental breaking changes in minor version updates, or with systems that don't install the latest package versions, or with systems that have renamed their libraries, or with systems that have stuck custom incompatible patches on top of their libraries.
If you're monitoring CVE releases for your statically linked dependencies, then when something breaks you push one patch and that's it, you're done. The alternative is that the problem never, ever goes away because people don't keep their systems up to date, and they have weird setups, and there's no patch you can ship to address that problem. You really don't want to be in a position where you're supporting multiple versions of the same library. When a security patch gets issued, you're going to do the same amount of testing regardless, so you might as well ship one version of the library at the end of that testing process that you know will work.
Aside from the support issue, there's also the fact that for security reasons you'd really like to fix most security bugs in your app yourself anyway, because, again, people don't keep their systems up to date, so it doesn't matter how quickly distros patch their libraries. If you don't ship an update that fixes the issue for everyone, then it won't be fixed for everyone. You can't trust users to update their own system libraries or not to forcibly maintain older versions, and for any app that needs to maintain a reputation of security (ie, most encrypted chat apps and password managers), you don't want your users to get hacked just because they were using a shared library ended up being vulnerable.
And even that's ignoring the fact that there are tradeoffs even for security fixes. If we're talking about a 10 year old game that's been abandoned, we'd like to have the ability to easily run it, even if we acknowledge that it might have some security holes.
But ultimately, this is really not worth fighting about. Prove Ethan wrong, try shipping dozens of commercial games with the strategy you propose and see whether or not your support requests go up or down. There's a reason game devs overall trend towards statically linking libraries. We're not doing it just to be difficult.
In fact a lot of indie games that support linux do it this way, setting LD_LIBRARY_PATH with a shell script and it works well.
If a 10 year old game has been abandoned, I don't mind testing or recompiling it for some security updates. I've done this for lots of open source games. I personally would never work on a closed source game, so most of what you/Ethan are talking about doesn't concern me. The situation changes when you stop trying to sidestep distro maintainers, and start committing to working with them. I get that it's inconvenient to do that and you're not just being difficult, but surely you know the trade-offs.
On Linux, relying on system provided dependencies, especially C++ ones, is guaranteed to break over time and over different distributions. So you would have to maintain your Linux version either way, shipping with your dependencies is just less insane.
In general, yes they will. You don't have to do much besides communicate to them that it's safe to bump a dependency and won't cause breakage. And you don't have to track down a dozen maintainers either, simply putting it in your release notes where they can find it is enough. The main reason why packages are dropped is because of lack of maintenance, possibly caused by absent or unresponsive upstream. That's what outdated means. Do you think packagers are going through all the trouble of dropping unmaintained versions of Python 2 on a whim?
Dynamic linking is probably your path if you're developing an app that is packaged and shipped with some distribution of linux and you use standard interfaces.
But yeah, a game would make this all wrong. With dynamic linking your game would last one release cycle. As soon as ubuntu 20.04 comes out, your 18.04 game would stop working. You'd get some weird shared library mismatch in your 100 dependencies and that would be that.
try this for fun:
cd /usr/bin
ldd mplayerBut I see no mention of those tradeoffs. Examples: what if the system-provided library is old or has bugs? What if you're delivering to multiple distributions (or even OSes) and you need a consistent behaviour? What if you need different versions of a library?
It never even mentions the reasons why people choose static linking, let alone attempt to refute them.
What if the 1.0.1 version of the library isn't 100% compatible with the 1.0.0 and the app now crashes?
Semantic versioning is nice in theory, but you can't rely on everybody getting in 100% correct and not introducing obscure bugs due to slightly different memory layout of structs.
It also makes testing apps more difficult as you now doesn't know exactly what code the clients are running as there can be an incredible high number of permutations of library versions.
Dynamic linking also makes it impossible to do link-time optimizations (e.g. inlining, dead-strip, locating code near the function that calls it)
- With Dynamic Linking, it's the job of the system admin (typically a distribution maintainer) to ensure that the system-provided library and application are compatible — and add patches if needed;
- With Static Linking, the package developer takes control of the whole experience; they deem more important to deliver a working application to the end user than to ensure compatibility with an OS.
This makes sense for isolated applications, e.g. video games. For a system's core stack (coretools, X11, etc.), the end user expects to receive security fixes in a timely manner. From the distribution's point of view, it seems simpler to have a single library to patch, recompile and push, than to have the maintainer of each package statically linked against said library release an updated version...
Dynamic linking as implemented by most UNIX-Like OSes has a huge drawback: you only have one system copy of any given library, so all your applications are stuck using the same copy. There's no easy way to have multiple versions (say, with different features) of the same library without giving them different names. This isn't really a failing of dynamic linking, it's a failing of the OS library management model.
Dynamic linking in general has some other drawbacks: you lose Link-Time Optimization, there's more overhead at load time (and on the first call with lazy linking), and a few other performance-related issues can crop up on occasion.
There's also the issue that replacing a dynamically linked library is a lot easier than replacing a statically linked one for an attacker, so dynamic libraries can allow for vulnerabilities to be introduced (though this also is an OS issue, and there are often good mitigations for it).
This is clearly not true. Libraries are versioned and one can have unlimited versions installed. Applications can link to the most specific or most generic version desired.
When using shared linking dynamic libraries can link some of their dependencies statically into the .so, using strictly local symbols, allowing the application to load it even if it already links some other version of the same dependency.
We had this problem at my workplace where we developed a component that needed to embed a relatively modern version of the Lua interpreter. When the team that developed the application that used our component tried to link it, they found out that some ancient component they also had been linking in for some completely different and unrelated purpose had a transitive dependency on Lua 5.0, released almost 20 years ago. For backwards compatibility reasons it was not an option to modify the old component to use the newer Lua version, and linking both Lua versions statically into the same binary is impossible. This would never have been a problem if we could have shipped our component as a shared library (which we already build and use for other internal applications), because there the Lua interpreter could have been linked into the shared library privately.
--redefine-sym old=new
Change the name of a symbol old, to new. This can be useful when one is trying link two things together for which you have no source, and there are name collisions.
--redefine-syms=filename
Apply --redefine-sym to each symbol pair "old new" listed in the file filename. filename is simply a flat file, with one symbol pair per line. Line comments may be introduced by the hash character. This option may be given more than once.
main.o
component_for_lua5.0.a (modified: referenced Lua symbols renamed)
component_for_lua5.2.a (with symbols left intact)
liblua.a (from version 5.0, modified: symbols renamed)
liblua.a (from version 5.2, its symbols left intact)
Edit: there is one pretty glaring downside to this approach though, which is that both the component we need to patch up with renamed symbols and the liblua.a are also used in other places, so they would either have to be built differently depending on the application, or we would have to maintain multiple versions of the same 5.0 liblua.a and change the build config/detect logic, for even more confusion and ugliness >_<
wouldn't symbol versioning allow to do something like this ? it needs a bit of modification of the Lua sources but that can likely be quickly automated with some bash script
annotate lua 5 functions definitions to look like
__attribute__ ((__symver__ ("lua_func@VERS_50")))
void lua_func_v50() { ... }
__attribute__ ((__symver__ ("lua_func@VERS_51")))
void lua_func_v51() { ... }
asm (".symver lua_func, lua_func@VERS_50");
Both him and the authors of glibc.
In the same way as containers per application is trendy now. So far it we are encountering all the drawbacks and limitations that used to be known long time ago and that once leaded to dynamic loading...
Except that for using third-party libraries and distributing apps or games, there is no reason to use dynamically linked libraries here and also in software distribution.
I guess if you talked to iOS / Android app developers who complained about dynamically linked third-party libraries that slowed app execution startup times, desktop app developers who wanted a single statically-linked executable that prevents their app being hijacked with a malicious dynamically linked library via LD_PRELOAD or DYLD_INSERT_LIBRARIES or in general developers or users don't want to have to upgrade other dependencies using dynamically linked libraries to install your software, then I guess one can say it would be better to use static binaries here.
Who's to say that a hacker can simply modify a .dll, .dylib or .so in your game and enable developer cheats to ruin it. The authors point on LD_PRELOAD alongside with 'security benefits' is nothing more than outdated and has now been intellectually bankrupted.
I'm sure there's a ton of multiplayer games out there that statically linked things like libpng and networking libraries, and by now are an open door into the systems of anyone playing online.
you know that even when they are dynamically linked, the developer just ships the dlls next to the app right ?
I LD_PRELOAD (and DYLD_INSERT_LIBRARIES) code into every process I spawn from the command line, except setuid executables (and sometimes even then–I jump through the hoops necessary to make this work), because it lets me extend compiled applications in arbitrary ways to better suit whatever I want them to do. Saying that my usecase is "outdated and…intellectually bankrupted" just because you're not aware of the very real and very convenient benefits of function interposing is the real ignorance here.
Surely a big reason containers became popular is because you can avoid all the massive headaches you get when deploying dynamically linked software.
I don't know, it just never clicked for me. Seems like a solution that appeals to people who like spending most of their time in config/build files than actually using a system, but maybe I'm just a curmudgeon/haven't used it right.
Quick real-life example: I had a program that heavily relied upon curl library and fork(), compiled (dynamically linked) and ran in debian 8 would work flawlessly, when ran over debian 9 would have a lot of memory leaks and sometimes SEGVs, when statically linked the program would ran the same way (good) in debian 8 and 9. It took me days to track down and find out that unlike in debian 8 in debian 9 the curl library uses a threaded dns resolver which of course is not a very smart thing to fork() after, so the new curl library had a new feature that messed up the program logic completely. So, no, I would say static linking is way better than dynamic because is CONSISTENT, the library functions you call are guaranteed to behave the way you think they will. Also the ltrace, LD_PRELOAD, etc doesn't legitimize the use of shared libs, why would you debug a release binary?
Yes, and all of your bugs can be introduced in a new single place as well!
Seriously though. People love static linking because it solves more problems than it introduces, despite its downsides.
I will argue that it’s easier to keep a system secure with static linking. You can upgrade the service that is insecure immediately. Or else you need to wait for a Big Bang deployment, and it can be held up by some random thing that isn’t even in the critical path.
Every single technical decision has trade-offs. You might make a case that you (usually) shouldn't do X in cases like Y, or maybe even that you (usually) shouldn't do X in general, but it's a very big statement to say that something is flatly harmful. Very, very few things are so simple. If you aren't Edsger Dijkstra, there's a good chance you're outing yourself as either inexperienced, or willfully ignorant, by claiming them to be so. Yet this meme has encouraged lots of people to do just that, by making dogmatism easy and catchy.
I think the main content dates from at least 2004 from when it was on Redhat's site: https://web.archive.org/web/20041221091001/http://people.red...
The Clarification looks to have been added in late 2006: https://web.archive.org/web/20061119024251/http://people.red...
- A bugfix to a library requires every program that uses it to be relinked. It's true for dynamically linked programs, too, they are relinked over and over. The solution to a forgetful admin is a freaking makefile.
- Security by obscurity, don't care. But if you care, addresses can be randomized during static linking, which should happen on the target system anyway (see previous point).
- Nobody cares anymore. Hard disks are enormous, memory is enormous, a tiny fraction of either is taken up by binaries. Unless you're on a microcontroller, where you don't have enough ram to even consider dynamic linking.
- No! Bad Ulrich! Terrible, horrible, awful design! Bad Ulrich! The "features" of glibc that require dynamic linking should be taken out and shot. The stuff that is too complicated to be implemented without dynamically loading code should be moved into a separate process. Just use dbus, it's pretty much needed for everything anyway. Or embed a scripting language. No, not the one implemented by ld.so!!
- Related, stop sabotaging statically linked binaries already, will you, Ulrich?
- Wrong! You can distribute objects and link on the target system. Which you have to do anyway, see above.
- I'm not sure how I feel about those 'hacks'. At any rate, if we want these hacks, we also get the complexity of the dynamic linker, some form of DLL hell and "Warning: gethostbyname() requires the exact version of glibc, etc..." I don't like the tradeoff, maybe because I don'd debug using LD_PRELOAD shenanigans.
- Bonus, from the clarification: What, fewer DSOs make startup faster? Didn't you just say, that with prelinking, dynamic linking was as fast as static linking? Did you just make that up?
I get the distinct feeling, Ulrich Drepper is on a crusade to eradicate static linking, and these "arguments" taste a lot like motivated reasoning. I don't know what's driving this guy, but things like the terrible implementation of the NSS cause actual problems in practice. Has he just found a shiny toy and refuses to let go of it? Does he get off on knowing that every program in existence runs a script on startup and he's in control of the interpreter (ld.so)? I don't know, but rationally, his arguments make no sense.
MUSL is great software, btw.
- LTO - small libraries/huge pages can link multiple libraries to one page - lower overhead on first call assuming you lazily link