Linux and GCC today have the ability to compile and run fully static executables, I don't understand why this isn't done...
Because when there's a security update to (say) OpenSSL, it's better for the maintainers of just that library to push an update, as opposed to forcing every single dependent to rebuild & push a new release.
This is only slightly more relevant for pure system administration scenarios where the machine is exclusively running software prebuilt by some third-party vendor (e.g. your average Linux distro package repo). Even then, unless you're doing blind automatic upgrades (which some shops do, but it carries its own set of risks), you're still hopefully at least testing new versions and employing some sort of well-defined deployment workflow.
Also, if that "security update" introduces a breaking change (which Shouldn't Happen™, but something something Murphy's Law something something), then - again - retesting and rebuilding a runtime environment for a dynamically-linked dependent v. rebuilding a statically-linked dependent is a distinction without a difference.
Today, with containers becoming increasingly the de facto means of deploying software, it's not so important anymore. The upgrade process is now: (1) build an updated image; (2) upgrade your deployment manifest; (3) upload your manifest to your control plane. The control plane manages the rest.
The other reason to use shared libs is for memory conservation, but except on the smallest devices, I'm not sure the average person cares about conserving a few MB of memory on 4GB+ machines anymore.
I think that's something of an exaggeration.
Yes, containers are popular for server software, but even then it's a huge stretch to claim they are becoming de facto.
And obviously video game distribution has looked like this since basically forever as well.
There's a file hundreds of megabytes large containing all the dynamically-linked system libraries on iOS to make your apps work.
Beyond containers / isolated runtime environments, the parent comment is correct about games (specifically of the console variety) being historically nearly-always statically-linked never-updated monoliths (which is how I interpreted that comment). "Patching" a game after-the-fact was effectively unheard of until around the time of the PS3 / Xbox 360 / Wii (when Internet connectivity became more of a norm for game consoles), with the sole exception of perhaps releasing a new edition of it entirely (which would have little to no impact on the copies already sold).
They do on XBox, Swift, iOS, Android sandboxes.
This assertion makes no sense at all and entirely misses the whole point of shared/dynamic libraries. It's like a buzzword is a magic spell that makes some people forget the entire history and design requirements up to that very moment.
Assuming you use containers, you're likely to not log into them and keep them up to date and secure by running apt-get upgrade.
The most common workflow is indeed: build your software in your CI system, in the last step create a container with your software and its dependencies. Then update your deployment with a new version of the whole image.
A container image is for all intents and purposes the new "static binary".
Yes, technically you can look inside it, yes technically you can (and you do) use dynamic linking inside the container itself.
But as long as the workflow is the one depicted above, the environment no longer has the requirements that led to the design of dynamic linking.
It's possible to have alternative workflows for building containers: you could fiddle with layers and swap an updated base OS under a layer containing your compiled application. I don't how common is that, but I'm sure somebody will want/have to do it.
It all boils down to whether developers still maintain control over the full deployment pipeline as containers penetrate the enterprises (i.e. whether re retain the "shift to the left", another buzzword for you).
Containers are not just a technical solution, they are the embodiment of the desire of developers to free themselves from the tyranny of filing tickets and waiting days to deploy their apps. But that leaves the security departments in enterprises understandably worried as most of those developers are focused on shipping features and often neglecting (or ignoring) security concerns around things that live one layer below the application they write.
I think they still make sense for base-system libraries, but unfortunately there is no agreed upon definition of 'base-system' in the wild west of Linux.
Static libraries do nothing of the sort. In fact, they make it practically impossible to pull it off.
There's far more to deploying software than mindlessly binding libraries.
Furthermore, installing that program will (again, in 90% of cases at least) not affect my overall system configuration in any way. I can be confident that all of my other programs will continue to work as they have.
Why? Because any libraries which aren't included in the least-common-denominator version of Windows are included with the download, and are used only for that download. The libraries may shipped as DLLs next to the executable, which are technically dynamic, but it's the same concept—those DLL's are program-specific.
This ability is what I really miss when I try to switch to desktop Linux. I don't want to set up Docker containers for random desktop apps, and I don't want a given app to affect the state of my overall system. I want to download and run stuff.
---
I realize there's a couple of big caveats here. Since Windows programs aren't sandboxed, misbehaving programs absolutely can hose a system—but at least that's not the intended way things are supposed to work. I'm also skipping over runtimes such as Visual C++, but as I see it, those can almost be considered part of the OS at this point. And I can a ridiculous number of versions of MSVC installed simultaneously without issue.
One program? How nice. How about 10 or 20 programs running at the same time, and communicating between themselves over a network? And is your program configured? Can you roll back changes not only in which versions if the programs are currently running but also how they are configured?
> This ability is what I really miss when I try to switch to desktop Linux. I don't want to set up Docker containers for random desktop apps,
You're showing some ignorance and confusion. You're somehow confusing application packages and the natural consequence of backward compatibility with containers. In Linux, deploying an application is a solved problem, unlike windows. Moreover, docker is not used to run desktop applications at all. At most, tools like Canonical's Snappy are used, which enable you to run containerized applications in a completely transparent way, from installation to running.
But isn't the reason to have this fully controlled and fully configurable environment to have a proof of interworking? Because when environment is in any form different you can, and people already do, say that it's not supported.
No, because there's far more to deploying apps than copying libraries somewhere.
Which is exactly the same selling point as for static linking.
Containers are popular because shared libraries cause more trouble than they are worth.
Diffing two docker images to determine the differences between builds would be far less onerous than attempting to diff a new deployment against a long-lived production server.
Shared libraries are shared (obviously) and get updated, so they're mutable. Linux systems depend on a substantial amount of shared mutable state being kept consistent. This causes lots of headaches, just as it does in concurrent programming.
You must have close to zero experience them because that's the norm on any software that depends on, say, third-party libraries that ship with a OS/distro.
Recommended reading: Debian's openssl package.
You can either employ an extremely disciplined SecOps team to carefully track updates and CVEs (you'd need this whether you're linking statically or dynamically) or you can use e.g. Debian to take advantage of their work to that end.
Isn't a containerized solution a good compromise here? You could use Debian on a fixed major release, be pretty sure what runs and still profit from their maintenance.
> Every single company that I used to work for had an internal version of Linux that they approved for production.
I can't deny your experience, but meanwhile I've been seeing plenty of production systems running Debian and RHEL, and admins asking us to please use the system libraries for the software we deployed there.
> Internal release cycles are disconnected from external release cycles.
That seems to me like the opposite of what you'd want if you want to keep up with CVEs. If you dynamically link system libraries you can however split the process into two: the process of installing system security updates doesn't affect your software development process for as long as they don't introduce breaking changes. Linking statically, your release cycles are instead inherently tied to security updates.
> We had to do emergency patching for CVEs and bump the versions in every service.
What is that if not tying your internal release cycles to external release cycles? The only way it isn't is if you skip updates.
> This process do not depend on Debian's (or other FOSS vendor's) release cycles and the dependencies are explicit, therefore the vulnerability assessment is simpler (as opposed to go to every server and check which version is installed). Don't you think?
I don't know, going to every server to query which versions of all your software they are running seems similarly cumbersome. Of course, if you aren't entirely cowboying it you'll have automated the deployment process whether you're updating Debian packages or using some other means of deploying your service. Using Debian also doesn't make you dependent on their release cycles. If you feel like Debian isn't responding to a vulnerability in a timely manner, you can package your own version and install that.
I'm talking about the operating system that's pretty much a major component of the backbone of the world's entire IT infrastructure, whether its directly or indirectly through downstream distros that extend Debian, such as Ubuntu. Collectively they are reported to serve over 20% of the world's websites,and consequently they are the providers and maintainers of OpenSSL that's used by them.
If we look at containers, docker hub lists that Debian container images have been downloaded over 100M times, and ubuntu container images have been downloaded over 1B times. These statistics don't track how many times derived images are downloaded.
I am convinced that Drepper's insistence on dynamic linking has set the linux desktop useability and developer friendliness back literal decades.
Now it seems even the most trivial of apps needs more than that just to start running, and on a workstation, less than a year old with 4 cores of i7 and 32GB of RAM, I still experience lots of lag and swapping (fast SSD helps, althougn not much) doing simple things like reading an email.
According to Activity Monitor, right now:
• 4.26 GB are being used by apps
• 19.52 GB are cached files
• 8.22 GB are just sitting idle (!)
Now, I'm not running anything particularly intensive at the moment, and I make a point of avoiding Electron apps. I also rebooted just a few hours ago for an unrelated reason.
But the fact is that I've monitored this before—I very rarely manage to use all my RAM. The OS mostly just uses it to cache files, which I suppose is as good a use as any.
I do that personally too, but in a work environment that is unfortunately not always possible --- and also responsible for much of the RAM usage too.
What I failed to mention was that the rootfs is also eating into that (ramdisk). In your case I'm guessing your rootfs was on disk.
https://answers.microsoft.com/en-us/msoffice/forum/all/teams...
There's a screenshot in there showing it taking 22GB of RAM. I've personally never seen it go that high, but the 10-12GB of RAM that I have seen is absolutely ludicrous for a chat app. Even when it's initially started it takes over 600MB. Combine that with a few VMs that also need a few GB of RAM each, as well as another equally-bloated Electron app or two, and you can quickly get into the swapping zone.
I remember comfortably browsing webpages with lots of large images and animated GIFs in the early 2000s, with a fraction of the computing power I have today. Something has become seriously inefficient with browser-based apps.
Probably the GGP said they experience lag while "doing simple things like reading an email." Now, maybe GGP meant to add "while I'm sequencing genes in the background", but since that was left out I can see how it would be confusing! :)
Could someone estimate how much software nowadays is bloated by duplicated modules?
It doesn't alter the fundamental point: shared libraries save both persistent storage and runtime memory.
Which is a significant fraction of everything even if you call simple like printf.
> It doesn't alter the fundamental point: shared libraries save both persistent storage and runtime memory.
I fail to see the argument for this. Dynamic linking deduplicates dependencies and allows code to be mapped into multiple processes "for free".
My dynamically-linked executable is 8296 bytes on disc. My statically-linked executable is 844,704 bytes on disc.
So if I had a "goodbye world" program as well, that's a saving of about 800KB on disc.
Now one can argue the economics of saving a bit under a megabyte in a time where an 8GB microSD card costs under USD5 in single quantities, but you can't argue that it's a relatively big saving.
At runtime, the dynamic version uses (according to top) 10540 KB virtual, 540 KB resident, and 436 KB shared. The static version uses 9092 KB virtual, 256 KB resident, and 188 KB shared.
I haven't investigated those numbers.
https://stackoverflow.com/questions/3430400/linux-static-lin...
I've also linked this Zig post into that list (and happy to add further languages if you can provide a link that shows that they have good out-of-the-box static linking support).
I never realised people were moaning about shared libraries.
(e.g. I have Firefox running under Capsicum: https://bugzilla.mozilla.org/show_bug.cgi?id=1607980)
Looking at how a browser, an IDE, and a few compilation processes will gladly chew through 8GB of memory... it’s not necessarily horrible, but this is a modern contrivance.
Admittedly you could put that on the Kubernetes folks, but the same problem doesn't exist with glibc.
Other have mentioned the other points: runtime loading (plugins), CoW deduplication and thus less memory and storage.
Also known as plugins.
It's not a design flaw, it's a feature.
Dynamic linking was added around Slackware 2.0 timeframe.
$ cat hello.c
#include <stdio.h>
int main() {
printf("hello world!\n");
}
$ gcc -o hello hello.c
$ ldd hello
linux-vdso.so.1 (0x00007ffff9da0000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fed449d0000)
/lib64/ld-linux-x86-64.so.2 (0x00007fed45000000)
$ gcc -o hello hello.c -static
$ ldd hello
not a dynamic executableIf you're the kind of person who wants static linking then you really don't want these features.
The real problem is that statically linked programs under Linux don't (didn't?) support VDSO, which means that syscalls like gettimeofday() are suddenly orders of magnitude slower.
In the end, we had to do a kind of pseudo-static linking - link everything static except glibc.
glibc is GPL licensed, and the GPL explicitly forbids statically linking to it unless your code is GPL too.
Thus any non-GPL project has it's license tainted by the GPL if you statically link it.
It's not a technical limitation, it's a legal one.
It's FUD.
See here -
https://www.gnu.org/licenses/gpl-faq.html#LGPLStaticVsDynami...
Dynamically linking a GPL library is the same as statically linking a GPL library; the resulting executable must be GPL-licensed.