An alternative to shared libraries (2008) | Hacker News Reader

An alternative to shared libraries (2008)

kix.in

34 pointsentelechy16 comments

> I like static linking. But code these days is getting extremely complex and bloated, so people needed an alternative. Instead of focusing on making their code more cleaner and lean, they started thinking about they can share this huge piece of complex and bloated code across several applications. If you think about it, if your code is small and clean, you wouldn’t feel the need for shared libraries.

One person's "bloat" is another person's "critical feature". When I see this word thrown around, I'm always left worried about what is going to come next. And then the author proposes avoiding shared libraries in favor of what sounds like a single-system microservices model. Such a model doesn't resolve any of the versioning issues of shared libraries but does add in all of the issues faced by a distributed systems combined with dramatically higher invocation overhead.

The stuff about plan 9 is interesting - but I'm left wondering why exactly this is being reposted now.

I've done mercifully-little C(++) programming in my life, but the very idea of dll's has always seemed extremely fragile to me. Depending that deeply on the system configuration for basic functioning... ylech.

I also can't help but wonder if they're obsolete these days, given how incredibly cheap disk storage is? Can anyone who works in that sector speak more to that?

These days I mostly work in JavaScript, and say what you will about NPM, at least it provides a consistent, reliable, central strategy for retrieving and organizing external dependencies (assuming you haven't just built them into your bundle). The same is true of Maven for Java, to some extent. It seems like at least in Linux, the package manager will get your DLL's for you and coordinate all that, but as far as I can tell there's nothing similar for Windows? Do Windows programs that rely on DLL's just have to manually check what's already there, whether it matches their needs, and copy files over during the installation process if necessary?

I develop small to medium size Windows applications. It's much easier for me to simply put all the DLLs I need in the installer. Just copy everything into the installation directory. Since Windows looks in the immediate directory first, that eliminates most potential conflicts. There are some things you might install onto the system like C++ redistributables, but even recently I have just started copying those DLLs into the installation directory too. Most people like this because the application directory is fully standalone and can be copied around from PC to PC.

Which totally defeats the purpose of dll's, right? I'm not judging; this just seems like a pretty condemning argument against the practice as a whole

It's not. DLLs also provide a way to share compiled code between programming languages that are not binary compatible.

At least one C++ library that I rely on (that produces an incredible number of template instantiations in the object files) ultimately results in a 2 GB shared object file for the debug version (generating DWARF info for 300,000+ objects takes up a lot of space). The test suite for this project compiles and runs about 5,000 executables that link, in some way, to that giant shared object library. Statically linking the test suite would consume on the order of terabytes (I am not sure the exact number; I have never tried) of disk space which just is not feasible for a workstation, so dynamic linking is the only reasonable option. Dynamic linking also makes testing easier since I do not have to recompile tests as often.

That said, DLL hell is definitely real (as anyone who has ever used windows knows). Things are generally better (though not perfect) with OSs like Debian where dependencies are centrally managed across the whole system. In general, doing C++ development, I have found it advantageous to dynamically link: recompiling one library does not necessitate recompiling everything that depends on it.

Why don't you run your test suite with dynamic linking, then? I've certainly worked at large, successful companies with enormous C++ codebases where unoptimized tests are run with dynamic linking and production binaries optimized, stripped, and linked statically.

> Statically linking the test suite would consume on the order of terabytes (I am not sure the exact number; I have never tried) of disk space which just is not feasible for a workstation

I read complaints like this a lot. Have you ever tried using -ffunction-sections -fdata-sections and --gc-sections as in :https://elinux.org/images/2/2d/ELC2010-gc-sections_Denys_Vla...

I find static linking to take up far less disk space when done properly, but I keep seeing comments like this.

MSVC linker also supports removing unused functions/data from the output binary: https://msdn.microsoft.com/en-us/library/bxwfs976.aspx. These can be applied irrespective of compiler optimizations, so you should still be getting good PGD debug symbols.

Haven't tried it personally, but it's worth a try.

So an alternative to shared libraries is RPC over FUSE-like transport?

It is a sane idea and may work well, but I think the page overcomplicates it. Why bother with filesystem -- having cryptofs implement "getattr" and "readlink" seems like a total waste. Why not use the things designed for RPC?

For example, one may use raw unix sockets (this is how Xorg, and pulseaudio work), or D-Bus (this is how disk mounting in modern desktop linux works), or HTTP-based transports (this will be known as "microservices" then).

Or just… executables? My shell is "dynamically linked" to every executable on the system. Git is "dynamically linked" to my $PAGER. My IDE is "dynamically linked" to my compiler toolchain (which is itself composed of several pluggable components which are "dynamically linked" to each other).

The low overhead, easily composable, encapsulated and abstracted nature of processes is one of the things that made Unix an unusually functional operating system in the first place. Shared libraries are great, but I think a lot of the time people end up producing and linking a .so when defining an old-fashioned shell-compatible interface would be perfectly good and far less trouble.

This proposal is crazy town.

If you think about it, if your code is small and clean, you wouldn’t feel the need for shared libraries.

Not so. The key feature of DLLs is to enable software components to evolve independently. When Apple changes the way a button looks in UIKit, all apps get the updated behavior, because UIKit is dynamically linked. This has nothing to do with the app size or its hygeine.

Replacing dylib calls with filesystems and fread/fwrite is absurd. IPC is much harder than library calls. There's the obvious expense of turning every library call into multiple system calls. It's also less flexible. When the interface is a function call, the function can switch implementations from eager to lazy, add or remove caches, etc. But when your interface is fread() you must provide your data up front. Switching a field to computed on-demand is a breaking change.

But the versioning issues are the worst. Reading from Plan9 /dev/mouse returns 49 bytes. I can't add a new field without breaking apps. The equivalent of adding a method in an OO language is now a breaking file format change in this scheme. The post attempts to addresses this:

With filesystems, it’s trivial to add functionality without breaking applications depending on older versions of your FS. That’s because all the compiler sees is a bunch of fopen/fread/fwrites and is not going to complain if the version of the filesystem changes because it doesn’t know.

The app can't break because the compiler can't detect the breakage. Huh?

Alternatively, if you’re thinking of modifying the behavior of your filesystem; consider providing a version file in the root of your FS right from the beginning. Applications would then write the version number they expect to be working with in that file as a way of initializing the filesystem

This sounds very bad:

1. Every API needs to be able to ingest and emit all past versions of themselves, in perpetuity. This is obviously a huge maintenance burden.

2. This does nothing about the reverse problem: how does an app run on multiple versions of the OS? Instead of, say, using reflection to test if a method is present, apps must be prepared to parse the product of every API they use against every OS version they want to support.

3. Applications do not have a single "version number they expect." Apps are built out of multiple components written at different times which consume different versions of the same API. There's no affordance for that.

So we're forced to use some structured data format with named fields, rather than raw bytes. How did this come out of a desire for a "lean, efficient and small" solution?

Exactly. It’s utopian “freedom” of tyranny like libertarianism. If every app bundled all of its own shared libs, and the FS deduped files and shared library loader deduped memory pages, there would be still be isolation, with sharing AND add the ability to upgrade any piece of software without having to maintain forests of NIX packages to or Habitat individual special libraries to reference count. And, to delete an app, just delete the app, which decrements dedupped files and deletes singular ones... no shared components to clean up! ZFS lz4+dedup plus GNU stow

And where, f.e., SO/DLL allocated memory supposed to live? In SO only text segment is marked ro and mapped shared between many processes. Data (whether is it .data or malloced) is in the particular process memory and is NOT shared

As Docker has proved to me, people don't care about saving space anymore. The time of shared libraries has come and gone. And yes I realize you can use shared libraries in Docker, but at that point it's acting more like a static library oddly enough.

Side note - anytime I see Drepper mentioned I cringe a bit. I don't like to badmouth people, but I can't think of a worse figure in recent open source history.

Reply on news.ycombinator.com