Hacker News Reader

Comment by mittermayr | Hacker News Reader

Would be cool to get a bit of a "why this matters" intro on repos like this. I clicked through the details, but left wondering if I have any potential use for this. Can I compile this onto a USB stick and use it as a throw-away boot Linux for maintenance tasks? Can I cross-compile this for embedded devices? What is the statically-linked advantage here? Not trying to minimize the effort, I would actually love to see more such things, but felt a bit left out as I couldn't figure out why exactly it exists.

View on HN

It seems like your definition of "why this matters" is "how can this pick me up as a user" but for some projects "why this matters" is "to see if this works as well as we think it could" or similar more abstract goals. If the goal of the project is to try to be as small and simple of a Linux distro as possible by trying a different take and seeing how it pans out (which it seems like this is) then it's accomplishing it's goals whether or not it tells you how you could use such a distro were it to pan out.

I don't think you're disagreeing with each other - a bit of intro text especially helps if the project is abstract.

The description of "I built this to try something cool, but maybe someone could find a use for it!" is just fine. But it would be great to know. Likewise: "We built at company Y this for use case x and intend to support it" also tells me what is going on.

As someone who uses linux casually, I often experience pain when I download a program, try to run it, and it fails because some dependency is not installed or is not the right version. So you try to figure out how to install it and it in turn needs a couple of things.

All of this works fine if whatever package manager your distro uses has the program you want, but the package managers don't have evertying!

So I'd love an ecosystem where things tend to be statically linked unless there is a good reason not to.

> So I'd love an ecosystem where things tend to be statically linked unless there is a good reason not to.

Me too, but I'm not sure this project brings us closer to that goal.

I don't need the base OS to be statically linked—in fact, that's where static vs dynamic linking matters least, because it all comes preinstalled. What I want is for everything else which I may want to install on top to be available as statically-linked binaries.

Since most common applications have moved to the web, and the biggest Linux end user deployment by far, Android, was created from fresh beginnings, the idea of the user being concerned with minutia of binaries is already far along a long road to the grave.

This right here is what's wrong with the application space. The idea that this could be true leads to terrible software design and slow, bloated applications. In the current landscape is it perfectly possible (and I'd say even usual) to do most of your work and processing on local system binaries (that aren't a browser or a browser with a different name).

It’s no one’s fault really. As long as people pirate software, it is economically Darwinian / inevitable that all commercial software will migrate to centrally controlled subscription gated servers. This includes open source software, if you correctly count how much of it is repackaged and sold to people this way.

It has nothing to do with pirating. That sort of predatory mercantile behaviour has happened basically ever since humanity invented money. It happens because unscrupulous people are allowed more power than they should have. Stop blaming everyone else for their bad decisions.

Eh, but if that was completely true I'd be using ChromeOS, or something similar. Perhaps some day I will—but as of 2020, I still find myself popping into a lot of native apps day-to-day.

99% of the apps I use aren't on the web at all

> Since most common applications have moved to the web

No, just please no!

This is true, though perhaps there is some performance benefit.

Check out kiss linux, another interesting project: k1ss.org/install

I'm considering trying it: having the simplest distribution baremetal, and keeping the complexity "outside" and "contained" (docker for servers, flatpack for desktops and laptops, etc.) makes sense to me

> So I'd love an ecosystem where things tend to be statically linked unless there is a good reason not to.

Well, every time there is a bug fix in one of your dependencies you/your distribution would have to recompile everything that depends on it. Which will use quite some resources.

So to actually use this you would have to reinstall (nearly) all programs once a week or so (no matter if source or binary distribution). While that is quite possible with modern internet connections, I am not sure if it solves all the problems you might think it solves. After all, even nowadays application authors could provide static packages, but many don't do it.

After using quite a few distributions over the years, I like Arch Linux best at the moment, as the combination of binary distribution, with frequent updates, and the source based AUR, with the very large amount of packages, suits my needs very good.

>Well, every time there is a bug fix in one of your dependencies you/your distribution would have to recompile everything that depends on it

Well no, I could opt not to apply the bug fix if it isn't affecting me. This again is sort of serve mentality, where of course you want to get a bug fix in as quick as possible in case it is a security issue. A desktop user, maybe you don't.

>After all, even nowadays application authors could provide static packages, but many don't do it.

Right, and really what I would like is for application authors to just static link more, more often, on linux. Or at the least ship their programs with installers that apply the needed deps if they are not already there! That the OS itself is statically linked doesn't really matter, but perhaps it would inspire an ecosystem that does this.

> I like Arch Linux best at the moment

Probably suggest arch linux to someone complaining about having to track down dependencies does not make sense, since such a person is more interested in the OS just working rather than making the OS a bit of a hobby that that they want to customize.

Which, is why I mostly use windows.

> Probably suggest arch linux to someone complaining about having to track down dependencies does not make sense, since such a person is more interested in the OS just working rather than making the OS a bit of a hobby that that they want to customize.

having used Red Hat, Mandrake, Ubuntu, Debian... Arch Linuxis by far the most "just works" distro I've used

Do you really mean Arch Linux or something based on it like Manjaro? Pretty sure Arch Linux doesn't even come with an installer.

there are plenty of installers of archlinux - I use https://anarchyinstaller.org/ for instance

But do any of those come with the ISO available at https://www.archlinux.org/download/? I don't see how you can say something "just works" if you have to go out of your way to choose an installer for it.

ArchLinux is not defined by the ArchLinux ISO ? it will be arch as long as it installs packages from the arch repos, which is what those installer do. So you just burn this iso to an usb and be happy: https://gitlab.com/anarchyinstaller/installer/-/releases

Sure, in the beginning you have to spend some time to set it up, but due to the rolling releases it has a great longevity. And talking about application integration, I prefer the package management style (even through AUR) greatly to the chaotic download some binary and execute it way windows does it. Over time your windows always gets so messy because every device manufacturer seems to think that it is cool, to bundle his device driver with hundreds of megabytes of non-sense-ware (last weekend I installed some Logitech driver on Windows, because the default Windows driver didn't work (on Arch I didn't have to install anything extra, just saying... yes, anecdotal evidence)).

>Which, is why I mostly use windows.

Oh, Windows can have dependency hell. And even worse is when you have a WinSXS blowout and you're trying to figure out why that subdirectory is using up 80GB of space for an unknown reason.

DLLHell

Then recommend Manjaro, not an OS that “just” loads them down with ads, tracking, forced updates, and more unless — and to some extent still even after — they customize it.

At the risk of starting a distro war in the comments, I'll have to recommend Pop!_OS over Manjaro.

Pop has broken itself less often then Windows for me, while Manjaro has broken itself slightly more.

> every time there is a bug fix in one of your dependencies you/your distribution would have to recompile everything that depends on it

Back in the dark ages, some mainframe operating systems like MVS normally stored executables as object files, and would link the program every time you executed it.

Has the on-disk space savings and updateable libraries of dynamic linking, without most of the complexity. I believe the original motivation was to allow an executable to run at an arbitrary address on systems without virtual memory though.

I would be surprised if there was not a way to take a dynamic executable, "add in" all the required libraries, and return a fat static executable

If you know of such a utility I'd like to see it, because I've looked and never found it. The closest I've seen are things like AppImage.

Shell scripting and ld tools should allow you to make that yourself.

EDIT: people are apparently entitled to solutions for their problems...

> Care to elaborate?

> Show us a script then! Your internet fame awaits.

I'm not interested in fame, or in reinventing the wheel.

Apparently, you are unfamiliar with this process, so TLDR: there are basically 2 approaches, in order of complexity: 1. working with ldtools or 2. running the executable to find the various parts then dumping a binary, (3 if you consider putting the libraries in a directory then run with LD_LIBRARY_PATH=/thisdir myexecutable)

Let me google some existing tools for you:

https://github.com/systems-nuts/dynamic_to_static

http://bitwagon.com/jumpstart/jumpstart.html

http://statifier.sourceforge.net/

https://www.ucc.asn.au/~dagobah/things/make-static.html

https://web.archive.org/web/20130114222319if_/http://dagobah... with patch and explanation on https://stackoverflow.com/questions/17390722/how-can-i-conve...

A better approach from 15 years ago is slinky, so that you can "update" the libraries if needed: https://www.usenix.org/legacy/publications/library/proceedin...

There's nothing fancy in there. It just requires some effort.

Care to elaborate? Because I'm not aware of any tool that could be used to combine a binary and all its .so dependencies (and all of theirs recursively) into a single statically-linked binary.

It exists. I came across at least one such project last year, but I don't recall the name. :(

I have used exodus to do this in the past. https://github.com/intoli/exodus

> Exodus handles bundling all of the binary's dependencies, compiling a statically linked wrapper for the executable that invokes the relocated linker directly, and installing the bundle in ~/.exodus/ on the remote machine. You can see it in action here.

That isn't really what is being described, or at least not what I'm referring to. It's more like a self-extractor.

You are right. My mistake. I remembered it wrong.

My admittedly poor understand of macOS application bundles, is that is basically how they work (modulo certain Frameworks in Library folders)

It was for personal operating systems (and ancient mainframes) that had a multitasking OS but without any MMU support in the hardware. Relocatable executables were required so that you could load multiple executables at different addresses within the single address space of the memory. This is now called Position Independent Code[1].

Modern MMUs meant that the OS would define specific addresses for use by a program, so for example all ELF programs in a distribution would have the same starting address in memory hard coded into the executable (one virtual address within a process is mapped to a different physical memory address for each process by the MMU).

[1] https://en.wikipedia.org/wiki/Position-independent_code

There’s a difference. Position independent means you can move the executable anywhere and run it.

Relocatable means the binary has enough information to fix any absolute addresses in the code after that move. That’s extra work, and means you cannot have the same physical memory mapped to different virtual addresses in different processes, and that you have to either swap out the relocated memory, or repeat the relocation when you swap in the code again (I’m not aware of any OS doing the latter, and don’t see a nice way to do it, but it is a possibility).

https://en.wikipedia.org/wiki/Position-independent_code:

“In computing, position-independent code[1] (PIC[1]) or position-independent executable (PIE)[2] is a body of machine code that, being placed somewhere in the primary memory, executes properly regardless of its absolute address.”

https://en.wikipedia.org/wiki/Relocation_(computing):

“Relocation is the process of assigning load addresses for position-dependent code and data of a program and adjusting the code and data to reflect the assigned addresses.”

> Well, every time there is a bug fix in one of your dependencies you/your distribution would have to recompile everything that depends on it. Which will use quite some resources.

This is far for more clear cut. Often some dependency bugs, might not impact my program. Even worse, sometimes by fixing a bug my program does not care about, a dependency might break my program in some subtle way. The author of the program is the best qualified person to make the upgrade/not upgrade decision.

So long as I'm not the one building it, its fine. My internet can redownload everything. It also depends on what libraries are being used. musl does generate smaller binaries than glibc. and if you look at the massive project that is sqlite, building software should only take a couple of seconds. Of course, whether anyone actually goes to such lengths for easy compilation is another matter entirely.

https://sta.li/

You won't have to recompile everything if you cache your build artifacts (e.g., object files); you'll only have to relink which is quite a lot better.

Letting software have dependencies then making the user manage them in the off chance there is some bug in the foundational mostly frozen libraries should be an obviously bad decision at this point.

No other environment forces this kind of complexity on their users and almost no non-enthusiast user wants to deal with this structure.

If you need exotic libraries, you install them into /usr/local/lib with GNU Stow for secondary package management. Then point your loader there first when starting programs that depend on them. Your distro stays pristine and the program runs so everything is good.

Actually that's one problem I've never had with Linux. Even using stuff that isn't in the default package manager seems to just work.

Hot take: A great tragedy of the GNU/Linux ecosystem is the fact that ABI+API is still not figured out, and the most common interface between programs is the C ABI, which is embarassingly primitive even by 80's/90's standards. Some people in the FOSS community just want to leave things as-is to hinder propiertary software, and it's the same story with device drivers. You can debate the merits rightfully so, but then there's still companies pushing out binary blobs which break every few kernel updates. As a FOSS developer it's an eternal fight between good and evil with no winner in sight, as a propiertary developer it's pain in the ass to maintain old and new software between all the permutation of Linux distros, and as a user I get to cry because of the lack of popular software and backwards compatibility. Snaps and flatpaks are an ugly hack, literal ductape around this very fundamental problem, and clearly not the solution. GNU/Linux should have adopted COM and/or an adequete proglang with stable ABI a long time ago, and should have tried to control wasted effort put into duplicate software ecosystems (KDE and Gnome).

That's a searing hot take, to be sure.

There is no alternative to the C ABI. If you look at stable ABIs in other programming languages, they're either identical to or thin shims around the C ABI! Even COM, with the caveat of calling convention on MSVC.

Regardless of ABI stability you also need software to have API stability for stable distribution. The two solutions to this problem are to statically link everything or package your shared objects with your software. Then to launch the software you need a shell script that overrides the loader search path and move on with your life.

The ultimate solution is the .app/.bundle paradigm of MacOS. Everything should copy that.

The other issues of packaging software like entitlements/capability based security and code signing are similarly orthogonal. If you're shipping big software today, it needs to have everything it needs and to ignore whatever is on the user's system unless they explicitly request an override.

The real tragedy of GNU is that the interchangeability of software was always possible, package managers just managed to make it accidental and explicit. Too many foot guns there.

> Then to launch the software you need a shell script that overrides the loader search path and move on with your life.

> The ultimate solution is the .app/.bundle paradigm of MacOS. Everything should copy that.

All ELF systems support the $ORIGIN RPATH macro. It lets you build and package an application exactly like, if not better than, you would a macOS bundle. Specifically, it let's you specify at compile time a library load path relative to the invoked binary. You'd specify it as $ORIGIN/../lib for the canonical bin/, lib/, etc layout.

Awesome tip, thanks! I didn't know this existed.

Example: GCC had been known that GNU worried to add plugin API.

https://lwn.net/Articles/301135/

I haven't done native development professionally, but it seems to me like dynamic linking is a leftover from a time when memory and even disk space were precious, isn't it? Are the savings really still worth all the breakage? What is Docker but a way to force static linking on programs that were built for dynamic linking?

As a primarily Linux developer I'm super jealous of COM. It reminds me a lot of dbus with more legacy cruft but also a lot better designed.

As a former Windows developer, I am thankful that I never have to touch COM ever again.

Damn, guess the grass isn’t greener.

Stable ABIs don't seem like a good solution though. It is preferable that the system running the code should make the decisions on how best execute it. Especially as hardware becomes more specialized and diverse. Additionally, guarantees of stability necessarily increase maintenance burdens, so they should be avoided whenever possible.

IMO, the best solution would be something like SPIR-V. That is, programs are shipped in an intermediate representation which a platform-specific compiler would reduce to the final program. That way no stable ABI is needed, there is less maintenance for distros and toolchains, and end users still get high quality binaries.

Of course there are issues with protecting the intermediate representations of programs from reverse engineering. But the combination of strong DRM with obfuscation should satisfy most software vendors. After all, most companies are comfortable shipping Java based programs.

I believe Apple sort of does this for iOS: developers submit LLVM IR to them and they compile it to send to end users

We should aim to not use Snap, please! :/

> So I'd love an ecosystem where things tend to be statically linked unless there is a good reason not to.

Security: a good distro provides security updates for shared libraries. With static linking it becomes too burdensome.

It's a joy to deploy. I loved the IBM Blue Gene/L. Like AWS Lambda, except instead of a zip you deployed a static linked binary.

Looks like the readme was updated 3 hours after your message, and does a great job of explaining the ideas/advantages.

Reply on news.ycombinator.com