Exodus – relocation of Linux binaries–and all of their deps–without containers | Hacker News Reader

Exodus – relocation of Linux binaries–and all of their deps–without containers

github.com

307 pointspabs365 comments

I often use exodus to move software onto a Synology NAS (after hosing my installation in the past through haphazard use of alternative package managers) and it has been incredibly helpful. It's great to be able to bring over utilities without fighting with docker, building appimages, or the like.

Thank you so much for it!

It just works most of the time, which is wild given how diverse the environments it operates in can be. One piece of software I have struggled with however is w3m.

   Wrong __data_start/_end pair
   Aborted (core dumped)

I get the above error messages which I understand to be related to libc, but I cannot get around it. Is this something anyone has seen before?

I'm pretty sure __data_start/__data_end is the symbol for the first/last initialized data in an ELF binary, so possibly a conflicting glibc or a conflicting or missing library

This probably won't work on binaries that contain references to system files necessary in order to run the binary that aren't code. You usually can't detect them other than running the program and watching it die, or running the program with strace and looking for the files it's trying to stat/open.

In the past what I've done when I have to copy a binary somewhere, is statically compile it in a Docker container and export the install files, then copy those over. I have about a half dozen tools prepped like that (gdb, curl, strace, busybox, etc)

You could also use a package manager utility to list which files are needed, assuming the target binary was installed that way. (pkgs.org lists the files every package installs, or locally you could run something like `apt-files`).

Maybe check out the --detect flag from the README that claims to do just that?

Oh, wow! They've thought of everything.

That will detect files in a system package, but not files in a system package depended on by a system package depended on by a system package. It depends on how the distro packages deps and how apps use them. There are apps which use non-linked files that are only provided in extra packages. And even if you walk the whole dependency graph for a single package and export all listed files, you miss files that are created and updated during the setup steps of a package install (which are not listed in the package).

As long as the automatic way finds most things, and there is any way to manually include arbitrary things, that is actually perfectly great to me.

I don't understand what the complaint or expectation is here.

Just trying to put out there to expect undefined behavior or failures. Users might not understand how applications work and so might expect more than is possible. The strace trick is a nice workaround but I didn't see it in the readme

You usually can't detect them other than running the program and watching it die, or running the program with strace and looking for the files it's trying to stat/open.

You can actually pipe the output of strace into exodus and it will include the files that were accessed by the program in the bundle. For example:

    strace -f nmap --script default 127.0.0.1 2>&1 | exodus nmap

Oh. That's pretty cool

Could this be used to relocate Android-native libraries to other platforms? Even if the architecture matches, the bionic libc calls is what I found more challenging.

The tool moves runs the application and all of its dependencies in a container for compatibility, but on Android with bionic that would probably imply moving all of libc with the app. I don't think that'll go down as easily, as the entire bionic framework is integrated quite deeply with the rest of the system.

For "simple" bionic libc calls, an LD_PRELOAD shim might be enough for many binaries, though. You'd need to translate all the bionic libc calls to your system's libc calls, but that might just be easier than it seems because the level of compatibility between the two. Bionic is actually a subset of POSIX libc, so you should be able to map all calls to your normal system calls.

I don't know how well-maintained the project is, but https://github.com/Cloudef/android2gnulinux might just do the trick for you.

I've tried using https://github.com/libhybris/libhybris directly earlier. Will try this.

Awesome. I wonder if this could be used to make packages that "just work" for NixOS while no one does it "the Nix way".

I'm not sure you can easily link to the bundled libraries

nix already has something similar with pkgs.buildFHSUserEnv it creates a regular liking environment that can run regular binaries

Virtual machines, then lean containers, now this... I predict that in a few years we will rediscover static binaries and we will finally find closure.

After this, we will get back to shared libraries, because updating that libpng would require updating all those static binaries. This is already happening with many go and rust programs.

No? Assuming all processes on a machine _want_ to use the same shared library is what leads to isolation mechanisms.

No need anymore. Both bandwidth and storage are dirt cheap.

Linked libraries became obsolete the moment hard drives became huge.

Only if you don't care about security.

And became relevant again the moment people stopped using HDs in devices that run Linux ;)

And eventually maybe it'll all converge back to the rational way to do things: a base set of shared libraries provided by the OS that maintain strict ABI compatibility, and everything else static.

Right until you want to run more than one release on the same machine...

I think the end goal is what we all want - be able to do any of the options - single binary to external libraries and anything in between easily. Not with the one way going in fashion and the other out but both supported and accessible.

The cool thing would be multi-host static binaries i.e. binaries that host multiple "apps" and its dependencies and use commandline option to launch specific "app" from the binary(or without commandline option, provide a app list from which the user can select that app to run)

So you want in the extreme to package a distribution as unikernel?

Or add a binary launcher in front of a squashfs image of a live distro?

Something like that could be done with AppImage I guess.

So… busybox?

> The cool thing would be multi-host static binaries

My first thought was someone is thinking about something like:

https://ahgamut.github.io/c/2021/02/27/ape-cosmo/

There is also something similar that can make binaries that run even bare metal or as EFI apps. But I can't find at the moment. (Maybe someone else has the link?)

alacarte staticity

I think not. The landscape of software packaging is too unpleasant and chaotic, the goals of its participants--users vs package collection (distro) maintainers vs. software authors--too misaligned.

Instead, what I think (hope) will happen is a simultaneous increase in prevalence of two things: "effectively static" software releases (whether that's an actual static binary or a container/flatpak/exodus-bundled folder/whatever), and sandboxing/isolation tools at the OS level, to prevent the statically linked dependencies in installed programs from causing harm to the system.

There will probably always be exceptions made for software that by necessity must be integrated tightly with the whole computer (window managers etc.), but I don't think people are going to rediscover dynamic linking en masse.

The last discovery is to add all the container security boundaries to a super lightweight container called a process.

I've never seen that linker trick before to avoid rewriting rpaths! Very cool

How does this work when locating libraries recursively though? It seems --inhibit-rpath only applies to the current executable:

    # Create a.out <- one/libx.so <- one/sub/liby.so
    $ echo 'int g(){return 42;}' | gcc -shared -o one/sub/liby.so -x c -
    $ echo 'extern int g(); int f(){return g();}' | gcc -shared -o one/libx.so -x c - -Lone/sub -ly '-Wl,-rpath,$ORIGIN/sub'
    $ echo 'extern int f(); int main(){return f();}' | gcc -x c - -Lone -lx '-Wl,-rpath,$ORIGIN/one'

    # works fine
    $ ./a.out
    $ echo $?
    42

    # can't locate library through rpath with --inhibit-rpath
    $ /lib64/ld-linux-x86-64.so.2 --inhibit-rpath '' ./a.out
    ./a.out: error while loading shared libraries: libx.so: cannot open shared object file: No such file or directory

    # *does* locate liby.so through rpath!
    $ /lib64/ld-linux-x86-64.so.2 --inhibit-rpath '' --library-path ./a.out
    $ echo $?
    42

Tried it, didn't work.

Arch:

  $ exodus /usr/bin/git > ./foo

Ubuntu:

  $ ./foo
  Installing executable bundle in "${HOME}/.exodus"...
  Successfully installed, be sure to add ${HOME}/.exodus/bin to your $PATH.
  $ ~/.exodus/bin/git --version
  fatal: cannot handle x as a builtin

“If you ever try to relocate a binary that doesn't work with the default configuration, the --detect option is a good first thing to try.”

From the README file…

I get literally the same error with --detect. And honestly when the README says "transferring a piece of software that's working on one computer to another is as simple as this" you can't blame me for taking that at face value.

I'm the author and I agree that you're correct in assuming that what you ran should work. I just tested this with git on arch and I was able to reproduce the issue. I'll look into why this is happening and hopefully push up a fix soon, but I also invite you to try it out with another binary in the meantime. There seems to be something particular about git, and I think you'll have better luck trying it with almost any other ELF binary.

My guess is that it’s due to git looking at its own base name to figure out which command it’s supposed to run as, kind of like busybox does.

Yep. Source of that error message. https://github.com/git/git/blob/master/git.c#L879

Fun anecdote about that: at work we added a wrapper around nvcc to point it at the right compiler, and renamed the original "nvcc" binary to ".nvcc-wrapped". But nvcc looks at argv[0] to print out its name in the --version output, and it truncates anything after a period (presumably to handle things like "nvcc.exe"?). And CMake's CUDA detection looks at nvcc --version. So CMake went down a really weird path where it knew that nvcc existed but didn't really believe it was nvcc, which was extremely confusing until I looked at some log output and went "wait, why isn't nvcc printing its own name".

Oh hi, thanks for replying! So funny enough, git was literally the first thing I tried, because it was easily the first thing I could think of where being able to move later versions of it to earlier versions of Linux would've made my life easier. I'm not sure if anything else really falls in this category for me. But on your suggestion, I just tried Python 3.9, clang++, and g++, and didn't get errors for any of them. It's pretty nifty! Thanks for writing it.

clang++ also is just a symlink to clang, and clang uses the filename to change the language mode to c++.

So why does that work?

clang just checks if the name starts with clang++ probably.

I checked this, the logic is as follows. The following suffixes are mapped to the equivalent driver flags:

    {"clang", nullptr}
    {"clang++", "--driver-mode=g++"}
    {"clang-c++", "--driver-mode=g++"}
    {"clang-cc", nullptr}
    {"clang-cpp", "--driver-mode=cpp"}
    {"clang-g++", "--driver-mode=g++"}
    {"clang-gcc", nullptr}
    {"clang-cl", "--driver-mode=cl"}
    {"cc", nullptr}
    {"cpp", "--driver-mode=cpp"}
    {"cl", "--driver-mode=cl"}
    {"++", "--driver-mode=g++"}
    {"flang", "--driver-mode=flang"}

`clang++-x` matches none of these. Then the same logic is applied to the name with any of "0123456789." trimmed from the right. Still no match. Then the trailing `-component` is stripped, and `clang++` matches.

It works by chance...

See https://github.com/llvm/llvm-project/blob/c22b110612600b0d0a... and https://github.com/llvm/llvm-project/blob/c22b110612600b0d0a...

TIL: if you symlink ++ -> clang it works as a compiler in C++ mode.

> TIL: if you symlink ++ -> clang it works as a compiler in C++ mode.

Note that this is only about the compiler driver mode, i.e. the default libraries, search paths, include directories, etc. The source language is determined by the file extension unless specified with e.g. `-x c++`.

See my comment in the parent thread but basically Git commands like ‘git thing’ actually dispatch to lower level commands, usually with the name git-thing (I’m not a git expert, just something I’ve noticed). And it seems that git builtins are dispatched into by calling the git binary with a different basename, and this conflicts with your convention of adding -x to binary names.

Is having the -x necessary in filenames, or could you add a subdirectory and use the same original file name for a bundled file?

The docs explain that internally ~/.exodus/bin/git is actually a shell wrapper that calls ~/.exodus/bin/git-x (which itself might be a symlink). The -x suffix is added by exodus.

My guess is that git is parsing argv[0] to try and determine which porcelain command to launch and gets very confused by the -x because it is not a git builtin command.

So anything that relies on the name of the executing file will break.

Git is extended by adding `git-$commandname` executables to $PATH. So `git-x` would extend git with a new command called `x`, i.e. `git x`.

What are the pros and cons compared to AppImage?

This tools seems easier to use. But the high level result seems quite similar.

Would it maybe even make sense to integrate both tools? I think AppImage is compressed so has an advantage in that point.

There is also outrun: https://news.ycombinator.com/item?id=26504131

Haven’t tested it myself yet but always wanted to

You can use ~ in PATH? This seems to go beyond normal shell expansion - even `which` returns a path with ~ in it, so it must be aware of the syntax.

The tilde is usually parsed/replaced by the shell, so yes it you can use it for path definitions in the shell. The parsing is disabled by single quotes however. A lot of my colleagues were confused by that.

In this case it is not parsed by the shell. Double quotes disable parsing just as well as single quotes for ~. Try running the command in the video and then run "env".

Lot of work going on in this area at the moment.

I still

apt update; apt upgrade

And look for config in /etc

Linux is a community, sharing space is important to me.

Wonderful! Thank you!

Why not just use docker?

Because you sometimes want to distribute binaries, not entire container images?

Can't "entire container images" be nearly the same thing as a binary? (I'm thinking to things like distroless based images)

Container boot time currently prevents this from being true in many scenarios

Sometimes you can't run Docker (maybe you don't have root permissions to install and start the service). Someone else in the tread said that they're using in their Synology NAS, that don't have access to a package manager.

Docker also will have a much bigger overhead since you basically need to include a whole distro instead of just a few libraries. And I particularly find Docker applications to be sufficient slow to start that I never want to use CLI tools via Docker.

I believe something like a static build of crun might work, if the kernel is new enough. https://github.com/containers/crun

Reply on news.ycombinator.com