Systemd replacing ELF dependencies with dlopen | Hacker News Reader

Systemd replacing ELF dependencies with dlopen

mastodon.social

113 pointsklooney90 comments

The interesting part is: https://github.com/systemd/systemd/pull/32234

It's nice to see a proposal that fits so neatly within existing infrastructure and norms. I just hope they either standardize or drop the extensible type field idea instead of letting it turn into a semi-documented mess other projects can't easily adopt.

If they are "optional", why do initrd generators need to know them? This truly has all the hallmarks of a systemd "solution"..

A dependency being optional on a technical level doesn't mean that it's optional on a functional level for everyone. `libfido2` mentioned in the pull request is a great example: if your rootdisk is encrypted using a FIDO token, you need that library in the initrd; but otherwise you don't. By making the dependency optional and initrd generators aware of that optional dependency, people that need it will have things just work, while others don't need to install the library.

Deleted comment

You know that you use a token to unlock your boot device. You don't necessarily know the specific set of libraries needed to enable the feature, though.

Because the optional dependencies actually do things. By default an initrd-builder would probably copy all deps including optional ones but it now enables you to choose not to do that as well, an avenue that was previously not available.

I don't understand the initial motivation for converting regular dynamic library dependencies to dlopen dependencies. How does that help with reducing the footprint?

It makes the presence of those libraries optional: you no longer need them to execute the relevant tool at all. It'll just mean you can't use features which depend on those libraries. For libraries that are only pulled in for less-commonly used features it makes a lot of sense (the specific case they are doing it for is generating the initrd, which needs a copy of any libraries used by anything running in the initrd, which is almost always going to include systemd, but the systemd in the initrd is very unlikely to use any of these optional features)

dynloading libraries can have some side effects.

Which the recent xz attack used to mess with `sshd`, even if it never actually used functions from the library.

dlopen loading only loads the library *when it's actually used*, and doesn't include everything and the kitchen sink on startup.

If the libraries are used seldomly enough, it might also allow to make more dependencies optional in package management.

I think it's also relevant that the xz exploit made use of the fact that by running code before main, they could modify areas of memory that later get turned read-only. Any library that does get loaded with dlopen can of course still attack the process it's in, but it has less tools available to it for evading detection.

So... I repeat the GP.

What you gain is that the vulnerabilities will be harder to track down?

The specific way sshd was infected would not have happened with libxz as dlopen library.

Debian's sshd only uses libsystemd for the notify api. I.e. it doesn't need any feature that uses libxz. If it's dlopen()ed, it does not need to be loaded into the process context to use an unrelated feature.

FWIW, IMO upstream systemd should split their monolithic library and allow users to pick better that way, but this has other implications on DX.

> FWIW, IMO upstream systemd should split their monolithic library and allow users to pick better that way, but this has other implications on DX.

FWIW, upstream systemd has the opinion that no-one should load the library for startup notification, instead they should use the well documented api and just write a message to a socket.

Indeed, that's what the Debian maintainer of OpenSSH did soon after the quick security fix. He replaced the dependency on libsystemd with some hand-made code that notifies systemd by socket. https://salsa.debian.org/ssh-team/openssh/-/commit/cc5f37cb8...

That's not what they said in their update to the documentation[0] last week, which says that "although using libsystemd is a good choice, this protocol can also be reimplemented without external dependencies".

It calls it a "good choice". Did they say somewhere else that no-one should do it despite it supposedly being a good choice?

[0] https://github.com/systemd/systemd/pull/32030/files

Yes- https://mastodon.social/@pid_eins/112202696589971319

sshd should not have used libsystemd in the first place for the trivial notification. And the ifunc stuff is its own security nightmare. Papering over this by dlopen-ing some libs in libsystemd does not address the deeper issues.

FWIW, IMO upstream systemd should split their monolithic library and allow users to pick better that way, but this has other implications on DX.

They've done the exact opposite afaict. Libsysd used to be split up, now it is monolithic.

Upstream systemd should just cut the unused crap from their notification protocol and promise it's stable.

They probably couldn't change it anymore if they tried to.

The systemd notification protocol has been stable and documented as such for years (probably even a decade at this point): https://systemd.io/PORTABILITY_AND_STABILITY/

The whole protocol is "write READY=1 to the socket found in the NOTIFY_SOCKET environment variable".

> What you gain is that the vulnerabilities will be harder to track down?

No, they're just as easily tracked.

What you gain is that you can refuse to install optional dependencies because they are now optional when they used to be required.

That's a fairly big deal.

Of course, in practice all those optional dependencies will be installed anyways because of other things in the distro needing them. You can fairly object that not much changed, at least for now.

A better approach would be to eliminate a lot of these dependencies somehow. Another approach would be to sandbox the dependencies that cannot be eliminated.

It meant that had the bad xz version been shipped to distros, only some people would been vulnerable instead of everyone. That is valuable.

Though only with this particular approach to the backdoor. If systemd had always had this approach (or distros hadn't patched sshd to link it in), the attackers would have focused on a different path from delivering malicious code to widely-used distros which executes in a priviledged context to network RCE.

Wouldn't this systemd feature add a convenient centralized point of attack to inject libraries? Not as open at the user level, but similar to a LD_PRELOAD kind of vulnerability.

Not in a way that isn't otherwise accessible, IMHO. I mean, if you're really concerned about injected vulnerabilities into high-trust software (and you should be!) you should be suspicious of any dynamic linkage at all. But if you're going to do it, doing it late and under affirmative control is almost certainly the right choice.

It's because of how dynamic linking/loading works on Linux. An ELF dependency means that symbols in the library can override symbols in the binary or in other libraries. That doesn't work when the library is loaded with dlopen().

To be honest, liblzma using systemd seems like an unimportant detail more than anything else.

On my laptop there's 141 binaries in /bin/ that link against lzma; an abbreviated list with similar entries omitted below. And that's a fairly minimal Void Linux system – the list for an average Ubuntu system is probably a lot longer. Sneaking in via kmod, grub-install, udevd, or xbps-install seems just as viable as sneaking in via libsystemd. Perhaps a little bit harder, but not much.

It will remain problem as long as you have a single command that runs your malicious code as root. Being directly linked against sshd was convenient, but there's all sorts of devious shit you can pull once you've got root, and it doesn't take that much more effort.

  % for f in /bin/*(*); ldd $f 2>&1 | grep -q lzma && print $f
  /bin/amdgpu-arch
  /bin/bsdcat
  /bin/bsdcpio
  /bin/bsdtar
  /bin/bsdunzip
  /bin/c-index-test
  /bin/clang-*
  /bin/clangd
  /bin/diagtool
  /bin/ffmpeg
  /bin/ffplay
  /bin/ffprobe
  /bin/find-all-symbols
  /bin/grub-editenv
  /bin/grub-install
  /bin/grub-mkimage
  /bin/grub-mknetdir
  /bin/grub-mkrescue
  /bin/grub-mkstandalone
  /bin/kmod
  /bin/lspci
  /bin/modularize
  /bin/mpv
  /bin/nvptx-arch
  /bin/pp-trace
  /bin/pskctool
  /bin/qemu-*
  /bin/rtorrent
  /bin/tiffcp
  /bin/tiffdump
  /bin/tiffinfo
  /bin/tiffset
  /bin/tiffsplit
  /bin/udevadm
  /bin/udevd
  /bin/update-mime-database
  /bin/xbps-alternatives
  /bin/xbps-checkvers
  /bin/xbps-create
  /bin/xbps-dgraph
  /bin/xbps-digest
  /bin/xbps-fbulk
  /bin/xbps-fetch
  /bin/xbps-install
  /bin/xbps-pkgdb
  /bin/xbps-query
  /bin/xbps-reconfigure
  /bin/xbps-remove
  /bin/xbps-rindex
  /bin/xbps-uchroot
  /bin/xbps-uhelper
  /bin/xbps-uunshare
  /bin/xmlcatalog
  /bin/xmllint
  /bin/xsltproc
  /bin/zstd

Wouldn't dlopen make you more vulnerable to runtime library attacks? Instead of having dependencies defined at the build/start up time, they will be loaded at runtime on demand. So the dependency library could be changed at any time and you wouldn't have any idea what code you are loading at this time?

So upgrading XZ could even attack already running sshd too?

I would be more inclined to have static linked libraries instead of having lazy loading of them for preventing this kind of attacks..

> So upgrading XZ could even attack already running sshd too?

With the specifics of how the xz injection attack worked, actually it wouldn't. libsystemd would have been loaded to implement the service-started notification, and then there is no reason to use anything further and therefore nothing to trigger the loading of the xz library.

Which is kind of the point of this change: the goal is to load libraries on-demand instead of eagerly, so if there is nothing demanding the library load, the library is never loaded.

Lazy loading would have achieved the same thing with less codebase churn. However, the `dlopen()` thing means that the dependencies are then optional as far as the packaging system go, and therefore you can run with fewer dependencies installed. Also, lazy loading can be turned off with an environment variable -- we would need a way to indicate that a dependency is optional in order to have something like always-lazy loading for it.

There's also "filters" and "auxiliary filters", which can be used to make dependencies optional without having to go to `dlopen()`. However, IIRC filter functionality is very bare-bones or non-workin in the Linux link-editors and ld.so.

Loading at runtime is a feature. You're supposed to lock down the environment for execution so the right libraries get used. The trouble with statically linking is that you have to relink everything to update. You also can't easily audit which versions are in a binary. It seems appropriate for some small userspace programs but not for complex software like services with many dependencies.

I think this isn't motivated by security at all. That it messed up the backdoor was a complete coincidence.

That's correct, as evidenced by the fact that the systemd change to dlopen() liblzma was proposed and merged before the backdoor was discovered.

This seems a bit nitpicky: you just move the threshold for where you might load a backdoored library from first exec to first use which dlopens that library (which might even just be at exec anyway, I haven't checked the source to see how systemd handles it). I think the focus on the specific means that the xz backdoor used to get into sshd is misdirected: it's code that is run in all kinds of priviledged contexts and has thousands of different ways to turn that into RCE. If sshd didn't load it directly the developers of the backdoor would have simply taken a different route.

> So upgrading XZ could even attack already running sshd too?

Doesn't that work both ways, without dlopen updating to the patched version would still leave your sshd running with the old backdoored one.

They are doing dlopen() wrong.

Instead of dlopen()ing libfoo.so.7, you should build a private wrapper library, normally linked with -lfoo, and then dlopen() that private wrapper. Then you get dependency tracking for free.

For some definition of free, that includes writing and maintaining a private wrapper library.

This seems a lot like a workaround for our tools not supporting optional dependencies. Let's fix that instead.

Uhh, how does that get you dependency tracking?

Sigh.

On the one hand, this seems like a slight improvement.

On the other hand, while it’s a bit of a slog, it’s possible to actually sandbox dependencies. libxz, for example, has well defined inputs and outputs. It’s possible to load it in such a way that all it can do is consume inputs and produce outputs that depend on those inputs. (And allocate memory and waste CPU, but that’s DoS at worst.)

A shared object that is loaded in the same thread/process as the rest of your application is not sandboxed from your application.

In can be with RLBox[0], which compiles the code to WebAssembly and then back to C. It's used in Firefox to sandbox various format parsing libraries.

[0] https://rlbox.dev/

That's true. But both in-process and out-of-process sandboxing are doable, as is writing code in a manner that is provably (for various definitions of provability) free of various classes of side effects.

I thought mastodon supported long posts. I thought breaking these things into a chain of tiny links was a twitter pathology.

It's configurable by the instance maintainers.

Thanks. That awful thread would have made a fine blog post. I can’t imagine why anyone would switch that misfeature on.

This seems like kind of a janky workaround when what you really want is linker support for weak/on-demand linking.

Right, as Lennart points out macOS has supported this for years. And adding a DT_WEAK_NEEDED to ELF wouldn't be that hard. The glibc maintainers even work for Red Hat. Feels like systemd is the wrong place to solve this.

We could just implement a DT_NEEDED_WEAK already instead of using these awful hacks.

what i do miss is a libsystemd-notify. Doing dlopen() is just fixing the wrong prob.

I agree that a simple self-contained .h header file would solve it in a better way.

But I stopped trying to push for minimalism. Sadly.

At least Poettering acknowledged that Systemd's dependency creep is a problem. [edited out]

The PR for changing compression libraries to use dlopen() was opened several weeks before the xz-utils backdoor was revealed.

https://github.com/systemd/systemd/pull/31550

Then its good the systemd team changed their minds even if its a decade late.

"Jia Tan" was pushing hard to get systemd updated to use the new xz because he saw this change in progress and wanted to get it in a release before this went through.

No they weren't. Jia Tan has never interacted with the systemd developers in any way as far as I know.

It isn't systemd's decision what version of xz-utils to use, it's the distro's decision. And Jia Tan did push the distro maintainers to update xz-utils, but the systemd developers have absolutely nothing to do with that, so your statement is incorrect.

You're correct, and I was overly vague. The key observation is "Kevin Beaumont speculates that knowing this was on the way may have accelerated the attacker’s schedule" Here's the exact wording, from Russ Cox:

2024-02-29: On GitHub, @teknoraver sends pull request to stop linking liblzma into libsystemd. It appears that this would have defeated the attack. Kevin Beaumont speculates that knowing this was on the way may have accelerated the attacker’s schedule. @teknoraver commented on HN that the liblzma PR was one in a series of dependency slimming changes for libsystemd; there were two mentions of it in late January. https://research.swtch.com/xz-timeline

See also previously on HN: https://news.ycombinator.com/item?id=39916125

Do you have a source for that? The backdoor was added to a minor release without a SONAME change, and generally such updates are done by distributions and not something upstreams concern themselves with.

It was mentioned in the recent Oxide and Friends podcast with the guy who, discovered the backdoor. Maybe there is a link there somewhere.

I'm not going to listen to a 1.5 hour podcast to find the exact quote, but someone must've misspoke or misunderstood. I did some searching in the meantime, but there's no evidence at all of "Jia Tan" interacting with the systemd developers. They pressured distributions to update xz-utils, but not systemd upstream.

That may be true, but that wasn't the claim. The claim was changes in upstream systemd would limit the timeline for them. Not that they had interaction with each other.

The podcast is interesting either way.

The claim here was this:

> "Jia Tan" was pushing hard to get systemd updated to use the new xz

Which is different from your claim "changes in upstream systemd would limit the timeline for them"

I can't edit my original comment, so I added the following correction https://news.ycombinator.com/item?id=40031620

Apologies for any misunderstanding.

You could link the systemd issue.

Why does an init replacement even need a compression library built into it? At some point the bloat is more risk than feature.

I guess this has to be said, for the another billion times, systemd is not and has never been 'only' an init replacement. And even earlier 'init' systems did more then init.

Just because you call it an "init replacement" doesn't only make it that, it describes itself as a "system and service manager" which should clue you in, that (de)compressing stuff is something that is commonly done during system management.

Reply on news.ycombinator.com