Nasty Linux systemd root level security bug revealed and patched | Hacker News Reader

Nasty Linux systemd root level security bug revealed and patched

zdnet.com

28 pointsCrankyBear26 comments

For many of us, our biggest beef with systemd is that it has no planned limitations to its scope. (This has already been discussed to death so I will try to be brief.) Let's look at the actual current code size. I just cloned the systemd Github repo (git clone https://github.com/systemd/systemd.git) and this is what I got:

    $ wc `find . \( -name \*.c -o -name \*.h \) -print`
    [ ... ]
            37       89     1100 ./src/xdg-autostart-generator/xdg-autostart-service.h
    677446  2077830 23953433 total

(Simple wc includes blank lines and comment lines and we could be fancier but wc will do.) Now compare the same thing with OpenRC (git clone https://github.com/OpenRC/openrc):

   $ wc `find . \( -name \*.c -o -name \*.h \) -print`
    [ ... ]
      56    242   2209 ./src/rc/_usage.h
   17263  53243 419162 total

So 677,446 lines vs. 17,263 lines.

Lennart just keeps adding to systemd and refuses to say when he will finally stop adding to it. How many skilled humans on this planet are available to audit those 600k+ lines of systemd code and are actually auditing it? (And how many work for intelligence agencies?)

Most of the new features are being done in separate daemons from the init. The lines of code relevant to only the init are in src/core, so your comparison would probably only make sense if you compared that folder.

>Lennart just keeps adding to systemd and refuses to say when he will finally stop adding to it.

I'm not sure I understand, most projects only stop adding code when development is done. So the answer would probably be "when people stop using it." Are you a distro maintainer? If you want a stable version with fixes backported, you can use this: https://github.com/systemd/systemd-stable

>How many skilled humans on this planet are available to audit those 600k+ lines of systemd code and are actually auditing it? (And how many work for intelligence agencies?)

I'm not sure I understand this either, are you asking how many C programmers there are in the world that are able to perform code review on a C program for Linux, like systemd? And what subset of those C programmers work for intelligence agencies? It might be worth answering those questions, but I'm not sure how that is related to systemd specifically. If you look at the systemd github (or the github for any other big C program), you will see a good number of people already publicly reviewing patches, so you could get started there.

Until formal verification becomes common, something the size of systemd is always going to have security bugs that intelligence agencies will know about before the rest of us. If you don't want North Koreans hacking your server you would do better to opt for non-systemd.

Sure some people might know about bugs before others, and may seek to abuse that knowledge, but I don't think that is limited to intelligence agencies, and I don't think that has anything to do with the size of the project either. If you have some studies and data that goes over all the risk factors here, I'd love to see it.

>If you don't want North Koreans hacking your server you would do better to opt for non-systemd.

This seems to be fear-mongering and doesn't seem to follow at all from what you said either. There are a lot of other large projects used by the various Linux distributions, systemd is not out of the ordinary here. For an example of what you're looking at here, debian bullseye has 1.17 billion lines of code in total: https://sources.debian.org/stats/

Of course you would have to do a deeper dive to see just how much of that is expected to run as root or with CAP_SYS_ADMIN, or in kernelspace.

> The lines of code relevant to only the init are in src/core, so your comparison would probably only make sense if you compared that folder

  $ wc `find src/core \( -name \*.c -o -name \*.h \) -print`
  [...]
      81604  249802 3001043 total

So 81,604 lines for src/core vs 17,263 for OpenRC. So why does systemd need 677,446 lines of C code in its entire tree if src/core is 81,604 lines?

I suppose the answer is that IBM/Red Hat make money shipping systems that they and they alone have the expertise to support and fix. Making systemd's tree over 600k lines so that Red Hat employee Lennart is pretty much the only human being alive who understands it all is conducive to that goal.

> So 81,604 lines for src/core vs 17,263 for OpenRC. So why does systemd need 677,446 lines of C code in its entire tree if src/core is 81,604 lines?

Do you want a real comprehensive answer to this? You can just skim through the code to systemd to see what it's doing. Just in my experience, systemd has a lot more features than openrc; you could also read their respective documentation to get a full picture of what they do differently. All the other additional lines are for other optional daemons.

I have been using systemd in debian for years and nobody I know has needed to pay Red Hat. That's just not really how open source works at the distro level. To the contrary, I've seen a good number of debian users/developers contributing their expertise upstream to systemd, as well as fedora users, arch users, etc.

Maybe your argument still holds, but this is a misleading comparison. The systemd repository contains a lot more than just an init system (e.g. systemd-networkd, which is an optional network manager).

The appropriate comparison would be for the systemd the init system vs OpenRC.

Why does systemd need an optional network manager? Will this optional network manager become non-optional in the future?

It doesn't need it, I think it was just something that was interesting to have. I have not seen anyone saying there were any plans to make it non-optional, and that would probably not make sense to do that, because there are a lot of other network daemons that people use with systemd.

I presume because they are using a mono-repo for all the related projects. A GitHub repository doesn’t have to map to exactly one output. Just because the code is in an adjacent folder doesn’t mean that it’s used in all the targets.

If it became non-optional, then sure you can include the LOC then.

The upstream blog post is definitely better for avoiding the derails repeating rants about systemd:

https://github.blog/2021-06-10-privilege-escalation-polkit-r...

polkit is a separate daemon which is not just used by systemd so this is interesting from the perspective of whether similar problems might affect systems which don't use systemd:

https://gitlab.freedesktop.org/polkit/polkit/

Just about my favorite thing about using Ubuntu on Windows via WSL is that it does not use systemd: it uses a minimalist systemd shim instead that implements enough of systemd's interfaces to keep other essential dependencies working. So on WSL Ubuntu you have init for PID 1:

  $ ps aux
  USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
  root         1  0.0  0.0   9952   764 ?        Ssl  Jun15   1:06 /init

I'm sick of Microsoft telemetry and I would like to go back to native Linux, but non-systemd options are very limited:

- Devuan is most promising, being 'Debian without systemd', but the project has mountains of work to do and few volunteers to do it, yet so far seems to be managing. I'll be keeping my eye on Devuan....

- Alpine Linux doesn't have glibc, which is a very good thing except that so many precompiled Linux binaries depend on glibc, so you are stuck doing time consuming builds for some programs you will need.

- Gentoo and Artix are rolling releases, so no long term 'you only need security updates' option for servers that just need to do one job and do it well.

EDIT: I wrote Arch instead of Artix, as a commenter below pointed out. Artix is basically Arch without systemd.

Arch is primarily a systemd distro.

I think it sounds like you should seriously sit down and think about why don't you want systemd, but answer some other questions first. You're running windows with all of its telemetry and longstanding security and performance issues but a different init system on Linux is holding you back? That doesn't seem right, I would sit down and honestly think about what you're trying to accomplish with your machines and why. And why does linux vs windows help vs hinder that goal? Then why does glibc or systemd hinder that goal?

I think most people's objections to systemd sound similar to google's objection to systemd. Their objection amounted to "complexity in PID 1" and security. Around the same time they pushed android binder into the kernel, which is just yet another reimplentation of IPC. That spawned some very serious security issues. If they really didn't like the current IPC systems, they probably could have (and should have) implemented one of them on top of AF_UNIX sockets in userspace. So at the end of the day, they apparently don't take "security" as seriously as their objections to systemd would have you believe. What it really boils down to is that they don't like the piece of software for some vague hard to define reason. Ok sure, but that isn't going to help you accomplish your designs since it doesn't inform you on what direction your design should go. I would try to articulate why I don't like it very precisely, and then back it up with data. I can make that argument for windows for example. If I can't do that then my objection was probably emotional and irrational.

I compared systemd vs. OpenRC code size in another post, but here is Musl vs glibc code size:

  $ git clone git://git.musl-libc.org/musl; cd musl
  $ wc `find . \( -name \*.c -o -name \*.h \) -print`
    [...]
        7      11      94 ./src/unistd/_exit.c
    104637  330017 2883975 total

  $ git clone https://sourceware.org/git/glibc.git; cd glibc
  $ wc `find . \( -name \*.c -o -name \*.h \) -print`
    [...]
        49      216     1647 ./wctype/wctype_l.c
   1389414  6823487 52867225 total

So 104,637 lines vs 1,389,414 lines. There are more human programmers on this planet who can audit 104,637 lines than can audit 1,389,414 lines. What is glibc doing that is so terribly useful that it needs to be 13 times the size of Musl? It's just standard GNU bloat. And why does GNU awk (gawk) need to add network socket programming support? Isn't this beyond the scope of what awk does well and does best? Keep adding features, keep increasing the attack surface....

Do you think a manual audit of 100k lines of code will uncover multi-threading or memory issues or any other security issue?

Experience has shown that for c code for the first two, no it won't. And for any language code, manual audits won't find issues in the last category either. It may help, but it won't excise them all (or even close to all). To get them out, you need real world testing. The system that sees more real world usage, probably gets more testing. Any security analysis would want to take this into account also.

My earlier point about "what do you want to get out of your systems" is that you alluded to the reliability of debian being desirable. Distro maintainers by and large switched to systemd since it saved them a ton of work. They could then spend that extra time on the rest of the activities of being a distribution. So debian could focus on being more debian-like instead of debian-like + maintaining a creaky init system. So if what you want is debian, and you see very few distros like it that also don't have systemd, think about why and if what you're asking is detrimental to the other goal of what you want.

FWIW, I've maintained embedded linux distros at a couple companies, at my most previous employer we used systemd, the current one we don't. It isn't the right choice in all scenarios, but it is a lot of the time.

From what glibc developers have told me, glibc supports a huge number of systems, and a lot of the "bloat" in glibc comes from supporting various architectures and operating systems accumulated over the last 30 years. In general, code is not deposited in glibc at random, so somebody did see a good reason to do whatever it is at the time it was done. Musl is nice but it's not a strict replacement for glibc in all cases, and the musl developers seem to be pretty honest about that, for example here: https://wiki.musl-libc.org/functional-differences-from-glibc...

Also in my experience, the attack surface cannot be strictly reduced by removing features -- if users want a specific feature, they will just get it from something else, so at some point one needs to have code on the machine somewhere that does it.

ofubd8kc: Right, so straight and such superficial comparisons are wrong for this reason. Most of the glibc code isn't used for any given compilation. Similarly for systemd, most of the code is being compiled into binaries outside of systemd-init, for things like systemd-networkd. Whether or not systemd-networkd should just be a separate git repo is a good question, but it isn't really relevant. A similar question would be should glibc carry around support for these other systems?

Similarly, for systemd-init, the comparison should be, the other init system + all the code for the init scripts + all the code for all the utilities that are called inside the init scripts not counting the program being started. You should be looking at the system as a whole, not just a part of it.

I can’t firewall systemd from making network socket calls. So, YEET!

Artix and Devuan, it is.

I'm not sure what that means or how the second statement follows. You can't firewall the kernel from making network socket calls either, because the kernel is what implements the firewall. Of course the service manager is going to have higher privileges than every other process on the system, because it has to supervise all the other processes.

systemd makes various network socket calls.

Like DHCP

Can you expand on this? Systemd doesn't make network calls, so presumably you mean services started by systemd, which there exists an API specifically so you can do things like start a firewall before the network is brought up.

One word, DHCP.

The article has somewhat of a clickbait title. The bug was actually in polkit, which is available on every single one of those distributions you mentioned, and is a dependency of a bunch of other low-level daemons. It happens to be an optional compile-time dependency of systemd, but a distro can ship a version with polkit disabled and just use sudo, if that's what was wanted.

WSL is probably very similar to other containerized Linux scenarios, where you might want a small init to function as a sub-reaper within the container. I don't think that is much of a specialized case at all, as far as inits go anyway.

Don't forget about Void Linux.

I use it daily and it's a pleasure.

It baffles me that Void is so obscure. It's everything good about Arch, plus a better package manager and larger repositories and no systemd. I even put in on my appliance Raspberry Pis because they boot in 4 seconds.

However it is rolling release which it sounds like OP doesn't want.

Slackware, Void, GuixSD, MX Linux, PCLinuxOS, GoboLinux, and CRUX are some of the more well-known Systemd-less distros. There's also all the BSDs.

Reply on news.ycombinator.com