Hacker News Reader

Comment by cyberax | Hacker News Reader

systemd binary is around 1.5Mb, with the shared libsystemd it's around 2.5Mb. Both can be reduced by not including compressors (the infamous libz), audit libraries, SELinux, etc.

The minimum binaries size that can be obtained by trivial removals is around 2Mb.

OpenWRT should just stop being stubborn and integrate it. They dropped support for 4Mb devices loooooong ago, and recently bumped the minimum specs to 16Mb.

> systemd does not work well with musl which is very popular in embedded linux.

Systemd can work with musl with a small compatibility shim, I did it about 9 years ago. Somebody did that again: https://catfox.life/2024/09/05/porting-systemd-to-musl-libc-...

View on HN

> OpenWRT should just stop being stubborn and integrate it.

Why? If it fails to provide benefits commensurate to the effort required to integrate it, then there's no reason to do the work.

OpenWRT is for routers and access points... devices which usually have a fixed [hardware] configuration, and very little need for the absurdly-complex (and in my professional experience, often subtly buggy) event-driven systems that systemd provides. "Every system everywhere gets to use the same unit files" isn't a significant benefit in practice... EVERYTHING that needs nontrivial pre-/post- start/stop verification and/or configuration has systemd run a shell script (or other such helper) anyway... so you only get to just write a unit file and call it a day in the most trivial of cases.

OpenWRT is a mess. It started as a small distro for routers, but it stopped being that long ago.

Now it can host telephony services, full-blown Docker orchestration systems, etc. So you get all the pleasures of complex systems running on antiquated infrastructure.

> so you only get to just write a unit file and call it a day in the most trivial of cases.

What "start/stop verification"? Give examples, please. Systemd natively supports device dependencies, network status, other services, etc.

It's a very rare case when you _need_ shell in systemd.

> Now it can host telephony services, full-blown Docker orchestration systems, etc.

What features that systemd provides that aren't provided by openrc, runit, monit, openrc's init system, etc, etc, etc that are needed by software like Asterisk, Docker/runc, Kubernetes, and similar?

If what OpenWRT is running works fine to bring that up, then systemd doesn't have any relevant compelling features that aren't provided by another init system. Switching away from something just because it's "antiquated" is a great resume-building tactic, but it's a pretty poor strategy when your objective is to keep things that have been working working with a minimum of effort.

> What "start/stop verification"? Give examples, please.

One example (out of many) are pre-start, pre-stop, and pre-restart configuration file correctness verification so that a typo or thinko doesn't bring a service down.

A concrete example of this is Gentoo's running of 'named-checkconf' in BIND 9's service file [0]. Gentoo's BIND 9 systemd unit file [1] tries, but probably does not do the same thing... I don't know if 'ExecStartPre' runs before systemd brings down a service on a 'restart' action, but I bet you that it definitely does not run on service stop.

I just checked and Ubuntu 22 does NOT do this sort of checking for its bind9 package, which is pretty wild.

> It's a very rare case when you _need_ shell [or other such helpers] in systemd.

Then you're either running "seatbelts off" (maybe without even knowing that you are), or you rarely deal with anything nontrivial. This need comes up a LOT in my day-to-day.

[0] <https://gitweb.gentoo.org/repo/gentoo.git/tree/net-dns/bind/...> [2]

[1] <https://gitweb.gentoo.org/repo/gentoo.git/tree/net-dns/bind/...>

[2] You might look at that file and be like "wow, TL;WR", but do try to notice how much of that file is dedicated to creating useful, human-readable status messages and error messages with good corrective instructions. If you care at all about your sysadmins, that complexity has to go somewhere, and that somewhere cannot be inside a systemd unit file... systemd just doesn't provide the features required to do that (and it would be kind of insane for the devs to even try).

This article explains the security features of systemd that make chroot superfluous: https://www.redhat.com/en/blog/mastering-systemd

And here's an example of how a network service can be confined in its own network namespace: https://www.cloudnull.io/2019/04/running-services-in-network... - with zero shell needed, btw.

> What features that systemd provides that aren't provided by openrc, runit, monit, openrc's init system, etc, etc, etc that are needed by software like Asterisk, Docker/runc, Kubernetes, and similar?

Dependencies, watchdogs, start only when the given network interfaces are available, declarative configuration so that can be partially customized, etc.

> If what OpenWRT is running works fine

Unless you try to update it. And end up with merge conflicts in the /etc directory. And then it stops booting, and you have to get in via the serial port.

No, OpenWRT is most definitely not fine. It's stuck in the past, and doesn't support automated rollback, configuration management, etc.

> A concrete example of this is Gentoo's running of 'named-checkconf' in BIND 9's service file [0]

Wow, what a... But named config has always been a disaster zone.

With systemd _none_ of that crap is needed. It can use its own namespacing and cgroup support to ensure that BIND9 can't read outside of its allowlisted tree.

If you want to run /usr/bin/named-checkconf, then it's just an ExecStartPre directive in the unit file. And unlike SysV that just fails to run and forces you to dig into the logs, systemd provides you a nice status saying that the pre-exec had failed.

> You might look at that file and be like "wow, TL;WR", but do try to notice how much of that file is dedicated to creating useful, human-readable status messages and error messages with good corrective instructions.

Yeah, and hanging indefinitely, forcing me to do a trip to the datacenter at night (true story): https://lwn.net/Articles/619565/

Ah, you're that CyberAx. Right.

So, briefly:

1) That's a list of systemd features. Some of which are nice to have, but none of which are needed for the software I mentioned.

2) On-upgrade merge conflicts for entirely-unmodified-from-their-stock-settings system configuration files are why I gave up on Ubuntu. Systemd doesn't solve anything here.

3) Systemd doesn't handle BIND configuration... or any non-systemd application configuration. And most nontrivial init systems (with systemd as one notable exception) have facilities for printing info, warn, and error messages in service startup files rather than demanding that you pick through the application logs to find out why startup failed.

OpenWRT is VERY sensitive to firmware size. I developed for a device with 16 MB flash that had both a normal (~11 MB) and recovery firmware (~3MB) on flash. The rest of flash was used for persistent storage, bootloaders, and calibration data. This is very common for older routers and APs which is the whole reason that OpenWRT exists.

Well, OpenWRT doesn't even officially support devices with less than 16Mb of flash anymore: https://openwrt.org/toh/views/toh_available_16128

> The rest of flash was used for persistent storage, bootloaders, and calibration data.

So you couldn't spare around 1Mb of compressed flash or 2Mb of uncompressed flash for systemd?

> This is very common for older routers and APs which is the whole reason that OpenWRT exists.

That hasn't been the case in quite some while.

And it won't matter anymore, there is no price difference between 32Mb and 64Mb NOR or NAND flash. It's literally the same cost.

> So you couldn't spare around 1Mb of compressed flash or 2Mb of uncompressed flash for systemd?

Absolutely not.

> That hasn't been the case in quite some while

These routers and APs still exist and are in use. Yes, they are 10+ years old, but they're still out there. One of the main goals of OpenWRT is to support legacy networking hardware that vendors abandoned.

> Absolutely not.

So basically, you're dealing with devices that use obsolete hardware and are designed to be extra-brittle (no safety margin), with a built-in planned obsolescence (what if the new update requires just a bit more space for "calibration data"?). Got it.

sysvinit indeed fits perfectly. Make the system extra unreliable, to provide a stimulus to buy a newer version.

> One of the main goals of OpenWRT is to support legacy networking hardware that vendors abandoned.

And OpenWRT abandoned this goal. OpenWRT also can do nothing about firmware vulnerabilities in the old WiFi chipsets.

Moreover, 10 years ago, systems with 32Mb were already common.

OpenWRT doesn't use sysv init, it has something called procd (~120KB compressed). Its very reliable, and they have a large install base on mission critical devices.

> And OpenWRT abandoned this goal.

They have not. Individual devices get too old to be supported, but there are plenty of 16 MB devices they still support. Its true that basically every device now has 32MB or more of flash, but there are still plenty out there that don't.

> with a built-in planned obsolescence (what if the new update requires just a bit more space for "calibration data"?). Got it.

The calibration data is from the factory to tune the wifi radios, it doesn't change since its specific to that individual device. I really hate to reply like this on HN, but you don't seem to know what you're talking about. The kinds of constraints you deal with in embedded networking are different than that in general computing. There's a reason systemd wasn't used, and there's a reason OpenWRT devs went through the trouble of developing their own init system. Just because you can't fathom those reasons doesn't mean they don't exist.

> OpenWRT doesn't use sysv init, it has something called procd (~120KB compressed). Its very reliable, and they have a large install base on mission critical devices.

procd is only slightly better than sysvinit, it has the same ideas, and only adds simple and incorrectly implemented event triggers. And yes, I spent days debugging issues that it caused.

E.g. a USB drive for the logs becomes slow and changes the startup order, so now openvpn starts before the interface acquires the IPv6 address. Boom, broken IPv6 routing.

> They have not. Individual devices get too old to be supported, but there are plenty of 16 MB devices they still support. Its true that basically every device now has 32MB or more of flash, but there are still plenty out there that don't.

And systemd can live just fine at 16MB. I ran it on RouterBoards with 32MB (the smallest ones available) in 2014. It has grown a bit since then, but it can be slimmed down (remove compressors, auditing, SELinux/AppArmor, etc.).

> The calibration data is from the factory to tune the wifi radios, it doesn't change since its specific to that individual device. I really hate to reply like this on HN, but you don't seem to know what you're talking about.

The entire firmware for WiFi chips that were available during 16MB era is around 500KB-1MB. So it must be some other mysterious "calibration" data.

So you're thinking about more and more corner cases, that require the stars to be just right for the size of systemd to matter.

> So you're thinking about more and more corner cases, that require the stars to be just right for the size of systemd to matter.

These weren't corner cases, they were everyday reality. We had to intentionally slim down the firmware size to get it to fit comfortably since we added quite a bit of application code. Like I said before, we got a recovery firmware down to ~3MB, and the normal firmware (without our application) was ~8MB. Every megabyte counts at that point, so having a 1MB init system would be a non-starter. We were buying commodity hardware, so we did lots of shopping and there were still plenty of 16MB APs on the market 10 years ago.

You're right its not a problem if you buy new hardware, but OpenWRT is used to support older devices with limited hardware specs. Its one of the reasons people install it, to give new life to old hardware (rather than being forced to buy a new device).

By the way, I use and like systemd on full size computers. I actually know of a product that runs a full rpm based distro (with systemd) on an embedded device and it works great. But systemd isn't one size fits all.

Reply on news.ycombinator.com