Hacker News Reader

Comment by bhaney | Hacker News Reader

> boots in 1/3rd the time of OpenRC on the same system

Wow, that's a suspiciously impressive difference. I was under the impression that OpenRC and systemd support similar parallel service startup features and usually boot systems in approximately the same amount of time, with maybe a slight edge to systemd. 3x speed makes me wonder if there's something else going on, like the current early systemd port having some bugs that cause it to incorrectly skip some important work.

Either way, super cool. What's the likelihood that this ends up getting upstreamed?

View on HN

> I was under the impression that OpenRC and systemd support similar parallel service startup features

It's not enabled by default in OpenRC because it's not stable.

> WARNING: whilst we have improved parallel, it can still potentially lock the boot process. Don't file bugs about this unless you can supply patches that fix it without breaking other things!

https://github.com/OpenRC/openrc/blob/ea310b2e580a25038f9592...

I'm amazed projects like that are still active.

This isn't a case where systemd is ahead of the curve, every other OS also starts services in parallel.

Plenty don’t, e.g. BSDs. Bunch of embedded GNU/Linux systems. Etc.

I meant OSs that have consumer market share. Eg on people's desktops and phones. Nobody cares about BSDs and a bunch of embedded GNU/Linux systems

A lot of people care about BSDs and embedded Linux systems. Such as: Netflix (FreeBSD), anyone with a Linux-based router or cable/DSL modem, and so on.

That said, parallel startup is possible on those systems too. There was an out-of-tree patch for parallel rc(8) on FreeBSD some time ago, and dinit is parallel, etc. I think OpenRC should move towards making parallel startup stable and supported so that those systems can do that as well.

Almost everyone's router is an embedded Busybox/Musl/Linux system. There are billions of them running every day.

they don't use openrc though

Alpine has market share. It's just targeting a different market space than what you're measuring.

> consumer market share

Some Alpine linux derivatives are kinda want switch to systemd and hence the porting

OpenRC doesn’t have huge consumer market share, at least in the context where this sort of thing matters. For regular Linux desktops, most of the distros use SystemD.

It is a simple little startup system with some dedicated users. Open source projects don’t need huge user-counts to justify their existence. It isn’t a profit center, people just do this stuff for fun, because they enjoy tinkering, or maybe they coded it up and decided to just share it out of altruism.

Folks who evangelise these "just as good as systemd" solutions always hide a key factor.

There is a reason systemd took over, its massively supported and widely tested.

It couldn't be massively supported until it took over, though? It's more than a little platform effect going on, it's not all by virtue of systemd being just all over great. It has some great ideas but it's an entire world unto itself which feels very NIH-ish.

Edit: I think it serves as both a warning and an inspiration. Don't let a duct tape solution (such as for instance boot via rc shell files) live for too long, or it might be replaced by something baroque.

Ideas are just 1% if implementation. Systemd (while being a little problematic in beginning) has had huge amount of amazing work being done in it.

Doesn't mean the result is all that good.

Having dealt with it for some time now, the amount of "good idea, bad implementation" I hit is too damn high.

And it's nigh impossible to replace for various reasons, including at one point critical interfaces that various user-important programs depended on being a constant treadmill of updates, lack of documentation, and hidden dependencies on internals.

It's true. systemd has a lot of good ideas with questionable implementations. I'd love networkd if it didn't have silly decisions made.

And it has some questionable ideas with good implementations. For example, I hate the journal, but admit it is implemented well.

At the end of the day, it's about tradeoffs and using what suits the system at hand. For some people that will be systemd, and bringing systemd to Adélie and other musl libc distributions means that it can be used by those people.

Call me old-fashioned, but I also hold out hope that at least some of those questionable implementations could be fixed if only someone with the desire would write a patch and send it upstream. Bringing systemd to musl means the people in musl land that aren't knee-jerk anti-systemd maybe might be more enthused to do just that, improving it for everyone using it.

Do you have specific examples or just general grievances rehashing past claims?

Can't specific examples be rehashes of past grievances? It hasn't changed much. The log is annoying and sometimes freezes while it compresses or something, for instance. That's old.

Depends on whether its already been resolved?

Various, but don't have time to rehash all of them - the following are just things that I was either bitten by fresh or constantly suffered from in last year:

- The utterly broken in places interface (one of the worst offenders is systemctl show non-existing-unit dumping you a screenful of systemd unit parameters instead of "this unit does not exist")

- Bonkers defaults the moment you step outside of the minimum case for new unit - spent ridiculous amount of time dealing with that trying to make a self-restarting service with sane values for restart timer etc.

- Lack of documentation resulting in having to read the source code (which is horrible code, IMO), which I can only figure as being result of built-in assumption that any resource consumed by systemd is exclusively for systemd use. Which causes problems sometimes (my specific case was totally undocumented values used by systemd secrets mechanism wrt to TPM2, which caused hard to diagnose errors)

But systemd was already widely supported not even 2 or 3 years after its original release. The reason it built its network effect is because it was better. Sure, a lot of it is also due to the classic Redhat NIH and their marketing push for their own solutions regardless of if other stuff already exists, but in this specific case systemd was also very very useful from the get go.

It wasn't widely tested until it took over either.

The improved boot times are likely largely due to systemd's parallelization, OpenRC's support is buggy and OpenRC overall is known to be quite slow.

Thankfully, there are plenty of other non-systemd inits besides OpenRC :)

Yes, it's massively sponsored as well. So it went heavily imposed almost everywhere even when a quite big chunk of the Linux community deeply disagreed about the imposition.

People seem to not appreciate that Lennart Poettering managed to get systemd from concept to shipping in RHEL 7 in under four years. At that point I'm not even sure anyone at Red Hat had even asked for it, they had already transitioned to Upstart for RHEL 6 after all. Whatever your opinion of Poettering may be, you have to admit that he has tenacity.

So it went heavily imposed almost everywhere even when a quite big chunk of the Linux community deeply disagreed about the imposition.

There is a substantial selection bias in the complaints towards systemd. At the very least it seems that Red Hat's customers didn't have a problem with it. For most Linux users, it was a minor change, but for distro maintainers it was a massive relief[1].

Again, Lennart Poettering actually took the initiative to develop systemd and get it adopted. By comparison, detractors of systemd did not develop a competitive alternative. Thus, it's hardly a surprise that systemd ended up steamrolling the competition. Ultimately systemd is what users deserve because no person or company bothered to make something better.

[1]: https://bbs.archlinux.org/viewtopic.php?pid=1149530#p1149530

There is a lot to hate about systemd, but in the end anything like it is very hard and few people have the time/energy to attempt something.

As it happens way too commonly, the loudest in OSS communities are ones that have zero stake in building or maintaining the projects they criticize. It's easy to just rant if everyone else has to pick up the extrenalities.

Your comment can be read as either for or against systemd.

I'm leaning that it's against systemd but I'm not completely sure. :)

I allow you to interpret it in a way that makes your day better :D

The reality of my situation is my job frequently involves meeting people where they are - and most people are on systemd. Issues with systemd have caused and continue to cause me significant headaches - sometimes they are one-off but major events, sometimes they are recurring frustrations because of specific design decisions. I've included a list with some examples of both elsewhere in these comments.

I don't want to build a competitor to systemd because I fundamentally disagree with the design philosophy that has led to the proliferation of a huge number of systemd-* replacements for other things that already worked fine for all of my use cases. For situations where I have complete control over the infrastructure and don't need to worry about junior engineers that only have experience in the systemd ecosystem, I don't run it, and my life is lower stress because of it. I don't even particularly care much about the init portion of it, though a lot of the improvements are things that I don't really care about, like boot time - if I'm in a situation where a server's boot time has some impact on our overall uptime, etc., then things have gone wrong in a very big way - but if it was just a new init system, I'd have minimal complaints.

But in general, it's one of the reasons that I have been glad to shift my professional focus away from being very specific to the Linux and compute side of things to broader cloud platforms, etc.

So far most of the complaints about systems are really all about the fact that there is no real competitor to systemd. The argument that systemd is difficult to replace says more about the state of the rest of Linux user land, than it says about systemd, since most of the benefits of systemd are easy to replicate.

Writing another init system isn't exactly something most people would want to spend years of their life on. At best, it'd take a few years to reach some limited parity with where systemd is today. Furthermore, most of the systemd detractors fundamentally object the philosophy of systemd. So how can they meaningfully innovate over upstart and/or runit? It's either developing systemd-lite or a more elaborate ball of scripts. Even then, there is the issue of convincing distributions to adapt your new init system.

The reality is that systemd arrived at a time when Linux distributions were dying to move off of sysvinit. Poettering struck when the metal was hot and he delivered a comprehensive tool that greatly reduced the burden on maintainers. Sure, nothing lasts forever, but it will be many years (if not decades) before systemd is slated for replacement.

If you are trying to convince me that systemd happened out of the creativity of Mr. Poettering and Red Hat just followed him... well I have to say I don't believe that. But anyway, for me it really doesn't matter. It's the beauty of free software.

As strange as it might sound, that’s basically what happened.

I mean, let’s assume that Red Hat leadership initiated the systemd project, then we are immediately confronted by a number of problems. First of all, if Red Hat planned to ship systemd with the launch of RHEL 7 in 2014, then why did development on systemd only begin in early 2010? It seems rather irresponsible to intentionally delay development of the new init system for your flagship product, particularly when you’ll be supporting it for the next 15+ years.

Second of all, why would Red Hat hand the systemd project to Lennart Poettering, a developer who was primarily responsible for PulseAudio? Moreover, Poettering was already notorious for being an outspoken member of the Linux community. So why would Red Hat management go so far out of their way to choose such a divisive figure lead their project?

Also, why would Red Hat management seek to imitate launchd of all things? Why not SMF or some other “enterprise” solution instead?

Last of all, upstart development was primarily funded by Canonical and it had proven itself on RHEL 6. Therefore, replacing upstart with systemd meant shifting the maintenance burden back onto Red Hat in one of the very few areas it could actually save on it.

Now, if you still have your doubts than here it is from the man himself: https://web.archive.org/web/20181108025744/https://thenewsta...

Again, what Poettering accomplished with systemd is astounding.

> heavily imposed

Not really. The community Linuxes which aren't corporate-sponsored (Arch, NixOS) were the first aboard the systemd train.

Systemd really does make a distro maintainer's job a thousand times easier.

As an intermediate level linux user (never seriously daily driven a distro, but I do a lot of ops work and am comfortable getting in the weeds)... I really have come to prefer systemd over e.g. upstart/sysinit.

I also recently moved to networkd and resolved when setting up some new nixos boxes and greatly preferred the way those worked when doing vlans and split-DNS respectively.

The inflection point was GNOME 3.8 not working properly with anything other than systemd and it being night impossible to implement compatible alternatives for some of the interfaces involved.

Right! If you wanted new Gnome you needed systemd - illusion of choice.

Systemd always is ahead with new features. Which is nice if its a feature you need, and double worse if it crashes your machine, like logind sending your machine into a bugged suspend. Which makes systemd linuxes unusable for newbies on some hardware.

I used OpenRC because, not knowing about logind.conf, i had no other choice.

Yet systemd doesn't have an equivalent to runit's sv once. Also, everything's so opaque that I can't figure out why nftables ruleset isn't loaded on boot, but systemctl restart works correctly (all on Ubuntu 24 server without any kind of frills added).

> Yet systemd doesn't have an equivalent to runit's sv once.

I'm not familiar with runit. Is this approximately equivalent to 'systemctl start service' when service uses the default of Restart=no ?

Yes, but, as the name implies, only once, and selectively at runtime. If you have a service and issue sv once on it, it will not be restarted, but its configuration is not changed so it will run normally after system restart.

That's a one-shot service[1].

RemainAfterExit=yes is what achieves the effect (i.e. if the process terminates it is regarded as still active for system-dependency purposes / status unless you purposefully mark it otherwise).

[1] https://www.redhat.com/sysadmin/systemd-oneshot-service

> Yet systemd doesn't have an equivalent to runit's sv once.

It's a design difference - systemd assumes services know whether they are intended to run once or restart under various conditions. If you want that behaviour it's just a matter of a tweak to the .service file.

> Also, everything's so opaque that I can't figure out why nftables ruleset isn't loaded on boot

Has it been disabled? ("systemctl enable [service name]" will enable it)

> Folks who evangelise these "just as good as systemd" solutions always hide a key factor.

No one in these comments was "evangelising" and no one was hiding anything. If you want to have a technical discussion about the merits of various init systems you can do it without inflammatory language and without reviving debates that almost everybody is sick of

> There is a reason systemd took over, its massively supported and widely tested.

Partly due to that it's funded and forced upon by corporate. Any product can excel if resources are available.

Nobody forced us, Arch Linux, to adopt systemd.

That choice did get me to switch off an arch derivative to Devuan.

Also, the purported benefits in this post (which I guess is authoritative for the project?):

https://bbs.archlinux.org/viewtopic.php?pid=1149530#p1149530

Haven’t panned out for me.

8: logind being repeatedly and completely busted (in 2022-2024) got me to try Debian again. 9: Debian taking 10’s of seconds to boot (just like manjaro), when devuan does it faster than a monitor sync got me to switch to devuan.

(And, yes, since logind regularly wedges, I spend lots of time rebooting).

Hotplug worked fine for me way before systemd existed, so that’s not a benefit either.

> There is a reason systemd took over, its massively supported and widely tested.

Yes. The people behind it held key positions at Red Hat, freedesktop.org, etc. I'm not calling them some shadowy cabal pulling strings behind the scenes, but it would be silly to discount the political sway the people that started the project have.

Being widely-tested also hasn't keep it from producing system bricking issues, such as this debacle just a few months ago:

https://github.com/systemd/systemd/issues/33349 https://github.com/systemd/systemd/pull/33383

Or that time that systemd mounted efivarfs as rw making it possible to brick your motherboard

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Or the time back around 2012ish when an update caused the dhcp lease renewal portion to get wedged and took many thousands of servers offline until the service could be manually restarted or the machine rebooted.

Or there was that time when OpenSSH was compromised because they pulled in systemd components to handle notifications: https://www.openwall.com/lists/oss-security/2024/03/29/4

Then there are all the weird choices that are inconsistent with the rest of the ecosystem, like showing asterisks when typing in passwords instead of nothing: https://github.com/systemd/systemd/issues/8495

Even formerly simple things like log collection for systems not hooked up to Splunk or similar is now a chore. Most distros not longer include rsyslog by default, and pulling log files out of the journal is a painful process. Before, if I wanted to rip out the past week's logs for analysis, it'd usually mean just grabbing some gzip'ed files that were relatively small in size. Journalctl files are significantly larger in size for similar timeframes. OK, so how about I just interact with the journal directly and tell it to dump just the parts I care about? This is quite resource intensive on smaller VMs running in cloud providers and can severely degrade the overall system performance, so now I have to artificially slow things down and get it piecemeal or schedule it to run during off hours when the performance hit won't matter.

I work with systemd daily. It is enough to make me wish I had to work with systemd never. I'm not some sysvinit purist, either - I like SMF on solaris quite a bit!

> Either way, super cool. What's the likelihood that this ends up getting upstreamed?

High: <https://github.com/systemd/systemd/pulls?q=is%3Apr+author%3A...>.

Did Lennart change his mind? https://lists.freedesktop.org/archives/systemd-devel/2014-Oc...

I don't think Lennart changed his opinion, if you look further in this thread[1] Lenart wasn't ever against musl per se, it is more like "we use non-POSIX features from glibc that we found useful, and we are not going to reinvent them, so either convince musl to implement those features or carry patches fixing the issues". That I kind concur with him, instead of workaround missing functions it may be better to just write a compatibility library that complements musl with the missing functions. This way, any program that needs those functions could just import this library if they're compiling in a non-glibc system.

I am seeing that lots of the fixes proposed by the author are things like "missing import because glibc exposes some symbols by mistake and musl does not". And this is definitely the kind of fix that upstream would accept[2].

[1]: https://github.com/systemd/systemd/issues/10130

[2]: https://github.com/systemd/systemd/pull/34066

it may be better to just write a compatibility library that complements musl with the missing functions

It exists. See gcompat.

https://github.com/Stantheman/gcompat

FWIW, that is an unofficial mirror repo and very old. gcompat lives at https://git.adelielinux.org/adelie/gcompat.

Cursed github monopolizing bing seo for open source projects.

> This library is designed to be used for binaries that are already compiled against glibc. It does not contain any headers, and cannot be used to build software that requires glibc. It is instead recommended that any software that requires glibc APIs be modified to become more portable.

This is exactly the opposite that I was talking about. What I was talking is a library that expose glibc compatible headers to compile software that uses Glibc-isms with minor or zero modifications, but this instead is a compatibility layer for binaries and doesn't expose headers (probably for proprietary software that can't be recompiled).

Less than a day later and everything in this list is already merged. Congrats!

Do those PRs represent everything necessary to compile systemd against musl, or are there more PRs coming to finish the job?

Reply on news.ycombinator.com