Why systemd is a problem for embedded Linux | Hacker News Reader

Why systemd is a problem for embedded Linux

kevinboone.me

208 pointssynergy20300 comments

I couldn't disagree more: I've worked with lots of embedded devices running systemd, and it solves many more problems than it introduces. The community is also quite responsive and helpful in my experience.

I won't pretend there aren't occasional weird problems... but there's always a solution, here's a recent example: https://github.com/systemd/systemd/issues/34683

Memory use is irrelevant to me: every embedded Linux device I've been paid to work on in the past five years had over 1GB of RAM. If I'm on a tiny machine where I care about 8MB RSS, I'm not running Linux, I'm running Zypher or FreeRTOS.

Small typo: Zephyr https://zephyrproject.org/

That "occasional weird problem" is because systemd is not designed to be used with other software. While you technically still have the choice to roll your own non-systemd initramfs, it will be an uphill battle.

Not uphill at all; try buildroot

> That "occasional weird problem" is because systemd is not designed to be used with other software.

That's not true, systemd has an explicitly documented and supported initrd interface: https://systemd.io/INITRD_INTERFACE/

It's actually really easy, there's rarely a reason for the initrd to be more complex than a single 50-line busybox script on an embedded device with built-in drivers.

"really easy" except for the hotplug events that don't get to systemd and cause your problem.

I got an immediate answer with a solution from the upstream developer when I asked about it: I can't imagine how that could have been easier to solve. And it was also trivial to hack around by leaving the entry out of fstab and mounting it in a script.

Debian, Ubuntu and all derivatives thereof use initramfs-tools, which does not use systemd in the initrd, and things work just fine

Ubuntu is literally trying to replace it with dracut as we speak.

https://dracut-ng.github.io/dracut-ng/developer/compatabilit...

Dracut is used both in Void Linux and on Alpine without systemd and with busybox.

It even runs continuous integration with musl based containers.

Alpine uses mkinitfs per default, tho.

Not an uphill battle at all; Void Linux does this by default and has for many years.

Void Linux dropped systemd because it wouldn't work with a different libc than glibc. Which i would add as next point in my systemd lock-in list.

The embedded people will naturally look at busybox. I saw it running on a credit card scanner at Old Navy a few years ago.

It has an init:

  $ /home/busybox-1.35 init --help
  BusyBox v1.35.0 (2022-01-17 19:57:02 CET) multi-call binary.

  Usage: init

  Init is the first process started during boot. It never exits.
  It (re)spawns children according to /etc/inittab.
  Signals:
  HUP: reload /etc/inittab
  TSTP: stop respawning until CONT
  QUIT: re-exec another init
  USR1/TERM/USR2/INT: run halt/reboot/poweroff/Ctrl-Alt-Del script

On an embedded system, that will be a strong contender for pid 1.

This is what Alpine Linux use, Alpine also uses OpenRC for service startup.

> every embedded Linux device I've been paid to work on in the past five years had over 1GB of RAM

That is almost by definition not an embedded device. There's a reason we have vfork().

Well, you can gatekeep all you want, but it's increasingly practical and common to have what would have seemed like an absurd amount of RAM a decade ago on things like toasters.

You have been living in a strange world if you’ve been getting away with 1GB in the average consumer IoT device for the past 5 years.

That’s not typical at all. I’ve done a lot of work with production runs in the millions. There is no way we’d put that much RAM on a device unless it was absolutely, undeniably required for core functionality.

I've asked in the past, and been told that a even a 2x-3x difference in the amount of RAM made such a negligible difference in cost it was decided to go with the larger amount. I frankly have a hard time understanding how that can be true... but I can't really imagine why they wouldn't be honest with me about it.

> I've asked in the past, and been told that a even a 2x-3x difference in the amount of RAM made such a negligible difference in cost it was decided to go with the larger amount

That doesn't pass the sniff test. Look at retail RAM prices. Certainly the magnitude of the price is quite different than buying individual RAM chips at quantity, but the costs do scale up as RAM size goes up. Hell, look at RAM chip prices: you are definitely going to increase the price by more than a negligible amount if you 2x or 3x the amount of RAM in your design.

Also consider the Raspberry Pi, since the article mentions it quite a bit: RAM size on the Pi is the sole driver of the different price points for each Pi generation.

The retail price of a finished product has very little to do with the cost of individual components and more with profit margins or customer segmentation.

At quantities of 100, 512Mbit of RAM is $1.01 [0]. 1Gbit of RAM is $1.10 [1]. 2Gbit is $1.05 [2]. 4Gbit is $1.16 [3]. It is only at 8Gbit that prices substantially increase to $2.30 [4].

So no, at those sizes price really does not change all that much, and installing 512MB of RAM instead of 64MB only increases the product's cost by $0.15. It's a commodity made on legacy processes, things like packaging, testing, and handling far outweigh the cost of the actual transistors inside the chip.

[0]: https://www.lcsc.com/product-detail/DDR-SDRAM_Winbond-Elec-W...

[1]: https://www.lcsc.com/product-detail/DDR-SDRAM_Winbond-Elec-W...

[2]: https://www.lcsc.com/product-detail/DDR-SDRAM_Samsung-K4B2G1...

[3]: https://www.lcsc.com/product-detail/DDR-SDRAM_Samsung-K4B4G1...

[4]: https://www.lcsc.com/product-detail/DDR-SDRAM_Samsung-K4A8G1...

At a company I worked at they explicitly told us they will do anything to avoid upgrading the hardware from 1GB to 4GB because it increases costs. They would rather we optimize the software to use less RAM than upgrade the hardware.

I remember arguing as well with people about 0.10$ components. They told me it was a no go, not even a point of bringing it up. Sometimes even a 0.01$ is a big deal.

Yeah, prices do indeed go up beyond 1GB - but we're talking about systemd needing 8MB of RAM. With small RAM chips there is more variation between parts than there is between sizes - hence the linked 2Gbit chip being cheaper than the 1Gbit one despite both of them being the cheapest option at LCSC. Those 8MBytes of systemd might push you from needing 1Gbit of RAM to 2Gbit, but it isn't going to push you from 8Gbit to 64Gbit - and as shown 1Gbit and 2Gbit don't meaningfully differ in price.

There are a lot of other factors involved with respinning hardware which make an upgrade a lot more expensive than simply a BOM increase. I can definitely understand why an existing product wouldn't be upgraded, but for a new product going to a bigger memory chip is a far smaller hurdle. The added software engineering time for working around RAM limitations can easily outweigh any money saved on the BOM, with choosing a smaller chip ending up being penny-wise pound-foolish.

And indeed, an extra $0.10 or even $0.01 can be a big deal. But those cheap systems usually aren't powerful enough to meaningfully run Linux in the first place: just because you can technically hack a $1.00 RP2040 or ESP32 into running Linux doesn't mean it is a good idea to actually do so. If your product is both cheap enough that it represents a significant fraction of your BOM and high-volume enough that you can afford the engineering time in the first place, why not use a purpose-built embedded OS like Zephyr?

Even Apple produced laptops with 8 GB RAM just recently, which they sold for hundreds of dollars with huge margins (AFAIK). If you're going to produce something with $50 cost, 1GB RAM cost will be meaningful.

In my experience production people will eat your soul for a single resistor if they can cut costs on it.

That RAM is unified though, not a good comparison.

Also, just because something holds true at large numbers doesn't mean it scales all the way down. Either due to economies of scale, or the negligibly different architecture/components at that size.

The RAM is ordinary LPDDR5 organized into what is de facto just a large number of memory channels. It's not HBM or anything exotic, the cost of the RAM chips themselves are the same cost they are anywhere else.

That is the Apple tax on everything the fruit company sells, they always push the margins as far as fans are willing to pay for.

Typically in IoT you'll count RAM in kB not MB and definitely not GB. See STM32 H5/H7/L4/L4+/U0 as an example.

Typically in IoT you'll use an soc that actually supported some kind of network connection

Depends.

In Automotive (i.e. telematics devices) you'll want a separate MCU for CAN-bus. For example, if you are doing Request-Response model you'll want to make use of the built-in filters. Besides, it is unlikely that a modem would support the CAN interface.

In Cellular IoT you'll prefer a separate MCU as it is much easier to port to different hardware. For example, you can hook-up the module via UART and use CMUX (AT/GNSS/PPP) and you'll cover 80%+ of the modules available in the market with very minimal specific implementation layers to enter into these modes.

Huh?

Been a while since I was in the digital signage space but a lot of the equipment runs of the shelf RK3288 plugged into commercial displays. 2GB of RAM was pretty common. IIRC's though LG's WebOS TV's have minimum 2GB of RAM in the digital signage space directly built into the units themselves. I believe Samsung Tizen based units has similar RAM.

My router has 1GB of RAM in it. But even my cheapest routers have 128 to 256 MB of RAM. The Cisco 9300 Catalyst switches have about 8 GB of RAM, but switches with beefy amounts of RAM are getting pretty common now, even if somewhat pricey.

Yeah there's massive swathes of embedded space that's tiny. But the higher end stuff isn't exactly out of reach anymore. The RK3288's IIRC ran about $20 a unit at the time before I left the industry.

A decade ago, I had to settle for 512MB of ram for my Windows XP desktop.

Three decades perhaps. In 2014 it was Windows 8, SSDs, 5th gen i7 (Haswell), 8-16GB of DDR4 ram. Even the iPhone 6 came with 1GB of ram.

There's a huge chunk of people out there, not able to afford the latest.

I see you, but win XP was literally end of life 10 years ago.

So what?

Indeed, but still wrong.

In 2014 i got myself a second- or third-hand thinkpad X220, released in 2011, off eBay. It came with 8gb ram (two 4GB sticks) but it supported 16GB ram as well (two 8gb sticks).

The laptop (Asus A8Jc) I got when I was a teenager in ~2005 came with a dual core intel cpu and 1GB ram. So "512mb desktops" are way older than that.

But desktops with those specs are <$100, probably less than $50.

If they can't afford those specs then you're approaching the group of people who can't afford a computer in the first place.

Yes, that's about 3 billion people. They use whatever they can afford. Usually that's whatever is very old, because new software doesn't run well except on very new hardware which is more expensive.

I recently wanted an MP3 player for an art project. Local stores don't sell mp3 players anymore, they only sell smartphones. So I bought an MVNO smartphone for $40. When I charged it up and tried to use it, I thought maybe it was broken, because it would take 10-30 seconds to load a settings menu or app. Nope, all these bargain carrier-branded phones are that slow. The hardware is [somewhat] old, but the new Android OSes run like molasses on them. It was like going back in time. Remember how Windows 98 would make your hard drive screech for a good couple minutes as it struggled to juggle the swap memory so you could open MS Word? That's the experience with most software today with "affordable" hardware even a few years old.

So using Windows XP is often the only choice, if you don't have a lot of money, like 1/3rd of the planet. (And it's not just the third world. 59% of American households with K-12 school kids don't have a working computer, or it works too slowly to be useful)

This is simply so misleading thats its nonsense.

So you bought a new phone for $40, and it was a POS?

My kids use my old iPhone 7, which is in the same price bracket and is nothing like that. Its fast enough for Roblox, Minecraft, and certainly fast enough for a web browser.

I have an old Dell USFF that I use for server purposes, but its a Skylake (so newer than what was the original conversation), with ssd and 16Gb, and that was <£50. That can boot with systemd in under 5 seconds. It can boot to the full Gnome desktop in under 6 seconds. Firefox can start and get the Office.com site up in less than 3 seconds.

Because thats what were talking about.

> But desktops with those specs are <$100, probably less than $50.

I just checked eBay. Yes they still are.

That was the amount of RAM in my Athlon Windows XP multimedia PC, bought in 2002, by 2006 my newly acquired ThinkPad PC RAM was already measured in GB.

Three decades ago was 1995 and Windows 95 ran on 8MB of RAM.

Had vfork, unless you're on some not-BSD where vfork has remained relevant the last three decades?

DESCRIPTION vfork() was originally used to create new processes without fully copying the address space of the old process, which is horrendously inefficient in a paged environment. It was useful when the purpose of fork(2) would have been to create a new system context for an execve(2). Since fork(2) is now efficient, even in the above case, the need for vfork() has diminished. vfork() differs from fork(2) in that the parent is suspended until the child makes a call to execve(2) or an exit (either by a call to _exit(2) or abnormally). ... HISTORY The vfork() function call appeared in 3.0BSD with the additional semantics that the child process ran in the memory of the parent until it called execve(2) or exited. That sharing of memory was removed in 4.4BSD,

there are 2 types of embedded systems: those that ship 1 million units, an and this that ship 50. if you're shipping 1mil units, you need to optimize RAM size, but if you're only shipping a few, them it's not worth squeezing everything down as long as it doesn't break your power/cost target. there's a ton of devices out there that literally just use a cheap smartphone as an "embedded" CPU because that way, Google has already done 90% of your R&D for you

I was just thinking "there is no way this person works on embedded devices". Then I read the last paragraph where you bring up "over 1GB of RAM". Explains that.

> every embedded Linux device I've been paid to work on in the past five years had over 1GB of RAM. If I'm on a tiny machine where I care about 8MB RSS, I'm not running Linux, I'm running Zypher or FreeRTOS

The gap between “over 1GB of RAM” and 8MB RSS contains the vast majority of embedded Linux devices.

I, too, enjoy when the RAM budget is over 1GB. The majority of cost constrained products don’t allow that, though.

That’s said, it’s more than just RAM. It increases boot times (mentioned in the article) which is a pretty big deal on certain consumer products that aren’t always powered on. The article makes some good points that you’ve waved away because you’ve been working on a different category of devices.

> The gap between “over 1GB of RAM” and 8MB RSS contains the vast majority of embedded Linux devices.

Of all currently existing Linux devices running around the world right this moment? Maybe.

But of new devices? Absolutely not, and that's what I'm talking about.

> The majority of cost constrained products don’t allow that, though.

They increasingly do allow for it, is the point I'm trying to make.

And when they don't: there are far better non-Linux open source options now than there used to be, which are by design better suited to running in constrained environments than a full blown Linux userland ever can be.

> It increases boot times (mentioned in the article) which is a pretty big deal on certain consumer products that aren’t always powered on. The article makes some good points that you’ve waved away because you’ve been working on a different category of devices.

I've absolutely worked on that category of devices, I almost never run Linux on them because there's usually an easier and better way. Especially where half a second of boot time is important.

> But of new devices? Absolutely not, and that's what I'm talking about.

The trouble with "new" is that it keeps getting old.

There would have been a time when people would have said that 32MB is a crazy high amount of memory -- enough to run Windows NT with an entire GUI! But as the saying goes, "what Andy giveth, Bill taketh away". Only these days the role of Windows is being played by systemd.

By the time the >1GB systems make it into the low end of the embedded market, the systemd requirements will presumably have increased even more.

> there are far better non-Linux open source options now than there used to be, which are by design better suited to running in constrained environments than a full blown Linux userland ever can be.

This seems like assuming the conclusion. The thing people are complaining about is that they want Linux to be good in those environments too.

> There would have been a time when people would have said that 32MB is a crazy high amount of memory

Those days are long gone though, for better or worse.

We live in the 2020s now and ram is plenty. The small computers we all carry in our pockets (phones) usually have between 4 and 16g GB ram.

That's entirely the point. In the days of user devices with 32MB of RAM, embedded devices were expected to make do with 32KB. Now we have desktops with 32GB and the embedded devices have to make do with 32MB. But you don't get to use GB of RAM now just because embedded devices might have that in some years time, and unless something is done to address it, the increase in hardware over time doesn't get you within the budget either because the software bloat increases just as fast.

And the progress has kind of stalled:

https://aiimpacts.org/trends-in-dram-price-per-gigabyte/

We've been stuck at ~$10/GB for a decade. There are plenty of devices for which $10 is a significant fraction of the BOM and they're not going to use a GB of RAM if they can get away with less. And if the hardware price isn't giving you a free ride anymore, not only do you have to stop the software from getting even bigger, if you want it to fit in those devices you actually need it to get smaller.

I recently looked up 2x48GB RAM kits and they are around 300€ and more for the overclockable ones. That is 3€ per GB and this is in the more expensive segment in the market since anyone who isn't overclocking their RAM is fine using four slots.

The end of that chart is in 2020 and in the interim the DRAM makers have been thumped for price fixing again, causing a non-trivial short-term reduction in price. But if this is the "real" price then it has declined from ~$10/GB in 2012 to, let's say, $1/GB now, a factor of 10 in twelve years. By way of comparison, between 1995 and 2005 (ten years, not twelve) it fell by a factor of something like 700.

You can say the free lunch is still there, but it's gone from a buffet to a stick of celery.

> We live in the 2020s now and ram is plenty. The small computers we all carry in our pockets (phones) usually have between 4 and 16g GB ram.

I do not think the monster CPUs running Android or iOS nowadays are representative of embedded CPUs.

RAM still requires power to retain its contents. In devices that sleep most of the time, decreasing the amount of RAM can be the easiest way to increase battery life.

I would also think many of the small computers inside my phone have less memory. For example, there probably is at least one CPU inside the phone module, a CPU doing write leveling running inside flash memory modules, a CPU managing the battery, a CPU in the fingerprint reader, etc.

> The gap between “over 1GB of RAM” and 8MB RSS contains the vast majority of embedded Linux devices

The gap between 16MB RAM and 64MB RAM doesn't exist, though. Literally doesn't, the components have the same cost down to a cent in the BOM.

And if you can have 64MB, then there's systemd's own true memory use (around 3-4MB) is completely immaterial.

except thanks to availability crisis hitting the industry for the past decade you have to go with the 4mb sometimes

just look at wifi routers. in the usa and China they are all sold with 64 or 128mb ram. south America and Europe they are all 16 or 32 for no clear reason.

Do you have some examples? I have a very hard time imagining a modern Wifi router supporting the latest standards and IPv6, admin web interface and so on running on 16 MB of RAM. I also have issue with "wifi routers in Europe are all 16 or 32 MB of RAM". In what decade?

My ISP provided router also does VPN, VoIP, mesh networking, firewalling, and it's towards the lower end of feature set (as it's offered for free and not a fancy router I bought).

Are you talking about devices from the early 2000?

My TP-Link MR3020 from around 2015 only has 4^H 16 MB of ram (4 MB flash) and thus cannot even run OpenWRT anymore.

I’ve still got two MR3040. TP-Link hasn’t released any update for them in years. You can run an older version of OpenWrt on them, but there’s no real point. These things don’t even support 5GHz WiFi.

Device with very nice design. I still keep it as decorative even when its a brick.

I've got a few devices that only support 2.4 B/G. They're not in common use, but using an equally legacy router is the only way for them to connect.

2015 was also almost 10 years ago.

So i guess its a brick now and there is nothing we can do about it.

Using something that's already been produced (good) is not the same as selling dead end e-waste that's so underspecced it's barely working new (bad).

That thing was overstaying it's welcome for two years even then by staying on WiFi 4. 802.11n was adopted 15 years ago.

every. single. one.

pick any modem from linksys or dlink or netgear. then buy one in south America and compare what's really inside

look at all the revB on openwrt wiki, sometimes ram lowers. sometimes arm cpu change to mediatek. often the wifi chip changes from qualcomm to rtl. and it's always the revisions sold outside of usa and China in the observation fields.

> south America and Europe they are all 16 or 32 for no clear reason

I don't know where you're getting your data from but it's clearly wrong or outdated. These are the most often sold routers in Czechia on Alza (the largest online retailer) under $100:

- TP-Link Archer AX53 (256MB)

- TP-Link Archer AX23 (128MB)

- TP-Link Archer C6 V3.2 (128MB)

- TP-Link Archer AX55 Pro (512MB?)

...

- Mercusys MR80X (256MB)

- ASUS RT-AX52 (256MB)

https://www.alza.cz/EN/best-sellers-best-wifi-routers/188430...

"Best sellers" usually means "best advertized because of worst sales".

> It increases boot times

Is it really the case? On desktops it is significantly faster than all the other alternatives. Of course if you do know your hardware there is no need for discovering stuff and the like, but I don't know. Would be interested in real-life experiences because to me systemd's boot time was always way faster than supposedly simpler alternatives.

When Arch Linux switched to systemd, my laptop (with an HDD) boot times jumped from 11 seconds to over a minute. That 11 seconds was easy to achieve in Arch’s config by removing services from the boot list and marking some of the others as supposed to be started in parallel without blocking others. After the switch to systemd there was no longer a such a simple list in a text file, and systemd if asked for the list would produce such a giant graph that I had no energy to wade through it and improve things.

Later, when I got myself a laptop with an SSD, I discovered that what my older Arch configuration could do on an HDD is what systemd could do only with an SSD.

Arch changed to systemd in 2012, at which point systemd was 2 years old. It surely had quite a few growing pains, but I don't think that's representative of the project. In general it was the first init system that could properly parallelize, and as I mentioned, it is significantly faster on most desktop systems than anything.

it was only faster if you started with bloated redhat systems to begin with. but yes, it was the beginning of parallelism on init...

but the "faster boot" you're remembering are actually a joke at the time. since the team working on it were probably booting vms all the time, the system was incredible aggressive on shutdown and that was the source of it. something like it reboots so fast because it just throws everything out and reboot, or something. i don't really care much for the jokes but that was why everyone today remembers "systemd is fast".

It mandates strict session termination, unlike the unsustainable wild west approach of older Unix systems. Proper resource deallocation is crucial for modern service management. When a user exits without approval of "lingering user processes," all their processes should be signaled to quit and subsequently killed.

i think the "unsustainable wild west" of sending sigterm, waiting and sending sighup was very good because it was adaptable (you were on your own if you had non standard stuff, but at least you could expect a contract)

Nowadays if you start anything more serious from your user session (e.g. start a qemu vm from your user shell) it will get SIGHUP asap on shutdown, because systemd doesn't care about non service pids. but oh well.

...which is where the jokes about "systemd is good for really fast reboots" came from mostly.

The old way has literally no way to differentiate between a frozen process and one that just simply wants to keep on running after the session's end, e.g. tmux, screen.

It's trivial to run these as a user service, which can linger afterwards. Also, systemd has a configurable wait time before it kills a process (the "dreaded" 2 mins timer is usually something similar)

which was fine for everything that didn't need a watchdog. systemd on the other hand still lacks most common usecase and people bend over backwards to implement them with what's available. ask distro maintainers who know the difference between the main types of service files...

> In general it was the first init system that could properly parallelize

I'm not sure what you mean by "properly" but didn't initng and upstart (and probably some others I can't recall) do the parallel stuff before systemd?

I switched to systemd when Arch switched and from the get go, it was massively easier to parallelise with systemd than with the old system and that was with an HDD.

Systemd already parallelises by default so I don't know what insanely strange things you were doing but I fail to see how it could bring boot time form 11s to 1 minute. Also, it's very easy to get a list of every services enabled with systemctl (systemctl list-unit-files --state=enabled) so I don't really know what your point about a giant graph is.

Running things in parallel isn't going to make the disk faster... with a HDD I'd think it is actually even more likely to make the disk slower.

We don’t have to talk in hypotheticals here. Booting time benchmarks from the time systemd was released are everywhere and showed shorter boot times. It was discussed ad nauseam at the time.

I mean, this anecdote only tells us that if something is configured poorly it will behave poorly.

If you're working in the embedded space it's surely worth a little bit of time to optimize something like this.

Same for me. Incredibly useful on the bloated sensors running Linux.

All the smaller systems with no RAM run on baremetal anyway. There's no room and no need to run Linux or a threaded RTOS. Much less security headaches also.

Systemd trolls on HN are really out of hand. This isn't even credible.

Anything mass produced is going to be pressured to reduce BOM cost, RAM capacity is still and will continue to be a prime target.

OpenEmbedded/Yocto, Devuan and Gentoo provide multiple init systems.

systemd CVEs: https://ubuntu.com/security/cves?package=systemd&limit=100

Rust PoC: https://github.com/KillingSpark/rustysd

> Rustysd is a service manager that tries to replicate systemd behaviour for a subset of the configuration possibilities. It focuses on the core functionality of a service manager, not requiring to be PID1 (aka init process).. core systemd functionality is not very hard to replicate.. the advantages of having a systemd-like service manager can be brought to many other platforms.. (like alpine linux, commonly used in docker containers and small vms) and freebsd.

How many systemd CVEs are memory or thread safety related?

I can't tell if this is rhetorical but... it looks like quite a few on the list. (Note: the CVE list is long but it looks inflated / full of not-serious issues.)

* CVE-2022-3821 : off by one error leading to buffer overflow

* CVE-2022-2526 : use after free

* CVE-2021-33910 : "Memory Allocation with an Excessive Size Value" (not sure if this qualifies... not interested enough to read the source)

What do memory and thread safety have to do with the project's stated goals? I for one would love to avoid systemd's arbitrary requirements while retaining compatibility with upstream (e.g. PID must be 1).

I'd bet you get more CVEs and actual remote exploit entry points by systemd forcing mdns, bonjour, upnp and other zero conf hacks just because that was the work the systemd team was doing in rh before.

remembered another one. They use the same code for VM and bare metal. Including the easy-login convenience hacks.

So now anyone wanting physical access to your host, just have to intercept plain-text communication with the BIOS (after secureboot did its thing), and reply with a new root password when the OS request bios key 11 or something. evil maids are living the dream.

Is that with systemd logind? Why does logind interact with BIOS?

https://vincent.bernat.ch/en/blog/2021-startx-systemd

it's before logind. it creates arbitrary files in the root filesystem and is intended to write to the user authorized keys. so it works later with sort of login or remote access

Rust would have the same issues since you'd need to use unsafe code to achieve certain things systemd does.

> The more fundamental problem is that the people who most like systemd are distribution managers. (...) a distribution manager doesn’t have to maintain, integrate, and support a whole bunch of system utilities from different sources – systemd provides everything in one huge build.

That’s actually a great lesson in optimizing your product for adoption.

it's also why corporate software is so shit - it's shiny in all the ways it needs to be shiny for management to make the decision, but it's a pain for the actual users.

the trick in both cases is to know your customers!

Turns out, many people use systemd after what, 15? years of doomsday predictions :-)

Unfortunately not by choice

i don't notice anything good systemd does for me, just where it interferes, and that's a constant annoyance, a few days a week.

Can you link some upstream issues you raised about the big annoyances?

not many people enjoy bikeshedding.

Random example. They randomly broke suspect-then-hibernate then the community manager gaslit people into trying to make them believe they don’t actually need the feature in the way it’s used and then silently acknowledged that their fix is broken after all.

This is a common pattern by the way. But easily the most irritating thing for me is the simple issue that you used to be able to just go to /var/log/lastlog to see what failed during the last boot and that just the other day I had to spend hours figuring what broke in the system that made journalctl not log properly. In the end I had to reinstall all currently installed packages.

https://www.reddit.com/r/archlinux/comments/zczdnq/systemctl...

So there was a bug and bad interaction and everyone on non-rolling system didn't even notice, because it's been fixed in 2 months? (Nov to Jan) It sucks, but it's been fixed for almost 2 years at this point.

For the last boot you can use "journalctl -b -1" as long as you enabled persistent logs. If you ended up reinstalling everything, I don't think we can say whether that was a systemd related issue or not.

I was curious what the current daily issues are.

The linked GitHub issue is one of the most aggravating exchanges I've ever read: https://github.com/systemd/systemd/issues/25269.

I don't understand how you can read someone telling you that they want to suspend and then hibernate after a set duration, but A) not understand why this would be desirable, B) not understand that this is compatible with also hibernating at low battery, and C) not understand your own lack of comprehension.

What the issue author is asking for is also the default behavior of Windows (which is developed by his employer) on laptops.

Yeah, that's pretty shitty behaviour, invalidating all the good-faith feedback and going so far as to call it "trolling"?? Ridiculous. There's no excuse to be so hostile when handling bug reports.

> The linked GitHub issue is one of the most aggravating exchanges

Hyperbolic much?

Because if you actually read the whole issue, here we have one maintainer acting out of line of refusing to process the issue until another one (Yu Watanabe) interjects, says there is an issue and actually commit a patchset adding the option asked for.

I cannot think of any exchange I've read that aggravated me more than this one.

I'm hedging my bets with "one of" because there might be something I've forgotten, but that specific series of events is like reading a transcript of The Trial if it happened in real life. It is seriously extremely annoying to me and I'm both intrigued and concerned that it doesn't even rank in your list.

The cgroups changes [0] made my life more difficult for a while. "all the low level bits will go away without replacement".

[0] https://lwn.net/Articles/555922/

The thing is, the good things are the things you don't notice; the things that just work.

They didn't used to "just work" like that before systemd.

System boot is faster. I can have fewer things started up in the background, and instead have them start up when needed. Restarting services works consistently. Every application doesn't have its own bespoke and half baked management scripts which don't work half the time. They don't all have to invent their own daemonization support and logging; you just log to the journal.

And that's just core systemd. Things like systemd-resolved give me proper support for split-DNS when using VPNs. systemd-networkd gives me consistent, powerful configurable networking setup that works across distros.

Is it perfect? No, I do still have some complaints with it. But it's a hell of a lot better than what it's replacing. I wouldn't ever go back to sysvinit or upstart, and many other parts of systemd are compelling alternatives to the things they replace, like systemd-networkd over ifupdown.

>the good things are the things you don't notice; the things that just work

sorry, no, at least in any meaningful sense, because init already just worked for me.

and when something didn't, I could find how to fix it in a way that was transparent and I understood and could even modify or make better, without being connected to the internet looking for cargo cult incantations on stack overflow

init was a tool; systemd is an adversary

Yeah, the two systemd components GP mentioned have given me no end of trouble.

Okay, I'll bite.

What kind of problems do you have with the part of systemd that replicates sysvinit /every other day/?

My most common issues with systemd are related to those long timeouts when something at boot/shutdown is not working as intended, and unexplained/unexplainable changes to the order of boot of some components. For the former I have given up playing whackamole with all the timeouts you need to reconfigure, for the latter I didn't even try because I know that there's something peculiar about my setup that will never work nicely with systemd, there's simply not enough systems configured like that for upstream to care. I have accepted this new reality, but I know that before systemd I was able to fix any highly customised setup of mine, now I have to avoid that and minimise tinkering/hacking.

> now I have to avoid that and minimise tinkering/hacking.

I think this right here hits at the crux of the issue.

There are people who like systemd because it's integration-tested with itself and its own defaults, so if you never change those defaults you don't have many problems.

Then there are people who don't like systemd because if you do have to change any of its defaults, it often doesn't go well. And, of course, the latter behavior as a box users are expected to live in is poisonous, because everyone is being conditioned to be passive and uniform.

Yes, I think your analysis is spot-on.

"See Figure 1".

> And, of course, the latter behavior as a box users are expected to live in is poisonous, because everyone is being conditioned to be passive and uniform.

No, it's not, this assumes that the other camp are all idiots.

Mainstream distros should be rock solid and boring.

Linux never got anywhere with the standard distros because they're all so different.

Tinkering is a different mindset (and has a different place) than professional engineering work.

Systemd is basically engineering, SysV & co. were basically old school tinkering/hacking.

In the same vein, don't be creative with bolt sizes. Be creative with what those bolts allow you to achieve. Be creative at a higher level. That should be the nature of humanity.

> Linux never got anywhere with the standard distros because they're all so different.

This is eliding the difference in what you want to be standardized.

Take system logging for example. You definitely want some standard interface to do it so all the different daemons and things can implement it once regardless of which system logger the distribution is using. But once it passes that data to the other program, it's under a different dominion of control which should operate as a black box with respect to the program generating the data and the rest of the system.

Because that's how you prevent ossification -- which is engineering. You want to make it so the other component can be improved or substituted. One system logger is designed to store the logs on a central server instead of the local machine, another supports modules so other people can easily write log-parsing scripts. And when you come up with a new idea for a third, you don't have to be on the systemd team to publish an independent implementation that other people can use as a drop-in replacement, instead of (not in addition to) the default one, without requiring it to be separately integrated with a dozen moving targets in the systemd repository.

What you're trying to do is to allow this:

> In the same vein, don't be creative with bolt sizes. Be creative with what those bolts allow you to achieve. Be creative at a higher level.

You want to standardize the bolt sizes, i.e. the interfaces between components. What you explicitly don't want is to replace bolts with having everything epoxied together.

But that's what happens when you e.g. integrate the logging system with the service manager, which is the sort of thing people are complaining about.

We've had Unix for what, close to 55 years now? Nobody standardized and got widespread adoption for those standards, which BTW, especially when you look at corporate interests regarding the web, are easily sabotaged.

So I'd rather have the epoxy open source standard than no standard.

And now we're back to this:

https://news.ycombinator.com/item?id=42039273

There are things that systemd does better than SysV and other things it does worse and you can want it to stop doing the things that are worse while still doing the things that are better.

> it often doesn't go well

Plenty of people changes systemd configuration all the time and it just goes fine. You live in fantasy.

Even op is basically saying: “my issue with systemd is that I dislike the timeout configuration of some services but I stubbornly refuse to change these configurable timeout durations because it would show that the problem was myself and I prefer blaming systemd.”

It takes no time whatsoever to get a boot graph with each services name and starting time. That’s an actual feature documented in the manual of systemd which solves OP issue. But of course it would require actually understanding something new and everything new is bad, isn’t it?

> Plenty of people changes systemd configuration all the time and it just goes fine. You live in fantasy.

"since I have never experienced what you say, it must be fantasy"

> Even op is basically saying: “my issue with systemd is that I dislike the timeout configuration of some services but I stubbornly refuse to change these configurable timeout durations because it would show that the problem was myself and I prefer blaming systemd.”

This is a perfect example of toxicity; I have been successfully using systemd for years and I am entitled to point out what I dislike, I do not have to love everything of it, it's not a religion nor a cult. Your reply tells more about yourself than the topic of the discussion at hand.

> It takes no time whatsoever to get a boot graph with each services name and starting time. That’s an actual feature documented in the manual of systemd which solves OP issue. But of course it would require actually understanding something new and everything new is bad, isn’t it?

You're missing the point, the problem is not changing timeouts but preventing failure and achieving an overall deterministic behaviour out of your system, without ignoring failures. But I refuse further eating these baits, you seem more interested in creating some flames rather than having constructive discussions.

> I have been successfully using systemd for years

You are writing that you have been unsuccessfully using systemd for years because of the annoying timeout. I'm replying that it's entierely your fault because it takes seconds to make these timeouts disappear.

That's not toxicity. That's calling out some non sense on the internet.

> My most common issues with systemd are related to those long timeouts when something at boot/shutdown is not working as intended

That’s an issue with a daemon, not systemd. Anyone who used NFS saw that routinely on SysV init during the era when Red Hat distributions shut down networking before ensuring that the network mounts were unmounted.

I regularly see it also on a system not using NFS, and it seems related to console seats. Never went to the bottom of it because it's sporadic/non-reproducible.

Yes - my point was simply that the shut down process tells things to stop but most sysadmins have war stories about that not working well for all kinds of things.

The problem is that there isn’t a universally correct way to do this: if my web server has hung, SIGKILL is what I want to get the system back in a usable state as quickly as possible but if it’s a file system, database, etc. you have questions like losing data which aren’t trivial to answer (e.g. if it’s a transient error, waiting for it flush is good but if my storage had an irrecoverable error I might want write off that data as lost and focus on getting the service back up).

    /etc/systemd/{user,system}.conf.d/dont-wait.conf:
    [Manager]
    DefaultTimeoutStopSec=5s

There, done. That's all the timeouts you'll ever need. Go and complain to your distribution they are setting the wrong defaults, even desktop environment(s) already recommend[1] setting it to a low value.

You can stop hating on systemd now, everything you needed was in `man systemd-system.conf` all along.

[1] https://community.kde.org/Distributions/Packaging_Recommenda...

> There, done. That's all the timeouts you'll ever need. Go and complain to your distribution they are setting the wrong defaults, even desktop environment(s) already recommend[1] setting it to a low value. You can stop hating on systemd now, everything you needed was in `man systemd-system.conf` all along.

Here's another example of cultism repressing any dissent.

Let me make a bullet points list for you:

* I do not hate systemd, I have never stated that, I have been using it for possibly longer time than you and love most of it

* I am entitled to write about what I don't like, you can disagree and move on, we all need to do this exercise on a daily basis

* there are cases where systemd will change the order of dependencies during boot, that's by design because systemd works in a way that tries to achieve states. It's not really enforcing a graph with order

* in a sufficiently complex system, this constitutes a source of non-determinism and it is basically undebuggable: you see the failure, learn about the corresponding configuration, change it and hope that you did the correct change (you have no way to test this until next random occurrence)

That's all, it's based on my experience; I write plenty units on a regular basis and 99% of the times everything goes very well.

I've got the waiting ages for shutdown feature on reboot, so I tend to force reboot it. I never bothered trying to fix it because firstly I don't know where to start and secondly I only reboot after updates once every few months.

well, systemd is doing things that init did not do, so that's not a replication, but if I kill something, it should stay dead till I restart it; if I dismount a volume, it should stay dismounted; if I set the permissions on something, they should not reset back. Don't treat me like I'm the anomoly.

"I'm afraid I can't do that, Dave."

> They didn't used to "just work" like that before systemd.

They used to "just work" for _decades_ before systemd. Like, everything.

I see reasons for systemd as PID1 and it's great in that role, hands down.

Everything else looks like a malignant growth to me, including much praised journald and DoA pieces like resolved.

> systemd-resolved give me

gives me #1 source of troubles on any desktop computer I have ever ran under the systemd era. I feel like resolved is specifically designed to be removed from fresh installations, sort of like a transparent peel on new devices' screens.

In my strong opinion, resolved is the most brain damaged part of systemd. Unconfigurable, unmaintainable, unmanageable.

> systemd-networkd gives me

Oh, there is no point in learning systemd-networkd. By the time you do Ubuntu folks will swap the set of the network initialization tools once again and you'll have to start over. Thankfully, 'apt install ifupdown' still works kind of like `apt install upstart` worked a few years ago.

i don't think many distros use networkd. they all ship it disabled with nm instead

Turns out many people don't care to change their distribution defaults.

Exactly, because I don't care about the plumbing in my house unless it's really bad. Systemd is opinionated and quirky but it does the job just fine for 99.999% of people and it provides a standardized approach to things just by virtue of how it works, and there is a lot of value in that.

The real question is would anything else also be able to do 99.9999% of the job with less code and better semantics.

People seem to think it is.

Poettering actually took his idea to completion.

A lot of people claim the same, but the proof is in the real life pudding.

Don't forget that geeks idealize perfection and especially minimalist perfection.

> Poettering actually took his idea to completion.

One of them. The others didn't work out so well. That's a curiosity in and of itself.

> A lot of people claim the same, but the proof is in the real life pudding.

Those claims are true. Those systems exist. People use them. Every day. I've got tons of business riding on them. Systemd's defaults are simply a non starter for any real production environment.

> Don't forget that geeks idealize perfection and especially minimalist perfection.

They prefer a system which leaves the options open. Systemd removes them in the name of simplistic cohesion. It's all based on the flawed idea that linux will ever be a popular desktop operating system if you just make it work like windows does.

It misses the point on what computing is entirely, but hey, as long as it's "complete" implementation of this flawed vision I guess that's good enough for some people. And yet we're still no where near windows.

What do you imagine this proves or disproves?

How many people eat McDonalds? Or buy in to countless other things that can be trivially demonstrated to be against their own interests.

I don't know about anyone else but I never heard the critiques of systemd to be based on any doomsday predictions.

Everyone knew it would function, and that it even serves one use case better than before.

The critique was only ever that it serves one purpose better at the expense of all others, and is a downgrade in flexibility and functionality from what already existed.

> How many people eat McDonalds?

If you’re already starting like this it’s obvious that the following text will be non-scientific non-sense.

As a reminder: There is no such thing as “healthy food” and “unhealthy food”, there is just healthy and unhealthy nutrition.

Still baffles me that people think it would actually be legal to sell actual unhealthy food, i.e. poisonous food.

Also, “organic food” is just a marketing scam. If you think your nutrition is better off if you stick to nature, you’ve missed a large part of human history where people starved to death by the thousands every year because they exclusively relied on “organic food”.

systemd runs on billions of systems worldwide and everyone who ever professionally maintained enterprise servers understand what a bad approach the crude hack that SysVInit with its collection of shell scripts was.

And yet, you’ll still find many of us that think systemd is worse, but for different reasons. (I think the food analogy was lost)

It is legal to sell unhealthy food, ie poisonous food. Sugar is a poison, drugs are poisons. You can legally sell escolar a fish that if eaten too much is poisonous.

What is poisonous food? You can sell plastic that breaks down into the environment and people are drinking it in water, eating it in fish. Is that legally sold poisonous food?

Sugar is not, in fact, a poison.

As for drugs:

> Alle Dinge sind Gift, und nichts ist ohne Gift; allein die Dosis macht, dass ein Ding kein Gift ist.

  - Paracelsus, 1538

https://en.wikipedia.org/wiki/The_dose_makes_the_poison

Found the Hackernews who hasn't yet read Lustig's research into the hepatotoxicity of sugar. It's as bad for your liver as alcohol.

Sugar is indeed poison.

Personally, I like to add a bit more science and reasoning to my understanding rather than rely on quotes from someone in 1538.

Everything is a poison. It's simply a question of dosage.

I grokked the quote and my response is still that it’s a cute way to hand-wave away the fact that poisonous food exists.

Sugar is not poison. It can be, in certain situations, "bad for you". But sugar is found naturally in a variety of fruits and vegetables and has been a staple of the human diet for all of history and pre-history. There's no way to describe it as "poison".

The sugar being discussed is not natural sugars but refined processed sugars, that are absolutely poisons.

Booze has been a staple of diets for a long time too. Doesn’t mean it’s not poison just because society is stupid enough to embrace it.

Do you have any evidence of this or is this just your belief? I think, if it's just your belief, you should say that.

Do I have any evidence? As in have I performed studies on processed sugar and gathered evidence? No. I do not have evidence.

A quick web search will gladly provide evidence if you actually are interested.

Here’s a HN thread on an NPR article: https://news.ycombinator.com/item?id=26126183

Here’s a Canadian news agency article: https://www.cbc.ca/news/health/how-toxic-is-sugar-1.1894262

If you’re just going to knock everything off as “just your belief” then no amount of evidence will convince you.

You seem to be a very confused person. I don't even know where to begin with all of this randomness.

Do you actually not understand what is wrong with McDonalds food? Or countless other similar low quality but legal products?

Do you actually not understand that terms like "healthy" and "poison" are not binary but a spectrum? And that every single thing has some aspects of both?

Do you actually not understand how a bag of sugar is not poison and legal to sell, and yet, if you ate one for dinner every night would not be good?

Do you actually not understand the point that popularity does not by itself prove or disprove anything? That great masses of people frequently voluntarily choose the inferior option for all kinds of reasons that aren't even all some form of lack of choice?

Do you actually not understand that that you are attempting to speak for "everyone who ever professionally maintained enterprise servers" TO some of them who quite robustly do not say any such thing?

Ignoring the food portion of the above:

> systemd runs on billions of systems worldwide and everyone who ever professionally maintained enterprise servers understand what a bad approach the crude hack that SysVInit with its collection of shell scripts was.

This is your opinion and not one held by all. It also ignores the various BSD system initialization scripts, which are explicitly not "SysVInit" related, and have been in use for decades.

It also ignores the diverse (if small) set of Linux init systems... 'openrc' being the one I'm most familiar with, as a Gentoo Linux user.

I'm so sad I wasn't paying attention when Debian was choosing init systems... I would have burned a TON of time getting anything they claimed in good faith was missing in 'openrc' into it, had I known that that clusterfuck was going on.

Non organic food is very obviously a poison for the vast majority of Earth's life.

See : the current apocalyptic and accelerating global biomass destruction directly caused by industrial farming.

Please explain to me how farmers who are making organic food are performing a scam.

The fact that you don’t like how organic food is marketed does not mean it is full bore a scam. It’s a real agricultural practice that exists. Organic eggs require far more freedom for chickens for example.

I'm not a systemd hater, but I use it mainly only because my distro of choice (Debian) switched to it. If Debian chose something else, I'd be fine with that too. I'm a user out of necessity (or perhaps inertia), not out of choice.

what a weird comparison...

I think what he's running into is "the unix philosophy".

It is basically about small tools, limited in scope, that solve a problem surgically and comprehensively. They work together by mixing and matching to solve the problem.

systemd is sort of like if microsoft word took on system init. In the beginning it was small, but now it is web based and does videoconferencing.

I don't know, it is sort of like busybox, replacing other system components in a limited Minimum Viable Product way, but more like limited function, not limited resources.

> systemd is sort of like if microsoft word took on system init. In the beginning it was small, but now it is web based and does videoconferencing

And it's slow. Windows 11 boots in 5 seconds on my laptop, Ubuntu 22 used to take about a minute before I finally made the switch.

That almost certainly has little to do with systemd itself, and more to do with which services are enabled on boot.

It's due to systemd in that systemd makes it too fiddly to figure out or change which services are enabled on boot. (I used to know a way to disable certain services on boot under systemd, but it doesn't work any more, and I've reached a state of learned helplessness at this point)

`systemctl disable` and `systemctl mask` work today just like they worked decade and half ago.

I don't know if they're the recommended way of doing things today, but they weren't what was in the manual when I was learning.

They are there from the start and were always the recommended way to do things in the documentation. I mean the command is literally disable.

The command to list services enabled on boot is also completely esoteric: systemctl list-unit-files --state=enabled

My system boots systemd in 2 seconds from the firmware handover to my code fully running. And this includes device initialization and a DHCP lease renewal.

On VMs you can get to within 200ms for a full system.

You can use `systemd-analyze critical-chain` to visualize the boot process. The slowdown is likely caused by something unexpected.

Windows 11 probably uses Fast Startup for those speedy boot times, which is to say that it's not a "cold" boot. Apples and oranges and all that...

> Windows 11 boots in 5 seconds on my laptop

Try measuring a reboot instead. On modern Windows versions with “fast startup” enabled, the “Shutdown” button is a lie: it logs you out and hibernates the system instead of actually shutting it down. (This is important if you dual boot, because it means you can’t safely access the Windows partition in the interim.)

> the “Shutdown” button is a lie: it logs you out and hibernates the system instead of actually shutting it down

I'm okay with that. How do I enable this on Ubuntu?

Are you seriously complaining right now that systemd doesn't have a specific feature no other init for linux has either, while at the same time complaining it is too complicated and does too much stuff?

I'm asking a question

I think the problem is that small tools are missing that "comprehensively" part.. trying something as simple as getting full system startup log (something very useful on headless system!) - it was basically impossible until systemd came. Only gentoo tried to do this for a few years, but then that functionality broke.

Same goes for service startup - why does start-stop-daemon discards stdout/stderr instead of logging it? This would be trivial to fix, but somehow this was not available until upstart (and then systemd) did it.

This seems like the motte and bailey thing.

Some of the oldest init systems made decisions that were sensible at the time but not so much now, e.g. when storage is hundreds of thousands of dollars per GB you don't want to log everything, when it's tens of cents per GB you do. But you can obviously make a non-monolithic init system that causes output to be stored to a log instead of being discarded.

So then systemd comes in, solves some of the old problems and independently, unnecessarily introduces new ones by being a huge monorepo that lacks clean and stable interfaces between its own components, inhibiting anyone else from providing a viable competing implementation of an individual component.

Then people complain about the latter and the defense that comes back is of the former. But that's no defense -- we could instead be making the good change and not the bad change.

> lacks clean and stable interfaces between its own components, inhibiting anyone else from providing a viable competing implementation

That's exactly what Microsoft did back in the era with Word and others, just like GP stipulated.

> systemd is sort of like if microsoft word took on system init. In the beginning it was small, but now it is web based and does videoconferencing.

It's actually a suite of tools that do solve individual problems comprehensively. Not as small as the programs in coreutils, but still not a huge monolith.

I don't agree with comprehensively.

for example timesyncd is barebones and not super accurate when it comes to timekeeping, while chrony and ntp actually manage the system time.

similar - systemd-boot is barebones, systemd-networkd is barebones (but has gotten more MVP-featured in a busybox way), etc

Maybe 10 more years and everyone will be used to things, and people won't be arguing about chesterton's fence.

https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton's_...

> It's actually a suite of tools

It's a suite of tools created with hate to the unix philosophy. That's something that all of them have in common. There was no reason to create systemd timers as cron worked just fine for decades.

> with hate to the unix philosophy

It's a suit of a tools each doing one thing and communicating together via a message bus. How exactly is that hating the "unix philosophy" whatever that means? From where I stand, it's extremely close to the way Unix has ever done things. The design is even directly lifted from the boot system of a certified Unix.

To me it looks exactly like ann opposite of suite of independent tools.

Okay. How? They're all individual binaries that just happen to use the systemd name. You realize systemd isn't just one big program, right? It's like 30 programs, all independent of each other, with integrations to other programs.

I believe they were not created to solve real problems. No one wanted a cron replacement, really.

Thus they are created out of political or ego reasons.

This is purely speculative. From my talks with sys admins, all the ones I've met greatly appreciate systemd. Also timers and cron have different requirements, notably timers can do much more. Also timers is purely optional and most distros DO use cron.

There's a major disconnect, I've seen, from what systemd haters proclaim and what is reality. Nobody is forcing anything, you don't have to use 99% of systemd, it's not a monolith, and many people do want the features. To be clear, this isn't my opinion - this is the truth. You can find all this out by perusing the repos and mailing list.

> greatly appreciate systemd

Really? I have yet to see one. Kids, maybe. Experiences sysadmins may appreciate systemd as PID1, but almost everything else is pure cancer.

> There's a major disconnect, I've seen, from what systemd haters proclaim and what is reality.

I can certainly agree with that.

> Nobody is forcing anything

... until distro maintainers decide to move crucial tasks to systemd-timers from cron, change resolved to systemd-resolved, switch from ntpd to whatever its called, remove sudo in favor of another systemd thing, etc. But sure, you are free to not use any. Except the only *nix thing left is the Linux kernel. Soon to be systemd-kernel.

I've seen a lot of complaining without any facts.

I've been building and automating data centers for years now. Systems has greatly simplified my life. I use all of the systems tooling; timesyncd, networkd (which is actually does _more_ unlike what is stated above, go read the man page), timers (better than cron, again go read the man page) resolved.

Not only does it solve a lot of problems, the configuration file format consistent across the tools, and the configuration is consistent _across distros_. I can write large swaths of automation that just works on most modern Linux distros.

Anyway, I'm not one to argue on the Internet. But I'm absolutely tired of hearing that's it's a dumpster fire. Sure, there are legitimate complaints and use cases where it might not be a good fit, but that holds true for all software. But, there is a reason so many distros switched to it. It's objectively better than the way things were. Those of us that deal with this stuff day in and day out are not going back to the way it was.

Cron is probably the worst example you could have chosen. For my uses cases that, you know, involve manually running jobs to see if they actually work, cron has been broken for decades and is still to this day basically unusable. Systemd timers work far better than cron.

If it's broken for decades, what took us so long to change it and who gave systemd swamp the honor to provide the one and only right solution?

> I think what he's running into is "the unix philosophy".

"Unix philosophy" is throwing crap at the problem, until it kinda-sorta works on the developer's machine.

I'm conflicted on the name/issue. I don't have any issues with Linux on embedded systems, because I'd use a minimal init with that one single app it should be running. (Or as mentioned in other comment, build your minimal distro yocto-style) RPi is a low spec general computer at this point. The idea of running gnome on an embedded system just doesn't make sense in my mind...

RPi took the form factor that was used for industrial control boards and made it into a usable desktop... which people use as a desktop (or home server). But it created a segment which really is neither. And a lot of the software just isn't written with that segment in mind. Maybe as it gets more popular, people will pay more attention. But currently it seems more of an expectations issue.

https://chimera-linux.org/ https://chimera-linux.org/docs/faq#what-is-the-projects-take... Is going a really interesting direction with it's service management and login / seat management. Fwiw, I think it's author also has a more gounded perspective on systemd then many others. For Chimera the main issue is systemd's high volume use of gnulibc and gcc extensions, which is a porabiluty issue for Chimera which uses a LLVM toolchain and musl libc (and a bunch of really cool hardening and compiler engineering)

I like the direction Chimera is taking; the lead is an ex-Void maintainer and there's clearly a lot of inspiration from there.

I’ve never used it as an init replacement, but runit (https://smarden.org/runit/runit.8) is great for supervising applications and services.

- It’s built into BusyBox and is very lightweight - Configuration is dead simple (everything is basically shell scripts in the filesystem; no special config language) - Includes logging functionality with rotation - Easily controllable from other applications using named pipes - It’s an almost perfect embodiment of the Unix philosophy: a series of small, single-purpose executables (svlogd, runsv, runsvdir, etc), with no strange config file formats or protocols.

Regarding udev, it’s entirely possible to run an embedded system without it if your device has a fixed set of peripherals, doesn’t need to handle hotplugged hardware, and uses a fixed set of kernel modules. In Yocto, you can define a dummy device manager recipe and boom, no udev. As a bonus, removing udev can shave several seconds off your boot time.

For what it's worth, runit is the default on Void Linux.

Wait so systemd is a problem for embedded Linux because... it uses 250MB of RAM? So it's only really a problem for systems where using a mainstream distro probably wouldn't even be a consideration anyway (yocto seems to be popular, as mentioned above). I'm not seeing a strong argument here. At the risk of inviting a parade of "actually, one time at band camp I saved the day with a first-gen raspberry pi zero", how many embedded systems in 2024 are really RAM-limited?

> how many embedded systems in 2024 are really RAM-limited?

A lot. Plenty embedded Linux stuff still ships with 256 MB or 512 MB RAM, and the wishlist for features in the software running on top is always getting longer than initially planned...

But of course the average has moved up, and it's not unusual to see systemd in embedded systems either (while I don't have a number at hand, the 250MB number seems off to me). The space is big, and different constraints and rules at different points in there.

Seems like a complaint the author could help solve by extending the general purpose systems to their choice of hardware.

I remember when I used to run gnome 2 (i.e. mate) on a machine with 256mb of ram. It was a full experience and it worked with youtube videos and so on. What are we even doing .

Sure, back when videos were at most 480p/720p that's feasible.

I'm not saying software is any more efficiently written these days but I do think it's important to recognize that just the act of pushing more pixels on its own requires more RAM.

Honestly, no. Prior to 2009 there were almost no 720p videos on the platform. In 2009 Windows 7 came out with a minimum requirement of 1GB RAM (Vista in 2007 also already recommended 1GB). What I'm trying to say is, 1GB wasn't much even before 720p got common. The amount of people watching 720p on systems with less than 1GB has likely always been minuscule.

Thats a weird question, every embedded system that is a consumer product will end up with limited RAM simply because less RAM is a $1 saved on the BOM. For most systems, RAM is a choice, and for many of them someone will ask at the end "how much can we get away with".

(Your WiFi router probably has 256 or 512 MB of RAM)

I argue that the extra $2.50 (or whatever) to expand to 4GB is worth being able to use a stock Debian + systemd.

Except those $2.50 for customer will turn into $2500000.00 for the hardware vendor. And they would rather keep the $2500000.00 and let it be someone else's problem.

If you're shipping 1 million units, don't use Linux. Done.

You are implying hardware vendor is the one who got to choose the OS, but that's often not how it works.

Who's forcing it on them? Or, are they systemically incapable of evaluating what their suppliers are providing? Sounds defeatist to me.

Spending hundreds of thousands of dollars in additional bulk cost for the privilege of nothing other than adding systemd to the device, what a steal!

$2.50??

(Almost) All the routers that people have in their homes - the ones running OpenWRT - have between 64-256 MB RAM, and no swap. Raspberry Pis are not "embedded", they are cheap, low power consumer devices.

Actually, modern routers typically have 512MB RAM. And RPis absolutely are embedded, not only are they often used for one-off embedded installations due to the 2-day lead time from Amazon, but also the compute module is very popular in industrial control.

The memory comparison isn’t really fair as SysVInit relies on external utilities for starting and stopping processes.

At the very least, SysVInit requires a shell for running init scripts which systemd doesn’t need for native service files.

Real embedded developers wouldn’t stand for the overhead of modern stuff like SysVInit.

Those 'real developers' you're talking about wouldn't stand for the overhead of a kernel either

> I wish I knew the solution to this problem

Ooh ooh, I know! It's to improve systemd for the use-cases outlined and build a really great open source init/process manager.

The systemd maintainers are pretty particular about things that aren't a "supported use case", so even if you bother to do the work, you stand a solid chance of having your work be wasted. In my professional experience, "unsupported use case" effectively means "I don't want to work on that". [0]

Also, at $DAYJOB, we run into mysterious systemd failures and misbehaviors at least once a year (usually for things that should be super straightforward). Every one of these failures and misbehaviors we've run into has been met with a "Wow. That's weird. Sucks to be you, I guess." with a dash of "No, we won't be updating the documentation to mention that." if the actual behavior is contrary to the documentation.

In short, while it may be true (I really don't know) that the systemd project welcomes changes and patches, IME it's pretty clearly true that if you're working on a problem that the project management doesn't understand, and/or doesn't want to think through, your work won't be considered. [0]

[0] Which, fine, it's not my project, so I don't get to tell you what to work on. Doesn't mean I have to like it or recommend that people use it.

    > we run into mysterious systemd failures and misbehaviors at least once a year

Can you share an example?

> Can you share an example?

The most detail my NDA permits me to get into is that our most recent failure was that the systemd cron replacement gets in a state where it refuses to ever schedule a scheduled task and that the systemd folks were like "aw, shucks" about it.

An example from my company - sometimes on a few of the many thousands of identical devices in field something fails due to bug or hw failure. After recovering the device we get the dump of all info we deem interesting, including all logs and journals. In a small but not insignificant numbers of such archives the journal has multihour gaps in the logs. And they are gaps, so there are logs before the gaps and after. I think we never succeeded to reproduce or fix this issue in the lab, and relied on duplicating logs to the file on the device fylesystem (not optimal due to storage size and wear of the ssd memory).

PS: personally I like systemd more that the script mess which preceded it. But the are some outstanding issues with it to be improved.

I ran into this issue on an embedded system before it was raised / fixed:

https://github.com/systemd/systemd/issues/8398

FWIW, I'm generally a fan of systemd. That's a pretty minor issue in the grand scheme of things and it's not really "mysterious" -- the behaviour was consistent but didn't line up with the documentation.

In my experience, almost every time I hit an issue with systemd, it's when I'm interacting with it at a (relatively) low level. e.g. Monitoring / controlling units via D-Bus. There is theoretically enough documentation to do that, but it's often incomplete or ambiguous and you're left trying to figure out systemd's behaviour via experimentation.

But as I said, this isn't _really_ a dig against systemd. Even if monitoring service state transitions via D-Bus is finnicky, it's a heck of a lot easier than the alternatives with busybox init or whatever.

> Ooh ooh, I know! It's to improve systemd for the use-cases outlined and build a really great open source init/process manager.

People already did the latter (sadly no-one used it because there was no mechanism that forced distros to adopt it), and would do the former if the systemd maintainers would let them.

I have no philosophy issue with systemd. The biggest show-stopper (on embedded Linux) for me is its install size. We have edge devices with only 64M RAM and 128M NAND flash. For an OTA update (via Bluetooth sometimes) only ~20M in total, adding ~5M systemd plus its dependencies doesn't make sense. 5MiB is roughly equivalent to the entire Linux kernel image on those devices.

Among the components of systemd, the part that has the lowest added value for our use is dbus. It's designed primarily for desktops and useless for most of our systems, but unfortunately, it is also a hard requirement for systemd.

The BSDs may be more to your liking. They're a lot more minimalist than Linux now.

In the embedded world, BSP/driver support is the main thing that matters. Linux is undefeated in that area for embedded purposes.

I thought there was a compatibility layer available that made it possible to run Linux drivers on at least FreeBSD.

That's for userland, not drivers. Linux doesn't have a stable interface for drivers so a compatibility layer would be a nightmare to maintain.

That's Linuxulator, but there's also LinuxKPI: https://wiki.freebsd.org/LinuxKPI

In embedded you should choose hardware that works. Should is key, all too often the, hardway guys don't ask they just give software something.

> In embedded you should choose hardware that works.

I've never seen it approached with that kind of incompetence, in my professional life. Embedded is a system thing, not a hardware thing. You pick the system that will work. Ignoring the software side of things is ignoring the majority of the problem in embedded work. I would have agreed with you 20 years ago, but it's 2024, and why RPI is so often used for prototyping.

Sorry, wrote the original comment on my phone. The sentiment is close enough, but generally things are not quite that bad. I've seen hardware ask, but the person they asked didn't know how to check how much memory we actually needed. Hardware also failed to tell that person that half the memory they were asking about was dedicated to the GPU and thus not available for application code. I've also seen them choose hardware that seems to have great drivers and they can point to others in our industry who are shipping with that, only to latter hit a major bug that means we can't use it without that fixed - but we are too small a customer to be able to demand they fix bugs that only affect us.

RPI is often useful for prototyping because it is cheap enough, but if you are not a very large company nobody will talk to you. This is both you can't get support from Broadcom, and worse your supply management cannot get contract signed (for small companies this doesn't matter you take what you can get, but larger companies often demand better support than they can get from pi)

> With care, though, a minimal installation of systemd does run on a low-resource ARM board like the Pi 3. In fact, it will run on a board with 1Gb RAM, perhaps even lower.

Come fucking on. I ran systemd on a 32Mb MIPS board (RouterBoard from Mikrotik) just fine.

The RES column is incredibly misleading in this case, because systemd is the first app in the system. So it gets charged for loading the entire glibc and the supporting cast of libraries.

If you want to look at the true usage, check the RssAnon value:

> root@kmipsrv:/proc/1# cat /proc/1/status | grep RssAnon

> RssAnon: 3220 kB

You can also pare this down a bit.

Similarly for journald:

> root@kmipsrv:/proc/1# cat /proc/231/status | grep RssAnon

> RssAnon: 640 kB

So in reality, systemd works just fine for the vast majority of embedded platforms. If you can spare 3Mb of RAM, then you can run a full-blown dependency management system with reliable recovery, logging integration, and event-driven device management. Soon with support for verified boot with hardware root-of-trust.

This should be the top comment.

systemd itself is also large in storage size(8MB?), which is why openwrt has its own procd replacements, and yocto is not default to systemd either due to this. additionally, systemd does not work well with musl which is very popular in embedded linux.

systemd binary is around 1.5Mb, with the shared libsystemd it's around 2.5Mb. Both can be reduced by not including compressors (the infamous libz), audit libraries, SELinux, etc.

The minimum binaries size that can be obtained by trivial removals is around 2Mb.

OpenWRT should just stop being stubborn and integrate it. They dropped support for 4Mb devices loooooong ago, and recently bumped the minimum specs to 16Mb.

> systemd does not work well with musl which is very popular in embedded linux.

Systemd can work with musl with a small compatibility shim, I did it about 9 years ago. Somebody did that again: https://catfox.life/2024/09/05/porting-systemd-to-musl-libc-...

> OpenWRT should just stop being stubborn and integrate it.

Why? If it fails to provide benefits commensurate to the effort required to integrate it, then there's no reason to do the work.

OpenWRT is for routers and access points... devices which usually have a fixed [hardware] configuration, and very little need for the absurdly-complex (and in my professional experience, often subtly buggy) event-driven systems that systemd provides. "Every system everywhere gets to use the same unit files" isn't a significant benefit in practice... EVERYTHING that needs nontrivial pre-/post- start/stop verification and/or configuration has systemd run a shell script (or other such helper) anyway... so you only get to just write a unit file and call it a day in the most trivial of cases.

OpenWRT is a mess. It started as a small distro for routers, but it stopped being that long ago.

Now it can host telephony services, full-blown Docker orchestration systems, etc. So you get all the pleasures of complex systems running on antiquated infrastructure.

> so you only get to just write a unit file and call it a day in the most trivial of cases.

What "start/stop verification"? Give examples, please. Systemd natively supports device dependencies, network status, other services, etc.

It's a very rare case when you _need_ shell in systemd.

> Now it can host telephony services, full-blown Docker orchestration systems, etc.

What features that systemd provides that aren't provided by openrc, runit, monit, openrc's init system, etc, etc, etc that are needed by software like Asterisk, Docker/runc, Kubernetes, and similar?

If what OpenWRT is running works fine to bring that up, then systemd doesn't have any relevant compelling features that aren't provided by another init system. Switching away from something just because it's "antiquated" is a great resume-building tactic, but it's a pretty poor strategy when your objective is to keep things that have been working working with a minimum of effort.

> What "start/stop verification"? Give examples, please.

One example (out of many) are pre-start, pre-stop, and pre-restart configuration file correctness verification so that a typo or thinko doesn't bring a service down.

A concrete example of this is Gentoo's running of 'named-checkconf' in BIND 9's service file [0]. Gentoo's BIND 9 systemd unit file [1] tries, but probably does not do the same thing... I don't know if 'ExecStartPre' runs before systemd brings down a service on a 'restart' action, but I bet you that it definitely does not run on service stop.

I just checked and Ubuntu 22 does NOT do this sort of checking for its bind9 package, which is pretty wild.

> It's a very rare case when you _need_ shell [or other such helpers] in systemd.

Then you're either running "seatbelts off" (maybe without even knowing that you are), or you rarely deal with anything nontrivial. This need comes up a LOT in my day-to-day.

[0] <https://gitweb.gentoo.org/repo/gentoo.git/tree/net-dns/bind/...> [2]

[1] <https://gitweb.gentoo.org/repo/gentoo.git/tree/net-dns/bind/...>

[2] You might look at that file and be like "wow, TL;WR", but do try to notice how much of that file is dedicated to creating useful, human-readable status messages and error messages with good corrective instructions. If you care at all about your sysadmins, that complexity has to go somewhere, and that somewhere cannot be inside a systemd unit file... systemd just doesn't provide the features required to do that (and it would be kind of insane for the devs to even try).

> What features that systemd provides that aren't provided by openrc, runit, monit, openrc's init system, etc, etc, etc that are needed by software like Asterisk, Docker/runc, Kubernetes, and similar?

Dependencies, watchdogs, start only when the given network interfaces are available, declarative configuration so that can be partially customized, etc.

> If what OpenWRT is running works fine

Unless you try to update it. And end up with merge conflicts in the /etc directory. And then it stops booting, and you have to get in via the serial port.

No, OpenWRT is most definitely not fine. It's stuck in the past, and doesn't support automated rollback, configuration management, etc.

> A concrete example of this is Gentoo's running of 'named-checkconf' in BIND 9's service file [0]

Wow, what a... But named config has always been a disaster zone.

With systemd _none_ of that crap is needed. It can use its own namespacing and cgroup support to ensure that BIND9 can't read outside of its allowlisted tree.

If you want to run /usr/bin/named-checkconf, then it's just an ExecStartPre directive in the unit file. And unlike SysV that just fails to run and forces you to dig into the logs, systemd provides you a nice status saying that the pre-exec had failed.

> You might look at that file and be like "wow, TL;WR", but do try to notice how much of that file is dedicated to creating useful, human-readable status messages and error messages with good corrective instructions.

Yeah, and hanging indefinitely, forcing me to do a trip to the datacenter at night (true story): https://lwn.net/Articles/619565/

Ah, you're that CyberAx. Right.

So, briefly:

1) That's a list of systemd features. Some of which are nice to have, but none of which are needed for the software I mentioned.

2) On-upgrade merge conflicts for entirely-unmodified-from-their-stock-settings system configuration files are why I gave up on Ubuntu. Systemd doesn't solve anything here.

3) Systemd doesn't handle BIND configuration... or any non-systemd application configuration. And most nontrivial init systems (with systemd as one notable exception) have facilities for printing info, warn, and error messages in service startup files rather than demanding that you pick through the application logs to find out why startup failed.

This article explains the security features of systemd that make chroot superfluous: https://www.redhat.com/en/blog/mastering-systemd

And here's an example of how a network service can be confined in its own network namespace: https://www.cloudnull.io/2019/04/running-services-in-network... - with zero shell needed, btw.

OpenWRT is VERY sensitive to firmware size. I developed for a device with 16 MB flash that had both a normal (~11 MB) and recovery firmware (~3MB) on flash. The rest of flash was used for persistent storage, bootloaders, and calibration data. This is very common for older routers and APs which is the whole reason that OpenWRT exists.

Well, OpenWRT doesn't even officially support devices with less than 16Mb of flash anymore: https://openwrt.org/toh/views/toh_available_16128

> The rest of flash was used for persistent storage, bootloaders, and calibration data.

So you couldn't spare around 1Mb of compressed flash or 2Mb of uncompressed flash for systemd?

> This is very common for older routers and APs which is the whole reason that OpenWRT exists.

That hasn't been the case in quite some while.

And it won't matter anymore, there is no price difference between 32Mb and 64Mb NOR or NAND flash. It's literally the same cost.

> So you couldn't spare around 1Mb of compressed flash or 2Mb of uncompressed flash for systemd?

Absolutely not.

> That hasn't been the case in quite some while

These routers and APs still exist and are in use. Yes, they are 10+ years old, but they're still out there. One of the main goals of OpenWRT is to support legacy networking hardware that vendors abandoned.

> Absolutely not.

So basically, you're dealing with devices that use obsolete hardware and are designed to be extra-brittle (no safety margin), with a built-in planned obsolescence (what if the new update requires just a bit more space for "calibration data"?). Got it.

sysvinit indeed fits perfectly. Make the system extra unreliable, to provide a stimulus to buy a newer version.

> One of the main goals of OpenWRT is to support legacy networking hardware that vendors abandoned.

And OpenWRT abandoned this goal. OpenWRT also can do nothing about firmware vulnerabilities in the old WiFi chipsets.

Moreover, 10 years ago, systems with 32Mb were already common.

OpenWRT doesn't use sysv init, it has something called procd (~120KB compressed). Its very reliable, and they have a large install base on mission critical devices.

> And OpenWRT abandoned this goal.

They have not. Individual devices get too old to be supported, but there are plenty of 16 MB devices they still support. Its true that basically every device now has 32MB or more of flash, but there are still plenty out there that don't.

> with a built-in planned obsolescence (what if the new update requires just a bit more space for "calibration data"?). Got it.

The calibration data is from the factory to tune the wifi radios, it doesn't change since its specific to that individual device. I really hate to reply like this on HN, but you don't seem to know what you're talking about. The kinds of constraints you deal with in embedded networking are different than that in general computing. There's a reason systemd wasn't used, and there's a reason OpenWRT devs went through the trouble of developing their own init system. Just because you can't fathom those reasons doesn't mean they don't exist.

> OpenWRT doesn't use sysv init, it has something called procd (~120KB compressed). Its very reliable, and they have a large install base on mission critical devices.

procd is only slightly better than sysvinit, it has the same ideas, and only adds simple and incorrectly implemented event triggers. And yes, I spent days debugging issues that it caused.

E.g. a USB drive for the logs becomes slow and changes the startup order, so now openvpn starts before the interface acquires the IPv6 address. Boom, broken IPv6 routing.

> They have not. Individual devices get too old to be supported, but there are plenty of 16 MB devices they still support. Its true that basically every device now has 32MB or more of flash, but there are still plenty out there that don't.

And systemd can live just fine at 16MB. I ran it on RouterBoards with 32MB (the smallest ones available) in 2014. It has grown a bit since then, but it can be slimmed down (remove compressors, auditing, SELinux/AppArmor, etc.).

> The calibration data is from the factory to tune the wifi radios, it doesn't change since its specific to that individual device. I really hate to reply like this on HN, but you don't seem to know what you're talking about.

The entire firmware for WiFi chips that were available during 16MB era is around 500KB-1MB. So it must be some other mysterious "calibration" data.

So you're thinking about more and more corner cases, that require the stars to be just right for the size of systemd to matter.

> So you're thinking about more and more corner cases, that require the stars to be just right for the size of systemd to matter.

These weren't corner cases, they were everyday reality. We had to intentionally slim down the firmware size to get it to fit comfortably since we added quite a bit of application code. Like I said before, we got a recovery firmware down to ~3MB, and the normal firmware (without our application) was ~8MB. Every megabyte counts at that point, so having a 1MB init system would be a non-starter. We were buying commodity hardware, so we did lots of shopping and there were still plenty of 16MB APs on the market 10 years ago.

You're right its not a problem if you buy new hardware, but OpenWRT is used to support older devices with limited hardware specs. Its one of the reasons people install it, to give new life to old hardware (rather than being forced to buy a new device).

By the way, I use and like systemd on full size computers. I actually know of a product that runs a full rpm based distro (with systemd) on an embedded device and it works great. But systemd isn't one size fits all.

I can only say that the obvious answer here is to throw effort into supporting & improving Gentoo.

I've had very nice experiences using systemd for developing embedded systems. Laying out all the unit files and just slapping the files in an overlay makes it super simple.

The RAM usage isn't being calculated correctly. You can easily run systemd on very limited systems.

Try s6 or runit, and mdev from busybox. Mdev is not a "full featured" udev replacement, but that's sort of the point. It does just enough to get the job done and is extensible with scripts.

If you want to do the size of embedded where this matters, you also probably want a smaller libc like musl, and then buildroot and busybox are your friends.

I run systemd-based Debian just fine on 512 MB RAM BeagleBoneBlack SBCs with 4GB emmcs. Now emacs, that's a problem due to the large install size.

> Now emacs, that's a problem due to the large install size.

I was running Emacs on a 486 with something like 16 MB of RAM. Don't remember the HDD size.

The Emacs executable I built from source is about 28 MB. Honestly I don't know what else is needed besides that executable that'd be really big.

Is it really that problematic that it wouldn't work with 512 MB of RAM and 4 GB of storage?

I think the problem is the Debian packaging, which has hundreds of MB of elisp or something.

https://packages.debian.org/sid/emacs-common

(yes, it does fit, but it takes up way more space than I'd like...)

If it can run X11 on a directly connected display, it's probably not embedded spaces ;). Don't take this comment too seriously, however raspberry pi is definitely at the "more desktop-like" end of the embedding pool.

As for systemd, it's an interesting set of tools, and one can if one wishes pick which ones are active in a system, if that system is complex enough to be running Linux (or other compatible OS) to begin with.

The article fails to mention another aspect of systemd that interferes with emmbedded dvelopment: it's utility applications (primarily systemctl, but all the others as well) are intended to be run on the booted system for which the control is being performed.

When configuring boot media, like an SD card, for an embedded ssystem that is not the running system where the configuration is occurring, this is an impediment. There is systemd-firstboot, etc, but this is not as convenient as just being able to set config options on the mounted (non-booted) embeddded media.

I've never liked it, I still don't like it, and I think the number of people in this camp is understated by the article.

However, I am still running it. As other posts have mentioned, switching distributions is a major hasstle, especially if you've built tools using the distro's architecture. For me, this is archlinux.

Although, I am in the process of testing and migrating to void linux. Which is systemd free, and hosts ARM and x86 binary package repositories.

Can you be specific about what cannot be configured on mounted embedded media?

systemd follows the drop-in config file model where configuration snippets are placed into directories (like .d/ directories in Debian). It should not be necessary to run utilities from the target system in order to configure it on mounted media.

Drop unit files for services, sockets, filesystem mounts, timers, etc onto the mounted media and they will be detected when the target system boots.

> When configuring boot media, like an SD card, for an embedded ssystem that is not the running system where the configuration is occurring, this is an impediment. There is systemd-firstboot, etc, but this is not as convenient as just being able to set config options on the mounted (non-booted) embeddded media.

systemctl --root /path/to/sd/card/ <...>

Most of our tools that adjust files work really well offline. systemd-tmpfiles, systemd-sysusers, systemctl and so on all support --root= and --image=.

I really don't know what the problem is suppoaed to be.

Lennart

As for a solution to this, wouldn't it be possible to use systemd to build a targeted init script? Use the expertise embedded in systemd to bring up the system you're targeting, recording how it does that and then removing systemd, leaving the artifacts?

Why keep this huge monolithic system running in the background after it has served its purpose?

Also, with ai, would it be possible to redraw the lines within systemd between the separate domains/concerns, then redistribute the functions back to where they conceptually belong? Then you can refine those changes to make them more standardized and universal instead of interdependent and centralized.

We (https://github.com/husqvarnagroup/smart-garden-gateway-publi...) are using systemd on devices with 128 MB of RAM and are happy with it. It solves so many problems one usually has to fiddle with, totally worth a few MBs of RAM.

You could checkout void and alpine.

I'm quite annoyed by the bland statement in the introduction of this article: people make about it don’t stand up to much scrutiny, even the technical ones

Basically the author says that everyone else has stupid rants that doesn't stand but himself has valid claims so that in the end we have a proof that there are a few valid technical rants!

That being said, the main problem with systemd from the beginning is that it is a cancer. It was explicitly designed to prevent as much as possible the modularity and intercompatibility. Kind of opposite to the POSIX way even I think.

Despite what is pretended, doing the things would probably not have impacted features, like boot speed but it was key for them to shove through our throat the usage of systemd in a "all or nothing" way.

And udev or logind are good example of that.

I think it's fair to say the majority of people don't really care about any of those things as much as having a system that just works -- which was part of their whole argument regarding why package maintainers love it so much and end users really don't tend to care.

I would assume people using Linux more or less care to an extend

And systemd replace all POSIX init system because they were all bad or systemd do the same thing better or faster, your point is a good argument against following POSIX religiously

It is unfortunate that after all this time the basic reasons why systemd was introduced and has been so successful are pervasively misunderstood.

> The majority of Linux users are uninterested in the pros and cons of systemd. A small number are violently opposed to it, and a small number are violently opposed to those who are opposed to it.

Starting out by framing systemd in terms of opposition deliberately steers attention away from the issues that drive its adoption. The tone is moderate, but this is not good faith argumentation.

> I think there’s little argument that the main target for systemd is a general-purpose computer, with a modern, integrated graphical desktop

That is one of the targets. The main target is servers that run the web of services typically required to back up modern applications. What if an application depends on a resource server which itself depends on a remote filesystem being mounted and an event service running. Theoretically that can work with System V init, but how tricky is that to put in place and make robust? Administrators for systems vending complex interdependent services are the main target and always have been.

> Unfortunately, what makes systemd good for general-purpose desktop applications

Get it wrong and then run with it. Failing to understand what is driving adoption of systemd leads to arguments about it that don't really go anywhere.

> The more fundamental problem is that the people who most like systemd are distribution managers. Sure, there are administrators who like it, and defend it vigorously; but most end users and administrators don’t really care.

Not distribution managers, but system administrators. They are the ones with the complex needs that systemd serves. As such they advocate for it and because of the lack of working alternatives distribution managers switch to it in order to please the largest number of clients. Thinking of systemd in terms of desktop systems and distribution managers means misunderstanding why it is there in the first place.

I used it heavily at work. Then moved to a shop that does exclusively containers and realized how incredibly immature engineers’ understanding of the problem space is.

Container startup ordering, startup dependency management, socket handling and all things timing and network related are frankly a complete mess in modern infrastructure. Even, surprisingly in well maintained projects like k8s. I find myself shaking my head at problems solved by systemd a decade ago far too often.

TBH, my biggest gripe is that systemd doesn't play nice with musl or ulibc, I would like to not have to ship a cron, a supervisor, a xinetd equivelant, and a binary just to react to inotify/dnotify events. Though openrc how has a supervisor.

Big systemd on embedded devices is actually A GOOD THING! Imagine what Gentoo, Buildroot/OpenWrt or even Busybox alone could do with so much memory after removing all that bloated OS!

Excellent plan, go to the other inits, bug bounties are what I do over christmas and init systems and hand rolled init scripts are a great source of security flaws.

Taking notes on which distros dont run systemd, for when I want to make some money over christmas.

No need to stop at non-systemd inits!

https://security.gentoo.org/glsa

https://security.archlinux.org/issues/all

^ (ctrl+f systemd)

https://ubuntu.com/security/cves?package=systemd&limit=100

https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=Systemd

For me as a user, systemd is far better than old init systems. The logging alone is worth it to switch to it. Add more ram to the embedded board and welcome to the future.

> The logging alone is worth it to switch to it.

Weird. In my professional life, I've found systemd's logging to be woefully lacking when compared to systems that just run every daemon through syslog (or -even more retro- "each daemon writes stdout & stderr to disk, which then gets sent to your log sink of choice").

What I usually find is that a ton of information never, ever makes it into journalctl, so I always need to go to the actual log files on disk. [0]

On top of that, journalctl just fucking giving up when logs have been rotated is totally bogus. (If this has been fixed, then the fix hasn't made its way to the systems I work with.)

ALSO, have they fixed the issue where minor data corruption just totally fucks your log file to death... rather than what happens with regular log files and you get an unreadable section that's bracketed by good data?

[0] Yes, you could reasonably say "Well, those services are not correctly configured to log to journalctl!". But my point here is that I see this happen so often that journalctl is like my tenth stop for log messages, rather than my first.

I think author is missing on the Devuan-maintained udev fork.

Not really embedded, but one of the easier classes of wins I've had over the years is moving prod servers to a non-systemd Gentoo. Random (rare) 200ms latency spikes and whatnot just disappear, and the system jitter drops down to the tens of microseconds range, significantly improving the qualitative performance of basically every piece of software I care about.

Did you ever dig in to see why that changed? I'm surprised that in a steady-state (not booting or restarting services) systemd/not-systemd would have an impact.

A little bit. The tldr; is that there are a lot of periodic activities including log rotation, automatic updates, time syncing, waking up to check if such things were currently needed, .... Those are generally nice (aside from automatic updates, which are sometimes good but I think usually more harmful than whatever they're fixing), but systemd doesn't know anything about what my userspace demands on the hardware are, and it's generally happy to hog system resources if it doesn't do so for long. For anything quasi-realtime, I can certainly hunt down and squash everything systemd is running at the wrong time, but in practice it's much easier to install something else, ensure no processes are running other than the ones I care about, and move those concerns to userspace.

> For anything quasi-realtime, I can certainly hunt down and squash everything systemd is running at the wrong time, but in practice it's much easier to install something else, ensure no processes are running other than the ones I care about, and move those concerns to userspace.

Should these issues be fixed by CPU/IO schedulers? Are there any systemd realtime tasks involved?

Now that the realtime patch is in mainline Linux, that approach might make more sense going forward. I don't think systemd has any meaningful realtime tasks by default. My current job has a different set of performance constraints (only need low latency something like 99% of the time, so systemd and noisy cloud neighbors and whatnot are fine), so I probably won't be looking at that part of the stack in any depth any time soon.

So pretty much, Gentoo+systemd and Gentoo-without-systemd aren't the same OS? One does auto updates, auto time syncing, auto logrotating etc, and apparently the other doesn't?

Or is the fact that systemd is the one doing it that much worse? Does systemd-timesyncd take more time than chronyd launched by some other init? I'd be surprised.

I like systemd sometimes. Other times it confuses me. Particularly the way cgroups work. The way splicing works. Why oracle mem usage is still uncontrollable when I set user splice restrictions. Idk, I’m still learning it. I don’t like the terminology it uses sometimes too like “snapshot.” When I first read about systemd snapshot I was like oh cool you can snapshot the system. No. Also some esoteric things it supposedly supports don’t work sometimes. I find the idea of systemd targets and fstab having anything to do with each other anathema. Idk.

Meh, we are not in 2021 or 2018 anymore.

Eudev and elogind ARE mantained and used in voidlinux and gentoo, and anyways being that embedded it doesn't seem clear to me where would the "hard dependency of a very essential software on some systemd-thingd" happen.

https://www.gentoo.org/support/news-items/2021-08-24-eudev-r...

https://github.com/eudev-project/eudev

https://github.com/elogind/elogind

"I'm using a thing in a way it's not meant to be used and it sucks"

Is this even a serious post.

It is a serious post; you've just summarised it appallingly.

This is anti-systemd FUD. Implying that applications which work without systemd are going to stop doing so. Suggesting that Linux distributions adopted systemd because it's good for distributors and not because users actually want it. We are "forced" to use systemd's udev implementation. Please.

All of those things are trivially true. What are you disagreeing with?

There are extremely few use cases where your app may want to integrate with systemd. Out of those, you get again a small percentage that can't be trivially made optional. (99.9% is just optional notification and socket passing) Outside of large projects which are interested in full system integration like gnome, nobody breaks apps to not work without systemd.

Except none of these are true, they're just things people have made up to conceal the fact they oppose systemd for ideological reasons. There's little to no technical arguments against systemd - it's fast, much more robust than what came before it, and services 99% of usecases.

Users DID want systemd. Ask any sys admin, most much prefer systemd.

Also systemd doesn't "force" anything. Most distributions don't even ship all systemd projects, just a couple.

systemd-init IS small. The idea that systemd is a big ole monolith is just not true, and one glance at the git repo reveals that.

oh definitely not, and it's a matter of record. Red Hat and Poettering wanted systemd in order to control more of the platform; this is why they absorbed in udev (which had been a separate repo), came out saying that there are certain parts of the system that, despite all this being open source, were forbidden to modify or replicate, and stacked the debian voting. Systemd at the time sucked, incredibly, even worse than it does today, and routinely bricked or crashed systems. The adoption was forced and 100% political.

systemd is, of course,today a continuing shambling feature factory behemoth in which hundreds of product managers try to shoehorn in more mandatory features in order to cement grip on the platform. That's why you get this ridiculous dns nonsense, this ridiculous container nonsense, this ridiculous cron nonsense, hostnamed (!), the list is endless.

The technical argument is that it's a giant tasteless bag of ever increasing poorly implemented scope. The rest of Unix, by comparison, is generally not that way. If you pick up a BSD or even an alpine, you'll find that you don't need a bunch of badly written, hacked together garbage in order to get your work done. The system can be entirely readable and repeatable without cramming a bunch of cruft and poor decisions into pid 1.

The idea that systemd doesn't 'force' anything is, of course, hilarious. Systemd is designed to try to be the ultimate mandatory dependency for essentially everything on the system. That's the way Red Hat wanted it to be, and that's the way it is.

Every time someone mentions systemd, some random apologist dutifully trots out the idea that systemd-init is small and therefore systemd is not a monolith, checkmate. Of course, journald, libudev, localed, logind, hostnamed, homed, networkd, resolevd, systemd-boot, systemd-bsod, systemd-nspawn, timedated, timesyncd, tmpfiles, udevd, and all of the other array of dumbass bolted-on second-system-effect-driven product-managed nonsense somehow don't get mentioned.

> Red Hat and Poettering wanted systemd in order to control more of the platform

Speculative and ideological. From a technical perspective, I don't care about this.

> despite all this being open source, were forbidden to modify or replicate

This isn't true - you're allowed to modify or replicate any parts of systemd and always have been.

> systemd is, of course,today a continuing shambling feature factory behemoth in which hundreds of product managers try to shoehorn in more mandatory features in order to cement grip on the platform

I'm not sure you understand what systemd is.

systemd is not a piece of software, systemd is an endeavor. Many, many projects are under "systemd", as in they have the name, but they are all individual binaries. Practically 0 distros include all systemd binaries, because they're optional. Many projects already existed before systemd, like gummi boot, and were just given the name.

> The rest of Unix, by comparison, is generally not that way

This is untrue, as systemd follows unix principles. systemd-init does one thing and one thing only - it's only the init system. systemd-networkd is just the network daemon. systemd-journald is just the logger. And on and on. Despite what you may think, they ARE optional, and distros mix and match constantly.

> without cramming a bunch of cruft and poor decisions into pid 1

Again, out of the dozens and dozens of binaries under the systemd umbrella, only one (1) runs under PID 1. This simply isn't true.

> Systemd is designed to try to be the ultimate mandatory dependency for essentially everything on the system. That's the way Red Hat wanted it to be, and that's the way it is

Speculation, ideological, and not evidence backed.

> Of course, journald, libudev, localed, logind, hostnamed, homed, networkd, resolevd, systemd-boot, systemd-bsod, systemd-nspawn, timedated, timesyncd, tmpfiles, udevd, and all of the other array of dumbass bolted-on second-system-effect-driven product-managed nonsense somehow don't get mentioned

Yeah... because those are separate programs. Maybe you just don't understand how computers work, but these programs are unrelated. They talk over IPC, they're not even linked together on any systems. You can take or leave any of them, and many distros do. Even Debian, the supposed systemd cocksucker, doesn't include half of these.

"these programs are unrelated" ok, enough feeding the systemd troll. Good luck with all of whatever that is.

There is another aspect to the memory footprint: Does systemd fit into the L1 cache? Does sysvinit?

Both x86 and Linux (page cache) rely on caches for performance.

I'm not sure that fitting a init system in L1 cache should be a priority when it would be swapped out almost immediately on doing further work, aka the purpose of the device.

Systemd, Wayland, DBus, and other modern Linux "solutions" are all problematic for the same reason: they are complex monoliths that are extremely difficult to replace, in part or in whole. Systems often won't work without them, because the entire ecosystem has grown dependent on them. You literally can't run a modern desktop without a bunch of "shims" whose sole purpose is to fake being systemd.

The best designs, sadly, don't win out. The ones that win out are the ones that serve the interests of the largest players. And the largest players are usually RedHat, Google, and other vendors, who just want to get "their" needs met and move on. They couldn't give a fig about compatibility with some other random tiny project, or an embedded device. And those people working on tiny projects aren't in the room when "standards" are decided for the entire community.

There are some hold-outs. Slackware went a long time without it, but next release will unfortunately use systemd. I've run Alpine Linux (systemd-free, so far) as my laptop OS for years. I'll probably switch to another distro that's more user friendly, is easier to contribute to. But it's embarrassing and annoying that we have to pretend to be systemd-compatible just to run a desktop. We should call it "GNU/Systemd/Linux" since it's now one big monolith.

Where have you heard that Slackware will be using systemd next release? I have been using slackware for years, and this is news to me.

There's nothing named "systemd" in the FILE_LIST [1] for Slack64-current (which is what eventually becomes "the next release" at the appropriate time.

So it would seem the GP has been taken in by some rumor somewhere.

[1] https://ftp.ussg.indiana.edu/linux/slackware/slackware64-cur...

I thought I saw in the last release a note that the next one would come with systemd. It could be I hallucinated it. Maybe I'm becoming a real LLM!

RELEASE_NOTES [1] contains no mention of systemd (or at least does not contain that string).

README.TXT [2] also does not contain the string "systemd"

ANNOUNCE.15.0 [3] also does not contain the string "systemd"

If such a 'note' were included in the last release (15.0) it should have been in one of the above three files. So there is nothing in the release that appears to indicate a shift to systemd.

[1] https://ftp.ussg.indiana.edu/linux/slackware/slackware64-15....

[2] https://ftp.ussg.indiana.edu/linux/slackware/slackware64-15....

[3] https://ftp.ussg.indiana.edu/linux/slackware/slackware64-15....

Slackware will never implement systemd. Father would not allow it.

Wayland seems like an odd entry in that list. It's mostly a strugle for people because it's not the monolith X was and now we have dozens or hundreds of different applications from different vendors to cover a subset of the functionality that was built in. In that sense X was the complex monolith that is extremely difficult to replace.

Also unlike the complaints about systemd in the article, Wayland is well suited to embedded. Automotive Grade Linux was an early adopter. It's desktop usage that has taken longer.

That is an odd one. While X is definitely an ugly monolith, Wayland is microservices. I think that speaks volumes...

Reply on news.ycombinator.com