Systemd 250 Released
lwn.net
lwn.net
Come up with a commercial solution to a simple problem. Ex: file sharing or password management. Make it slow, insecure, privacy violating, lose people's data. Make millions, become a hero.
This is not about the problems systemd solves, but how they solve them and how they act while solving them.
Linux itself was mostly unusable when it first started. And a kernel is even more important.
There's a distinction between labeling it as a hobby project and something ready for production.
Linux started as a hobby OS, and being buggy is OK in that regard.
systemd is way more buggy than it should when it's deemed production ready. This is my point.
I'm using Linux for a long time, and I know the meanings of unstable, buggy and experimental; as well as production ready.
I don't hate or I'm not against systemd, I'm against the attitude behind it.
Systemd is exceptionally voluminous, fast moving, feature-laden (which is part of why it is so exciting to so many people), and buggy.
My only real big pet peeve with Systemd is the documentation is somewhat lacking..., like it's all there in the manual, but its laid out poorly and makes it hard to understand. The interaction between unit files and dependencies is also quite convoluted when you start digging into how things are ordered. There's a ton of unit file options that enable weird conditional behavior. The moment you need to start comprehending the difference between Wants vs. Requires and not hooking into running when multi-user.target is reached (the most common situation) you are in for a world of trial-by-error testing.
Systemd is great for starting simple background services on simple desktop and server environments. It becomes a real pain in the ass once you get beyond that.
And the worst is that the tooling to test is very limited ( non-existent). I had a problem with conflicting/circular ndependencies ( my service had to be launched before networking but after dbus, which is just impossible because dbus needs networking itself), which worked 9 times out of 10. It was nearly impossible to debug, i had to use the graph tool that shows you visually how much time each service took to start, and completely by chance i noticed that dbus started after networking, which led me to look into it.
Thing is, it also forcibly changed a bunch of things by introducing defaults that were not at all what people expected. They might have been better overall, but it still caused some nasty surprises.
Let me try and explain a bit more concretely -
An easy example (not a hypothetical one, btw): if you have a physical server with multiple hard drives and one of them doesn't start, old school unix would boot everything it -could- boot anyway. SystemD changed that behaviour to "if a filesystem doesn't come up, and you don't set an option to tell systemd it's ok if that doesn't happen, don't finish booting".
Now, imagine you're somebody who has a physical server somewhere that has a big cache disk that you use a cheap drive for because if it fails, well, you've lost some cache space. Now imagine your server used to run a pre-systemd setup, and when you upgraded it to a systemd using version of the same distro you left all your settings as-is because everything seemed to be working fine. Now ... imagine that drive dies during a reboot ... and so the server doesn't finish booting, and so you can't even ssh in to it to figure out what's going on, and the only way to figure it out is to get physically in front of the console.
If that server is four hours' drive away, you might not be very impressed.
I still, overall, think systemd is pretty nice. But while some of the people annoyed with it are annoyed with it on a purely aesthetic/principle sort of basis, there are definitely some decisions the authors made that caused -very- surprising results for people who had been running servers for years already, and some of the dislike being thrown around is very much understandable.
I still have to find a way to let it digest sshfs filesystems that may hang at boot without stopping the whole boot process.
(I don't particularly blame him for the specific non-booting bug, but I do blame him for the arrogance. Everyone makes mistakes, but most people learn from them and develop a bit of humility. Pottering continues to break people's computers every couple of years)
- Gnome started to depend on systemd features. Now non-systemd systems become second class citizens.
- You used to be able to freely swap out window managers, taskbars, and so on, because everything was working with common standards. In the modern world, everything is integrated in your wayland compositor, and if you want to change the way window decorations look, you will also have to change everything else. Systemd is similar, it wants to take over so many responsibilities (task scheduling, DNS, session management), and if you want to swap out one part it is going to be difficult.
- Pulseaudio, NetworkManager, Polkit: Written in the same spirit, in part from the same people. Work great if everything works as expected, if they break it is a nightmare to figure out why. But code depends on them, so they become de facto standard on desktop linux.
My Linux workstation and servers are not something I want to tinker with, I want solidity and stability and a fragmented user space ecosystem is not the solution.
If I want the stabilty and flexibility of Linux (the kernel) and the GNU command line tools, and the flexibility to choose a UX paradigm that I prefer and visuals that I like, I go to desktop Linux. In that sense, choice is the USP of linux (linux as in: desktop distribution). If I loose the flexibilty, I go back to Windows or Mac. Opinionated ("not fragmented") systems like Gnome only work if you're fully on board with their choices. If not, then there is really no benefit for me over just sticking with windows.
Now, on a server the situation is of course different, and stability is paramount. I like for example that services are declarative and not imperative in systemd. But on a desktop, I don't really care.
Again, no. You call it choice, I call it chaos, it was anarchy, and that's the default state for any loosely-coupled systems. There is no manifesto, no underlying philosophy that decided that yes, Linux is about choice. Don't conflate lack of organisation with choice.
Linux has always been about creating a libre UNIX-like kernel for x86 processors. And converging toward the same set of libre techonologies isn't against the Linux spirit.
I still don't understand why people keep harping on about choice. I've used Linux full time since 2001, and the anarchy has just made a libre desktop an utopia because everyone wants every software to support every single option, with no understanding that the number of bugs scales exponentially with the number of choices.
Because they don't want the choices other people made for them.
So everyone who uses or develops alternative init systems like runit and openrc and alternative DEs like KDE and package managers like apk and pacman should just stop and work on and use systemd, GNOME, and flatpak instead because that is the One True Way of using Linux on desktop?
Quite a power trip attitude I'd say. I can't imagine why I would use Linux if choices I don't agree with were shoved down my throat.
Ok I will give that a go.
That reminds me when will Wayland be able to support disparate displays say one big display on 4k with 30 bit colour and two others on QHD and 24 bit colour?
I'm too tired to write them all over again, but here's a rundown, from me.
(1) Xorg replacing XFree86 and making video "just work"
(2) Pulse Audio replacing OSS and making audio "just work"
(3) systemd replacing the dog's breakfast of scripting madness each distro built from scratch instead of collaborating
It amazes me that the same person was responsible for leading 2 of the 3 changes.
I for one welcome an increasingly unified operating system core, instead of the random competing and overlapping components we used to have. Sure they worked 97% of the time, but the other 3% when they didn't could be a royal PITA. Especially when two waring components wasted time pointing blame at each other instead of focusing on solving end user issues.
Which goes completely against what Linux stands for - collaboration. The "scripting madness" as you call it was actually completely transparent - if the scripting was bad, you were free to fix it. But now you have to issue a bug report when systemd does something stupid and it's not even your fault to Lennart and his gang only to get it dismissed with "no a bug, won't fix".
Also, Pulse Audio only "just works" because the distribution maintainers do that for you. It's still a stinking pile of "don't no how, but it just works".
Keep telling yourself that Lennart does a great job. Stockholm syndrom much?
Yes, but not how you'd imagine. I spent years as a hard core Linux fanboy. Advocating for Linux. Declaring Linux the future when many thought Windows would obviously eliminate all competition. Volunteering for install fests and wrestling various distros onto unwilling hardware.
I also spent 9 years teaching and writing training materials for RHEL, SUSE, & Ubuntu. As a result I dug deeper into core system details than most. (How many remember SUSE's attempt to achieve faster boot times by introducing an optional alternative to normal init that dynamically generated a makefile to execute init scripts in parallel? That was one of many strange rabbit holes I got to explore.)
Thankfully, I've gotten over it. I still prefer to keep Linux near the center of my career, but I've been able to adapt as "the cloud" has rendered my old school sysadmin skills less relevant. Occasionally I call on them to rescue a pet, but today I find "herding cattle" and going "serverless" much more enjoyable.
And you?
Have a look at the contributor stats[0] for systemd and then repeat that with a straight face.
> Lennart and his gang
Also known as the Linux Plumbers[1]. You know, the guys that actually tie everything together and that enjoy the well-deserved trust of most distributions and users.
> dismissed with "no a bug, won't fix".
Usually with very good technical reasons.
> The "scripting madness" as you call it was actually completely transparent - if the scripting was bad, you were free to fix it.
You're still free to fix it; systemd is open-source.
Maybe pulseaudio and systemd only breaks 2% of the time rather than 3% of the time. But when they do break, you can't understand what's happened and you can't fix them, and even if they worked yesterday they won't necessarily work today. If that was tradeoff I was happy with I'd use windows or OSX.
No, I used to patch bash scripts, patch the deb package, deploy to prod, then have to do it all over again when the init script changed upstream in the distribution. Now I just have my configuration management change the options I need in the unit file and never touch it again, so now I sleep while unattended-upgrades works for me.
Also, I could not do anything with a bug inside firefox, the kernel and a litany of other complex programs (mind that they are complex due to essential complexity) so I don’t see system booting an exception to that.
If something critical launched by your init system isnt coming up, I'd way way rather be using systemd. There's know wwys to inspect, check out what ran, what didnt, see what dependencies are failing. Check the common logging system. In the dark old days it was a nightmare of different daemons, each with their own custom init scripts, & little common reporting. Every problem was unique & required unique diagnosis.
I dont get what the complaints are. Things work great now. There's great diagnostic tools. Most of them with json/machine readible output. Distros sometimes do break various daemons but systemd and pulseaudio are pretty much bulletproof, and if they worked yesterday, i absolutely have faoth they'll work today. I detest so much that this kind of casual easy convenient Fear Uncertainty g Doubt shade throwing persists, that we so casually malign while making no assertions that could be refuted. There's a spectre of doubt that sabotages the good, that seeds disbelief, and it's never justified, never backed up by anything concrete or real or debateable: it's all just this haunting image that it's not going to work. It's seditious.
Seems like software history revisionism to me.
PulseAudio let us have a common stack instead. There were bugs (PulseAudio was probably a bit buggy at first, and also hit audio driver bugs because of the new ways of using them). The transition was rough and PulseAudio is not perfect too but is it there for reasons and solves many issues.
Now it is being replaced by PipeWire, probably for the best (more efficient, can replace JACK too, making sound management much easier on Linux), but PipeWire leverages experience from PulseAudio.
I lived the transition to PulseAudio and it went well for me and probably a lot (the majority?) of people. Some things could probably have been handled better, especially the (too early?) transition. I think PulseAudio still is/was a good thing.
The reason so many people had issues with PA was not PA but the brokenness of the underlying system that PA exposed for the first time when trying to use all these drivers in a systematic way and build functionality on top of it. It took years of fixing all the drivers and PA - unfortunately - got a lot of undeserved flak.
PA is the reason that Linux has a halfway decent audio subsystem, being able to deal with all the modern devices; thank you, PA devs.
I find it hilarious that people are festive about PA being replaced by Pipewire - if PA hadn't paved the way, the Linux driver landscape would not be in such a good shape today and Pipewire would 'fail' in exactly the same way that PA 'failed.
That's not my point tho, my point is that PA didn't sweep in and save us, in fact it at first pushed us back A LOT. And claiming otherwise is a disservice to ourselves when we consider additional radical change. Such as Pipewire and and Wayland.
That argument I don't quite follow. Would you consider rephrasing it? My question is; how did PA set back Linux sound if it did expose all those flaws in the audio driver in the first place?
If you care about real world usage, PA set us back years when Ubuntu made it default.
But that's the crux, isn't it? Linux sound was not not-working because of PA. Linux sound was not working because it was fundamentally broken with drivers that had wildly different behaviors. Everyone was hacking around broken drivers all the time, I distinctly remember how I had to recompile the kernel back then to get some custom-fixes for my setup in because the driver was so kaput.
Only with the introduction of PA was there finally a way to test and evaluate the drivers in a systematic fashion.
So, in short, my point is this; any software that - in this broken state of Linux audio - tried to introduce a software that made available the features the drivers proclaimed would have run into exactly the same problem as PA. Because PA didn't introduce the problem, PA made it visible.
It's like blaming the doctor for diagnosing the illness.
So, while your experience was that sound suddenly wasn't working anymore because Ubuntu configured the default setup to use PA, for lots of people that sound never worked this was fine - because their drivers weren't broken.
Anyone who has ever plugged in a usb headset or connected a bluetooth headset or plugged an hdmi output into their laptop knows the value of pulseaudio. It's role as a modular, pluggable, controllable audio intermediary layer is invaluable: it has a great control panel that lets you say, ok app, start sending your output somewhere else now. Even if you only have one soundcard ever: it let you adjust per app volume! Utterly basic need. To propose that users ought have been fine without these basic capabilities is insanity. Alsa was insanity.
Alsa was a dead, impotent, lifeless husk. It let apps output audio to a device, if you knew to try various hardcoded strings like hw:1.0, hw:2.0, hw:2.1. But that was all alsa was good for. One couldnt open a control panel & change the volume of an app. One couldnt send an app to a new output (bluetooth, hdmi, usb audio). Trying to get a sound card to pick between various output modes, for optical output for example, was hell: i spent literally days getting optical audio going. Everything required careful scripting in a shitty awful .alsarc dsl to do anything good.
Pulseaudio made it all much easier.
I have no clue why anyone would valorize alsa. It was incredible finnicky, super hard to configure. It feels like certain sects just want to spread poison against linux & freedesktop. There have always been reactive, outraged camps. Eternally dismayed at progress, at better. I just can't take seriously the thought that alsa was an appropriate & adequate tool for end users, in any way though. It's ridiculous: nothing worked without careful careful delicate scripting & trial by error, and the system was utterly static, unable to respond to change. Alsa was not an end user system, unlike pulseaudio: it was an api for software. Pulseaudio changed the game. I am so tired lf hearing pretenses that alsa was ok or enough. It never was amything remotely adequate. The nonsense hate has to die. Progress was required. I saw it work flawlessly on many systems even in the earliest days. It wasn't & still usually isn't required. I don't see what anyone could be asking for. It just seems like a need to sow doubt.
While I get that some people are uncomfortable with breaking with traditions, I absolutely do not get all the hostility directed towards this craftsman programmer.
Systemd broke my ability to login, once. That was fun.
Now yes, it gives you more functionality, but it was a pain until it got there. Systemd was just Pulseaudio all over again (except now in init, so if it breaks your whole system can't boot).
Now, yes. initd is jurassic. Systemd got stable eventually and has gone beyond it. And this is more complicated than it seems. But still, it's the attitude that made things hard.
Not really
> For versions 3.28 and earlier, the GNOME desktop environment contained components that required systemd-logind[5] (among them, notably, its display manager, GDM). However, as of version 3.30, Gentoo's packaging of GNOME allows it to work once again with sysvinit + OpenRC as the init system
It became a question of "are you going to have GNOME 3.8 or later, or not"
I for one am very grateful for Lennart/Red Hat's vast contributions. Leading modern Linux.
Sure it isn’t perfect, but what software is?
And let me ask a question, if systemd is crap, why does every major distro use it? If it was truly crap then surely the distro builders would use the alternative?
Because it's tightly coupled to everything (particularly to the few profitable parts of linux - most open-source is maintained on a shoestring, so it's pretty cheap to take control over development of even vital low-level infrastructure). This is how systemd bypassed the bikeshedding problem of there being half a dozen replacement inits but no clear consensus, with the result that everything relied on a lowest common denominator. But it's toxic for the long term, because it undermines the whole point of using linux if you can't swap out pieces or maintain your own forks of individual components.
You can, though? And many distros do by default.
Don’t like systemd-boot? You can replace it with grub.
Don’t like systemd-networkd? You can replace it with networkctl. Want to use it but prefer to handle Wireguard tunnels with wg-quick? Fine!
Don’t like systemd-resolved? You can replace it with dnsmasq.
Don’t like systemd-cryptsetup? systemd won’t care if some other system takes care of your LUKS partitions.
Etc, etc.
It's ridiculous to imagine that no program is allowed to depend on SystemD.
There are multiple libc implementations. libc independence is important and valuable. You'll probably find a "hard dependency on glibc" list in the documentation of e.g. Alpine.
> Or "hard dependency on Bash"?
Ubuntu will have a list from when they switched to ash as /bin/sh. Again, multiple implementations and independence are important, and something the linux community generally cares about.
You could make another SystemD implementation if you really wanted. The point is that there are plenty of APIs - even ones with a single implementation - that nobody bats an eye about programs having a "hard dependency" on, but suddenly for SystemD it's apparently a big issue?
It's bullshit technical excuses to hide the real reasons people object to SystemD, which are more embarrassing.
No you can't, because they don't offer stable interfaces as a matter of deliberate policy.
Or maybe the maintainers are I don’t know do you have links to show this is the case?
Systemd is used by all major linux flavors because it’s the best solution for the problems it solves.
The distro maintainers are too smart to be stuck with systemd against their will.
And it’s linux …. make a better solution, if it’s truly better it’ll be used.
Systemd was minority pretty much till GNOME 3.8 introduced super-hard dependencies on it through an interface that was supposedly documented (Ubuntu sunk a bunch of money trying to keep alternative implementation running).
For many a distro, it became a question - "do we include GNOME, or do we drop it?" and that was impetus to move to systemd
> This is really unfortunate, but GNOME 3.8 does not require logind. I discussed the non-dependency of logind+systemd on #gentoo-desktop and why they thought different. Apparently GDM 3.8 assumes that an init system will also clean up any processes it started. This is what systemd does, but OpenRC didn’t support that.
Systemd was adopted because it was (and still is) simply superior in many aspects to previously available process managers, because application developers had it up to here to maintain n different init scripts for n different distributions and distribution packagers didn't want to have to write custom init scripts for each application anymore.
As an example, check out the discussion around the adoption of systemd in Debian[1].
[0] https://blogs.gnome.org/ovitters/2013/09/25/gnome-and-logind...
And no, not in "GDM doesn't clean up after itself", which was given as explanation for the infamous "systemd now defaults to destroying everything in a session on logout" change few years later. (Which GDM could require anyway earlier, but apparently somehow it wasn't enough)
There is a lot of good ideas that went into systemd though, and I'd agree that a lot of anger is in how badly they are implemented (the only reason I do not complain more about systemctl is because upstart managed to be even worse). In my copious free time, I've been tinkering at alternative, but even if I disagree at the format of systemd service files, it includes a parser for them.
If you by 'it' mean the fact that the distributions switched to systemd, I would be declined to discount the speculation that 'it was because people did report in trying to run GNOME 3.8 without logind [...]'. Do you honestly think that Fedora, Arch or any of the other distributions would not try to evaluate other strategies of getting GNOME 3.8 to run? Replacing the process manager and PID0 is not an endeavor chosen lightly - it is a massive undertaking that every distribution would have weight very carefully. "Oh, there are people having trouble running GNOME 3.8 without systemd present" seems not to be enough reason to go through all the pain of switching to systemd.
It's rather that systemd offered something that systemd offered something that the preceding mess of shell scripts and conventions didn't offer; consistency and reliability, as much as you might doubt that. Have a look at the Debian general resolution with regards to systemd and the provided arguments by the proponents[0].
> I've been tinkering at alternative
I would be very glad if your alternative implementation were even modestly successful. And I'd be thoroughly thankful if it managed to replace systemd.
Not because I think systemd is badly implemented (I've yet to see any actual evidence in that regard from any detractor) but because systemd needs competition and I would prefer to see some well established and stable protocols between the individual parts of systemd. That will only happen if said protocols get also provided by alternative implementations.
With regards to you re-using the systemd configuration format; that's probably a smart move. Systemd is at this point a well established fact and anyone trying to best it at its game will have to make is as easy to migrate as possible. Hurdles like a new file format might well be too much for an indifferent user.
As for supporting configuration format - it's an obvious move, though more as "import" rather than only source of truth (different internal data models), and the format... is not exactly good (mostly due to lack of sane defaults that start to hit you once you wander beyond functionality that can be summarised as "SysV initd plus dependencies")
Looks like we're still in disagreement about this. As far as I can tell, there was never a 'hard' dependency of GNOME 3.8 on logind; logind was an optional dependency[0].
> beyond functionality that can be summarised as "SysV initd plus dependencies"
I would attribute that to the lack of similar "service manager" functionality in other "boot managers". That comes down to the hesitancy of these "boot managers" to embrace the role of "service managers". There's tons of functionality that the systemd project (not necessarily the pid0 binary) introduced to make services run consistently and reliably. They paved the way in that regard and we're still waiting for contenders to catch up.
[0] https://mail.gnome.org/archives/desktop-devel-list/2013-Marc...
If Big Mac is a crappy food, why billions are served with it?
I like systemd.
If Windows is crap, why does every system builder ship the majority of their machines with it? Why bother with a different desktop than Windows? Just because it's popular doesn't mean it's good.
Building a distro is a lot of work already, and avoiding systemd takes a lot of effort these days. Plus, there seems to be a lot of people who like systemd, so if you want curmudgeons like me (well, probably not me, I'm going to FreeBSD instead; I understand there may be some sort of fancy rc replacement in the works there, but I presume it won't break the things I expect to work, and it won't subsume half of base) and people who like systemd, then you've got to do twice the work, and everybody will ask you why you're pushing against the tide.
Because they get incentivized by Microsoft to do so. This is a well document part of their often illegal business practices.
Won't find systemd in Gentoo, Slackware or Alpine. Also it is a little interesting, just a little, that whatever imperative systemd has to exist is entirely absent in every SysV and BSD, and yet variants of SysV and BSD and thier hybrids still exist, somehow, no worse for the wear.
systemd-networkd will now automatically configure routes to addresses specified in AllowedIPs=. This feature can be controlled via RouteTable= and RouteMetric= settings in [WireGuard] or [WireGuardPeer] sections.
I really hope apt will notify users about this when upgrading.
sd-boot gained the ability to change screen resolution during boot-time, by hitting the "r" key. This will cycle through available resolutions and save the last selection.
Overall I learned to stop worrying and love the systemd, especially systemd-networkd. The maintainers can be impressively responsive in acknowledging and fixing reported issues.
Similar story with other components.
Having to read the release notes before upgrading to a new major version is a price I’m willing to pay.
Sincerely, someone who lost a week or two trying to get anywhere with a bit more complex setup and ended up going back to /etc/interfaces because it had a "show what commands will be run" option
What would the reasons be?
It’s an absolute classic anti-pattern any industry veteran would recognise. Your new-grad engineer, top of their class and incredibly smart, spends two weeks rewriting a core system in Idris without telling anyone.
So accordingly, I was delighted to see the following:
> [files in] systemd-homed home areas are now internally owned by the "nobody" user and dynamically mapped to the UID used locally.
> This makes migrating home areas between different systems cheaper because recursively chown()ing file system trees is no longer necessary.
All that to save a chown -R if I screw my drive into a new PC (or rebuild the current PC around it)?
If I move it to a non systemD system, you’ve just added the need to chown -R all my files because the new OS says they are all owned by nobody.
I don’t even believe for one moment that a recursive chown of 100k files takes any appreciable amount of time. Certainly not when it’s a one off. (5s for 66k files on my crappy VPS just now.)
In any case, the nobody user is a blunt but useful instrument for running daemons as a user that does not own any files. I don’t really want some janky daemon to run as a user which has write access to /home/gorgoiler.
[I have some guacamole prepared, to spread on my inedible but soon to be eaten hat, the moment someone links me to a diff showing the author of this change as Alan Cox.]
[Update: The hat and condiment survive, author is one L Poettering: https://github.com/systemd/systemd/commit/cf5115f6e531c65bbb...
What is the use case?
In the old days you would write a little script to set up the mapping for your USB drive, once it is plugged in. Linux provides the UID mapping feature, udev gives you a place hook your script, and you glue it together the way you want.
If your solution was particularly high quality it might get included as a package in the OS. Someone else might write a different solution that gets packaged alongside yours. Maybe they scratch the same itch in two different ways? Maybe theirs is just better?
Systemd on the other hand is judge, jury and package maintainer. What they decide upon is what gets included. They are the cathedral, where once there was a bazaar.
So I guess there is none.
With all due respect, but this is bullshit. The devs that started and are working on systemd are clearly veterans of linux system programming and understand quite well the ins and outs of initializing a linux machine. Unless by some chance you're a gray beard that was working on posix in the eighties, I would give the title of veteran to Lennart and the gang.
Now, I wonder what could be the reason for that? Maybe because some people just trow out unsubstantiated accusations and hearsay as matter-of-fact? Maybe. Who knows.
> If I move it to a non systemD system, you’ve just added the need to chown -R all my files because the new OS says they are all owned by nobody.
Looks like you completely misunderstood the use case. Congratulation.
At this rate systemd will probably become another new OS on top of Linux similar to Android.
> systemd-timesyncd does no clock discipline: the clock is not trained or compensated, and internal clock drift over time is not reduced. It has rudimentary logic to adjust poll interval but without disciplining the host will end up with uneven time forever as systemd-timesyncd pushes or pulls at whatever interval it thinks the near-term drift requires. It also can't assess the quality of the remote time source. You're unlikely to get accuracy much greater than 100ms. This is sufficient for simple end user devices like laptops, but it could definitely cause problems for distributed systems that want greater time precision.
https://wiki.archlinux.org/title/Systemd/Timers
They were always there, and they work much better than cron in my experience (you're not forced to use cron expressions because it supports things like human-readable time formats; they can randomize startup times, which is useful for backups; they can put additional constraints (like "start only if network is available"); and they can actually be controlled from a single place (systemctl list-timers) instead of cron files that can be spread across multiple configs and directories in /etc/cron*, and pretty much every single user profile).
It's a tiny slice of functionality and it makes a lot of sense to keep it in the service manager.
Try harder next time.
- single place control: crontab -e
- additional constraints: systemctl status && scirpt.sh
- randomize startup times: sleep ${RANDOM:0:MAX} && script.sh
The nice thing is that it integrates with the existing infrastructure (shell scripts). This means you don't have to add code to cron to add features.
I agree with the 10s of files in /etc/cron.daily, etc. But those are there for convenience (you can even remove them from /etc/crontab)
Example: I use restic backup. One timer runs every 24h or on the next boot if there where more than 24h since the last backup as this is a laptop. Also it waits some random amount so that not all script run at the same time after boot. If a backup fails (eg remote server down) it retires every 15min, but no more than one hour before I get an Mail with a detailed report.
All really easy in systemd. No scripting required.
sleep $random
for i in 1 2 3 4; do
backup && exit 0
sleep 15m
done
logger "backup failed"
exit 1
Yes systemd might be slightly easier, but that's because they put a lot of money into systemd, not because systemd is a better architecture. It's nice that they open sourced it (although it does give them a competitive edge), but we can't act like what used to be there isn't as good.
Someone could have packaged utility wrapper scripts in much less time than it did to add those features to systemd, and these wrapper script would be more widely usable (by init scripts, cronjobs, etc)
These corner cases take exponentially more time to get right than the first 80%.
"Someone could have packaged utility wrapper scripts in much less time than it did to add those features to systemd"
But nobody has, and I am exhausted to hear how easy it is without systemd without offering a non-hypothetical solution. You don't like systemd, then do not use it and move on. There are distributions without it.
I used logger because it's better practice to log and for the logger to send email, you can send email using the mail command or msmtp. The timing was omitted because it is trivial and didnt serve to illustrate the point, just do backup && touch /var/lib/lastbackup && exit 0, with a if at the start of the script like if [ -x "$(find -mtime -1 /var/lib/lastbackup)" ]
The mid-backup shutdown is a property of the backup script, you'll have to consider that even using systemd.
Sadly there are not many distros without systemd. Everyone is strained in resources and Redhat is pumping money into the ecosystem, so it's really easy to take over most of it. Like I said I don't dislike systemd, I think it adds value, but it's just not the best solution since it does not integrate with others well.
EDIT:
>You don't like systemd, then do not use it and move on.
But this is the issue with systemd: you can't. Want gnome? needs dbus, which needs logind which is systemd. Need libvirt? also needs dbus. Need Y? Needs systemd-tmpfs, which packages the whole systemd as a dependency. Same thing for systemd-udev.
The interlock is so brutal most distros are forced to switch because they can't handle the dev burden.
Now the system boots anyway, just as it would if the filesystem was entirely missing.
between ubuntu16 and ubuntu20 some unit file parameters have changed. I find it rather troubling that we can't maintain consistency.
It'll probably be fine, but I'll be Jack's complete lack of surprise when I'm taping servers back together after a DNS exploit gives some kids root access.
I wish it was possible to pull a hard drive from one Linux system, and plug it into another, and mount the ext4/etc. partitions in a "removable" mode which ignores permissions and mismatched UIDs, and grants full access to the files without needing to be root.
> We used to sit around in the Unix Room saying, 'What can we throw out? Why is there this option?' It's often because there is some deficiency in the basic design — you didn't really hit the right design point. Instead of adding an option, think about what was forcing you to add that option. (Doug McIlroy).
Sigh!
https://github.com/systemd/systemd/issues/481
Still, systemd is just great and it has made my life a lot easier.
See, see, taking lessons from C++. So someone requested a bug, and they got it. How could that be useful?
How could that be useful?
* automatic == permanent address
* dynamic == finite (time-limited) and possibly re-assignable to a different client
* manual == assigned by system administrator (and simply conveyed by DHCP)
Genuine question though: why didn't Lennart and his gang just fork some Linux distribution for their ideas? Why invade everywhere else instead? Anybody ask them that?
I still wonder about my original question though. What's so bad about homed?
I still won't use systemd, and to the extent of my ability I will fight the trend of the Linux ecosystem hard-depending on it. But it's a bit silly to be anything but grateful for open source and choice.
I propose: "UNIX-ish" or Sysd type Linux.
Shit, what am I thinking!
GNU LINUX was the older discussion and development.
SYSTEMD LINUX is likely most applicable nomenclature, given current development.
I used to care a lot more. Frankly, what I do now is make sure I can use an OS and that I am depending on more portable software where I can. I find myself in Windows, OSX, and Linux land, maybe now I should just say Systemd Linux land and pretty much have something other than the OS to worry about.
Ideally, the OS gives me few worries.
Many computer users run a modified version of the systemd system every day, without realizing it. Through a peculiar turn of events, the version of systemd which is widely used today is often called "Linux", and many of its users are not aware that it is basically the systemd system, developed by Red Hat. There really is a Linux, and these people are using it, but it is just a part of the system they use.
Linux is the kernel: the program in the system that allocates the machine’s resources to the other programs that you run. The kernel is an essential part of an operating system, but useless by itself; it can only function in the context of a complete operating system. Linux is normally used in combination with the systemd operating system: the whole system is basically systemd with Linux added, or systemd/Linux. All the so-called "Linux" distributions are really distributions of systemd/Linux.
(/s)
Systemd: Not that.
See https://news.ycombinator.com/item?id=29670980 for an example.
That used to be many discrete programs and shell scripts and files.
There is a lot of philosophy to consider from there.
I really should have expanded on that post, and at a minimum removed "discrete programs" from it, the distinction being the behavior in shell scripts and such and sequential behavior at boot time.
Fork/exec is the lightweight process inheriting of open i/o and memory state with copy-on-write which made Unix a joy to work with. It's a delightful mapping of kernel and userspace into complex inherited tree of processes sharing that origin state (or not. You can close it all down, change things.)
systemd is a complex mega binary which parses control files and subsumes several functions. It's big. It's not small like init. It's not fork/exec simple it forks control processes and monitors children and does a lot of work some daemons did themselves (getty, inetd) but others didn't do. And you get to monitor things. It's not unlike launchctl on OSX or other platforms. And after all /etc/rc is now 5 or more/etc/rcX.d/ complex substates of states to look for scripts, SAT solver complex dependency stuff between semi arcane shellscripts... blah.
PRO: everything is now much more consistent
CON: greybeards like me hate it (beardless btw but meh) because it's complex
gimme open-rc or gimme death
actually, sysv was fine too
open-rc solves the parallel-startup desire for people who want a faster boot and doesn't hide logs in binary formats which can only be interrogated with specialised tools - my primary reason for leaving Debian-based systems that I'd used for 16 years was having systemd forced on me and trying to deal with how to read good old system logging, not to mention 5-minute shutdown times, slower boot on the same machine than with Ubuntu's upstart or with open-rc.
> not to mention 5-minute shutdown times
5 minutes? Try total hang on shutdown, and often not starting up. That was my systemd experience. I traced it down to a NFS mount that no amount of tinkering would mount reliably pn boot. Granted that was several years ago, but soured me on systemd.
It's a bit of a monolith. People will say it's not, because it's made out of a bunch of smaller components, but when those components are that closely interlinked it stops mattering if they live in different code bases.
It also has a pretty wide scope. PID 1 is responsible for things like dealing with zombie/orphan processes and it's generally a good idea to keep it as minimal as possible, and keep the attack surface as small as possible. Systemd handles things like socket activation, logging, scheduling, etc. It handles all this in more or less one process, which means that when/if that process crashes or has security concerns it's a much bigger deal. It even has it's own docker-equivalent!
Really you want to keep PID 1 as minimal and secure as possible.
>Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.
They've made it hard to work with generic linux tools. For example they have their own custom log format that isn't text based. This means that linux tools like find or grep won't work on systemd logs. Generally systemd has replacements for those tools, like you can still grep but you've got to learn the specific systemd ways of actually getting the output and all that (note that a lot of distros seem to ignore systemd's logging provisions and just log to files like normal, for a bunch of stuff).
It's pretty complicated to reason about, one "unit" can be a bunch of different files spread out all over the filesystem, and it can be hard to get a good view of all the aspects that effect a unit without resorting to systemd-specific tools to figure out the relationships between things. For example if you want to edit a unit file the suggested way to edit a unit file is with something like `sudo systemctl edit ssh` or possible `sudo systemctl edit ssh --full` depending on the situation. Spreading files all over the place like that sort of betrays their enterprise roots, it makes some sense when you're working with very large teams but just having all the until files in one location (/etc/init.d) would be a lot more readable and manageable. Also having one file per unit, instead of having to have separate file for units and their dependencies (things like sockets) would be nice.
Another problem is that they broke chroot, which makes dealing with embedded systems that use systemd a pain if you're not using a systemd based host to develop on. They sort of force you to use systemd's docker equivalent instead of the more common chroot if you're dealing with systemd-based embedded systems. That means you'd better be running a systemd based host system as well.
---
Also I think some of us feel like it's being forced on us. Redhat/IBM funds or has control over a lot of the unix ecosystem, so when we see things like Gnome not working on BSD any more because it has a hard dependency on systemd, well it starts to look like an intentional.
Things like the systemd decision to randomly kill off screen/tmux sessions on user logout leave a really bad taste in some people's mouth. Honestly a fair bit of the dislike is just repeated instances like that, and repeatedly ignoring feedback as to why that's actually a pretty big problem, and generally being unwilling to compromise with the broader community.
https://www.reddit.com/r/programming/comments/4ldewx/systemd...
* lack of composability — This is what is really meant by the “do one thing” mantra. Plenty of UNIXY tools are bloated but they don’t feel like it because they can be called as filters. systemd is actually quite modular but the pieces aren’t designed to be standalone. Like how in Ansible is everything is a plug-in but it’s cumbersome to just run a single module.
* “better is worse” — one of the weirder faults of systemd is that it’s thoughtfully designed, handles edge cases gracefully, and does things the right way. Kinda odd to paint this as a fault but it’s the reason people like bash over powershell; Unix tools were mostly made by people solving their own problems and so are completely inconsistent, full of sharp edges, odd syntax and parsers, have plenty of unimplemented features but are fiercely useful. systemd is a tool not forged in battle and is designed by people trying to write good software over software that solves an immediate need. This isn’t to say that systemd isn’t useful — it’s genuinely transformative but software written out of frustration tends to be designed differently and it shows.
SystemD is not an interoperable technology. It insists on wrapping itself around what was already there, in the guise of making it better while also making it impossible to use something else.
Abstract example: imagine systemd-plated manages access to crockery in your kitchen, providing neat features like being able to quickly pick out plates by colour (the old system never did that!). However, if you take a plate for yourself instead of asking systemd for it then systemd will take the plate out of your hands, put it back in the stack, and say “error: plate not requested via systemd”. Now imagine everything in your kitchen has to go through systemd. Now imagine that systemd is one of the biggest and most rapidly changing machines in your kitchen that breaks a lot. You’d be pretty annoyed if you ever did anything weird enough to encounter these bugs. If you only ever want a blue plate, you’re fine.
Real example: Any day now we’ll be unable to use plain LXC on Linux because systemd (1) assumes you only manage containers via systemd and (2) clobbers any changes you make outside systemd, using the lxc-* tools.
Yes, some things was improved but it could be done in sane way - making things by design replecable, as it was always in UNIX way. But systemd is intentionally self absorbing, more - looks like it is intentionally layer above Linux. Now what we can do when one day Pottering or anyone from that IBM/Intel/MS cabal will merge Linux kernel uncompatible change ? And make Windows kernel "working flawless" - for some time of course. And with build in tracing. Or login right licensing ? :) That can be easily done today. And what plain users will tell ? "Works for me, what you talking about ??" Remember, some idiots already tried to sell plant seeds that you needed to re-buy every year. Business unchecked becoming evil.
Btw. about that "boot fast" :> Now it is dropped, of course. Also making system gooing down almoust instant is broken for years now.
My prediction is that in some time facepalm will be most common reaction when someone hears "systemD".
Also, your listed “features” are bullshit: logging in, audio, GNOME, etc have absolutely nothing to do with systemd. Even logging can still be forwarded to any log tool you would like, but having it implemented inside is a sane choice — you have to start taking logs when there is not even a file system available. You know how was it solved in the pre-systemd days? It was simply ignored!
System init is a very complex problem, systemd only gives you the essential complexity for the task.
Also RE poweroff speed: there is a single flag you can toggle to either kill lingering processes (which is usually not enabled because some hacky tools like tmux likes to live in the background and systemd killing it is bad) or wait a bit before. So yet again, systemd does the correct thing while before these were simply killed.
This isn't universal. For service management systemd is awesome, and has been way more enjoyable to use than any other system (Ubuntu's upstart, Solaris's SMF, FreeBDS, and old-school linux init). As a sysadmin my only issues with systemd have been occasional patches that required rebooting, but these have practically disappeared in the last few years.
Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
AFAIK none of these describes systemd.
Here’s a thought I haven’t evaluated fully: can the same thing be said about the Linux kernel?
At the end of the day I think some of the major points of criticisms of systemd (moving fast and breaking things for example) are valid, however I still see it as a net gain. Fortunately there are distros like Devuan, OpenWRT and Alpine for those who want something different.