A word about systemd (2016)
skarnet.org
skarnet.org
No idea why software tries to "daemonize" anyway. Seems to be some kind of weird tradition. It's always been the wrong thing to do. They should just run normally and output to standard streams even if it's connected to a terminal. The reasons are outlined here:
https://mywiki.wooledge.org/ProcessManagement
Ironically, systemd is one of few service managers which can properly track those weird "daemonizing" processes. It uses Linux's cgroups. It doesn't matter how many times they fork, they can't hide from systemd.
mydaemon & daemonpid=$!
- you ssh'd into server, restarted server, checked it worked. Then you logged out and server got killed.
- your server runs as root or power user
- your server output/error goes somewhere, not clear where
- your server's process got paused by SIGTTOU, because it was trying to write to console
etc...
The complex daemonization dance, with double-forks, closing random FDs, etc.. was required to avoid all this bad stuff before systemd days. Sysvinit/openrc folks came up with "start-stop-daemon" helper, but it was pretty terrible in general, especially in logging department (it had none).
The point is software should be written to make the service manager's job as easy as possible by keeping it simple.
I disagree that systemd is a proper service manager. Systemd is horribly complicated in comparison with what is actually needed to implement a proper service manager, as demonstrated e.g. by D. J. Bernstein's daemontools and their many derivatives.
Perhaps systemd is better these days, but a few years ago when I have tried systemd on one of my servers I have stumbled upon a bug that could be explained not by some random programmer mistake but only by a bad global architecture of the program. That bug made me mistrust the competence of the systemd creators, so I have never tried to use it again. Moreover, systemd does not provide any feature that I could consider useful and that is not available in alternative service managers, so there were no reasons for further experiments with it.
(That systemd bug was a bug that I have never seen in a computer without systemd, from the mainframes of 50 years ago to the latest computers, regardless of the operating system. Due to a race condition in systemd, sometimes the shutdown or reboot of a computer could fail, with the computer remaining stuck in a deadlock, i.e. in an infinite loop, without properly saving all state and closing all resources.)
You might not need the features, but someone does, and putting it all in one places means everyone can use the same tool, and nobody has to learn 10 different systems for slightly different tasks.
Hence the hate that systemd has, and always will have, as long as there exists a community of people who use Linux because it is a Unix-like operating system.
But, Linux is versatile enough that everyone can have their own little fiefdom. The real rage came in years ago when tools that were widely used by all suddenly became reliant upon systemd to function at all. Thankfully, in the time since, enough development has happened that mostly people are free to choose what they do and don't want to run again. The manner in which it was foisted upon the community, however, was one that hopefully will be remembered as being far more aggressive than should be repeated in the future.
But, I imagine that as long as Lennart Poettering walks the Earth, we likely will have yet another similarly awful episode.
Seems like we're in a pretty good place now, with separate development efforts by and for the people who are actually interested in anything but systemd.
-I can spawn/respawn daemons as someone other than root. SysV inittab runlevels are root-only.
-I used runlevel 4 for all my magic. Now I can have so much more magic.
-nspawn has let me craft containers from day 1.
-Path units are so much better than inotifywait (and somewhat better than incron).
-Do I prefer socket units to inetd? Mostly.
-Free automounter! Yes, crazy syntax, but gratis!
Do I wish it worked on BSD? Yes.
It IS functional - it does things that are helpful and important.
But the implementation is hard to swallow. It is just so far away from elegant, understandable and usable.
- mediocre config files - they decided that config files are declarative, which could be good, but it pushed simple (and visible) logic in /etc/init.d/* files into more complicated logic encoded in binaries. The simplistic config files seem to be more simple for systemd to implement, but harder to understand, maintain and use.
- binary log files. everything has to go through systemd to access them.
- responsibility creep - systemd started taking over more functions, and it doesn't seem to stop. Will systemd turn into busybox, with simpler poorer implementations of every single thing in the system? dnsmasq is similar. The thing is, busybox and dnsmasq are cut down implementations for smaller system that are alternatives to their full-featured brethren. systemd takes over the core of a system.
That's fine, and I'm OK with a world where we disagree about how usable systemd is, but I interact with the systemd config files and symlinks and logs pretty frequently, and have found them to be orders of magnitude more usable and understandable than /etc/rc#.d and /etc/init.d and "I wonder if this system puts sshd logs in /var/log/auth.log or /var/log/secure or somewhere else". Notably, I interact with a mix of Arch and Ubuntu and RHEL systems over the course of a week, and systemd means I can engage with all of the above using the same methodology.
In terms of creep... busybox is noteworthy for packing all of a system into a single binary. systemd doesn't do this. Their resolver and time daemon and networking and all the other bits and bobs are their own separate binary, and you aren't required to use them.
systemd is functional. It does the things you say. It unifies things, including config files and log files. It works for you.
I just don't think the way it did this is that elegant.
My busybox analogy should be more viewed as: what if arch/ubuntu/rh decided that busybox would replace all coreutils, etc on their systems today? All the same commands. Still usable. But something is missing...
I have the feeling that you are mourning the old, bad days?
It is actually way better understandable and manageable: /usr/lib/systemd is for distro-supplied and packaged units. /etc/systemd is for your local overrides, /run/systemd is for units created at runtime by various generators. (Then there are $XDG_(RUNTIME|CONFIG)_DIR/systemd for user units, for consistency).
So no more diffing config files at update time: the distro provided ones are always updated, your overrides are always preserved, cleanly.
> - binary log files. everything has to go through systemd to access them.
So send them to external syslog, that saves them as you wish. There is a point in indexing them (and being able to have a newline in the log message).
- responsibility creep
This is a matter of perspective. If you take it as "init system at boot", then yes, it might looks so. If you take as an service manager that runs/stops/reloads/monitors units, based on different events (startup, dbus activation, socket activation, timer, etc): what's the difference in the bulk of action, except for the event that caused that action? The service handling, unit definition, etc. are all the same, just the initiation event is different. In this light, it doesn't take over functions; it consolidates and unifies them.
Unlike dnsmasq or busybox, it is not in cut-down fashion, but exactly opposite. It replaces rag-tag of tools with one with more complete and tested implementation.
check out for systemd networking, the resolver, timesyncd, etc...
for example timesyncd IS cut-down and brute-force:
"does not bother with the full NTP complexity, focusing only on querying time from one remote server and synchronizing the local clock to it"
ntp is complex, but it does many things in safe and effective ways.
etc.
systemd-resolved, on other hand, is good. There's no other resolver stub, that has the functionality that resolved has. It is the only one for Linux, that can do per-link, split DNS properly.
Addendum: https://news.ycombinator.com/item?id=15228940 This approach clearly deviates from widely expected behavior, whether it is in conformance with applicable RFCs or not.
See, many people have misconceptions how it works; see the comments in thread you linked to. Add to it the fact that resolv.conf syntax cannot express what is expected of modern resolvers (not even with apple-like extension - `/etc/resolver/*`; it cannot react to links going up and down), it is basically just compatibility shim for broken applications (they are not supposed to use it directly); that's why systemd-resolved configures it to point to itself and handles the specifics for them.
That understanding has been wrong; it is an implementation detail of glibc and that's it. It could query them in random order, or round-robin them for each request; or time them how fast they respond and prefer the fastest, up until it starts timing out and then shake the order again. Some resolver stubs ask all of them and use the answer that arrives first. That's also valid. Since they all are returning the same answer, it doesn't matter. It also makes any argument for ordering superfluous.
So that "common understanding for many" is erroneous and those many should update it. The thread you linked to has exactly a discussion on this point, so in the years that passed since then, those affected many had ample time to fix up their assumptions.
Again, there can be other valid reasons for maintaining an order. Could be about latency, privacy etc.
I admit to not having read the full thread, and I apologize for any needless rehashing of arguments. It was just the first result confirming systemd-resolved breaking setups by its handling of what many (in that thread) consider expected behavior.
What I think you're primarily suggesting is that there are no formal standards prescribing glibc's behavior in terms of DNS resolution order. That is certainly true, and I don't disagree calling it an implementation detail.
1. symlinks: I think the systemd structure is very intuitive and the order of overwriting makes sense. It's new, but it makes sense, and we only have 3 different locations, each with a specific order and precedence.
2. Config files: I think keeping the declarative syntax is the secret sauce of systemd. The whole point is you can orchestrate processes without needing to write code. They're much easier to understand if you assume a knowledge point of 0, and they're much more maintainable (no logic), and they're even more portable. Bonus is they're much faster to execute by the init system. Yes, most of the logic is in systemd now, but you tell it what to do. Reading the documentation goes a long way to making unit files less cryptic.
3. Binary logs: yeah I agree just not a great idea. Always sounds nice on paper because you can do neat things with permissions and can have faster performance, but in reality just not necessary. They got too fancy.
4. Responsibility creep: from where I'm standing, not a problem. systemd isn't a piece of software, it's just a name given to a collection of software, like GNU. systemd-init doesn't have feature creep. Sure, now we have systemd-file-my-taxes, but that's optional and it's a whole different program that just uses the systemd name.
Yes, systemd services can do a lot of things. Except that before rc scripts did all those things, but without any unified and standardized primitives. systemd provides those for a lot of common use cases - and that's how it ended up doing so much - because our rc scripts used to do that. systemd actually does less because unit files are not scripts (but one can hook any external program in a lot of places, if they need to).
Also, as I understand it, systemd isn't a single piece of software either, it's a whole project with a bunch of semi-autonomous pieces, tied with common conventions and assumptions. It's difficult to define the scope for an umbrella project, and I believe individual pieces all have their scopes relatively well-defined.
The more systemd does, the better I like the power, features, consistency and functionality my linux systems have.
systemd is a beautiful work of art.
Software developers owe it to themselves to full understand what systemd has to offer, because if you really understand it then you'll find you can architect your systems around systemd and often let it to the hard work for you.
If anything, the old gnu/linux name should be dropped and replaced with linux/systemd to illustrate that in fact systemd is really a major part of what Linux is.
Also, what do you mean my "standardizing dnsmasq behaviour?" From my recollection of using it (and from manpage[0]), it's pretty basic. You (1) manually edit /etc/resolve.conf to point to localhost dnsmasq instance (2) use app-specific ways to configure nameservers, in dnsmasq's case it's an ad-hoc list of secondary resolv.conf files specified on command line, with no defaults. I don't think this is worth standardizing, as this scheme cannot even represent network interface -> dns entry associations.
Well, many people are confused about how DNS resolving works. Many of do not consider a problem, when they configure their resolv.conf to two different upstreams, each returning different answers (i.e. one responding for domains that the other doesn't) and then wondering, why stuff is broken.
This reminds me of teenagers of the last generations who preferred distorted mp3 sound to original because that's the sound they have grew up with. Teenagers of today prefer distorted wide-angle selfies instead of properly shot faces because again that's what they have grew up with.
Calling systemd a work of art gives me the same vibe.
As a long-time Windows developer (both apps and device drivers) substantially coming to Linux long after the decision to use systemd was made, I am impressed as heck by Systemd.
And was completely horrified by my very limited experience, as a user, with pre-systemd Linuxes. That was insane. Manually edit what in to which system file? And my system invariably didn't even HAVE that file.
Even writing systemd units is virtually effortless.
The Unix philosophy is about function, not form. Say you have a tool to send something over a network. In 1993, that tool can get by being quite simple. In 2024, that same tool now needs to be more complex, just to do the same 'one job' and to 'do it well'.
It's a consistent insistence that the 'single job' be defined today just as it was in 1980 that results in Linux requiring hundreds of packages to be even remotely usable. Oh, you want Wi-Fi? Well, that's not really the job of the networking stack, the job of the networking stack is to connect PDP-11s together...
Linux was(is?) the system. Shell scripts could configure the system, reasonably well. Systemd meant it was no longer possible to simply fix a linux distro that had it...a massive recompile and intricate understanding of many low level systems was needed.
No, you shouldn't need to know about PDP-11s to fix your buggy wifi, neither should you need to know a massive binary subsystem to configure your wifi either...
So as far as doing its "one job", it did/does it fairly poorly, even today. It is a lot more standard and it does get points from me in that respect, but it fails at being a configurable system completely.
What kinds of problems are you hitting where you need to recompile systemd?
Configuring the system with textfiles only works if the system anticipated all your needs, and to anticipate all needs is not only complex but error prone.
There have been times where I needed to do something that wasn’t available in the unit file syntax, so I just do it in a shell script and ExecStart that.
If so, perhaps it is less of a problem than it used to be.
It still doesn't fit every need but it's better if it can get out of the way when told to.
It seems to be made for, and works well for, a modern ephemeral infrastructure workflow.
Build stuff to work with an unmodified distro, wipe it and reinstall if it goes wrong.
The Unix philosophy is about function, not form. Say you have a tool to send
something over a network. In 1993, that tool can get by being quite simple.
In 2024, that same tool now needs to be more complex, just to do the same
'one job' and to 'do it well'.
WiFi absolutely should be a separate component; the unix philosophy is an excellent defense against servers in a datacenter trying to install a WiFi stack.
I counted no less then 64 related executables related to systemd on my system.
Accusing systemd of "not doing one thing and doing it well" is like accusing the GNU project of not following the Unix philosophy.
It is currently the only linux resolver stub, that supports link specific upstream resolvers. In the terms that user would understand, your split-horizon DNS works correctly over your VPN.
1993's sendmail is more complex than qmail. The latter uses a modular system, where each subtool does its job well, rather than sendmail's monolith design.
Systemd, particularly journald, violates this severely by snarfing any stream you want to send to /var/log or wherever and locking it in a binary database somewhere, that you have to use journalctl to access.
I have no problem with journald as an idea, but the audacity that it knows better than the user what to do with stdout/stderr is insane, and 100% anti-Unix.
Though I'm surprised that you say:
> Every time systemd comes up everyone says it's great and how much better it is
When my experience is usually that people complain about systemd and it being over-engineered and doing too much and having too many components (yes, I know that those projects are related to systemd, not part of it, I'm just saying what I see people mention as downsides).
People do use them. Multiple distros either default to one of those, or support it: Alpine, Devuan, MX Linux, Slackware, Void, Guix, Gentoo and others.
> or they don't add anything worth noting over SysVinit.
Not looked at this myself, but do they mostly not add something - dependencies, easier config or something?
Funny the author mentions Solaris because Solaris has SMF and other systemd-similar tools.
Linux pre-systemd looked like a kernel duct taped to a bunch of random thrown-together shell scripts... because it was. A sorry state of affairs that a bunch of very stubborn people were afraid to let go.
The simple fact that configuring networking out of the box is completely different on every distro and sometimes within distros depending on which tools were loaded, in 2024, is simply insane.
I'd rather have the chaos of Linux distributions than the tyranny of "standard" ways of doing things.
It is important to note, that if you want to do it, you have to roll up your sleeves and do it. Not to expect free manpower of others to provide it to you.
Or in another words: http://islinuxaboutchoice.com/
The biggest reason systemd is taking over is companies like Red Hat keep paying people to work on it. As time passes, systemd gets better, gets more features, makes the life of distribution maintainers easier and therefore gets adopted, and makes the lives of users easier too, mine included.
I'm making my own freestanding Linux thingy but I'm not under any illusions that it's going to replace systemd or any other software any time soon. I'm just one guy doing it in his free time because he enjoys it.
I really hate it when I see people complaining about systemd. The people behind systemd had a vision, they reimagined Linux user space and put the necessary work into making their vision a reality. They're an inspiration for me. Not only that, they took their work and they released it as GPL licensed free software. One would think the users would be more thankful. One would think people would not have moral objections to free software getting released to the world. Some technical criticism is certainly warranted but the sheer hatred that people display towards it and its developers certainly is not.
I agree that "Linux is about choice" is often misinterpreted. It's about my choice as a programmer and software developer to redefine the entire system downstream of the kernel however I want. Users are encouraged to themselves become technical and educated so that they too can enjoy this power. However, we're not about to drop everything and start maintaining some legacy system because people who aren't paying us keep insisting that it's how things gotta be done. We're programmers, we like it when things are made in our image.
Especially when many of the complaints can be summarized as, "I can no longer use my years of experience and knowledge about the old init system."
(The people who complain that systemd unnecessarily enlarges an install's attack surface I have more sympathy for.)
For the same reason, why Canonical forces netplan (consistency between desktop, using NetworkManager and server, using systemd-network), Redhat standardized on NetworkManager instead.
(I hate NetworkManager just like I hate netplan though. Both can get off my lawn.)
That's not really the case, though. The Debian folks had a very transparent discussion and vote about using it:
(Edit: Other anti-systemd rant sites sometimes mention these obliquely as "upstream dependencies" and "workplace-mandated", while ignoring the fact that those people have reasons to demand those things)
But HN really doesn't need to litigate this all again.
With SysV-init, it was/is necessary to modify /etc/init.d/servicenamed shell scripts to get consistent logging and respawning behavior. With SysV init it's not possible to add edges between things to say that networkd must be started before apached; there are 6 numeric runlevels and filenames like S86apache, which you then can't diff or debsums against because the package default is S80apache and Apache starts with A which alphabetically sorts before the N in networkd.
And then add cgroups and namespaces to everything in /etc/init.d please
Supervisord (and supervisorctl) are like s6. I don't think any of these support a config file syntax for namespaces, cgroups, seccomp, or SyscallFilter. Though it's possible to use process isolation with any init system by modifying the init script or config file to call the process with systemd-nspawn (which is like chroot, which most SysV-init scripts also fail to do at all, though a process can drop privs and chroot at init regardless of init system).
systemd works in containers now without weird setup. Though is it better to run a 'sidecar' container (for e.g. certbot) than to run multiple processes in a container?
Pre-systemd this was unthinkable.
(similar stories apply to non-systemd-proper parts of the systemd project, but I personally haven't used them as much)
There are some people who are vehemently opposed to systemd, for a variety of reasons with varying degrees of merit. I think it's fair to say that systemd's developers broadly ignored those people, figuring out (correctly) that if they convinced a majority of distribution maintainers, and made the user experience for the average user smooth enough, that was actually going to be successful. As it should be: distro maintainers get to pick how their distros work, and users determine which distros they use.
And systemd has a clear sell for those groups: distro maintainers can standardize their service scripts, often just using what an upstream package author can write agnostically to distro, and users can handle the majority of base system functionality without having to pick things like "which of these cron daemons should I install".
Which brings me to the question of which category systemd does not appeal to? I am guessing "minimalists" and "people who like things done the old way" - and those are simply not that numerous compared to others.
So, politics at its worst. Being ignored on a matter, especially when there is merit involved, can make someone a highly motivated hater.
I can’t stop you from hating, but if your pitch is that distros shouldn’t be allowed to make their own choices about what to include if you disagree… I guess your option is to start your own distro.
No linux distro was forced to adopt systemd. They did it on the merits, from their determination.
Systemd as PID1 is nice on merits. I wouldn't want to get back to sysv initscripts.
Everything else is incredibly hostile to practical unix philosophy and solely exists out of spite and politics. This is my opinion and I realize I'm in the minority.
Systemd as PID1 got its merits, hands down. Even coming from a person so destructive as LP, systemd as PID1 is still a good product.
Perhaps everything else is feature creep and architecture astronauts[0] type of things.
0 https://www.joelonsoftware.com/2001/04/21/dont-let-architect...
If the other binaries aren’t better for maintainers and their users, why are they replacing other tools with them?
Anyway. systemd vs unix philosophy is an insurmountable ravine and I chose to stay in what I think is sane camp.
To pick on cron, just from my own experience:
"I wonder where this scheduled task is configured" -> "Lemme check /etc/crontab" -> "nope, I guess maybe /etc/cron.d?" -> "Nope, lemme check /etc/cron.hourly/daily/weekly" -> "hmm, not there... maybe I want `crontab -l`... or it's running as a user, lemme check that list and run through `crontab -l -u $USER"
Finding out that I didn't notice a cronjob was failing because the stock methodology for notifications is email, and relies on the system having SMTP configured and having MAILTO set in cron.
Realizing a cronjob wasn't running because some, but not all, of the cron options have character limitations like "the file name can't contain any dots".
The total lack of dependency management, so anything I wanted to run @reboot needed to handle its own dependency checks and sleeps to make sure I didn't say, try to run puppet before the network was up.
On the networking side, I started using Arch when the config flow was "edit bash arrays in /etc/rc.local". I've used netcfg, netctl, and now systemd-networkd. systemd-networkd is easier to reason about, easier to debug, more powerful, and more portable between systems.
My question though: how did we manage to run infrastructure using cron for decades and did not feel any of that pain?
My take on the answer is this. Failures: we have /var/log/something for this and typically it's obvious something is not being executed. Many files: yes, there are different configs but don't you remember where you typically put your cronjobs? For example I always use user's crontab, so this is where I would look for the job spec.
Dependency check is one of the main reasons for systemd as PID1 if not the main, so hands down, that's valid and true.
But also what we consider to be painful has evolved. When all systems were to some extent a hand-gardened entity, maintained by some specific system administrator, it was easier for them to have a personal standard. It didn't matter if I used cron.d and you used crontab, as long as on our own systems we kept one method. Increasingly, servers are more ephemeral, more horizontally scaled, and more automated, and systemd timers are just easier to work with in that model.
In a decade or so, I'm sure we'll have further modified how we handle systems, and something else will come along that solves painful things about systemd.
And we're back at it: systemd seemingly has plans to eliminate cron. I was not impressed when I have been pulling my hair out on why certain jobs in Ubuntu still run despite being removed from cron files. Turns out, systemd now has 'timers'.
So, mark my words: a few Ubuntu releases down the line, cron will become an optional package and after a couple more it won't even be available.
Up next: sudo.
So if timers were gonna replace cron I'd imagine it'd have happened already. That said, those Canonical folks are pretty crazy, so who knows.
Sudo getting replaced by doas may not be the worst thing.
What happened?
See the last point.
I can't see that being a project that would be bullied easily. Especially not "bullied so that it was adopted before almost everyone else".
Yes, and that's okay. Whatever it is that came before it sucked so much I don't even remember what life was like before systemd.
Also, it's amazing that we even get to redesign Linux user space like that. That's something special. Not a single other operating system allows you to do this. You could boot Linux straight into your program if you want, without any user space at all. Linux is such a solid foundation it lets you swap out the entire user space, you can trash systemd and even the entire GNU stuff and just do your own thing. It's inspired me to start working on a redesign of my own.
That was possible from around day one of linux kernel existence.
You look up the manual entry on some Linux system call and it gives you the glibc documentation. You look up the init system in the manual, expecting details on what the kernel requires of PID 1, and it just gives you systemd documentation. This stuff is completely mixed up.
You look Linux up on Wikipedia and the article eventually starts telling you about standard C libraries as if they were part of Linux. In some articles there's even a diagram showing glibc wrapping the kernel, as if it was Linux's equivalent to the Win32 API.
https://upload.wikimedia.org/wikipedia/commons/4/45/Linux_ke...
In reality, you can trash that entire thing and remake the entire user space in Rust or Zig or Lisp if you want. No libraries are necessary.
a fair shake
I do not think it is charitable to them to say they think only Linux matters. If that’s the case why are they a FreeBSD developer?
The reality is UNIX is dead… these days I think you've basically got Linux and some rounding errors.
I couldn’t care less if Debian announced that Debian 13 is going to default to s6 or runit as the init system. I have no loyalty to systemd other than the Linux distros I use have integrated with it.
The linked article introduced me to s6 which is interesting. I do think there is value in simpler codebases a la wireguard, toybox, and doas. None of the hostilities towards one init project over the other have ever made sense to me but “the battle” was mostly fought before I got into Linux.
Might as well say GNU violates unix philosophy because it has text processors AND a shell.
A word about systemd - https://news.ycombinator.com/item?id=19077561 - Feb 2019 (17 comments)
https://web.archive.org/web/20240917143933/https://skarnet.o...
I can come up with a couple functions, but none of them seem like core things, more like tangential benefits:
- Startup ordering means you don't get log spam with "can't connect to X" until X starts
- Certain dependencies like mounts look the same whether they're "running" or not, so not starting before the mount is important. Or like, if a program enumerates interfaces or something, making sure all the interfaces exist first (via a dependency). Of course, a program could do its own checks and possibly be more stateless about it.
- As a standard, for packaging - you can distribute a service/socket file with your program and most distros can just plug it in and expect it to work (also thanks to having standard targets and directories).
Otherwise it seems like an ugly declarative DSL for gluing programs together, which you could do in bash or python or whatever other language too. Is reducing log spam really the point? FWIW I do think systemd is better than the alternatives ATM...
On any system, something has to start "first", and be in charge of figuring out what the heck happens next. On modern Linux systems (and I'm oversimplifying massively), the hardware hands off to the kernel which launches some kind of init system the first userspace thing. That could be sysvinit or upstart or s6 or systemd or any number of other things. You could write it in any language, though the nature of when it has to launch strongly incentivizes writing it in some kind of compiled language so you aren't having to make sure the right interpreter bits are in place first.
That thing then, depending on its level of complexity, gets to handle things like "what else do I launch" and "what order to I launch it in" and "what do I do if a service I started stops on its own".
Beyond that, your life is way easier if more than one distro uses the same thing to do this, so that an upstream package author can write "this is how an init system launches my service" and distro maintainers can use that, vs having to create and maintain their own scripts for how their init system launches a service.
So, the "point" of init systems is "something has to be in charge of userspace".
So the dumb response to your questions is if I have a list of services: "what else do I launch?" everything, "what order do I launch it in?" launch it all at the same time, "what do I do if a service I started stops on its own?" restart it. I typoed, but I really meant my first sentence verbatim: suppose init is a bash script that just does:
(while true; do service1 &> log1.txt ; sleep 5; done) &
(while true; do service2 &> log1.txt ; sleep 5; done) &
(while true; do service3 &> log1.txt ; sleep 5; done) &
It's largely touted that systemd's killer feature is that it represents services as a graph. But why is it important that it operates as a graph? Why is ordering startup important? Or does that not matter, and the real reason systemd matters is that it makes packaging easier (as you said, and as I said) and it has lots of built in conveniences? I have my guesses, but I'd like outside opinions.
Some examples:
On your desktop, while you launch programs through the window manager, things like the window manager itself, and also the network daemon, and time-keeping daemon, and the daemon that logs go to, etc etc, all need to be spun up by something.
Some of those daemons should restart themselves if they crash, and some of them shouldn't.
Some of those daemons misbehave, so it's nice to be able to tell them "don't eat all the RAM" that rendaw is trying to use to do whatever you're using the desktop for.
Some of those services shouldn't start until it's the right time in the sequence, but also you want booting to be fast, so where we can be parallel, it's nice to be parallel (this is where the graph comes in). And on the flip side, some of those services need to restart if a service they depend on restarts.
I've run s6 (the project built by skarnet, the author of the original post here) on a variety of systems. I would highly recommend it as a very high quality example of the kind of minimalist init system you're envisioning. And likewise, I'd encourage you to try it out! It's possible you'd find that for some/all of your use cases, the value of simplicity outweighs the value of the conveniences. But the reason that systemd has seen so much success on modern Linux distros is that for the average distro and average user, the conveniences in terms of functionality and standardization are worth it.
grep -R Restart /etc/systemd/ /usr/lib/systemd/
I guess different databases might require more manual intervention to recover?
(1) I want to stop "service1" for a few minutes, without affecting other services, how do I do this?
(2) I want to disable "service1" completely, or pass it an extra parameter, without affecting other services. How do I do this? Note you cannot edit the init file and restart, this'll restart all the other services as well.
(3) I just typed "my-package-manager install service4", and I want service4 to start right away (and naturally have full functionality - keep restarting on failure, save logs, etc..). How do I do this?
(4) service2 had a emitted too many log lines, and my log disk is full. How do I clean existing logs without restarting the service? (Note even if you remove the file, it'll still take up the disk space)
(5) You were debugging service3 performance, and noticed it has printed "ERROR out of foobars" to its log but didn't print the timestamp.. When did this happen?
(6) service2 needs to be started after NFS volume is mounted. service3 is responsible for cpu fans and should be started as early as possible. service1 will re-generate the userdb if the data directory is missing, so it should not be started automatically if that's the case.
(7) new version of service1 changes command-line and default restart interval. How do I apply this change to my heavily customized startup file?
I could go on and on. All of those are real problems, and before systemd, they were solved with tons of bespoke code. Either C/C++ code (making a trivial daemons complex beasts), or shell code of init system, with was typically highly distribution dependent. And of course it was fragile and full of bugs.
So maybe this is too reductive, but basically:
- Tools for mutation of a running system
- Dealing with hard dependencies that don't cause errors when unavailable (like nfs in your example, or fan control, but also enumerating devices as I mentioned earlier, or for security, firewall rules since firewall starts entirely permissive)
- Standard packaging (edit: er, composability via standard packaging, or modularity with standard packaging for modules)
I think the logging stuff (datetimes, clearing, etc) is probably more the domain of a logger like journald, whose purpose seems is more concrete to me.
This is pretty fundamental stuff. You can't run a program if the disk that it's sitting on hasn't been made accessible yet. You can't start a network service that listens on a certain IP address if that IP address hasn't been configured yet, or if the destination for service logs isn't ready, or if our service needs to talk to some other service like a database which isn't running yet. Etc etc. Booting up a modern system to a graphical environment requires hundreds of things to happen in the right order.
As for everything else, the project announcement blog post from 2010 does a pretty great job summing up the shortcomings of doing "the simplest possible thing": https://0pointer.de/blog/projects/systemd.html . You can certainly still do things that way if you want, but most people value the benefits gained from a more intricate approach.
I can see that being important for a desktop, where you're plugging/unplugging stuff and want startup to be fast.
> You can certainly still do things that way if you want
I think you're putting words in my mouth.
Debian and other distros asked their userbase which values to prioritize, and the users chose what systemd provides. Hope that helps explain "why does systemd exist / why have most distros chosen it?"
What we need is something simple that uses something easy like unit files. (Though I wouldn't choose the systemd unit file syntax.)
I hate systemd. I sincerely and strongly believe it to be the worst thing that happened to the world of Linux. Note: this is just my opinion, of course I'm obviously wrong, can I have it please?
Having said that I have to admit that the battle has been lost and there are no practically suitable server distributions clear from it, let alone desktop. So the only fight worth fighting is to delay destruction of other parts of the OS. In Ubuntu you can easily remove systemd-resolved, get rid of ssh socket activation, bring back cronjobs to crond and install a sane NTP client. Hopefully, we will still have sudo available for many more years at least as an optional package and then as a third-party deb. And thankfully we still have logs in /var/log and journald is not a hard requirement for anything. Yet.
Otherwise resistance is futile as systemd people excel at politics and seemingly no opinion or input has any influence their decision making process.
Again: I'm totally wrong here and systemd is good.
So this is yet another person who apparently thinks all the different daemons made by the systemd project runs in the same process.
>Remember sendmail, BIND, INN, and, definitely a better analogy, the early days of Microsoft Windows ?
Honestly, no, I know next to nothing about any of these. Someone wanna fill me in?
Providing DNS services on 127.0.0.1:53 has nothing to do with systemd. It's a convention used by NetworkManager, and associated network services. The very good reason being that network interfaces that are being brought up and down can choose to forward their DNS requests to 127.0.0.1, and allow a completely separate unit (resolved?) to deal with the details of where DNS requests get forwarded from there. (Or not, if they so choose).
Was there some OTHER burning compelling use-case that makes you want to use 127.0.0.1:53 for some other purpose?
"Additionally, systemd-resolved provides a local DNS stub listener on the IP addresses 127.0.0.53 and 127.0.0.54 on the local loopback interface"
https://www.man7.org/linux/man-pages/man8/systemd-resolved.s...
> Was there some OTHER burning compelling use-case that makes you want to use 127.0.0.1:53 for some other purpose?
The problem is... by default... resolvd routes ALL DNS REQUESTS to this stub by over writing resolv.conf with ONLY that socket and forcing you to configure resolvers elsewhere. And you have to manually reconfigure this if you want to change it and it forces you to get in the weeds way to much when all you want to do is add a DNS resolver.
https://www.man7.org/linux/man-pages/man8/systemd-resolved.s...
The confusing thing is why the heck is SYSTEMD an init system writing DNS resolvers? Seems out of it's scope in my opinion.
I was surprised resolvd existed when I was trying to troubleshoot a DNS error caused by it.
unless you want to use Arch at some point it will be the default for most Linux.
The systemd monorepo also provides systemd-resolvd which does, but this is optional and last I checked not included or at least started by default on Debian, for example.
If you really don't want it, it's super easy to disable:
systemctl disable systemd-resolvedFor previous versions I had to `chattr +x` as well because systemd stuff of course will not let me have my resolver.
`nameserver 1.1.1.1` is insane. Hope you don't ever have to use link-specific resolver.
Congratulation to breaking your system.
Ah. It might be a systemd-resolved.socket (or similarly named), allow socket activation. Systemd would warn you when you disable the service that it wouldn't be giving up the socket, that it could still be restarted. Because it's a great monorepo that's smart & courteous.
`systemctl status ssh` shows, that it is socket-activated:
TriggeredBy: ● ssh.socket
You would know to run netstat or lsof if you have expected something like that to happen. It certainly wouldn't ever cross my mind to have sshd socket-activated and systemd actively trying to hide the fact. This change by Canonical was out of the blue, totally unexpected.
Why am I forced into nothing but link specific resolvers?
Why cant I have a global default resolver anymore, with link specific as needed?
You can have default one. Chances are, if you have only one configured, it will be default. How are you forced? Who did that? How?
Oth, with resolv.conf syntax, link-specific resolvers are impossible. Even the apple-like extension in /etc/resolver, they cannot respond to link doing up and down.
Can you tell me how to have a default global resolver with resolvd and network manager?
Should be pretty straightforward right?
It's a pretty basic task.
I looked all through the Man pages and couldn't find a way to do it with network manager.
Then `resolvectl status` will show you each link, what DNS is configured for it, what domains it will resolve and which one is the default.
systemd's developers have formed a habit of deciding to implement something like an NTP client or DNS resolver, bringing a healthy dose of ignorance of how those services actually work, and then doubling down on their poor design decisions when problems are brought to their attention.
Those components become my problem when the distributions I choose to use choose to adopt them, and then I have to work around them.
Everything is a tradeoff.
Otherwise I could avoid Linux entirely if I so chose, and make the systemd consideration irrelevant.
Ultimately distros decide what software is on their distro and users decide what distros to use. So the solution here is to find a distro that meets your needs.
I was struggling with it while trying to troubleshoot a VPN connection because resolvd automates resolve.conf into a generated file from several other files and will overwrite any other functionality trying to add resolvers to resolve.conf, like an internal VPN company resolver for example.
Why is systemd even touching DNS?
Seems a little outside of it's scope.
This is kinda crazy to me personally when I found out resolvd was doing this.
https://www.man7.org/linux/man-pages/man8/systemd-resolved.s...
resolvd != systemd. It seems like you are mad that Ubuntu changed DNS resolver, but isn't that the actual job of a Linux distro?
1> in the systemd repo
2> if it's written and maintained by the same people who write systemd
3> if systemd is in the NAME
I don't really know what more is needed to say it a is part of systemd
Then you configure your DNS using NetworkManager, on respective connections. I.e. on your VPN, you configure the DNS on the other side of the connection, and (optionally) which zones it is supposed to resolve. (NetworkManager then configures systemd-resolved accordingly). The usage of this DNS then goes up and down together with your VPN connection. That's it.