I'm not a Linux expert, but I thought that systemd was generally hated?
Some reasons systemd is bad:
- Its a "big" for small docker containers (which is part of why a lot of people like Alpine Linux).
- It produces binary logs, which might not work with your workflow and is a kind of vendor lock-in if you don't script a binary-to-text script.
- the binaries are all a lot bigger in terms if SLOC and in terms of storage space than the solutions they replace. This makes auditing the code more of a chore and is a big argument against it in certain security environments.
- (my main gripe) systemd produces vendor lock-in from seemingly unrelated apps. A lot of packages which indirectly depend on a systemd functionality need to be patched to work on OpenRC Gentoo, for example.
Some arguments for systemd:
- systemd abstracts process management. This can be fantastic for scripting and security (again, context dependent)
- systemd by default kills services on log out (which is great for desktop - but can be an obstacle for servers). This is a good default behavior from a security perspective.
- systemd has some parallelization of init, making start up more predictable (because something falling over is less likely to nuke the whole procedure - again good for desktop)
- systemd provides a lot of tooling to stop you doing things the "wrong" way. For example, systemd-analyze-security provides configuration tips to harden your system
==========
In general, I would say that systemd is a very impressive tool and helped a lot of distributions standardise how they handle boot. I wish it did not impact downstream development the way it does though.
Aside size, last I looked it required a lot of hoop-jumping to get systemd to run inside docker at all.
Other software depending on Systemd isn't really Systemd responsibility. Its not systemd fault that almost nobody uses OpenRC.
Re: the binary logs - true, but the core point that its not text by default is still a (small) issue IMO. Not ideal default behaviour.
The nice thing about its binary log format is that it's organised by fields which are indexed for quicker searching and filtering. It's much easier and faster to analyse these logs than the traditional text-based ones.
Also, having journald authenticate the process that is sending it log entries, and the log sealing capability, are two features that can help guard against log tampering.
I think I am not the only one who is missing negative filters - check the log excluding audit messages.
- It's immediately compressed (without having to wait for a log rotation);
- It automatically rotates when a size limit is reached (no risk of filling the disk with logs);
- IIRC, it's deduplicated (repeated messages use less disk space).
All of these together means logs can be kept for much longer by default.
How do you do that? If you want non-binary logs, you can have journald tee log records to rsyslog; but the canonical log is still the binary log. Have I missed something?
Systemd will produce logs in its own structured representation and write them straight away as plain text in the logger of your choice. For all intent and purpose, you now have text logs.
Seems like a pointless complain to me. It’s arguing for the sake of it.
- It makes alternating between systems harder because so much is Linux + Systemd specific.
nohup ./long_batch_job.sh &
systemd broke that at some point, for no obvious reason. I think there is some nonstandard route to make systemd run something in the background, but end users shouldn’t need to talk to init.
Is there a guide that explains the bare minimum that typical end users need to know to stop systemd from breaking use cases that have worked for years? I would never expect a non-sysadmin to have read the “systemd-run” or “loginctl” manpages or know how to find them. I’m a nerd and even I don’t know how to get to the point of understanding what https://man7.org/linux/man-pages/man1/systemd-run.1.html is trying to tell me.
1. https://manpages.debian.org/stable/systemd/systemd-logind.8....
2. https://manpages.debian.org/bullseye/libsystemd-dev/sd-login...
That is bound to be a losing strategy. Systemd development (of breaking changes) is not stopping anytime soon. The simplest solution is to switch to an OS without systemd, like Devuan or Alpine or BSD.
That was proposed to nohup when systemd introduced the modification. It’s what nohup does on macOS.
A very productive discussion was happening but then the anti-systemd mob showed up with pitchforks, brigaded the discussion on the bug tracker and made sure nothing good would come out of it.
That’s when I personally decided Linux was a lost cause and definitely switched to macOS.
That is an strange course of action. So because you saw some low quality discussion on the internet, you switched from free software you can influence to commercial one you can't?
It’s not about low quality discussion. A significant part of the Linux community seems to me to be deeply toxic and that leads to what I consider to be suboptimal technical choices.
Right, good strategy.
> A significant part of the Linux community seems to me to be deeply toxic and that leads to what I consider to be suboptimal technical choices.
Do you mean toxic users influence some developers to make bad decisions? Any examples? Toxic users dumping on systemd may be one, but I don't think they have that much impact, systemd seems pretty successful in not giving away. Anyway systemd is a very specific case, this is not representative of all linux software projects.
https://superuser.com/questions/1372963/how-do-i-keep-system...
Systemd does a lot of good but they like to move fast and break things. Making tmpdir per process, their homedir changes etc are all good for security but tend to break things in unexpected ways
If you want to continue to run a process you have to start a service which is exactly that.
SIGHUP is not a signal to shut down, it's a signal that says the controlling terminal is gone. The action taken by the process at that point is the decision of the process and the user that launched it, not the init system. This is exactly what it is for.
Changing that without understanding what people used it for across UNIX vendors for decades, was dumb. And it showed off just how little the writers of the service knew about how users actually use their systems. Huge red flag.
It's not just random long-running batch jobs it killed either, it seems that it would kill all sorts of useful stuff people run for reliability. The irony of the init system killing screen/tmux sessions which people run specifically to keep their shell going in case of a disconnect!
Why do you assume they didn't know about this quite basic mechanism? They added 2 configuration options, one at compile time (to set the default) and another configurable by the user. In fact, distro maintainers didn't look up what it did and they just compiled it as is, resulting in the surprising behavior of killing lingering processes.
Also, it is an init and service management system and long-running processes are services. If you want to run something for "reliability" you should create a service for it, or just run the command with "systemd-run", to properly communicate with the responsible process manager what it supposed to do when the user logs out (and to actually make it reliable, e.g. you can then set it to auto-restart on failure, etc) Safely killing lingering processes and cleaning up after them is the correct default behavior, so systemd just tries to enforce that - they actually communicated with nohup maintainers to optionally add this systemd communication, so that "nohup whatever" would still work, but they were not open to it.
Even then, the default broke decades of standard behaviour so systemd failed users here.
> long-running processes are services.
Not all of them. A data processing/computation job running for tens or hundreds of hours is usually not a service. What is "long-running" anyway? Again, people used nohup for running their jobs for decades without being logged in. That is the standard usage of nohup, systemd broke it.
Paint it how you like, systemd broke decades-old, desirable behaviour for no good reason.
The attitude that everyone else is clearly wrong is unhelpful, and seems a red hat pathology. You see it with this and with Gnome, and it drives people away.
I never really understood this point — it’s not some proprietary format that has to be reverse engineered, you quite literally has both encrypt and decrypt source code available for every single version it has been released. You can also just pipe the binary output through the provided “decrypt” tool and then further use the usual unix tools if you wish.
But let’s add the advantages of this logging: systemd can log events from the very start of the boot process, which was not possible before.
From what I've seen using Alpine Linux for a few weeks on my Raspberry Pi, logs are written to /var/log/messages after the init process starts and launches the logging service. All logs before the init starts can be retrieved using dmesg? I'm not sure about this though, let me know if I'm wrong.
One of the things I haven't figured out yet is if traditional logging systems can easily do advanced log filtering like showing only logs from the current boot (like -b in systemd), previous boots (-b -1), and showing logs after a specific date and time (--since).
I manage this with clever usage of grep. You are correct in that there isn't a single --flag that will only show me those specifics.
I can grep my way through text logs as well but being able to get logs for different purposes using --flags is better user experience. I can always resort to to using grep, sed, and awk if I want to when using journald but the loss of these quality of life features make it hard for me to consider using a distro that does not have systemd.
People should just call systemd logs structured rather than binary because that’s what they are. Sure it means you need a shim if you want to view them as plain text but it’s not like we are talking about some unfathomable complexity here.
It's usually a signal that the person is not a practioner.
Systemd is the most important and well-developed Linux framework, besides the Linux kernel itself.
I'm not going to say I'm never guilty of it myself, but often asking even for the slightest bit of further explanation collapses that or otherwise just leads to repetition.
Systemd is really well designed and makes Linux workstation use predictable and more secure across distros in numerous ways. You can have any service you want now, even xorg/ wayland, as an unprivileged user. It basically removes all need to use root if used correctly.
Systemd however is however not designed for server use, but neither is OpenRC or any of the alternatives. Servers filesystems should contain a kernel, a shim init binary, and a static binary for a target service.
When you want to upgrade your fleet you compile a new bundle of those three things into a new filesystem image and boot servers from the new image.
So like containers, but heavier weight and you need to pre-allocate ram for every ~~process~~ kernel?
It sounds like you've reinvented the multiprocessing operating system.
The sandboxing is a convoluted mess few even know where to start learning to use. Lets say you successfully configure process namespacing, network namespacing, filesystem namespacing, cgroups, and apparmor or selinux all correctly to restrict every last syscall. You will still get burned by a crash, resource exhaustion, or security breach due to the next of a long series of implementation bugs in each of these arcane features.
Humans cannot create a large codebase of coherent and provably correct C code with memory safety and they really should stop trying.
Hypervisors are the only sandbox that works reasonably well and burning some extra ram is cheaper than most security breaches.
It sometimes randomly fails to boot, waiting for some strange condition, while I am not able to intervene via keyboard in any way. That is "well designed"?
I run Xorg as unprivileged user by executing startx, on sysvinit system. There is no problem that needs systemd solution here.
On a laptop things are much more dynamic. From sleep/hibernation/wifi-LAN-wifi (+/- VPN) switching/etc.... these fiddly bits were much harder to manage on Linux for DECADES then they are now. The reboot interval on laptops is huge compared to servers. An init system (plus all the extra plumbing) work well together.
On servers however; I HATE systemd. For logging, networking (and DNS) alone systemd wastes so much more of my time and gets in my way. I think the need for an 'improved' init system was always a red herring. I've never had a service race-condition, or other conflict that didn't take more then 10 minutes to fix in the 30 years I've managed *nix servers. There was one time where I was stumped why my VPN server kept failing to start at boot but came up fine if I restarted manually. My dyslexic ass inverted the script number (ie. 13 instead of 31 or something like that). That was just a face-palm embarrassment.
Production systems should never be changed or administered in place, only replaced. Servers should be immutable appliances. They do not need a package manager, or systemd, or ssh, or even a shell. Such things are developer tools and only belong in development environments like workstations.
A production filesystem should contain a kernel, an init binary, and either a container runtime or a static binary of your target application. At most a production init binary just needs to setup virtual filesystems and be a reaper for a target application binary.
That is a very restrictive work methodology (unikernels,exokernels). Although interesting and certainly useful in some cases (high security, rare updates), Bryan Cantrill was partially right on this - in the real world, we often need to debug stuff running in production and support paying customers who require that. Your idea of "production systems" and "servers" seems very specific and the "should" statements are often not true in practice.
Developer tools definitely belong on the server too, you need them to determine how to build your stack binaries in the best way for the OS and HW architecture there and keep them updated and secure.
If it was generally hated there would be a lot more support for the distros that don't have systemd. There isn't. Except for alpine, all of them are extremely niche, half of them are dead, the other half barely have enough people to stick around for a release a year. Even alpine is kinda niche, it's mostly used as a way to make lightweight docker containers, rather than as a distro that stands on its own, the fact it uses musl as its libc means you can't use it on a server that has nvidia gpus for machine learning, you can't use it on a desktop where you need a browser capable of DRM, you can't use it on a personal computer if you ever intend to install a video game etc.
I have -yet- to hear anyone in my life actually use something like Devuan, the systemd-less fork of Debian, in a production environment. Six years after their first release, instead of standing on their own as a distribution, they're still deeply angry and obsessed with systemd and this is the level of professionalism they exhibit on social media : https://twitter.com/DevuanOrg/status/1586963662295687169
Of course, one of the twitter comments underneath is "systemd macht frei".
One of the things the official devuan account retweets : https://twitter.com/jaromil/status/1544618996833583104 "It's 2022 and the master of systemd is now working for Micro$oft https://phoronix.com/scan.php?page=news_item&px=Lennart-Poet... this the last bit @phoronix omits. This is a classic cybernetic strategy: to become the master of a feedback loop you create and continuously adjust the framework in which nodes interact."
I think that says enough about the sort of irrational people we're dealing with?
People who hate systemd are very, very vocal, and highly unproductive.
Microsoft hasn't been the competition for while. They might not be "part of the team", but they definitely have more to win by linux being alive than dead. I'd put them more along the lines of IBM or Oracle, huge corps trying to somewhat discreetly steer linux in a path that suits them.
FreeBSD & OpenBSD would happily disagree.
The Devuan founder is truly the most impressive person in the universe.
"The most viewed research thesis of all time published by Univ. Plymouth pearl.plymouth.ac.uk/handle/… by Devuan's founder @jaromil" https://twitter.com/DevuanOrg/status/1587168838071816193
My personal opinion is that proper dependency based init systems are amazing but I'm not fond of some slightly superficial aspect of systemd (syntax, parameters, logic, config files).. to the point I think in a few years someone may make a cleaner variant.
Meanwhile millions are using it to get shit done without much drama. It’s been years since I filed a bug on systemd, but that’s what grown ups do when inevitable bugs are encountered, not make themselves and everyone miserable because they deeply integrated some dubious 1970s Unix philosophy into their personality.
When pipewire replaced pulse, despite pulse being well established and a core desktop component, there was very little churn and few bugs. People not impacted did not mind at all that a major component was changed, or that it did many things and bundled three completely different protocols into one, arguably not very reminiscent of Unix Philosophy™
I would confidently predict that a change as large as systemd, to a component as old, and that causes as much churn (rewriting, relearning) would follow a roughly similar hate curve, only distinguishing itself by whether it was more buggy or less buggy than the predecessor.
It's churn people hate, not change. The lesson I learned is that you can make large changes to the system, it's only hated when it forces humans changes.
I ran systemd very early on because it gave me some things I liked from SMF but I completely understand why so many people hated it. It's a (suite of) program(s) with good ideas that was hampered by extremely poor maintainership and relation with other open source projects.
I've been using Linux since Slackware installed from a pile of floppy disks. I'm extremely happy with systemd (desktop and server) and view it as one of the most important UX advancements ever made in the Linux sysadmin space. It replaced a smorgasbord of broken nonsense with a unified and thoughtful system that is objectively superior to what it replaced. I suspect a lot of the extreme reaction to systemd is not just resistance to change, but based on an emotional attachment to the rag-tag heterogeneity it made obsolete.
Setting common sig handlers, restart on exit, etc. are just one-liners with unit files. In the previous one, SysV, it was an annoying incantation especially if you were a newbie, like me at the time, or product engineer who was dabbling. You could easily screw it up. Some non infra engineers cursed the SysV files and loved the SystemD unit files.
Now I use dynamic user and the other security features in my unit files and wonder how many lines it would have taken to do that in SysV, so the complaints are laughable at this point.
I have trouble believing that people who loath it so much have actually sat down to read the documentation.
It might be sarcasm?