- Unified logging subsystem
- On-demand launching, memory limits, and general resource management
- Advanced security features, i.e. sandboxing, DynamicUser
- Well thought out and implemented given the constraints (feels like Linux quality)
In general it represents a shifting of code from daemon developers (who previously handled daemonization themselves) to the system.
On-demand launching is good for performance: work can be deferred from boot and inactive daemons can be stopped. This is critical on battery and RAM constrained devices
Unless you have some particular attachment to the NT kernel, there near zero reasons to use Windows today.
If you use a computer for actual industrial work, you will hit one of these problems very quickly.
The worst offenders are "industry standard" software. 3ds Max. Ableton. Photoshop. Altium. MS Office. Visual Studio. Go check WineDB and see how many versions of these are rated "Garbage".
I say again - if you attempt to rely on WINE to do real industrial work on Linux using Windows tools, you are virtually guaranteed to hit a showstopper somewhere almost immediately. It's just not viable.
Linux developers can often do wonders, but they still do not have a crystal ball. If that hardware is niche+closed+expensive I don't see how they can produce working drivers or software. Manufacturers not releasing documentation are the problem, not Linux, or BSD or any other non supported OS.
Indeed. They can't. Which is why Linux can't drive that stuff.
I fully agree with your assessment of whose fault it is. That wasn't the question.
[0] and of course the efforts of the WINE project and proton.
Don't succumb to fanboyism.
> you should notify resolved of the domains that LXD can resolve
You'd have the same issues if you replaced systemd-resolved with e.g. dnsmasq. Split DNS always needs resolver configuration.
For people using dhcp, yes. The surprise of systemd-resolved was it taking over the file on a typical server that didn't use dhcp.
It could use with some more netns awareness but its pretty great at the current stage
It lets you configure DNS Server per link and resolve only certain domains via a specific DNS Server configured from a link.
e.g. multiple customer specific wireguard Tunnels that resolve the specific project related domains, but dont resolve all your other DNS.
I keep wondering about this year of the Linux Desktop. So often I see it used as a snide towards Linux users (not by you). I think it shows naivety.
What does it mean? Is it about having a stable easy to maintain and full featured Linux Desktop. I really do think we have reached that point. Especially now that many workflows have moved to the web I am having a hard time coming up with use cases for which Linux would really not be suited. Even gaming has become very viable.
Or does it mean that Linux for overtake the market share of Windows and macOS. For that I would caution to be careful about what you wish for. Being an underdog brings an added benefit of having more flexibility and the option to think outside of the box. For Linux to overtake Windows it would slowly turn into Windows itself in my opinion.
Damning with faint praise. The success of Linux gaming is due to the efforts of projects that re-implement Windows APIs, and things moving to the web are obviously not targeting Linux either.
I have trouble believing that people who loath it so much have actually sat down to read the documentation.
It might be sarcasm?
Some reasons systemd is bad:
- Its a "big" for small docker containers (which is part of why a lot of people like Alpine Linux).
- It produces binary logs, which might not work with your workflow and is a kind of vendor lock-in if you don't script a binary-to-text script.
- the binaries are all a lot bigger in terms if SLOC and in terms of storage space than the solutions they replace. This makes auditing the code more of a chore and is a big argument against it in certain security environments.
- (my main gripe) systemd produces vendor lock-in from seemingly unrelated apps. A lot of packages which indirectly depend on a systemd functionality need to be patched to work on OpenRC Gentoo, for example.
Some arguments for systemd:
- systemd abstracts process management. This can be fantastic for scripting and security (again, context dependent)
- systemd by default kills services on log out (which is great for desktop - but can be an obstacle for servers). This is a good default behavior from a security perspective.
- systemd has some parallelization of init, making start up more predictable (because something falling over is less likely to nuke the whole procedure - again good for desktop)
- systemd provides a lot of tooling to stop you doing things the "wrong" way. For example, systemd-analyze-security provides configuration tips to harden your system
==========
In general, I would say that systemd is a very impressive tool and helped a lot of distributions standardise how they handle boot. I wish it did not impact downstream development the way it does though.
Aside size, last I looked it required a lot of hoop-jumping to get systemd to run inside docker at all.
Other software depending on Systemd isn't really Systemd responsibility. Its not systemd fault that almost nobody uses OpenRC.
Re: the binary logs - true, but the core point that its not text by default is still a (small) issue IMO. Not ideal default behaviour.
The nice thing about its binary log format is that it's organised by fields which are indexed for quicker searching and filtering. It's much easier and faster to analyse these logs than the traditional text-based ones.
Also, having journald authenticate the process that is sending it log entries, and the log sealing capability, are two features that can help guard against log tampering.
I think I am not the only one who is missing negative filters - check the log excluding audit messages.
- It's immediately compressed (without having to wait for a log rotation);
- It automatically rotates when a size limit is reached (no risk of filling the disk with logs);
- IIRC, it's deduplicated (repeated messages use less disk space).
All of these together means logs can be kept for much longer by default.
How do you do that? If you want non-binary logs, you can have journald tee log records to rsyslog; but the canonical log is still the binary log. Have I missed something?
Systemd will produce logs in its own structured representation and write them straight away as plain text in the logger of your choice. For all intent and purpose, you now have text logs.
Seems like a pointless complain to me. It’s arguing for the sake of it.
- It makes alternating between systems harder because so much is Linux + Systemd specific.
nohup ./long_batch_job.sh &
systemd broke that at some point, for no obvious reason. I think there is some nonstandard route to make systemd run something in the background, but end users shouldn’t need to talk to init.
Is there a guide that explains the bare minimum that typical end users need to know to stop systemd from breaking use cases that have worked for years? I would never expect a non-sysadmin to have read the “systemd-run” or “loginctl” manpages or know how to find them. I’m a nerd and even I don’t know how to get to the point of understanding what https://man7.org/linux/man-pages/man1/systemd-run.1.html is trying to tell me.
1. https://manpages.debian.org/stable/systemd/systemd-logind.8....
2. https://manpages.debian.org/bullseye/libsystemd-dev/sd-login...
That is bound to be a losing strategy. Systemd development (of breaking changes) is not stopping anytime soon. The simplest solution is to switch to an OS without systemd, like Devuan or Alpine or BSD.
That was proposed to nohup when systemd introduced the modification. It’s what nohup does on macOS.
A very productive discussion was happening but then the anti-systemd mob showed up with pitchforks, brigaded the discussion on the bug tracker and made sure nothing good would come out of it.
That’s when I personally decided Linux was a lost cause and definitely switched to macOS.
That is an strange course of action. So because you saw some low quality discussion on the internet, you switched from free software you can influence to commercial one you can't?
It’s not about low quality discussion. A significant part of the Linux community seems to me to be deeply toxic and that leads to what I consider to be suboptimal technical choices.
Right, good strategy.
> A significant part of the Linux community seems to me to be deeply toxic and that leads to what I consider to be suboptimal technical choices.
Do you mean toxic users influence some developers to make bad decisions? Any examples? Toxic users dumping on systemd may be one, but I don't think they have that much impact, systemd seems pretty successful in not giving away. Anyway systemd is a very specific case, this is not representative of all linux software projects.
https://superuser.com/questions/1372963/how-do-i-keep-system...
Systemd does a lot of good but they like to move fast and break things. Making tmpdir per process, their homedir changes etc are all good for security but tend to break things in unexpected ways
If you want to continue to run a process you have to start a service which is exactly that.
SIGHUP is not a signal to shut down, it's a signal that says the controlling terminal is gone. The action taken by the process at that point is the decision of the process and the user that launched it, not the init system. This is exactly what it is for.
Changing that without understanding what people used it for across UNIX vendors for decades, was dumb. And it showed off just how little the writers of the service knew about how users actually use their systems. Huge red flag.
It's not just random long-running batch jobs it killed either, it seems that it would kill all sorts of useful stuff people run for reliability. The irony of the init system killing screen/tmux sessions which people run specifically to keep their shell going in case of a disconnect!
Why do you assume they didn't know about this quite basic mechanism? They added 2 configuration options, one at compile time (to set the default) and another configurable by the user. In fact, distro maintainers didn't look up what it did and they just compiled it as is, resulting in the surprising behavior of killing lingering processes.
Also, it is an init and service management system and long-running processes are services. If you want to run something for "reliability" you should create a service for it, or just run the command with "systemd-run", to properly communicate with the responsible process manager what it supposed to do when the user logs out (and to actually make it reliable, e.g. you can then set it to auto-restart on failure, etc) Safely killing lingering processes and cleaning up after them is the correct default behavior, so systemd just tries to enforce that - they actually communicated with nohup maintainers to optionally add this systemd communication, so that "nohup whatever" would still work, but they were not open to it.
Even then, the default broke decades of standard behaviour so systemd failed users here.
> long-running processes are services.
Not all of them. A data processing/computation job running for tens or hundreds of hours is usually not a service. What is "long-running" anyway? Again, people used nohup for running their jobs for decades without being logged in. That is the standard usage of nohup, systemd broke it.
Paint it how you like, systemd broke decades-old, desirable behaviour for no good reason.
The attitude that everyone else is clearly wrong is unhelpful, and seems a red hat pathology. You see it with this and with Gnome, and it drives people away.
I never really understood this point — it’s not some proprietary format that has to be reverse engineered, you quite literally has both encrypt and decrypt source code available for every single version it has been released. You can also just pipe the binary output through the provided “decrypt” tool and then further use the usual unix tools if you wish.
But let’s add the advantages of this logging: systemd can log events from the very start of the boot process, which was not possible before.
From what I've seen using Alpine Linux for a few weeks on my Raspberry Pi, logs are written to /var/log/messages after the init process starts and launches the logging service. All logs before the init starts can be retrieved using dmesg? I'm not sure about this though, let me know if I'm wrong.
One of the things I haven't figured out yet is if traditional logging systems can easily do advanced log filtering like showing only logs from the current boot (like -b in systemd), previous boots (-b -1), and showing logs after a specific date and time (--since).
I manage this with clever usage of grep. You are correct in that there isn't a single --flag that will only show me those specifics.
I can grep my way through text logs as well but being able to get logs for different purposes using --flags is better user experience. I can always resort to to using grep, sed, and awk if I want to when using journald but the loss of these quality of life features make it hard for me to consider using a distro that does not have systemd.
People should just call systemd logs structured rather than binary because that’s what they are. Sure it means you need a shim if you want to view them as plain text but it’s not like we are talking about some unfathomable complexity here.
It's usually a signal that the person is not a practioner.
Systemd is the most important and well-developed Linux framework, besides the Linux kernel itself.
I'm not going to say I'm never guilty of it myself, but often asking even for the slightest bit of further explanation collapses that or otherwise just leads to repetition.
Systemd is really well designed and makes Linux workstation use predictable and more secure across distros in numerous ways. You can have any service you want now, even xorg/ wayland, as an unprivileged user. It basically removes all need to use root if used correctly.
Systemd however is however not designed for server use, but neither is OpenRC or any of the alternatives. Servers filesystems should contain a kernel, a shim init binary, and a static binary for a target service.
When you want to upgrade your fleet you compile a new bundle of those three things into a new filesystem image and boot servers from the new image.
So like containers, but heavier weight and you need to pre-allocate ram for every ~~process~~ kernel?
It sounds like you've reinvented the multiprocessing operating system.
The sandboxing is a convoluted mess few even know where to start learning to use. Lets say you successfully configure process namespacing, network namespacing, filesystem namespacing, cgroups, and apparmor or selinux all correctly to restrict every last syscall. You will still get burned by a crash, resource exhaustion, or security breach due to the next of a long series of implementation bugs in each of these arcane features.
Humans cannot create a large codebase of coherent and provably correct C code with memory safety and they really should stop trying.
Hypervisors are the only sandbox that works reasonably well and burning some extra ram is cheaper than most security breaches.
It sometimes randomly fails to boot, waiting for some strange condition, while I am not able to intervene via keyboard in any way. That is "well designed"?
I run Xorg as unprivileged user by executing startx, on sysvinit system. There is no problem that needs systemd solution here.
Meanwhile millions are using it to get shit done without much drama. It’s been years since I filed a bug on systemd, but that’s what grown ups do when inevitable bugs are encountered, not make themselves and everyone miserable because they deeply integrated some dubious 1970s Unix philosophy into their personality.
If it was generally hated there would be a lot more support for the distros that don't have systemd. There isn't. Except for alpine, all of them are extremely niche, half of them are dead, the other half barely have enough people to stick around for a release a year. Even alpine is kinda niche, it's mostly used as a way to make lightweight docker containers, rather than as a distro that stands on its own, the fact it uses musl as its libc means you can't use it on a server that has nvidia gpus for machine learning, you can't use it on a desktop where you need a browser capable of DRM, you can't use it on a personal computer if you ever intend to install a video game etc.
I have -yet- to hear anyone in my life actually use something like Devuan, the systemd-less fork of Debian, in a production environment. Six years after their first release, instead of standing on their own as a distribution, they're still deeply angry and obsessed with systemd and this is the level of professionalism they exhibit on social media : https://twitter.com/DevuanOrg/status/1586963662295687169
Of course, one of the twitter comments underneath is "systemd macht frei".
One of the things the official devuan account retweets : https://twitter.com/jaromil/status/1544618996833583104 "It's 2022 and the master of systemd is now working for Micro$oft https://phoronix.com/scan.php?page=news_item&px=Lennart-Poet... this the last bit @phoronix omits. This is a classic cybernetic strategy: to become the master of a feedback loop you create and continuously adjust the framework in which nodes interact."
I think that says enough about the sort of irrational people we're dealing with?
People who hate systemd are very, very vocal, and highly unproductive.
Microsoft hasn't been the competition for while. They might not be "part of the team", but they definitely have more to win by linux being alive than dead. I'd put them more along the lines of IBM or Oracle, huge corps trying to somewhat discreetly steer linux in a path that suits them.
FreeBSD & OpenBSD would happily disagree.
The Devuan founder is truly the most impressive person in the universe.
"The most viewed research thesis of all time published by Univ. Plymouth pearl.plymouth.ac.uk/handle/… by Devuan's founder @jaromil" https://twitter.com/DevuanOrg/status/1587168838071816193
My personal opinion is that proper dependency based init systems are amazing but I'm not fond of some slightly superficial aspect of systemd (syntax, parameters, logic, config files).. to the point I think in a few years someone may make a cleaner variant.
When pipewire replaced pulse, despite pulse being well established and a core desktop component, there was very little churn and few bugs. People not impacted did not mind at all that a major component was changed, or that it did many things and bundled three completely different protocols into one, arguably not very reminiscent of Unix Philosophy™
I would confidently predict that a change as large as systemd, to a component as old, and that causes as much churn (rewriting, relearning) would follow a roughly similar hate curve, only distinguishing itself by whether it was more buggy or less buggy than the predecessor.
It's churn people hate, not change. The lesson I learned is that you can make large changes to the system, it's only hated when it forces humans changes.
I ran systemd very early on because it gave me some things I liked from SMF but I completely understand why so many people hated it. It's a (suite of) program(s) with good ideas that was hampered by extremely poor maintainership and relation with other open source projects.
I've been using Linux since Slackware installed from a pile of floppy disks. I'm extremely happy with systemd (desktop and server) and view it as one of the most important UX advancements ever made in the Linux sysadmin space. It replaced a smorgasbord of broken nonsense with a unified and thoughtful system that is objectively superior to what it replaced. I suspect a lot of the extreme reaction to systemd is not just resistance to change, but based on an emotional attachment to the rag-tag heterogeneity it made obsolete.
On a laptop things are much more dynamic. From sleep/hibernation/wifi-LAN-wifi (+/- VPN) switching/etc.... these fiddly bits were much harder to manage on Linux for DECADES then they are now. The reboot interval on laptops is huge compared to servers. An init system (plus all the extra plumbing) work well together.
On servers however; I HATE systemd. For logging, networking (and DNS) alone systemd wastes so much more of my time and gets in my way. I think the need for an 'improved' init system was always a red herring. I've never had a service race-condition, or other conflict that didn't take more then 10 minutes to fix in the 30 years I've managed *nix servers. There was one time where I was stumped why my VPN server kept failing to start at boot but came up fine if I restarted manually. My dyslexic ass inverted the script number (ie. 13 instead of 31 or something like that). That was just a face-palm embarrassment.
Production systems should never be changed or administered in place, only replaced. Servers should be immutable appliances. They do not need a package manager, or systemd, or ssh, or even a shell. Such things are developer tools and only belong in development environments like workstations.
A production filesystem should contain a kernel, an init binary, and either a container runtime or a static binary of your target application. At most a production init binary just needs to setup virtual filesystems and be a reaper for a target application binary.
That is a very restrictive work methodology (unikernels,exokernels). Although interesting and certainly useful in some cases (high security, rare updates), Bryan Cantrill was partially right on this - in the real world, we often need to debug stuff running in production and support paying customers who require that. Your idea of "production systems" and "servers" seems very specific and the "should" statements are often not true in practice.
Developer tools definitely belong on the server too, you need them to determine how to build your stack binaries in the best way for the OS and HW architecture there and keep them updated and secure.
Setting common sig handlers, restart on exit, etc. are just one-liners with unit files. In the previous one, SysV, it was an annoying incantation especially if you were a newbie, like me at the time, or product engineer who was dabbling. You could easily screw it up. Some non infra engineers cursed the SysV files and loved the SystemD unit files.
Now I use dynamic user and the other security features in my unit files and wonder how many lines it would have taken to do that in SysV, so the complaints are laughable at this point.
Well, maybe you would say Alpine is not intented to run heavyweight stuff like that and is mostly focused on security and containers.
That's exactly what Alpine is, the OpenBSD of Linux distros. OpenBSD likewise cannot be used as your production powerhouse as well.
Alpine runs musl libc. This makes it buggy with all sorts of software that rely on glibc-isms, that makes it incompatible with proprietary drivers like NVIDIA and make it harder to run proprietary/binary software on the userland.
musl is developed by people with that sort of break-everything attitude : https://twitter.com/RichFelker/status/994629795551031296
There is a world of pain that awaits anyone with the expectation that this sort of distro could be used on a workstation.
Musl is really meant to be run.. with software vetted by communities like https://suckless.org/
Meaning things like dwm, dmenu, st.
Which may or may not be one's cup of tea. It certainly ain't mine.
The glibc 2.26 release with the ucontext breakages made me stick with musl.
Plus, when i did multiarch support in Alpine later, i found that LDSO paths for musl are much more systematically and don't have name collisions like glibc.
There are some very unfortunate ideological choice made by its author but they are not the one you quote. The reluctance to give developers a way to detect Musl from C code would be a better exemple. It’s both annoying and counterproductive.
> Without systemd, bootstrapping a complicated desktop environment would be a hot mess
startx on tty1, i3 being launched from xinitrc. Same procedure that i use on any other distro or BSD.
What is a complicated desktop environment? Why should I run it? I want to run simple desktop environment, like Xfce. I run startx in tty, and get unprivileged Xorg with Xfce on sysvinit system. It's rock solid.
I've considered it a lot, but I'm still to attached to service activations and a few other features from systemd --user.
I've started managing my user services using systemd --user and portable services and it's just amazing how well the individual components work together to create fully isolated, containerized, per-user services.
I'm lovin' it and I'm not missing the hot mess of the dozens of shell scripts that I needed before.
In short it allows you to bundle your application in one file and have it run in a sandbox. Like a docker container, but without docker, managed by systemd.
https://wiki.archlinux.org/title/Systemd/User
User services are also a systemd thing; you can create systemd unit files (service descriptions) and have them run as a user. So as a user you can manage your own running processes using the same systemctl interface as system processes. If you are allowed to "linger" on a system, you can run user processes as a service, set them up and tear them down without root permissions.
I have several systems where I run i.e. webservices without having full root access. I build an portable systems image, deploy it to the server and then can manage that service with my unprivileged user account.
Systemd does all the usual things, like logging, supervision, resource quotas, and so on.
Right now my mom is nagging me to fix her Ubuntu laptop, which doesn't shutdown anymore because some of the units got messed up. I'm at the verge of telling her to install windows.