Why Did ArchLinux Embrace Systemd? (2016)
reddit.com
reddit.com
I'm not sure it's necessarily a better solution as it requires more manual interaction out of the box (namely creating symlinks by hand among other duties), but that might be argued as a feature. There appears to be a wrapper or two out there to automate away some of the tedium.
runit, s6 and also nosh are all daemontool-style init systems, which have a somewhat different idea how to do things than systemd all, but all are inspired by the original daemontools package.
s6 and nosh have decent overviews:
https://skarnet.org/software/s6/ https://jdebp.eu/Softwares/nosh/guide/introduction.html
The original daemontools: https://cr.yp.to/daemontools/faq/create.html#why
I'll let others decide whether these are better or worse than systemd. I go with whatever my distro would give me, since picking a different init system is too much work...
I still dont understand what issues are fixed with systemd and I am aware of many it has created.
The real issue isn’t with servers, it’s with desktop systems and laptops. When users plug in a USB sound card or plug their laptop into a dock, they don’t want to (and likely couldn’t) “easily script around” that; they expect an audio device, network connection, video card to be created without any user intervention, and go away when devices get unplugged.
Making a set of init scripts (and surrounding unload and reload scripts) robust against insertion and removal of any of many different devices isn’t easy even for init script gurus, and would lead to duplicated code for detecting and handling the various load/unload conditions.
That, I think is the primary reason Apple went with launchd. Systemd took that idea. There’s nothing wrong with that.
The main issues with systemd, IMO are a) initially it was buggy, and b) tremendous scope creep.
Bingo !
Folks who know, and I mean really understand how to run servers know how to script init.
Folks who use Linux on the laptop / Admin Ubuntu do not and systemd makes them able to do stuff "most of the time"
There is no reason to force systemd as the default for servers. You dont plug things into servers and in fact you dont want some service monitoring servers so someone can stick something in a server and start rooting around it.
Also you seem to be conflating dealing with devices with initialization of the system. Dealing with such devices has insofar as I'm aware always been handled by c code in and out of the kernel whereas at one time initialization of services and orchestration of components was handled by a scripting language.
This is to say that a bash script may have brought up different components and services but that c code that makes up the component is in charge of actually managing said device.
Its obviously feasible for both c (or any other language) code and orchestration of start up and shut down of services to be buggy but its not equally easy for end users to fix their own problems in all cases.
Ideally bugs get reported to those who are most capable and the underlying code is corrected however if said labor isn't paid to work on their project hours to do so may be in short supply and it may frequently fall on users to lend a hand.
This is a numbers game. If you have 10000 users and 10 of them are qualified to solve such problems you may be lucky to get the help of 1 to submit patches or post a solution to a problem on a forums for others to use before the upstream problem is fixed.
If 100 are qualified you are 10x as likely to find help to solve a problem. I have on many occasions come across a solution months or years before an official upstream solution came into being. You are suggesting that most sysadmins might find it challenging to solve their own problems. They don't always have to. Merely having comprehensible user friendly systems make it more likely that one of their fellows can help them do so and more likely that such solutions travel both laterally between users and hopefully upstream solving the problem for all.
The good bits that systemd brought into the mainstream such as cgroups and lxc do not require systemd. They can be implemented independently. The watchdog timers and unit files were not a new concept either and did not require systemd.
You can still use cgroups independently with systemd, but it's recommended that you use a service and the delegate option to have systemd create a cgroup hierarchy and also know that it shouldn't mess with it after creating it. This lets you have your cake and eat it too if you have a case that systemd doesn't handle (I'm working on a setup with this myself for a coding sandbox setup).
This is conflating the systemd project and they systemd init system. Most of the things you mention are seperate systems (systemd-resolved,logind, polkit, etc.) that are part of the systemd project and interoperate with the systemd init system, but are not part of the init system itself.
I use systemd across distros, and am now confused when I run into an old SysV system.
When I'm sufficiently advanced to bump into one of these corner cases, it will be a Good Problem To Have.
The market penetration of systemd is about as good an empirical validation as can be had. How soon until it's codified in Posix?
I'll readily admit that systemd isn't perfect, but my overall experience is that it is better in many ways to its predecessors, and did solve real problems.
And when I referred to "systemd hate" I was not referring to healthy criticism, but to those who take very chance to claim that systemd never should have been invented and their init system of choice is clearly superior. Rather than, say trying to make systemd better, or make their init system fix the problems that systemd was designed to remedy.
I had it silently take over my DNS . I even have the 'official' way check-boxed to not override DNS settings in Network Manager. Does so anyway. I even removed the symlink from /etc/resolv.conf and made it root:root . SystemD changed it next reboot.
And it's way more documented that your average init script sourcing distro-specific bash/sh libraries where you have to dig into and read the code just to understand how that thing works
Init scripts were undebuggable, un-understanable, documentationless kluge with text logs, and systemd has a perfectly functional format converter. I'm not seeing the regression here.
1) Init scripts varied by implementation very much, I have managed to successfully debug, make, read and otherwise modify/parse init scripts from basically every _major_ distro including OpenBSD and FreeBSD.
I am not particularly intelligent, so if I can do it, you can.
2) There is no reason to assume we go back to bash scripts per-se, there is a huge middle ground between a huge C program which execve()'s programs and watches them live, parses their output, logs it, enables socket activation, keeps an in memory DAG that it determines randomly on boot.... and bash scripts.
Good for you.
> And yes people do use Linux on the desktop
Good for them.
But the fact remains that this post still ignores why binary logs are bad for the server use case. They seem great for me in that role.
I think part of it actually stems from people that had problems early on with PulseAudio, and have developed a vendetta against Poettering. Whether that vendetta is justified or not, I don't know.
Personally systemd hasn't been a pain point. It's been far less of a headache than everyone at the office thought it would be.
SystemD is much the same way, I'm coloured by my impressions from Fedora21-22, where it.. didn't work great.
The issue is with poetterware, it's _incredibly_ difficult to discover /why/ it's not working properly, which is dangerous for an init system (or, an "everything between your application and kernel"-system), a failure rate of 0.0001% is still an unfathomably enormous number of systems; and due to how it works it lends itself to an incredible host of threading issues.
Hypothesis: C is so pathologically bad that it breaks brains.
As a sysadmin I wrote a small kernel module to trigger a panic on a certain condition hoping to capture some info in a dump several years ago.
Today I cant find sysadmins who can use a CLI properly.
The goalposts have moved inwards IMO
Not sure its bad thing either but it makes for interesting comparisons
These days, people calling themselves professional developers with years of experience won't even touch Emacs.
I picked it up several months ago after using Jetbrains IDE's for quite a long time, and honestly haven't felt the need for any of what Jetbrains brought to the table.
Mostly thanks to LSP.
If you're not a Java developer, Emacs is perfectly fine as a configurable extensible editor. With the additional caveat that its lack of multithreading can get in the way sometimes.
Until very recently having wrong permission on one of the networkd config files would result in networkd ignoring the rest of the perfectly readable network configuration files too, cutting off all network connectivity. (Systemd also requested the permission change iself.)
Currently there's a bug where if you have a bridge and all interfaces are disconnected from the bridge (I use hot-pluggable USB network interfaces) when you re-connect them, networkd will keep the bridge off. If you restart systemd-networkd it will fix itself.
I can name at least two instances already where I lost networking in my entire SBC farm, and had to pop out all microSD cards and fix the breakage manually, just because of systemd-networkd bugs. And I don't do anything crazy, just a single ethernet interface and two overlay wireguard networks, and mostly default config.
And fixing it yourself is not easy. I can do C coding perfectly fine, but I run 3 different CPU archs, so fixing it myself means compiling (3 times) and distributing my own systemd version to all my machines until upstream fixes the bug. It's not pleasant, nor easy.
I certainly understand people who miss having an ability to more readily script the system startup/network config.
> complex corner-cases that only a a very tech-savvy user could generate in the first place
I agree that an init system should work on complex corner-cases, but systemd remains popular because it works for most people most of the time.
Tangent: have you found an init system that you like the most? I've only used four or five init systems and don't have any super strong opinions -- maybe because I'm not running and SBC farm. :~)
I like systemd and use it everywhere (I use Arch Linux / Arch Linux ARM mostly). I just don't think it needs to be painted all pink as "Just Works[tm]" software.
I'm not sure why it did that, as I didn't have time to dig into the why while trying to deal with the result.
However, after trying SystemD for one week, I became convinced that the designers of SystemD are incompetent, so they cannot be trusted with a component of such importance for a computer.
I am normally a Gentoo user, but a few years ago I wanted to install Linux in a hurry on a small computer. With a slow CPU installing Gentoo can take several hours, unless you have a previously prepared image, so I decided to try ArchLinux.
The installation was indeed fast and without problems. However, SystemD behaved in an unexpected way. SystemD is advertised as booting quickly. It indeed booted quickly, but no faster than my optimized Gentoo systems.
On the other hand the shutdown with SystemD was very slow. This was a huge surprise, because never before and never after have I encountered any computer where the shutdown is annoyingly slow.
Moreover, sometimes the shutdown FAILED, which is also something that I have never encountered on any computer not using SystemD, and I have used thousands of computers, with many kinds of operating systems, including at least eight or nine UNIX flavors other than Linux.
The failure of shutdown was apparently due to some kind of race condition when some process was killed faster than some SystemD component expected and that SystemD component was still trying to communicate via DBUS with the defunct process and it blocked because its messages no longer reached the destination.
I suppose that this bug might have been corrected eventually, but this is a design error that I consider unforgivable, i.e. to conceive a shutdown sequence which depends on successful inter-process communication.
After seeing such a mistake I have lost any confidence in the abilities of the SystemD developers, so I have never tried it again.
If I had to guess, it's probably because I've never ran into any of the weird systemd problems that I see people report sometimes. For me, it's always pretty much just worked, and more often than not it natively has the functionality that I want.
My largest complaint is that I have to bounce across several manpages to write a comprehensive unit file.
Traditional RC does a SIGTERM to everything, and then waits a bit and then does a SIGKILL to everything. This means that if the SIGTERM failed to be handled properly, progress is still made.
Weird as in 3 minutes after someone SSHd into my workstation, X11 restarted. Until I tracked it down to a default systemd configuration I thought I was going insane.
Beyond the default configuration, which is somewhat defensible as NIS is probably not a typical case, it really bothers me that if someone else's login process has issues, everybody's logins get killed.
I am not in the vast majority of use cases, then, because systemd is a serious pain in my butt. But then, so is PulseAudio (which is why I make sure it's never installed).
My issues with systemd (both technical and as an andicator of the direction Linux is going) are severe enough that I'm actively preparing to move all of my machines over to BSD.
However, technical headaches aren't my largest objection -- my largest objection is that its getting increasingly obvious that as we move into the future and more applications rely on systemd-specific facilities, it's going to get more and more difficult to operate a Linux system without it.
You remind me of all those people who are going to move to Canada "any day now" because of Trump.
Avoiding that kind of attitude is why I moved to Linux in the first place. Nothing ever "Just Works", most systems eventually break down, especially if you're doing anything non-popular. You can't just ignore things when they break, having an easy way to fix things is one of the most important parts of complex software. Saying "It Just Works" indicates to me that the developers don't care if the system is opaque and hard to work with, which has been my experience with systemd.
Say you live in a city, and you have multiple options of transportation: walk, bike, scooter, tuk-tuk, bus. None of them are the most ideal, but you have variety, choices. They're all simple enough for anyone to learn how to use them quickly, and you can even hop from one to the other. But you think, I'd kind of like to be able to move around faster.
The city also isn't running perfectly. There are part-time job shortages, traffic jams, the education system is haphazardly put together. The city wants a way to solve its problems too.
Now say a consulting company comes in and convinces the city they should abandon the other options and adopt a new transit system. This system is fully automated, faster, cheaper. Not only that, but it also solves the job crisis, eliminates traffic jams, organizes the education system. Nobody who uses the transportation system will notice that they are using new transportation, because they're going to put out cardboard cutouts of the old transportation systems, so in theory, everyone should just be able to use it as-is. So the mayor, having heard the pitch, signs a contract: The company owns the city's transportation for life.
Only it's not as perfect as the consulting company claimed. Those cardboard transportation shims will move you around, but you can't choose your seat, you can't talk to anyone else, you have to use pre-approved payment methods and pre-approved routes. Sometimes your route will break down, and there's nobody on board to tell you what to do. Sometimes you get to the wrong destination, but you can't walk home anymore, so you have to just sit there until you figure it out. Sometimes it never shows up. You think, I just want to get around town; why is this so complicated?
You have no choice or control. You didn't vote for this, and you can't vote to change it. You could move to another city, but that is way too involved for most people, so they will hope for the best, deal with whatever pains there are, and learn how to navigate this new, more complex world, because somebody gave a good sales pitch to the mayor. And you think to yourself, "Didn't I move to this city to have more choices?"
So does any init system or process supervisor system that has more than trivially tiny installation base.
The entire point of developing a new one is to reduce the number of the minority cases when it does not "Just Works(tm)"
Look at monit. It "Just works(tm)" for 99.9% of the use cases but boy its single threaded, everything is sequentially controlled with multi-second granularity is infuriatingly bad if you are one of the unlucky souls that wants monit-like functionality where every group runs independently from the other groups.
If it just solidly didn't work at all it would have been annoying but this sort of kind of but never really consistently almost working is what makes you want to physically throw your computer through the nearest window.
Now it seems to work acceptably well save for the fact that guis still regularly have garbage interfaces and one weird stupid configuration.
Pulse by default remembers which device a particular app was outputting sound so you can switch the default device to headphones while say your browser isn't actually outputting sound via any tab and then you will open a sound producing tab and it "remembers" it was using the speakers and blares at full volume at 2AM. Thanks pulse.
This is configurable but who is going to figure that out?
I believe the relevant line is thus where the relevant part is restore_device=false
load-module module-stream-restore restore_device=false
Regarding sound configuration virtually nobody wants different programs to output to different sound producing devices. This might be an important use case for some users but if you have 5 sound producing apps the process of switching from speakers to headphones should not be to move each one with 3 clicks per move. I almost exclusively perform 3 operations I turn the sound up or down and switch to and from headphones whereupon I expect everything including newly created streams to end up on the new device. The first 2 have dedicated buttons the third does not so I had to make it so.
Absent a dedicated button fewer steps would be nice last I looked a few years ago it was complicated in all environments I tried. Having the right answer to get a pleasant environment be write a shell script and fiddle with config files is probably not optimal.
It still feels like many people hate systemd but the reasons that are presented all sound totally valid from a programmers' perspective.
Prior to systemd, the lack of a standard init/rc/service monitoring system meant that you could pick and choose 3 different solutions for those 3 problems. systemd is pushing for unifying not just those 3, but other systems too (gummiboot, consolekit, udev). As more software becomes designed to work just with the systemd ecosystem, more people who have love for one of the systems that has been subsumed will feel forced to switch.
Say you have a favorite fork. You eat lunch with it everyday. Now at your favorite restaurant for takeout, the manager says "We switched salad dressings, and the new dressing line isn't compatible with that fork, you'll need to use this one instead." The fork that the manager hands you may be perfectly fine (better in some ways, worse in others than your favorite fork), but it's not your fork. So you sadly stop going to that restaurant. But then you notice something: more and more restaurants are switching salad dressing suppliers, so now you have to either give up your favorite fork, or use only salads you make yourself. You're probably going to be mad at the salad dressing company.
>Systemd is included by default but not enabled. You can scan your MX system and discover files bearing "systemd" names, but those simply provide a compatibility hook/entrypoint when needed.
>MX Linux uses systemd-shim, which emulates the systemd functions that are required to run the helpers without actually using the init service. This means that SvsVinit remains the default init yet MX Linux can use Debian packages that have systemd dependencies such as CUPS. This approach also allows the user to retain the ability to choose his/her preferred init.
Source: https://mxlinux.org/about-us/
Ah! SimplyMEPIS (a live version of MEPIS) was the first gnu/linux system I booted on my (dad's) P3 desktop computer in circa 2003-2004!
As a distro and philosophy, Archlinux was bound to adopt systemd eventually. e.g. Maintaining you're own init system/scripts is not KISS. Creating a custom init file to adapt 90% of software from systemd to another init system is not KISS.
If you look for most decisions Arch took in the last years, most of them were focused in simplifying the distro and reducing the maintainer burden. So things like removing the TUI installer (does someone remember it?) and migrating to systemd. Even the fact that Arch ships with most packages with few or no changes compared to upstream is focused on it: reduce the burder on maintainer.
>> I haven't heard a good explanation of why systemd and not a different, new init system
> There is none. Systemd came along, we thought it was great, some of us were already involved with systemd and pushed the change. We could have done many different things, but we didn't.
> Runit was suggested by users at some point, and I think there is/was a runit implementation in AUR. However, runit was never considered in detail.
> First, we don't know if the other systems were really alternatives (at least I don't know). The answer is boring: Systemd solved many problems, it was there, it worked and we already used many of its tools in our initscripts at the time. There is no specific reason why we did not use $WHATEVER over systemd.
> About launchd: it was never considered. As I said elsewhere, systemd seemed to do what we want and we went with it.
I really like logrotate and syslog and not needing journalctl for stdout/err logs from services. It's one of the things that makes working with Docker containers so much nicer too, because you have sane commands to get logs for services; way easier than remember all the weird switches for journalctl.
I like NetworkManager (this still hasn't been replaced in systemd, right?)
I still use OpenRC and Runit on Gentoo/Void respectively. OpenRC does dependency management but runit not so much. When it comes to building out VMs and stuff, I see the use of systemd for mount targets, but like I said earlier, I really hate the implementation in target files. It's so hard to make drop-in replacements for systemd that can run the same target files because of just how far it goes into the system.
For VMs at this point, I honestly prefer minimal startup and running everything in Docker. Alpine or RancherOS seem to fit this bill nicely. If FreeBSD had a current Docker API implementation, it would be perfect.
For my desktop, I prefer OpenRC or runit. It's just easier to do things and I'm not fighting with the system layer like I am in Windows. Then again I use i3 and prefer to have more control over my hardware, so that's just my use case I suppose.
For someone who don't know, here is the video "The tragedy of systemd". This is a really good talk.
I do not consider this to be a good talk because it paints dissenters as "not embracing change" because.. they are stupid?
It also makes the argument that the only thing that exists outside of systemd is sysvinit.
He also seeks to make Lennart a martyr because of death threats. (which, btw, seriously it shouldn't even be said that that is unacceptable, some people are fucked up).
Oh, and systemd definitely didn't invent cgroups, an engineer at google did and they don't even use systemd to run GCP compute hypervisors.
Benno so thoroughly misrepresents those that have issues with systemd that I can't help but feel it was intentional.
I'm sorry, I had assumed I had seen the exact video, but I had just seen the talk from another venue.
I can at least sympathize with the arguments for systemd if the only other choice was SysV init scripts. As Richard Gooch once said, "These boot scripts, in the tradition of SysV, can do anything. They are flexible. They are scalable. They can run industrial-strength systems. Unfortunately, they're bloated and ugly. OK, maybe they're not ugly to everyone (not that I've ever heard anyone admit they're elegant, only 'it works'), but they sure are bloated. They are complex and hard to navigate." (http://www.safe-mbox.com/~rgooch/linux/boot-scripts/)
But the choice was never just between SysV init or systemd. Couldn't Linux have done something like FreeBSD's rc? Or I read about the daemontools family of init systems, like runit, which is what Void Linux uses. These systems sound like they solved SysV's problems without adding systemd's. That is, they are more declarative, easier to maintain, can handle complex dependencies, and can run things in parallel. But they remained unixy: they are small, file-oriented, and don't swallow up adjacent software.
As a nice counterexample, OpenBSD doesn't have stuff like udev or dbus usage during boot. The rc.d system they use is literally just a bunch of scripts and is dead simple and reliable.
I left Arch after the systemd transition and didn't look back, but I wish Arch devs and users well.
I assume given context that Safari and IE are right out. Chromium and variants are probably out? If Firefox and variants are out (not sure if all of them have the pulse dependency, because honestly your comment was the first time I learned about it - probably because I'm using Arch), what's left? Qutebrowser? Konqueror?
I guess implicit in the above is an assumption of certain (non-basic) functionality that many people would consider table stakes in a web browser. Given that choosing to use any software involves making certain compromises (like "if you don't use Chrome, some Google applications will likely not perform as well") I guess my underlying desire is to know which compromises you are willing to make, given that you aren't willing to compromise your desire to keep pulseaudio off your machine in order to be able to use Firefox.
On a scale I'm having trouble naming but that goes from about 1 ("my son bought me this iPad so I can see my grandchildren on Facebook") to 10 ("I wget webpages to my server then email them to my Trisquel laptop") your pulse/firefox constraint seems to me to be solidly in the upper half, which I respect, but which also makes me expect you are willing to accept compromises that I'm not (yet) in choosing a browser. Maybe I'm wrong, though, because there's a lot of browsers on this chart: https://upload.wikimedia.org/wikipedia/commons/thumb/7/74/Ti...
https://www.howtogeek.com/335712/update-why-you-shouldnt-use...
So if Chromium is out, then so is qute.
Regardless, every time these types of threads come about I feel like I _have_ to speak because there's a lot of bitterly divided people regarding this issue. (or, rather the people who are bitterly opposed and the people who don't care, think systemd is a net good in the vast majority of cases or those who think the backlash is very much unwarranted).
I'd like to say first off: SystemD solves real problems that nothing before it was solving, maybe it doesn't solve those issues in the way I personally like, but it definitely does and was one of the first to do so- so for this reason the original maintainers should be warranted respect, it's also a given that it's not the developers fault that this is so widely adopted or even forced. I see them as problem solvers first and foremost so any gripes I have with systemd as a thing which is foisted upon me is definitely not on them.
However, I do consider that systemd as a new -layer- between user land and kernel space (called "system space" by Poettering) necessitates a lot of breaking changes. I do not consider this by itself as a bad thing, but having a poorly defined interface for these things does not enable systemd to be replaced without some form of "systemd-esque" emulation in future.
Thus while we have improved from where we were, we do not have fertile ground to improve in the future.
Additionally; SystemD as a "modular" system is, in my opinion, a myth; even if you take for granted the fact that systemd+journald are the only core dependencies, you then quickly start to drag in more and more rapidly (udev and systemd-resolved being famous examples).
Some design paradigms are scary, for instance socket activation leaves PID1 listening for traffic on the network, by default bound to all interfaces. (a good example is to look at cockpit) this could be a very large footgun if there exists any form of exploit in systemd.. and since systemd is not written in a memory safe language, and the overwhelming majority of security bugs are memory safety bugs... this is mildly concerning.
There are other concerns regarding opacity of the dependency system, the fact that no-two-boots are the same and thus some race conditions are likely to never be solved by me. (a lowly sysadmin type who only knows enough C to make linked lists). The design very much reminds me of Microsoft Windows in many ways, but perhaps that was a good design to choose from?
FWIW: SystemD is amazing on my laptop, I am really happy with it. But for my servers I am very unhappy with it, but the major distributions chose to use systemd and I'm a sysadmin so I'm beholden to what the company considers stable and developers.
I very much begrudge the fact that I'm not allowed to have a dissenting opinion on this.
I also hate that when people talk about systemd vs "anything", invariably people assume I am talking about openRC or sysvinit, but there are much better init systems available today, such as Runit.
SystemD is not a panacea, it should leave fertile ground for replacement. If that cannot be done it should not be adopted because that is a scary amount of lock-in.
I'm a sysadmin and not a programmer
SystemD is amazing on my laptop, I am really happy with it. But for my servers I am very unhappy with it
I think this is what it comes down to. The end-user experience is pretty good, just like the end-user experience for MacOS and even Windows nowadays is pretty good. But that also means you could replace Systemd with a similar system built on similar concepts, and the end-user experience would still be pretty good.
Also, echoed in another comment, I'd just assume use Docker and/or K8s and/or dokku for most of the services I write anyway. Frankly, I care about writing the applications and services that solve problems for me, I don't want to spend as much time on system administration or CI/CD infrastructure as I already to.
In the end, it does a better job at most of the problem space in most ways compared to the alternatives. I have to google for it any time I actually interact with it anyway, which is what I did with older initd systems, so I'm not really that enthusiastic one way or the other.
As a sysadmin, I want something to be debuggable because I'm the one who gets paged when the failure rates increase.
Embrace the change :)
But writing everything in C in non-scriptable way may be a bit of over-reaction to the old ways of having everything scripted in the least efficient way possible.
I also write my init systems in C (not for generic Linux distro, mostly for embedded HW/mobile use), but I'm planning to make it so that I have a program in C that offers some primitives and the core logic will be implemented using an embedded JS engine, so that it's scriptable and fixable without re-compilation.