Until very recently having wrong permission on one of the networkd config files would result in networkd ignoring the rest of the perfectly readable network configuration files too, cutting off all network connectivity. (Systemd also requested the permission change iself.)
Currently there's a bug where if you have a bridge and all interfaces are disconnected from the bridge (I use hot-pluggable USB network interfaces) when you re-connect them, networkd will keep the bridge off. If you restart systemd-networkd it will fix itself.
I can name at least two instances already where I lost networking in my entire SBC farm, and had to pop out all microSD cards and fix the breakage manually, just because of systemd-networkd bugs. And I don't do anything crazy, just a single ethernet interface and two overlay wireguard networks, and mostly default config.
And fixing it yourself is not easy. I can do C coding perfectly fine, but I run 3 different CPU archs, so fixing it myself means compiling (3 times) and distributing my own systemd version to all my machines until upstream fixes the bug. It's not pleasant, nor easy.
I certainly understand people who miss having an ability to more readily script the system startup/network config.
> complex corner-cases that only a a very tech-savvy user could generate in the first place
I agree that an init system should work on complex corner-cases, but systemd remains popular because it works for most people most of the time.
Tangent: have you found an init system that you like the most? I've only used four or five init systems and don't have any super strong opinions -- maybe because I'm not running and SBC farm. :~)
I like systemd and use it everywhere (I use Arch Linux / Arch Linux ARM mostly). I just don't think it needs to be painted all pink as "Just Works[tm]" software.
I'm not sure why it did that, as I didn't have time to dig into the why while trying to deal with the result.
However, after trying SystemD for one week, I became convinced that the designers of SystemD are incompetent, so they cannot be trusted with a component of such importance for a computer.
I am normally a Gentoo user, but a few years ago I wanted to install Linux in a hurry on a small computer. With a slow CPU installing Gentoo can take several hours, unless you have a previously prepared image, so I decided to try ArchLinux.
The installation was indeed fast and without problems. However, SystemD behaved in an unexpected way. SystemD is advertised as booting quickly. It indeed booted quickly, but no faster than my optimized Gentoo systems.
On the other hand the shutdown with SystemD was very slow. This was a huge surprise, because never before and never after have I encountered any computer where the shutdown is annoyingly slow.
Moreover, sometimes the shutdown FAILED, which is also something that I have never encountered on any computer not using SystemD, and I have used thousands of computers, with many kinds of operating systems, including at least eight or nine UNIX flavors other than Linux.
The failure of shutdown was apparently due to some kind of race condition when some process was killed faster than some SystemD component expected and that SystemD component was still trying to communicate via DBUS with the defunct process and it blocked because its messages no longer reached the destination.
I suppose that this bug might have been corrected eventually, but this is a design error that I consider unforgivable, i.e. to conceive a shutdown sequence which depends on successful inter-process communication.
After seeing such a mistake I have lost any confidence in the abilities of the SystemD developers, so I have never tried it again.
If I had to guess, it's probably because I've never ran into any of the weird systemd problems that I see people report sometimes. For me, it's always pretty much just worked, and more often than not it natively has the functionality that I want.
My largest complaint is that I have to bounce across several manpages to write a comprehensive unit file.
Traditional RC does a SIGTERM to everything, and then waits a bit and then does a SIGKILL to everything. This means that if the SIGTERM failed to be handled properly, progress is still made.
Weird as in 3 minutes after someone SSHd into my workstation, X11 restarted. Until I tracked it down to a default systemd configuration I thought I was going insane.
Beyond the default configuration, which is somewhat defensible as NIS is probably not a typical case, it really bothers me that if someone else's login process has issues, everybody's logins get killed.
I think part of it actually stems from people that had problems early on with PulseAudio, and have developed a vendetta against Poettering. Whether that vendetta is justified or not, I don't know.
Personally systemd hasn't been a pain point. It's been far less of a headache than everyone at the office thought it would be.
SystemD is much the same way, I'm coloured by my impressions from Fedora21-22, where it.. didn't work great.
The issue is with poetterware, it's _incredibly_ difficult to discover /why/ it's not working properly, which is dangerous for an init system (or, an "everything between your application and kernel"-system), a failure rate of 0.0001% is still an unfathomably enormous number of systems; and due to how it works it lends itself to an incredible host of threading issues.
Hypothesis: C is so pathologically bad that it breaks brains.
As a sysadmin I wrote a small kernel module to trigger a panic on a certain condition hoping to capture some info in a dump several years ago.
Today I cant find sysadmins who can use a CLI properly.
The goalposts have moved inwards IMO
Not sure its bad thing either but it makes for interesting comparisons
These days, people calling themselves professional developers with years of experience won't even touch Emacs.
I picked it up several months ago after using Jetbrains IDE's for quite a long time, and honestly haven't felt the need for any of what Jetbrains brought to the table.
Mostly thanks to LSP.
If you're not a Java developer, Emacs is perfectly fine as a configurable extensible editor. With the additional caveat that its lack of multithreading can get in the way sometimes.
Avoiding that kind of attitude is why I moved to Linux in the first place. Nothing ever "Just Works", most systems eventually break down, especially if you're doing anything non-popular. You can't just ignore things when they break, having an easy way to fix things is one of the most important parts of complex software. Saying "It Just Works" indicates to me that the developers don't care if the system is opaque and hard to work with, which has been my experience with systemd.
I am not in the vast majority of use cases, then, because systemd is a serious pain in my butt. But then, so is PulseAudio (which is why I make sure it's never installed).
My issues with systemd (both technical and as an andicator of the direction Linux is going) are severe enough that I'm actively preparing to move all of my machines over to BSD.
However, technical headaches aren't my largest objection -- my largest objection is that its getting increasingly obvious that as we move into the future and more applications rely on systemd-specific facilities, it's going to get more and more difficult to operate a Linux system without it.
You remind me of all those people who are going to move to Canada "any day now" because of Trump.
I'm not sure it's necessarily a better solution as it requires more manual interaction out of the box (namely creating symlinks by hand among other duties), but that might be argued as a feature. There appears to be a wrapper or two out there to automate away some of the tedium.
runit, s6 and also nosh are all daemontool-style init systems, which have a somewhat different idea how to do things than systemd all, but all are inspired by the original daemontools package.
s6 and nosh have decent overviews:
https://skarnet.org/software/s6/ https://jdebp.eu/Softwares/nosh/guide/introduction.html
The original daemontools: https://cr.yp.to/daemontools/faq/create.html#why
I'll let others decide whether these are better or worse than systemd. I go with whatever my distro would give me, since picking a different init system is too much work...
I still dont understand what issues are fixed with systemd and I am aware of many it has created.
The real issue isn’t with servers, it’s with desktop systems and laptops. When users plug in a USB sound card or plug their laptop into a dock, they don’t want to (and likely couldn’t) “easily script around” that; they expect an audio device, network connection, video card to be created without any user intervention, and go away when devices get unplugged.
Making a set of init scripts (and surrounding unload and reload scripts) robust against insertion and removal of any of many different devices isn’t easy even for init script gurus, and would lead to duplicated code for detecting and handling the various load/unload conditions.
That, I think is the primary reason Apple went with launchd. Systemd took that idea. There’s nothing wrong with that.
The main issues with systemd, IMO are a) initially it was buggy, and b) tremendous scope creep.
Bingo !
Folks who know, and I mean really understand how to run servers know how to script init.
Folks who use Linux on the laptop / Admin Ubuntu do not and systemd makes them able to do stuff "most of the time"
There is no reason to force systemd as the default for servers. You dont plug things into servers and in fact you dont want some service monitoring servers so someone can stick something in a server and start rooting around it.
Also you seem to be conflating dealing with devices with initialization of the system. Dealing with such devices has insofar as I'm aware always been handled by c code in and out of the kernel whereas at one time initialization of services and orchestration of components was handled by a scripting language.
This is to say that a bash script may have brought up different components and services but that c code that makes up the component is in charge of actually managing said device.
Its obviously feasible for both c (or any other language) code and orchestration of start up and shut down of services to be buggy but its not equally easy for end users to fix their own problems in all cases.
Ideally bugs get reported to those who are most capable and the underlying code is corrected however if said labor isn't paid to work on their project hours to do so may be in short supply and it may frequently fall on users to lend a hand.
This is a numbers game. If you have 10000 users and 10 of them are qualified to solve such problems you may be lucky to get the help of 1 to submit patches or post a solution to a problem on a forums for others to use before the upstream problem is fixed.
If 100 are qualified you are 10x as likely to find help to solve a problem. I have on many occasions come across a solution months or years before an official upstream solution came into being. You are suggesting that most sysadmins might find it challenging to solve their own problems. They don't always have to. Merely having comprehensible user friendly systems make it more likely that one of their fellows can help them do so and more likely that such solutions travel both laterally between users and hopefully upstream solving the problem for all.
The good bits that systemd brought into the mainstream such as cgroups and lxc do not require systemd. They can be implemented independently. The watchdog timers and unit files were not a new concept either and did not require systemd.
You can still use cgroups independently with systemd, but it's recommended that you use a service and the delegate option to have systemd create a cgroup hierarchy and also know that it shouldn't mess with it after creating it. This lets you have your cake and eat it too if you have a case that systemd doesn't handle (I'm working on a setup with this myself for a coding sandbox setup).
This is conflating the systemd project and they systemd init system. Most of the things you mention are seperate systems (systemd-resolved,logind, polkit, etc.) that are part of the systemd project and interoperate with the systemd init system, but are not part of the init system itself.
I use systemd across distros, and am now confused when I run into an old SysV system.
When I'm sufficiently advanced to bump into one of these corner cases, it will be a Good Problem To Have.
The market penetration of systemd is about as good an empirical validation as can be had. How soon until it's codified in Posix?
I'll readily admit that systemd isn't perfect, but my overall experience is that it is better in many ways to its predecessors, and did solve real problems.
And when I referred to "systemd hate" I was not referring to healthy criticism, but to those who take very chance to claim that systemd never should have been invented and their init system of choice is clearly superior. Rather than, say trying to make systemd better, or make their init system fix the problems that systemd was designed to remedy.
Say you live in a city, and you have multiple options of transportation: walk, bike, scooter, tuk-tuk, bus. None of them are the most ideal, but you have variety, choices. They're all simple enough for anyone to learn how to use them quickly, and you can even hop from one to the other. But you think, I'd kind of like to be able to move around faster.
The city also isn't running perfectly. There are part-time job shortages, traffic jams, the education system is haphazardly put together. The city wants a way to solve its problems too.
Now say a consulting company comes in and convinces the city they should abandon the other options and adopt a new transit system. This system is fully automated, faster, cheaper. Not only that, but it also solves the job crisis, eliminates traffic jams, organizes the education system. Nobody who uses the transportation system will notice that they are using new transportation, because they're going to put out cardboard cutouts of the old transportation systems, so in theory, everyone should just be able to use it as-is. So the mayor, having heard the pitch, signs a contract: The company owns the city's transportation for life.
Only it's not as perfect as the consulting company claimed. Those cardboard transportation shims will move you around, but you can't choose your seat, you can't talk to anyone else, you have to use pre-approved payment methods and pre-approved routes. Sometimes your route will break down, and there's nobody on board to tell you what to do. Sometimes you get to the wrong destination, but you can't walk home anymore, so you have to just sit there until you figure it out. Sometimes it never shows up. You think, I just want to get around town; why is this so complicated?
You have no choice or control. You didn't vote for this, and you can't vote to change it. You could move to another city, but that is way too involved for most people, so they will hope for the best, deal with whatever pains there are, and learn how to navigate this new, more complex world, because somebody gave a good sales pitch to the mayor. And you think to yourself, "Didn't I move to this city to have more choices?"
I had it silently take over my DNS . I even have the 'official' way check-boxed to not override DNS settings in Network Manager. Does so anyway. I even removed the symlink from /etc/resolv.conf and made it root:root . SystemD changed it next reboot.
And it's way more documented that your average init script sourcing distro-specific bash/sh libraries where you have to dig into and read the code just to understand how that thing works
Init scripts were undebuggable, un-understanable, documentationless kluge with text logs, and systemd has a perfectly functional format converter. I'm not seeing the regression here.
1) Init scripts varied by implementation very much, I have managed to successfully debug, make, read and otherwise modify/parse init scripts from basically every _major_ distro including OpenBSD and FreeBSD.
I am not particularly intelligent, so if I can do it, you can.
2) There is no reason to assume we go back to bash scripts per-se, there is a huge middle ground between a huge C program which execve()'s programs and watches them live, parses their output, logs it, enables socket activation, keeps an in memory DAG that it determines randomly on boot.... and bash scripts.
Good for you.
> And yes people do use Linux on the desktop
Good for them.
But the fact remains that this post still ignores why binary logs are bad for the server use case. They seem great for me in that role.
So does any init system or process supervisor system that has more than trivially tiny installation base.
The entire point of developing a new one is to reduce the number of the minority cases when it does not "Just Works(tm)"
Look at monit. It "Just works(tm)" for 99.9% of the use cases but boy its single threaded, everything is sequentially controlled with multi-second granularity is infuriatingly bad if you are one of the unlucky souls that wants monit-like functionality where every group runs independently from the other groups.
If it just solidly didn't work at all it would have been annoying but this sort of kind of but never really consistently almost working is what makes you want to physically throw your computer through the nearest window.
Now it seems to work acceptably well save for the fact that guis still regularly have garbage interfaces and one weird stupid configuration.
Pulse by default remembers which device a particular app was outputting sound so you can switch the default device to headphones while say your browser isn't actually outputting sound via any tab and then you will open a sound producing tab and it "remembers" it was using the speakers and blares at full volume at 2AM. Thanks pulse.
This is configurable but who is going to figure that out?
I believe the relevant line is thus where the relevant part is restore_device=false
load-module module-stream-restore restore_device=false
Regarding sound configuration virtually nobody wants different programs to output to different sound producing devices. This might be an important use case for some users but if you have 5 sound producing apps the process of switching from speakers to headphones should not be to move each one with 3 clicks per move. I almost exclusively perform 3 operations I turn the sound up or down and switch to and from headphones whereupon I expect everything including newly created streams to end up on the new device. The first 2 have dedicated buttons the third does not so I had to make it so.
Absent a dedicated button fewer steps would be nice last I looked a few years ago it was complicated in all environments I tried. Having the right answer to get a pleasant environment be write a shell script and fiddle with config files is probably not optimal.