Problems with Systemd and Why I Like BSD Init
bsdmag.org
bsdmag.org
There is another post mentioning the desktop/web thinking and the server/sysadmin thinking.
My thinking - the server/sysadmins are right, but the majority are people in the desktop/web crowd.
I actually expect things to get worse once you rope in the container (docker/deis/kubernetes/mesos/etc) crowd.
The general attitude seems to be "this doesn't work for me, let me invent something new".
If you've spent anytime trying to develop on top of containers and have a deep systems background, you are resisting a hell we had in the late 90s and early 2000s...because "new shiny thing".
SystemD is endemic of the new generation trying to solve problems on top of linux. Rather than understand and address the problem, invent something monolithic that isn't better than what it replaced.
Oh then force it on distros and not be open to criticism.
I really hate this debate and probably should keep quiet.
Systemd solves so many pain points in all areas that I don't care about ideological debates.
And most of the touted "advantages" of sysvinit are, in practice, a nightmare. "people should just know what they're doing"? Well, too bad, nobody does! Distribution maintainers don't know how to write robust init scripts, software vendors don't know how to write robust init scripts, nobody does!
Taking away the "freedom" to introduce the halting problem to service definitions is, in practice, godsent.
"With BSD init, I can just edit the /etc/rc script and replace it with arbitrary shell code."
And now try to debug it one year later when you don't even remember that you did any changes to it. The shit I've seen on Linux systems' /etc/rc.local would give Lovecraft nightmares.
And the ability to sprinkle echo statements all over your init system does not make a system "easy to debug", unless you're a PHP "devop" who thinks that OOP is the source of all evil in the world.
"The way systemd does parallel startup bothers me. If service A depends on service B, they were traditionally started in sequence; A only starts after B is done starting."
Unless it doesn't, because init scripts _cannot_ reliably determine whether a daemon has started up. Grab, say, debian's sysvinit sctipts and count the amount of sleep statements sprinkled all over them. That's not sequential startup, that's a race condition waiting to happen!
"Why should systemd implement NTP when ntpd already exists?"
Because ntpd is a steaming, insecure pile of crap. And no, openntpd is not a solution, it's just a nicer problem. Why do I need an ntp server on a pure client machine? Just because ntpdate is an even bigger pile of crap?
"Being unable to shutdown the system."
As if that never happened with sysvinit. Badly behaving init scripts can cause a lot of havoc, at least systemd has a configurable hard cut-off.
"Debian should not have adopted it for at least a few more years; they’re supposed to be the slow, steady, and stable distro. "
Debian profits the most from ripping out the unusable joke that was their own "parallelized startup enabled" sysvinit. I've seen more Debian stable+sysvinit systems unable to boot up than I've seen Arch+systemd systems.
The ability to "simply edit init scripts so that it works" often results in nasty, broken hacks ending up in production.
What. Even if you disagree with systemd, literally anything is better than init.
What a peculiar thing to say. The init process does almost nothing -- it just spawns getty processes and launches /etc/rc. On FreeBSD, init.c is a mere 1737 lines of code. For all practical purposes init is just a boot loader; the important parts all come later.
init (the process) != init (the system), but you're right about the terminology, of course.
I'll grant that LSB sysvinit scripts aren't the easiest on the eyes, but they're hardly the shining example of what a scripted init system can look like. OpenBSD's rc scripts are far more elegant and just as featureful, IMHO.
https://svnweb.freebsd.org/base/head/etc/rc.subr?view=markup
Those scripts are completely different between SysV, old-style BSD, and BSD rc.d.
I agree that SysV init scripts tend to be a jumbled mess, but they could be replaced with the rc.d framework without throwing out init entirely.
/lib/systemd/system is identical to /etc/rc.d in terms of maintenance complexity
On the one hand you have the traditional server/sysadmin thinking about uptime and stability.
On the other hand you have the desktop/web thinking of rebooting often (or shutting down when not used, starting up before using) and being on the bleeding edge.
Another thing is that his mention about standard ways of doing something. I think that was why Freedesktop was set up in the first place.
But they seem to have gone massively off the rails almost from the outset, or it was simply too dominated by Gnome/Red Hat people (it was after all founded by once such person, Havoc Pennington) to really get things properly generic and modularized.
Asi write this i get to thinking about politics and thinktanks, and how they can make political decisions seem like they are scientifically based, when they really are rubberstamping preconceived political notions.
Similarly it may well be that Freedesktop exist to rubberstamp Red Hat/Gnome notions about Linux, by virtue of appearing to be a independent entity (similarly Gnome has its own foundation, but a large segment of its core developers are under Red Hat employ).
BTW, i seem to recall that while the BSD people had implemented most of the systemd APIs for various settings changes, they had run into a problem with logind.
Probably because logind depends heavily on PAM to get its job done, and PAM seems to fly in the face of basic unix-isms.
I actually find this line of thinking baffling... I don't remember the last time I actively chose to shutdown/reboot voluntarily. Spending time optimizing the boot sequence strikes me as a total waste of effort.
Modern computers support sleep and hibernate... why would anyone choose to reboot except to apply OS updates on certain operating systems that insist on a reboot?
I would much rather focus on uptime stability and flawless sleep/hibernate so we can encourage people to never reboot unless they're forced to.
That'd better serve phones, tablets, and other devices that share this infrastructure and need this capability, and it would cater to a public that is getting used to always-on, always-available technology.
The systemd devs have a thing about listening to the -sec community. You can see this in how while the Linux default is to treat namespace mounts at private, systemd mount them as shared unless told otherwise. This because the -sec people recommend it that way (allows a sysadmin to override mounts from outside the namespace, to avoid dos-ing apparently).
Then again they also have a bit of theater in them. For example they accepted a patch to lock the computer if a HID devices was plugged in. But while the original patch did it on any device, they modified it so that it only happens on the device ids of known mouse wigglers used in police computer seizure kits.
There was a series of two patches: one to logind to add a mechanism to tell all active user sessions to lock, and one to the udev rules provided with systemd to add a policy that plugging in a known mouse wiggler causes all active user sessions to lock.
(It wouldn't make any sense to lock the screen when inserting any HID device.)
Separately from that, logind also implements other policies for when to lock the screen, such as when closing a laptop lid without an external monitor connected.
The last security advisory affecting the linux kernel for Debian was 5 days ago (https://www.debian.org/security/2015/dsa-3426). There was another one about a month before that (https://www.debian.org/security/2015/dsa-3426).
Additionally because of the way shared libraries are loaded in Linux after you apply the patches you might have old, unpatched versions of (eg openssl) linked into currently running processes. There are a few ways to handle this but a restart is one very simple one that is guaranteed to work.
Consequently any Linux machine that has more than about a month of uptime is probably missing patches. 500 days of uptime corresponds to a huge number of unpatched vulnerabilities. So restarting is quite important and is unlikely to go away soon.
This idea that a restart is needed is Windows-style thinking.
> more than about a month of uptime is probably missing patches
Only when you don't know how to properly maintain a Linux box. A reboot is one way to handle patches. It is even recommended when you are changing anything related to the startup process.
For everything else, you simply need to restart any process using the old libs. If it wasn't a security patch, you can often simply wait for processes to finish normally.
I implied it but I didn't make it clear: debian's security team releases a kernel patch about once a month. This is where the ceiling on uptime comes from.
This is why I mentioned hibernation. Standby is ok only for a few hours at most on laptops.
It also puts the computer back into a known state, and gives you smaller PIDs/thread IDs/etc. again...
For sysadmins, boot times are getting increasingly more important. We're getting to the point where we can spin up a new VM in <100ms, thanks to (in part) systemd. This doesn't matter if you've got a few servers in a rack, but it does if you're spinning up thousands of VMs.
It will rearrange the boot sequence on a whim, it will keel over if dbus does, it will fail to boot on the smallest of unit errors (to the point of providing no means of remote access no less).
It may work in a cattle ranch, and it may work on someones laptop, but it damn sure does not work in a rack.
> On the other hand you have the desktop/web
I've understood that systemd is about modernizing the init/service system for modern usage scenarios.
Gone are the days of the 80's when hardware was static and always plugged in, these days you have laptops that get docked and undocked, plugged and unplugged, connected to different networks etc. You have NICs and sound cards appearing and disappearing (USB NICs and DAC's), not to mention hard drives and USB drives... All of this needs to be properly supported on modern desktops and laptops and tablets. That's what systemd does.
"Modern usage scenario's" should still include reliable boot and predictable behaviour. Systemd seems to have been developed under an assumption of infallibility, which makes recovering from a failing boot much harder. And a failing boot is much more common under systemd, because it has been designed to fail hard instead of fail safe.
I'm running Debian on seven systems, and only one of those is running systemd. The others simply won't boot with it, or are headless and I won't risk bricking them (see fail hard above). The one system that I do run systemd on, I've had to disable the lvm and luks generators because they would fail hard, and I've had to compile it from git because Debian's shipped versions of networkd and resolved are broken.
I agree with the article that systemd adoption was premature. Throughout the discussion, I got the impression that the decision was forced because of primarily the interests of desktop and service developers, with only a minor nod to system maintenance.
imnsho, "systemd" the project was not ready for inclusion. Systemd as PID1 may have been ready, but systemd the project does not allow systemd to run as PID1 without the rest of the project.
Yeah hotplugging has obviously been available for ages.
> You either love or hate the ambiguity of terms like "systemd"
Indeed, in this case I was referring to the whole framework and not PID 1.
Personally I've been running systemd on my systems since even before Arch Linux incorporated it officially and I've pretty much never had any problems. I especially haven't had any boot related issues with it. I even have multiple luks encrypted disks attached.
This has been done by Linux and udev for about 12 years now, if not more. Systemd is a recent player in this field. You don't need systemd to properly support hotplugging.
The other project could include all the rest, basically. Again, having a unified API for applications to e.g. get notified when the network connection changes, time changes or a USB flash drive is connected, or a new display, or whatever - that makes sense; having it effectively mixed in with the init system does not neccessarily. As a bonus, if it was an independent project, it might have been portable across different Unixoid systems.
For reasons totally unrelated, though, I installed FreeBSD on the new home server I got recently, and it was a really pleasant experience. Plus I installed OpenBSD on an old laptop I saved from becoming landfill, which was fun, too. There are good reasons to like BSD even if one gets along with systemd just fine.
Yes, RMS is pretty pegged on the autistic scale, but he's not wrong, and Snowden proved that GNU as an ideology is pretty much our only hope against nation states intending to deprive us of our freedoms.
GNU is the closest thing to a soul the software industry could have, demanding nothing less than freedom for the users of software and writers of software. Their (not) unix is a beautiful thing to look at, and companies like Novell/SuSE and Red Hat built their empires on its foundation.
Now, Red Hat is taking an active role in rewriting everything but the kernel, in order to push-out and/or marginalize the GNU crowd. If they were honest, they would have Poettering come out and state publicly that they are creating a replacement Operating System for the linux kernel. but they're not. They're wrapping themselves in the GPL (and the goodwill that comes with it) while trying to destroy Stallman's creation.
Red Hat has quietly declared war against GNU, and those that support it, embracing and promoting the watered-down (and far more profitable) "open source" concept as they go.
In that war, systemd is merely another salvo that (I think) began either with Havoc Pennington writing dbus off the back of Nat Friedman of Ximian's hard work and research, or their complete takeover of freedesktop.org. NetworkManager, Avahi, and PulseAudio being pushed into all the distributions, always long before they were stable (and all written by Poettering), proved with a paper trail that they had worked the kinks out of their embrace-and-extend plan long before systemd was ever mentioned on a debian mailing list for adoption.
I wish the author well, but some of us are gonna stick around to try and keep GNU strong against this new threat. We've lost Debian, Arch, and Ubuntu, but this war is far from over.
> They're wrapping themselves in the GPL
...while encouraging a development style that isolates processes so they use IPC when they should be simply linking to a library and making a function call. Systemd depends heavily on dbus, because it ties developers into this model.
This enables a rather convenient way for freeloading propriety software to use GPL code, without having to be subject to the requirements of GPL code. If you get some package to provide their services over dbus, you only have to "use" the software instead of linking against a library.
Even ignoring systemd's bad design that tightly couples together services that worked fine as separate specs for decades, and even ignoring the terrible "we don't give a crap if we break your environment for no reason" attitude that is common among systemd's authors, this end-run around the GPL is a very good reason to avoid systemd.
Now, I'm sure some people in this thread think I'm making crap up, being paranoid, or similar. To those people I say: when are you going to start paying attention? This is obvious when you look at the Linux community over the last 20 years. Stop being distracted by technical features instead of investing in your future. Are you going to be happy when all distros are de facto variants of Red Hat? Some of you might be, but if you're going to ignore the vast majority of Linux and Unix, could you at least fork the project instead of dragging the rest of the community down with you?\
Please tell me this is a parody?
by using shell script, it is easy to incorporate proprietary code into an otherwise copyleft product. The proponents of the "UNIX philosophy" that try to define the latter as writing lots of independent tools that can be combined in shell scripts, are probably just trying to make sure they also can work around the GPL in the future too.
That's why the gcc compiler suite doesn't provide ways for other applications to access its AST via stdout: it could be used to code completion in emacs, but also to write proprietary gcc addons that manipulate the AST and work around gcc's GPL license.
Somewhere along the line a kind of "takeover" happened. I am not going to say that the people doing this are "evil" or something as they at least contributed something that can be used for free. Difficult to argue with that. But I cannot stop myself having the feeling that there is some hidden agenda of interests behind it as well. For me this "takeover" looks similar to what happened to W3C where certain people (companies?) started to ignore them and created HTML5 instead. I really like the well-formness of XHTML where XML itself is very strict and has many built-in features which are still missing inside what replaced them in popularity, like JSON. The people of W3C knew what they were doing and strived for every change to be stable and future proof which can be a slow process. Too slow for some people it seems.
Of course I tried to use Systemd and was really happy with it till the moment it started crashing and taking some clusters down with it. I migrated 2 servers to the new Debian and everything went well. Even the LXC containers started just fine until I started to upgrade the containers to the new Debian as well. At that time I did not even know the author of Systemd was the same as the one that started Pulseaudio, but somehow these issues gave me the same feeling of uncertainty that reminded me of it.
For the servers I am now using Gentoo (as almost every other distro already switched to systemd) in combination with OpenRC without any issues. Compilation of source packages happens on a separate server where I can test them before deploying the binary packages I just build to the test and production servers.
KISS is a magic principle that just works. The Unix/Linux world should never let it go.
which problems is he talking about?
This includes for example environment control (env, ulimit, cgroups, chroot), auto-restarts, changing uids.
A minimal system as a base image for Docker, and then install by apt or curl (would never, never have used curl for installing software 5 years ago) into a service image. And a second Linux as a base system for running Docker (and nothing else). Start/stop is managed by Docker ( /Nomad/Swarm/...). Next step might be moving the base image away from Linux.
I can completely understand why systemd need their own timers as part of those APIs, I however cannot see the point of having their own NTP service. Maybe the old one was broken with systemd?
But that's just one valid point, not being able to see the necessity of timesyncd does not invalidate the underlying goal of systemd, nor does it warrant moving to another distro.
Because bottom line here, bsdmag is a BSD oriented zine so its focus is to promote BSD distros. I used to use 100% BSD 8 years ago but my migration to Linux is now complete.
The BSD license in my opinion hurts open source more than GPL. There are three major examples that I can think of where big corporations have used BSD code to avoid re-inventing the wheel and piggybacking on open source. Microsoft when they used some BSD userland tools, Apple when they built OS X and Juniper when they made JunOS.
I used to interpret these decisions as flattering, but they are in fact simply taking the code they need, with the license that allows them to do it. They would never take GPL code as freely because they would risk being locked in by its requirements.
1) Have you not considered that it was the intention of the developers of that code that it could be used in this way, without giving back? 2) Have you considered that the rest of the world benefits from the use of BSD code in Apple and Juniper products, even if they don't give any code back? Anyone who uses these products receives gets to use better code than what may have been written from scratch. 3) Apple and Juniper both give code back to FreeBSD 4) Even if they didn't, it does not hurt FreeBSD any to have companies use their code and not give anything back. 5) We're agreed that Microsoft, Apple, or Juniper would never have used GPL code in their kernels. So how exactly has the open source world lost out due to the existence of the BSD license?
iOS (based on BSD Unix) is proprietary. Android (based on GPL Linux) is open. iOS came first and seemed to be the huge winner. Then Google took GNU/Linux (GPL), built Android around it, and outpaced iOS by far.
Now the BSD developers have to pay (!) for their own (!) developments when they buy idevices. They have no option to modify anything. They are just consumers.
The Linux developers can take the code, tweak their devices (Cyanogenmod) and do a lot of other things with it. There is Android for x86 for instance.
BSD is good for hardware producers. GPL is good for hardware consumers.
Having said that...
It does have some nice things. It's very easy to create robust service startup definitions (units) without resorting to brittle shell scripts checking PID files and whatnot. Also, services start in their own resource groups for better resource allocation between them.
These alone have changed my opinion about systemd. These solve real sysadmin issues.
"Systemd will start them both at the same time, but buffer any messages A tries to send to B until B is ready."
It's not systemd which is buffering messages, it's the kernel's IP or UNIX domain sockets. (Also, this means that daemons don't need to run setuid root just to bind a low socket.)
"With systemd, the order is not deterministic, so you don’t know what’s going to happen next time you boot."
All you need to do is define the dependency graph properly, then any possible ordering will be fine. You might as well argue that "make -jN" is somehow harmful.
"A server spends several minutes in the BIOS during POST anyway, before the bootloader is even run; making the OS boot faster doesn’t change very much"
Never used virtualization or containers?
"That doesn’t feel modular to me." "Cleaning up or re-implementing ntpd is a good thing, but OpenBSD didn’t feel the need to make it part of init."
The systemd project is not (any more) an init system, it's a collection of tools to create a Linux-based OS from. http://www.openntpd.org/ says that OpenBSD includes OpenNTPD as part of its base system.
"OpenNTPD was primarily developed by Henning Brauer as part of the OpenBSD Project and gets released as a base component of OpenBSD every six months."
Clearly this is the same as what systemd does? Do we conclude from this that OpenBSD feels "not modular", since it has an NTP implementation in the same repository as its init, or is it only bad if systemd does it?
"Regular journal corruption that Lennart says is not a problem. It seems obvious that a non-transactional binary log is a terrible idea."
As opposed to plain text files, which magically survive having random blocks overwritten by hypothetical filesystem bugs or hardware issues without losing a single line of log content?
"Numerous reports of systemd hanging on boot because of something in fstab that violated POLA."
Does this refer to systemd implementing the "nofail" keyword exactly as it was documented? The author will be very surprised when he discovers that FreeBSD does exactly the same thing, except that the keyword is called "failok".
"To my knowledge, no one has ever lost data as a result of a bug in ZFS."
Does the author have access to Sun/Oracle's internal bug tracker where such customer bugs would be reported, in particular during the early years of ZFS's existence?
Well, you CAN replace OpenNTPD in OpenBSD (and you should if you need e.g. autokey), and it will just work™.
The issue with systemd is certainly not the bundling, it's the tight coupling which makes it a lot less modular than it claims to be.
Neither system will protect you from losing a single line of log. The problem with journald is that a simple corruption can make you lose the entire log file.
[0] http://homepage.ntlworld.com./jonathan.deboynepollard/Softwa...