Escape from systemd
davmac.wordpress.com
davmac.wordpress.com
I don't really know about any of the drama surrounding systemd but I've liked using it so far. It's easily the most attractive feature in Ubuntu 16.04 LTS.
Using journalctl for logs is nice because I don't have to keep looking around for where logs are stored and then tail them. But then again, I don't worry about log rotation and things like that. My guess is that journalctl takes care of it.
I'm sure I've barely scratched the surface of all the things it can do (like auto-restarting daemons, some security stuff, etc).
Am I the exception? Can't be so, if most linux distros have adopted systemd.
For me it is as well, I really miss it when working with a Ubuntu 14.04 box.
I have not found a single downside to systemd, really.
The journal is nice, especially on desktop systems, but it's way too slow to be used in any moderately busy system. The disk space utilization is also _prohibitive_ for archival.
We still rely on remote syslog because of this, but it's not without issues: the journal->syslog translation is lossy, and it's compounded with some "quirky" upstream decisions (purely cosmetic) to write unit descriptions in the log as opposed to the real unit name which make tracking down units harder if you don't have the actual journal file.
Journal files also get corrupted easily with unclean shutdown. In case of issues (from software to hardware), the local journal quickly becomes completely useless.
All in all, systemd is an improvement, but it lacks the debugging tools, performance and overall polish you would expect from a central piece of software such as init.
The internal APIs are mostly undocumented or self-referential. The DBus interface is opaque. Things gets muddy very fast as soon as you touch non-central components such as systemd-resolved or networkd.
I'm not a detractor, but there are many things systemd can still improve, but it feels we're kind of stuck. I'm quite happy if we have some competition here.
Huh? I thought that they introduced this weird binary log format precisely to avoid this kind of error. (I don't remember the exact reasoning, though.)
There have been a pile of bugs I've run into and reported where journald gets really confused; e.g. 'journalctl -b-5' won't actually go back five boots, it'll show some other content, or it'll give an error, but using --since will work fine. I haven't seen this problem recently, so the fix may be trickling out.
I have no idea what version of systemd Ubuntu 16.04 has, but in systemd-233 I'm not seeing aforementioned problems with -b switch and corrupt journal files.
well, he does say:
>There may be a bit of NIH syndrome leading to this decision; that’s ok, I can live with that;
>I don't really know about any of the drama surrounding systemd
I have only been a passive observer of said drama, it boils down to A) init was a stable, widely used, and well understood system that didn't need an update B) systemd takes on far too much responsibility, contrary to the unix philosophy.
personally, I have experienced only minor annoyance when having to re-learn familiar commands, though there seems to be a init compatibility layer that has made it pretty transparent to me. It'd probably have been more annoying if I were still an admin.
If it didn't need an update why did various parties try and replace it e.g. OSX launchd, Solaris SMF, Ubuntu Upstart?
Apple probably designed it so .plist files would be editable with some GUI tool.
It made all these things really easy, I'm not sure why everyone is so keen on doing everything with various bash scripts like we used to.
See http://uselessd.darknedgy.net/ProSystemdAntiSystemd/ for the System 5 rc fallacy and several others to avoid.
E.g. a typical issue is process restarting. Simple. Can be done in a couple of dozen lines. Except that solves nothing for most people: If your server is spinning in place trying to start your big-proper-process-monitor, and it's failing to log anything because rsyslogd or whatever is also just spinning in place, and the boot processes that should have mounted the filesystems needed to get things working is repeatedly crashing?
You can mitigate all of that, but now you're creating a separate mechanism (maybe as simple as writing to disk, but still) for early-stage logging, and a separate mechanism and your init needs to deal with things like throttling restart and handling them and so does a separate process manager, and so on.
In order to simplify things, there's duplication of functionality all over the place.
A huge part of the architecture of systemd is down to starting to pull at that thread and untangling all of these issues. Is it perfect? No. Is it the solution to all of the concerns they've uncovered? Probably not. But what we had before was so much worse, and I'm so happy someone have had a go at addressing them.
And frankly, it's great that animosity towards systemd has motivated so many people to finally try to address this issues in other ways. Maybe one of them eventually solves what none of them did before systemd forced peoples hand.
I mean, for my simple uses, what is wrong with `sudo service nginx restart`, or `sudo killall nginx`? For many people, it is really dead simple. I get that it is not simple for everybody, but that is why we should have choice, not one-size-fits-all.
Not sure, but I'm using SystemD and do this literal command nearly daily. What exactly is your point here?
[chris@f26h ~]$ sudo service smartd.service status
Redirecting to /bin/systemctl status smartd.serviceCanonical fought back for a while with upstart, but they lost.
See, RedHat got their assorted junk integrated tightly into GNOME (partly also KDE) and "legacy" options killed.
It is like a modern car engine which is just a sealed black box under the hood, full of electronics. I mean, for my simple uses, what is wrong with `sudo service nginx restart`, or `sudo killall nginx`?
Why do you think killall wouldn't work? It's present by default on my Fedora machine here...
That's a fallacy. Webkit is open source, doesn't mean it is not completely opaque to the majority of developers. I'm not concerned about the legal free/open source status here, but about how well a casual user can grasp and change things (that is someone who has to start, stop, maybe install services, and isn't a systemd developer).
I'm not bothered that the name of the tool is now `systemctl` of course. My problem is that is does way to many things. Want to list services? It shows you hundreds of pseudo-service-objects. They can get activated at any time by mysterious means. If you need this, it's great that it is available, but for me, the old ways (as many failings that they had) were good enough.
I'm not sure that's a good argument. If you have a medical problem, do you try to use it yourself?
Do you wish that you were able to fix your computer CPU when it breaks or do you just buy another? If you were to make your car/computer more fixable, it would probably be more expensive/less reliable/...
Whether SysVinit scripts or systemd scripts, you have to learn something to do either one. I just see it as a difference rather than which is simpler. But systemd can use SysVinit scripts anyway if you prefer it.
As for simple incantations, "sudo service nginx restart" works just fine on Debian with systemd. It's trivial to provide wrappers. But not exactly hard to learn "sudo systemctl restart nginx" as an alternative. There is a small but real learning curve to replace your old habits, yes.
As for "sudo killall nginx" it is indeed easy, but it's also an awful habit. Suddenly you're on a server with multiple different nginx instances (in different containers, perhaps; been there, killed those by accident; stopped using killall). Or you're applying killall to a process that will leave a mess. Or you have a server process that spawns children that double-fork and won't die just because you've killed the server.
That simple case is indeed simple. Until it isn't. And that's one of the many caveats where systemd goes far further than most of the alternatives by e.g. using cgroups to keep track of related processes even when they do stupid stuff to confuse you. In return you get to do "sudo systemctl stop [unit]" and know that not only will it be stopped properly - along with all related processes, but any other dependencies will get shut down cleanly.
We do have choice, but in most cases where the "dead simple" cases matter, it is a matter of relearning a handful of commands (with wrappers for some helpfully provided by some distros) for the dead simple cases to work, with the bonus that all kinds of ugly corner cases most people don't even think about just stops being a concern.
If your needs are dead simple, then systemd is dead simple too. When your needs are not dead simple, most of the alternatives can't solve them properly, and you often end up having to cobble together awful monstrosities. Or trying to figure out what in the world went wrong.
Also, I'm sure journald is useful, but I've had so many problems with it. For me as a Desktop and occasional remote computing, small server user, logs are ephemeral. Something goes wrong, I can quickly grep the logs which are plain text. I didn't really feel any advantage from the modern, binary, structured logs. Also, I spent hours installing something on WSL (Linux on Windows), finding that everything works except for the systemd logs, which are empty.
The command that controls services spits out hundreds of cryptic lines, services interweaved with all kinds of other objects, when I just want to list the running services. It has too many moving parts for my taste.
I don't need socket activation. I don't need a separate DNS resolver in my init system (unrelated note: I'd love more command line/GUI tools to be able to use a separate DNS resolver, work on separate interfaces, ...!), and so on.
Even if I will never use this guy's init system, if it is even a bit successful it will mean that people will stop assuming systemd, leading to a looser coupling, more interface stability, easier porting between systems, and more choice. So I think it's a great thing he is trying.
It's. Not. No idea why people keep repeating this. The DNS resolver belongs to systemd-resolved, is entirely optional and off by default. Nothing about it is in the init system.
And maybe you don't need socket activation, but there are plenty of valid use cases for it.
See: http://0pointer.de/blog/projects/socket-activation.html
If they want to change that perception, they will need to change the name (and hopefully stick a stable ABI around it, but that's unlikely to ever happen).
The DNS resolver does not belong in the init system or part of it. You just need a working resolv.conf.
By that logic why use Linux in the first place when Windows is so much more popular?
If it's anything like other projects from the same source systemd is great when it works and impossible to debug when it doesn't. Which is exactly what I came to OSS to get away from.
Old school sysadmins and those who actually get their hands dirty with Linux systems under the hood despise it for what it is. It's the windowsfication of Linux, in my - and many others - opinions. A giant PID 1 that controls everything, re-implementing such basics as ntp and rm, is imo entirely anti-unix-like.
Systemd is great until it breaks. Once it breaks though, good luck. They've re-invented the wheel multiple times over, and as time goes on more and more things become part of monolithic systemd. If you do happen to find a weird bug, prepare to get yelled at and be forced to vigorously defend your bug report. The joke in the office is when Pulseaudio support gets added.
Kernel is also a giant code that controls everything. Is this such a bad thing?
Small tools that do as little as possible, but do it very well, composed into bigger systems.
Philosophically identical to RISC vs CISC.
Even with the kernel there is an element of choice about what parts of it you do / don't use, e.g. you can use ext4, XFS etc. etc. and even have the choice of providing kernel modules so you can bolt on ZFS.
The kernel doesn't start to dictate who or what software does DNS resolution, for example.
systemd keeps taking over more and more control of various aspects of your system and effectively removing choice by making it hard to use alternatives because it starts to make assumptions based on the inbuilt versions.
systemd is presenting you with a "one-size-fits-all" scenario. The problem is one size never fits all.
One of the biggest strengths within the *nix ecosystem has been the fundamental understanding no two use cases are entirely the same, what works well for one purpose doesn't work well for others, and choosing the right software to meet those various purposes is invaluable. It's impossible to make software that is efficient for every single use case.
The freedom and possibility to choose the right tool for the right task is essential to designing and building new, innovative, or even just plain old efficient systems.
dnsmas
Well written and agreed with but I'd add management of legal risk and labor productivity.
You could compile a http server into the kernel, which would benefit microscopically in ways almost all people don't care about, then product tie it into other projects to successfully kill nginx and apache off.
Or you could simply run Apache in a non-monolithic way.
What happens if a submarine patent surfaces? Or a critical "must fix this instant" security bug hits?
Well, if you're running Apache and not doing anything too weird, you'll be back up and running on nginx in a couple minutes and the world yawns. Or you can upgrade/downgrade/patch apache without messing up your DNS service or NTP service or whatever else.
But what happens on a monolith? Well, you could ... I guess you decide if you're better off infringing the patent or replacing the ENTIRE monolith.
Those things happen. Eventually its inevitable that something in the monolith will become unacceptable to run, legally or from a security perspective, and then the entire monolith has to go, simultaneously. Life with separate tools is incredibly easier.
Embrace, extend, submarine patent, extinguish.
There are also labor productivity issues with monoliths. Best expressed via a standard HN automobile analogy, where technically I could completely restore a vintage jeep automobile using one tool, a swiss army knife. Simplistically, all automobile restorations should be done with swiss army knives to minimize the paradox of choice and minimize the massive expense of tool training. In practice it seems obvious I would infinitely prefer not restore an automobile using a single swiss army knife as my only tool. Even a really bad toolbox would be better than using one swiss army knife. Given the choice between using an old, slight imperfect yet historically proven highly successful toolbox vs only being able to use one brand new untested swiss army knife... thats the kind of thing that makes me glad I use *bsd instead of linux.
You can also do the same with systemd. Don't want to use journalctl? Fine, forward logs to syslog. You can argue that you can't disable journalctl, however depending of the feature you want in kernel you can't disable some modules thanks to inter-dependency.
> The kernel doesn't start to dictate who or what software does DNS resolution, for example.
Neither does systemd. I don't use systemd-networkd at all in my machine and it still works fine.
> systemd keeps taking over more and more control of various aspects of your system and effectively removing choice by making it hard to use alternatives because it starts to make assumptions based on the inbuilt versions.
I kinda concur with you, however I also try to understand why they're adding support for all those things. In general, there is a sound resolution to include something in systemd, even if I or you does not agree.
And remember, except for journalctl, systemd does not force anything at you if you just want its init system.
> systemd is presenting you with a "one-size-fits-all" scenario. The problem is one size never fits all.
It is not. All systemd implements are foundations to build bigger and more advanced functionality. systemd-nspawn is basically chmod on steroids, you can't call it a container system. However, you can use systemd-nspawn to build a container system like rkt. systemd-networkd is also too bare bones to be called a "network manager", etc etc.
> One of the biggest strengths within the *nix ecosystem has been the fundamental understanding no two use cases are entirely the same, what works well for one purpose doesn't work well for others, and choosing the right software to meet those various purposes is invaluable. It's impossible to make software that is efficient for every single use case.
See above response.
> The freedom and possibility to choose the right tool for the right task is essential to designing and building new, innovative, or even just plain old efficient systems.
Yeah, that is why systemd does not force you to use anything (except journalctl).
Even if systemd is at the very high standard of kernel reliability, it would be better design to break out the monolithic parts where possible. The kernel is the way it is because it needs to be, not because it's a preferable way of writing code.
If you're a programmer I highly recommend taking a quick browser through the systemd source code, it doesn't take long to see that it isn't really monolithic at all and is in fact a combination of individual tools which are combined to accomplish things. Just because they're all under the systemd umbrella and all in the same repository does not mean it's all one binary and that parts can't be swapped out.
Systemd has over the sprouted a whole bunch of parts that fit large (cloud) scale deployments.
Where things get messy is when all you want to do is to spin up a gray box on the shelf to handle some internal data, and at that point systemd, and a whole bunch of other "plumbing" that has sprouted within the Linux ecosystem in recent years (often under the Freedesktop umbrella, even though it seems to have a ill fit on desktops), becomes a beast to wrangle into submission rather than an assistant.
> A giant PID 1 that controls everything
> monolithic systemd
Technically speaking only small portion of systemd is running as pid 1. This has been explained many times: https://news.ycombinator.com/item?id=9515789 and I'm not going to explain this again.
> re-implementing such basics as ntp
Systemd-timesyncd is a minimal ntp client. This is not a replacement for ntp. You are free to use full blown ntpd if you want.
Think of it as a dns resolver inside your libc vs running proper caching dns server. It's the same situation.
> more and more things become part of monolithic systemd
Well, systemd is a particular answer to a set of problems that have been well known for a while. Building an alternative solutions to those problems is not necessarily "another systemd".
> drama surrounding systemd
The quick summary: many people are critical of the architecture and design choices and implementation details in systemd in particular, rather than that the underlying issues it's supposed to address are real.
> but I've liked using [systemd] so far
This isn't surprising, as some of the problems it addresses are real, and a pain. But there are other implementations that also address many of the same issues. Whether those would make you as happy as systemd is another matter.
I've come to accept systemd and actually like it in the sense that, for the most part, it just works and gets out of my way. The more I learn about journalctl's options for viewing logs, the more I like it.
The case of Linux going its own way that drives me nuts though is the replacement of 'ifconfig'. The replacement seems to be in progress, with some distributions shipping 'ifconfig' in addition to the 'ip' and others only shipping 'ip'.
EDIT: Other Unixes have also gone their own way with init systems to meet their own needs. MacOS, for example, replaced initd with launchd. Solaris, way back in 2005 with Solaris 10, replaced init with SMF[0] which does some of the things that systemd does on Linux.
[0] https://en.wikipedia.org/wiki/Service_Management_Facility
Solaris is Unix. Solaris shipped SMF over a decade ago.
OS X is Unix. OS X shipped launchd in 10.4, back in 2005.
At best I could say that the sheer scope of systemd is an example of Linux going its own way. SMF and launchd, for example, don't change how logging is done. I don't believe either of those include their own DNS resolver either.
People claiming systemd is some slapped-together piece of junk with bad architecture are probably wrong.
People who think they can write a better systemd on their first try are probably wrong.
This stuff is really hard.
So you are wanting to build a new init system from scratch and you haven't taken the time to see all the stuff that already exists? I mean if this is just a project you want to do go for it but if you are seriously trying to build something for others you know whats out there and why people choose one over the other beyond just systemd and sys V.
But in fairness Davin McCall has done more than most. There is a short writeup of xyr opinions of various systems, that comes with the Dinit source code. It gets a number of things wrong about several of the systems and does not go into as much depth as, say, the Debian Technical Committee members did. But it does show that M. McCall has at least looked, just as xe states.
Honestly, my impression is that this is how everybody that did something widely adopted acted.
I don't think it's desirable, bu there may be a problem with taking your time to review everything out there.
- an integrated inetd that can launch non-root processes - ability to chroot() any application at launch - inotify capability (path units) - consistent scripted application startup and control
Just about any UNIX could really use an init that can do all of that stuff (even if they don't have cgroups). If you can't, thanks for playing.
FreeBSD's init system works well enough, but there's no question that it could be better.
Problems or features? I haven't run in to any problems.
From a feature perspective I've run into the system policy bug where the name of a package has a minimal relationship with the name of the file read in /etc/rc.conf.d... For example, say you're installing the package named "isc-dhcp43-server", you need to add a file named /etc/rc.conf.d/dhcpd. Why not a file named something "obvious" like /etc/rc.conf.d/isc-dhcp43-server ? Sometimes the names are close enough to be Extremely annoying, like package name "logstash-forwarder" and /etc/rc.conf.d/logstash_forwarder. Obviously thats not a bug in the init system, thats a bug mis-using the init system. Or is that mis_using? Or misusingd-4.2?
Its the kind of issue thats super annoying one time when you write the ansible script and then you forget about it until next time you're writing an ansible script to do something else.
I have not run into any lack of features problems with freebsd init. Although I admit to having run some end user applications (not system stuff) using supervisord because the end users wanted a pretty web interface for restarting and logs and stuff.
An ideal init system would have known (or been taught) that services x, y, and z depend on NFS mounts /a, /b, and /c, so the system could have started mounting those in the background and let other services (like sshd!) start up as required.
As to your point about the service names, that bugs me as well.
* http://www.mail-archive.com/supervision@list.skarnet.org/msg...
C++ needs a lot more discipline and thought if you don't want unmaintainable code years down the line. Ironic given that abstractions introduced by C++ are partly to make code more efficient to write, extend and maintain.
Awesome, maybe I won't have to!
>C++
Whelp, nevermind.