Systemd: The Biggest Myths
0pointer.de
0pointer.de
* "Not our fault, it's that other service."
* "We are never going to support that. You don't want it anyway."
* "Well that will be fixed with systemd"
* "You can add the user to the audio group, but that's wrong and you should not do it. You should instead do that thing that the other guys are telling you not to do and that they will never support."
* "This will be fixed with systemd!"
So, I guess now the question is, when someone comes along and says, "I need to do X, I used to be able to do X before systemd was in use and now systemd is stopping me, how can I fixed that?" will systemd maintainers respond with things like the above? Will there be useful and thorough documentation, so that users can fix things without having to bug the maintainers?
>Completely wrong. The BSD folks are pretty much uninterested in systemd. If systemd was portable, this would change nothing, they still wouldn't adopt it
systemd is not portable, it wasn't made to be portable, it relies on far too many linuxisms. Therefore the BSD's don't care about it. It is not 'the BSD's don't care about it so we didn't make it portable.'
launchd (OS X) and SMF (solaris) weren't portable, either.
By the way, when was the last time a BSD operating system cared about making their init system portable to Linux?
I'm not sure why systemd not being portable to other operating systems is suddenly a big deal.
To support systemd, the number of configs you provide will go from maybe 4 to 5, not from 1 to 2! And, looking at the examples provided by ArchLinux, the systemd services look as if they are a lot less boiler-plate than the typical debian init-script.
https://wiki.archlinux.org/index.php/Systemd/Services (e.g. dropbear is very simple)
Because software being written that has systemd as a dependency will be Linux only software.
And this is news, and suddenly it's systemd's fault?
If you hate Linux-only software, software that depends on systemd is the last of your problems. Software that depends on glibc and the Linux kernel are probable the main offenders.
The same is true for a lot of Linuxisms. I have spent an awful lot of time getting software that states on the tin it is written for POSIX to run on BSD because its source code uses all kinds of Linuxisms.
The same holds true for people assuming that /bin/sh is bash, which may be true on certain Linux distributions but isn't so most other systems. I've started noticing the same issues with other things as well, such as the symlink from /bin to /usr/bin, /sbin to /usr/bin ... whereby software believes all software lives in /usr/bin when in reality that isn't the case for the BSD's.
1) Speed does matter. Fast boots are good. 2) The new journal is just better. Finding something in the logs is easier. 3) Service files are easier to write than init scripts and one can have more confidence they will work as intended as you need write very little configuration oneself. 4) Knowledge of dependencies means I never have to worry about starting dbus before gdm. It just does what is necessary to do the correct thing.
I personally want all of those things. I am really happy they came to Arch and Fedora. They both feel much more modern for it. Working on systems that use older init systems now feels archaic.
Let's move with the times. I remember the same set of complaints when we got NetworkManager. I remember the same complaints with PulseAudio.
Guess what? I now have systems with excellent networking and excellent sound.
If NetworkManager is an example of how great systemd can be, then I just became vehemently opposed to it.
And this is perfectly the reason number two why i won't switch to systemd (reason number one is there is no need for it). PulseAudio does not work. It sucks. The irc channel is full of clueless (but trying to be helpfull) people. And nobody cares. I want my Alsa back. But now it's too late, i hope systemd will either be better (doubt it) or never be this popular.
And, I cannot lie, I don't want it because I don't want another dbus. freedesktop, stop turning my linux into a single user os. Next thing is they want to abolish x11. Sure it's about time for x12, but wayland? I just don't understand the community that surrounds Linux anymore.
But I don't have any issues. I don't have high CPU usage. It just plain works since various years.
Fundamentally, systemd tried solving too many problems at once, in a ways that inadvertently annoyed people. It replaced so much of the core infrastructure that upgrading systems resulted in an admin experience that feels alien. Giving people new tools to deal with things like binary logging != instantly changing every admin's CLI muscle memory.
I'm not saying that systemd doesn't solve valid problems (the issues it addresses are truly quite important) - it just goes about it in a dramatically jarring way.
Not really...
I prefer reading "This is wrong, because ..." over "This is right, but ...", since it's more honest.
https://bugzilla.redhat.com/show_bug.cgi?id=461546
First, he denies that this is something people need. Then, when he is told that people need it, he says that they are just doing things the wrong way. Then, when people point out that this is neither unusual nor the wrong thing to want to do, he just repeats that this is not what was intended so too bad. He offers no advice on how to do things the right way despite being the only person who seems to think that everyone else is wrong.
------------------------------- Internal Server Error
TracError: IOError: [Errno 2] No such file or directory: '/home/lennart/svn/trac/pulseaudio/VERSION'
--------------------------------
That kind of sums up.
For data shared by sevaral users, I very much prefer some separate /data/service/instance/... structure. Also much easier to allocate sane directory permissions.
This doesn't answer your question of course. There is still the 'why' of it.
Why has this become the standard?
The homedir is a volatile place. Where users reside often. And make changes often. Any place humans change things is a place where stuff breaks easily and on a regular basis. What if Lennard calls 'rm' with a wrong switch? Or chown? Or chmod? The service receives pain.
Or he could decide that he no longer wants his homedir to be readable/executable by everyone on the system, so he chmod 0700 his homedir. Forgetting that this will render his trac instance unavailable.
Or what about sharing some responsibility? How is Lennard ever going to share the maintenance of the ticketing system with anyone else? What impact will that have on the ticketing system? Will it have to be migrated anyway to /srv or /usr/local or /opt? What configs will that affect? What about the webserver configs? What about file permissions, uids, gids?
What about backups? Are they configured for this specific service? If it is ever migrated to another location, do they have to be reconfigured? Or will /home/lennard just be backed up as before and no one will remember that trac is now in /srv/trac, making a restore improbable.
What about a reboot? Will the service come back up? What if you have to re-implement this service on another machine after this one explodes? Are you sure you did not forget to point /etc/rc.local (or some other hack) to your new machine?
These are only the reasons I could think of off the top of my head. And most of these are actual examples from the field.
A standard is a standard, because it works for more people than just you. The question you, and lennard, should be asking yourself is: How can these standards work for me as well?
(As you can probably tell, I grew up in Ops, not Dev. I have seen the pain of the 'works-for-me'-mentality.)
Also, cnvogel is obviously more consise than me. :-)
I don't know whether existing open bugs were moved over at the time though.
"Why knowing your hardware is important" http://www.youtube.com/watch?v=QUUdVFZBd5g
"Scalable Parallel Programming Techniques" http://www.youtube.com/watch?v=jfimI7UC9Pg
When a problem is easily solved by just an "apt-get remove X" it is easy to start hating the piece of software. No matter who was actually at fault.
By making Linux audio temporarily problematic Linux audio became a solved problem (at least in my mind, and the minds of many users of the major distributions that use it)
Stop it.
And yes I reported the bugs.
I realize that is all anecdotal and I could be one of the few people in the world who still have problems with it.
And everytime, without fail, audio starts working again.
That's pretty much where my analogy falls apart, so I'm ditching it.
Whether I'm capable of contributing to a project that does not, in its current state, suit my needs has nothing to do with whether I'm in a position to contribute to that project.
This is especially true when there is an alternative that I'm already accustomed to using.
Poettering has clearly the better arguments than this reactionary Draxinger die-hard.
Could you provide the counterarguments?
I don't have a gnome session in my login manager because I don't need i18n for the words "login" and "password". (although that would be totally possible without a whole gnome session). I don't have a braille line so i can't say anything about accessibility, but that should be (in a ideal unix world) a single printf to /dev/braille. Also the feedback is audible and visual, this seems to be no issue to me.
Edit: Plus saying X breaks Y so X is not needed is not a good argument. Because you have not addressed if X is needed or not, only said that Y breaks.
I think I can safely say that if X breaks Y, and I need Y than X sucks. Especially if it worked before X came. X made things worse.
/edit: I think that's why Poettering is hated so much. There was community on the Nixes, with their own ecosystem and their own way to do stuff. It's wasn't always beautiful or nice, but their had their way to solve stuff so it would a kind of blend in with the rest. And then Poettering wants to turn it into MacOS X or Windows and adds stuff new users probably appreciate, but the old users never misses. There is a cultural shock. And then it breaks backwards compatibility. It shouldn't suprise anybody that this product should rather be superb and not just medicore to accepted.
The problem with "it's not the UNIX way" argument is that a lot of people don't care about UNIX that much, they just want a good system. Poettering mentions in that video that all his, and his colleagues' work (pulseaudio, consolekit, udev into udisks/upower) has been done because there was an actual demand for those features. There is a reason why most distributions use their work even as people complain about them.
Is there really anything at all that a majority of Unix-derivative users can agree on?
Even without such an impressive track-record, people should at least give him the benefit of the doubt.
Even a cursory look into systemd design debunks most of the "criticism" people have against it.
Without issues? PulseAudio is loaded with issues. The only time PA works correctly is when you are doing things the way Lennart thinks you should do them -- which is basically the way desktop Windows and Mac OS X users are expected to do things. Another way to put things is this: Lennart's software designs dictate how users should use their computer, to the exclusion of all other use-cases. Doing something cool or unusual with PulseAudio is like pulling teeth.
So in a sense, you are right: PA is not perfect, and neither were ALSA, OSS, and other earlier approaches. That is not the issue; the issue is, is PA actually better? Are we actually able to do things now that we were having trouble with before? From where I sit, PA makes it easier to use things like D-BUS and various "Kit" systems to duplicate the desktop user paradigm you see in Windows or Mac OS X, and nothing more (and that does nothing for me, but certainly gets in my way when I try to do things like multiseat setups).
I remember trying to get bluetooth audio working in the alsa-only days. Or network sound with esd/ssh. Or figuring out how to route audio nicely apps. Or saving my laptop battery. Or many other features that were simply damn difficult to pull off.
People don't seem to remember how crappy the audio situation used to be, for users and for ISVs. I respect Lennart and the current PA maintainers a lot for tackling the problem head on – even if they didn't or couldn't do it perfectly – instead of just complaining about the sad state of Linux audio, especially in the face of all the criticism they got over it.
In exchange for that I put up with confusing audio dialog boxes, high audio latency, high cpu usage when playing audio, skips and crackles when playing games, arcane commands to enable loopback, and the occasional necessary restart of the pulseaudio daemon when sound stops altogether. I guess that's a reasonable trade.
Actually, serious audio work on Linux is done with JACK. PulseAudio is completely inadequate for serious audio work.
Latency vs. CPU usage is always a compromise. To achieve low latency, you need very small buffers for "rendered" audio, and lots of well-timed copy operations to the audio hardware. To just play some audio files efficiently, you want large buffers, because then fewer copy operations are necessary.
Here's a better explanation by Lennart Poettering: http://0pointer.de/blog/projects/when-pa-and-when-not.html
> Completely wrong. The BSD folks are pretty much uninterested in systemd. If systemd was portable, this would change nothing, they still wouldn't adopt it.
> 15. Myth: systemd could be ported to other kernels if its maintainers just wanted to.
> That is simply not true. Porting systemd to other kernel is not feasible. We just use too many Linux-specific interfaces.
So what happens to software written for Linux that can be (and currently is) ported to BSD when it starts to require systemd?
It's to interact with the lower plumbing - ie. set up hostname, locale, timedate, multi-seat etc.
If these interfaces are missing, the software should still work minus a few missing features. Applications could still #ifdef all they want to make things work on BSDs, no change there either.
Or the BSDs could make their own implementation of the plumbing interfaces. Lennart even provided a handy chart: http://www.freedesktop.org/wiki/Software/systemd/InterfacePo...
Honestly, it's about time we standardize and stop wasting time on these low-level things. I'd rather work on software higher up in the stack, than wasting my time working around trivial differences between the Linux distros (bonus points if they get adopted on other the BSDs too). But that's just me.
5. The systemd documentation is indeed very good, and that's probably one of the biggest drivers behind its adoption. However, it is also difficult. A big part of the pushback from people over systemd is that it also replaced syslog, and did so with its own custom binary log format. To quote from a forum thread I started shortly after updating my old system to systemd, "Getting smbd/nmbd to work again was a real adventure. Like other users reported, it would just silently fail when starting it from systemd. No error message when issuing the start command, and only a vague "failed" in status. I ended up having to track down Lennart's blog post on "systemd for administrators" to figure out how in blazes to extract anything useful from that cussed binary log system he invented. My first half-dozen or so attempts to get anything useful out of the log journal got exactly zero results; I finally got lucky on another approach..." (I ended up abandoning that distribution altogether after that and a number of other frustrations, and the response from the forums.)
13. The problem for BSDs isn't so much that systemd is or isn't portable to them; it's that some upstream software is beginning to require systemd, making that software difficult (or impossible) to port to BSD.
14. It seems weird to me to hear someone else decide for other people what is or isn't a "negligible" amount of work.
15. So ... systemd is in fact Linux-only by design. How does that jive with 13 again?
19. systemd may not "force" you to do anything, up until your distribution adopts it, pushes it as an update, and then you find yourself spending hours trying to figure out how to troubleshoot a problem that didn't exist before the update. Then it certainly is forcing you to do something.
Here's the problem in a nut shell as I see it: if systemd had been the default in Linux for the last ten years, probably the tool chain around it would be mature enough to meet everyone's needs, we would all be accustomed to the specific commands needed to control and interact with and debug systemd stuff, and if someone came along and proposed replacing everything with a syslog daemon and a pile of init scripts, there'd be rage and outcry. That is to say, I don't see anything inherently bad about systemd.
But, what is enormously frustrating is to have something that works, and be well adapted to it, so that if something breaks I know exactly where to look, and then have all of that be replaced by a foreign system that breaks old things in new ways and requires hours spent trying to figure out what the hell happened.
If the replacement system offers serious benefits over the old system, that offsets the pain slightly. In this case, I've yet to see what the actual benefits are; I have no idea what problems systemd is attempting to solve which are so severe, so immediate, so intractable that they require a jarring change to some of the fundamental parts of the operating system.
However, I run Linux in three locations and here's where systemd fits for me:
1. Servers. No use whatsoever. There are two power states: off and on. None of this is really an improvement over SVR4 init. It's just another damn tool to learn.
2. Desktops: No use whatsoever. I run mine like servers simply because if I need remote access, trusting stuff like WOL and ACPI is just silly on Linux.
3. Laptops: No use whatsoever. This is simply down to the fact that the power management on Linux i.e. hibernate/sleep support is a bag of shit. I run mine as power states off and on, much as 1 and 2.
Regarding boot speed - my laptop boots in about 14 seconds (SSD). Who cares about making it faster?
I find that a shell script is far more useful i.e. it doesn't enforce constraints on you which have to then be added as features to systemd. Plus shell scripts are generic tools i.e. learning how to write them will be of more use globally than learning systemd guts.
So basically, screw it.
The problem that this outlines is that Linux's power management, event and ACPI state management is crappy. We don't need another layer of crap over the top of it to make it work properly.
1. It is way easier to write a systemd unit for an in house application than it is to write a correct sysvinit script.
2. Everything runs in a cgroup making it easier to add resource limits and see which process is started by which service.
3. Easy to check which daemons are running and not.
4. Reduced startup time for containers.
1. Not really. Most sysadmins have a decent template init file sitting around or you just steal another one on the machine and modify it. They are also more flexible as some startup processes aren't quite as simple as systemd thinks (consider lockfiles, temp data purging, permissions etc).
2. ulimit / selinux - per process. cgroup whilst funky looking is YET ANOTHER disposable mechanism which will no doubt get canned in 5 years like ipchains ipfwadm etc.
3. service --status-all
4. Concurrent startup yes. That really doesn't make much difference on a server with large IO and CPU capacity.
2. cgroups can do way more things than ulimit.
3. True
4. Have not used enoug containers to know how much it matters in practice.
How? This is one of the things that have really annoyed me as all the systemd services have gone from service --status-all.
systemctl | grep running
systemctl --type=service
systemctl --type=service | grep running
I'm not linux admin anymore, but I when I was, I used various service status software thingies only to find out whether the system thinks the service is running, because they tend to be wrong, and just annoy you (not starting crashed server because it "is already running" or something similar). I think debian doesn't even track service states, but I'm not sure.
Of course, with systemd it will be even more fun.
For example: we use backuppc for automated network and remote server backups. The backuppc data volume is a TrueCrypt-encrypted drive. Creating an init script that checked to make sure that the volume was present and mounted before launching the backup daemon was fairly straightforward. I understand that I could still do this in systemd, but the catch is that I would have to do it with a shell script -- the same way I am now, except for a different, alien interface -- because the native system doesn't have support for things like that. (This is assuming that some combination of unit config parameters couldn't do it; the commands for finding the drive and checking its status were a little fiddly, and I honestly haven't tried to do this the systemd way. Still though: once you understand shell scripting, you can do anything on Linux.)
Learning a new tool which doesn't teach you anything new except how to use this particular tool, is mildly bad. That wasted your time.
No! These are not mutually exclusive. The goal should be to restart less ofter, but restart faster when it is necessary. Systems have to restart. Well run/maintained/designed systems may not need to restart ofter, but they do need to restart.
Consider: Many moons ago I was responsible for (among other things) the company's CRM systems. These were a couple of mid-range Suns that were extremely reliable. On an anual basis, the business operations department issues downdown cost estimates for business critical systems (these were used to develop inter-departmental SLAs), and estimated that the CRM database server cost the company about $250K per hour of down time, or, almost $4200 per minute. Or read another way: a 10 seat license for our software in a saturated (niche) market.
The point is, outages happen (no you can not account for all possible failure scenarios), and startup times matter.
Your're going to have to clarify, as this statement doesn't make any sense.
You're right though. In this instance, system startup was already highly optimized. In fact, the order of startup was shifted around to get Oracle (the DB in this example) up and operational as early as possible.
On the other hand, with the introduction of SMF, that "need to optimise" largely went away. Sure, SMF didn't make much a difference in this particular instance, but parallelisation does make a difference when you're talking about total system (not service) startup time. Further it can make a difference on system that run multiple services. If 5 services have the same dependency set, then a parallel startup means that services 2-5 become available earlier versus a linear startup.
Finally, shaving 5 seconds off of service startup _does_ make a difference. While the above example means $350, there's a few things to consider.
1) a $350 loss is stil a loss. From a business perspective, a loss, no matter how small, is still to be avoided. There's other considerations as well, such as the impact of the extra 5 seconds on my budget (the SLAs mean that the loss comes out of my budget).
2) In the realm of big enterprise, this is a relatively small outage cost. In fact, I've managed system with an order of magnitude larger failure cost (SLAs are bitch).
3) It's difficult to factor soft costs. Things such as customer confidence, indirect productivity loss, etc. 5 seconds, again, means services are available sooner, lessening the impact of these soft, and difficult to quantify, costs.
Faster restart time is irrelevant to me. I'm already designed to deal with outages. Reducing the impact of a restart is extremely important (making them fewer in number, smaller in scope).
But even when not business critical, it is nice that you can bring it up quickly.
My assessment on the systemd debate, though, is that it feels a bit like neophobia and the technical justifications (against systemd) are a bit on the weak side.
But, I've been meaning to write about this for a while: I think neophobia is starting to become an unnecessarily religious article of faith in the technical community. I run a small consulting shop, we support (or try to support) just about everything under the sun. What this means is that every single time there's a significant hardware change, my hardware guy has to be on top of it; every time there's a new mobile device UI change, or platform, or service or software, he has to get to know it right away; every time PHP or MySQL or Linode or any of a number of aspects of Linux changes, I have to be on top it; every time a new operating system version is released, we have to be familiar with it.
And every single one of those little ecosystems assumes that you have the time to sit down and read and digest their documentation or play with their new way of doing things. And, if you don't, or if you miss something, you're chastised by the community (or, worse, by your customers).
It is exactly like being in college and having every one of your professors assign 3 hours' work each night and then wonder why you're making a big deal of it.
To make things worse, when something goes wrong, you know as well as I do that you rely heavily on familiarity. You don't want to be looking things up in a man page trying to remember the specific incantation for a particular thing while your budget shrinks by the second. I was lucky that the smbd/nmbd fault happened on my personal system; if it had happened to one of our clients, who has everyone using a centralized smbd share, then smbd failing to start would have stopped work for the entire company.
So, yes, it's true that I am not a fan of change for change's sake. Lennart argues that that's not what's happening with systemd. Maybe he's right. But, it is still another massive change in something that I use and support on a daily basis, that is going to irretrievably consume a little bit more time out of my limited life, that is going to force me to throw out all of the old familiar tactics ("hmm ... new client, their MySQL isn't starting, dunno where MySQL logs are on this system, let's start with `grep -R mysql /var/log/∗` ... oh wait, this is a systemd system, that doesn't work").
From that standpoint, I feel like the technical justifications in favor of systemd are a bit on the weak side.
I see other developers/sysadmins here saying that sysadmins usually have a template initscript which they can use to author new ones, but that's really still a lot of work if you want to do anything not covered by the template and it leads to duplicate code. (It's literal cut-n-paste programming.) Initscripts expose a lot of implementation details. Even worse, they're different for every Linux distro so the initscript for your application has to be different for every distro you want to be on.
I think the most telling thing here is, that the distro maintainers, who are the ones who write the bulk of the initscripts for Linux systems, are the ones pushing for the adoption. It's making their lives easier.
See, to me this is just a "shit barometer". If upstream software requires systemd, then almost without fail it is shit and, at least outside the linux ecosystem, I am better off not using it.
6. Myth: systemd is not modular.
Not true at all. At compile time you have a number of ...
And ad-pulseaudionem is just like a ad-networkmanagerem or a ad-dbusiem: If your software design sucks and it does not work good enough that you don't have to care about this you will be told it sucks. And to me it seems that systemd is written in the spirit of dbus and nm.
"I’m not sure that this is a bad design, but it is most definitely not UNIX or anything like it."