The bad side of systemd: two recent systemd failures
utcc.utoronto.ca
utcc.utoronto.ca
Can we just agree to move on? I'm sick of the endless banter. If you like systemd, use it, if you don't like it, don't. If you think it needs more time to mature, stay away from it for a while and experiment occasionally. That's my plan at least.
The problem is that people are deciding whether or not they like systemd based on "do you like Lennart" or "do you like The UNIX Philosophy", because that's what people are making noise about.
The problems with journalctl are usability ones, which doesn't make them any less of a problem, but doesn't point to an underlying technical failure either.
The problems with systemd are basically stated as "a Fedora upgrade causes systemd to go bad." Is this a systemd problem or a Fedora problem?
... what kind of nonsense is that? Almost every library supports static linking to a basic degree at least. Heck, I statically link giant things like Qt and Boost every day at work.
I mean, you could, but now your staticly linked binary still depends on other dlsym'd code. So whats the point?
(Not trying to make an argument that this is good or bad behaviour)
Well, that is a surprise to me. And something of a concern.
* See for instance https://bugzilla.redhat.com/show_bug.cgi?id=915912#c18
Of course, this may not even solve the root cause of the problem this article talks about. It would make PID 1 a little more resillient though.
[0] With the number of different packages in a modern Linux, I don't expect everything in a distributions repository to work perfectly, but core stuff (like the stuff that gets you from a cold boot to a login prompt, etc) should be rock solid.
The points discussed here are serious, should be filed as bug reports, and should be acted on by the developers. But the article also just sounds like systemd bashing, software has bugs and poor thought out solutions - which can be fixed or improved - it's just code.
I could probably have written a similar article about many, many Fedora (and Debian) upgrades over the course of a decade or so about core components such as distro upgrades making the init system crashing, failing to start services, glibc upgrades screwing up servers, grub upgrades calling for a 2 hour car drive to fix and so on. But I didn't, I either filed a bug report, fixed the code, or was just too plain busy getting things back to working.
"The alternatives are certainly not perfect" is no consolation. At all. Especially considering the vast majority were never fairly considered to begin with.
That is so naive. No one chooses to use systemd. People choose a distro. It's being forced on long time users of distros without any of the users having a say.
Yeah, I can switch from one distro to another. That's a choice I have. But it's not so simple as you make it sound.
Switching distros is like moving to a new city. If you stay in the same state (or at least in the same country) it isn't too disorienting. But switching to something completely different (like on another continent, or going from an RPM-based to deb-based distro) will require some significant effort to adapt to the new situation.
We pick our distro of choice based on a number of factors. Just because it has some downsides doesn't mean we have to move immediately. But that doesn't mean we can't give (constructive!) feedback if there are problems.
I did. I've been systematically installing systemd on my Debian machines.
It's being adopted by distributions in accordance with the stated and implied wishes of the developers, maintainers, and users of the distributions.
The reason that (e.g.) Arch dropped support for legacy init so quickly was because they literally failed to find anyone who wanted to go through all the work of maintaining the old init system. If none of the users want to do the work, then nothing's being "forced" - that's the way community-organized FOSS projects work![0]
Same goes for Debian, who made the switch in accordiance with their normal development process (including a general resolution in support of systemd!). Debian has promised to maintain systemd-shim (supporting other init systems) through at least Jessie, and possibly longer, depending on whether there is enough demand for alternative init systems, and support from developers for maintaining it in future releases.
Ironically, the biggest threat to systemd-shim in Jessie's successor are people who claim they want to fork Debian - if all the people who hate systemd leave Debian, then nobody who wants to actually do the work to maintain the alternative init systems will remain with the Debian project[1].
[0] Even then, you can still find an AUR package for it (https://aur.archlinux.org/packages/initscripts-fork/), though as with everything in the AUR, it's subject to much less testing. Note that the package has only 27 votes - this means that only 27 people actually said that they care about moving this package back into the main distro.
[1] This may be a bad or a good thing, depending on who you ask.
Users != developers
I am not a developer. I'm an administrator. I know enough C to be able to poke around source code and understand what is happening when I get an error, but not enough of it or anything else to maintain an init system. A lot of admins don't know this much. A lot of desktop users don't know this much.
I'm not disagreeing with your overall sentiment here - but it's not so simple as "If the users of the product liked it, they'd maintain it" - sometimes they don't have the skills required to. Not everyone who drives is a mechanic, etc.
If you car requires more knowledge to maintain than you have, you can get a different car, pay someone to maintain it or get a bike. Those are the only options I see. I don't see how init is any different than a car. You have those same options. I'm sure someone somewhere is paying RH or canonical or in house devs/admins to maintain init scripts for their application stacks.
It isn't a matter of managing init scripts - that's easy. Continually updating a distro to use your init system of choice is a wholly different matter. I can change the oil in my car. I can't replace the transmission.
What I find quite unfair is that many people say that they don't have the technical skills to effectively contribute to the project, let alone develop or maintain any alternative, yet accuse the systemd team (or the GNOME team, or the NetworkManager team, or the XFCE team, or whatever) of producing sub-standard software. Given the self-asserted lack of skills, on which basis such a severe judgement is done?
Can we please go back to trusting each other, praising who actually do some useful work or at least avoiding open attacks to them? Technical disagreement is welcome, but it has to be backed by some skills to be useful.
And those developers and maintainers wanted to use systemd.
If, as a user, you disagree with the developers and maintainers of your choice distribution, you picked the wrong distro, and should either find developers and maintainers that align with your views or fork it and become a developer / maintainer yourself if nobody provides for your use case.
You cannot bitch on a high horse when not paying a cent for any of this about how the developers and maintainers are doing what they want with their distro. Well, you can, but nothing will ever happen - they already made their choice based on their needs and use cases.
No, we can't. Sorry but that's just how the world works. Just like every Go language post has "generics or crap" mentioned, every systemd post will have "The Unix Way" mentioned.
A price of freedom of speech is having to deal with self-filtering idiotic viewpoints. Worth it in my opinion, but I'm an anti-censorship for EVERYTHING person.
Have you tried removing systemd from latest Ubuntu? I didn't go very well for me. I imagine removing it from RHEL 7 won't go well over either.
> I'm sick of the endless banter.
Ok but "use something else" or "fork it" is not exactly solving the problem either.
This means that systemd is not insidious, it's valuable.
Ubuntu is not the Universal Operating System, that's Debian. This means that Ubuntu is definitely opinionated in its software selection, and rightly so. You couldn't replace Upstart and now Ubuntu relies on systemd. If you have the skills to fully evaluate the costs and benefits of mixing and matching random pieces of software, distribution like Debian or Gentoo may be a better fit for you.
Indeed, switching to sysvinit in Debian is trivial and the systemd team worked really really hard to make sure the transition is smooth even if one keeps switching back and forth. They should really be praised for the effort they've put in it.
Yes. When some developers are pushing down your (and mine) throat critical software that sucks _that_ much, that's the least someone can do.
Other developers picked up said package and added it to the package collection they are maintaining.
You choose to use the package collection maintained by said developers, which means that you trust their technical decisions otherwise you'd be running software from people you don't trust, which is not a great idea.
Nobody is pushing software down anyone's throat. Please avoid such hyperbolic mischaracterization, it really does not contribute anything to the discussion.
This does not however mean that I must silently accept their bad choices. If they were about to step out in front of a bus I wouldn't remain silent because I trust their judgement. Sometimes people you respect excercise poor judgement. This is one of those cases.
Feedback is a thing. You're giving it here. Why shouldn't I give it to the distros that certainly are trying to force systemd on me?
But there are proper places to discuss this issue with them, and HN is totally not one of those.
> Sometimes people you respect excercise poor judgement. This is one of those cases.
Sure, but have you considered that the one exercising poor judgement in this case may be you?
I don't know which distro you're using, but Debian has extensively discussed the issue and the first tech CTTE resolution made lots of valid technical points in favour of systemd. For sure it's not an obviously bad choice, and if your maintainers have chosen to ship it they may have valid reasons to do so. Not to mention that's still entirely possible to avoid systemd for whatever reason, at least if you're using Debian it's trivial.
> Feedback is a thing. You're giving it here. Why shouldn't I give it to the distros that certainly are trying to force systemd on me?
Right, but I'm here trying to give feedback to you for your actions in this thread. You're here giving feedback on HN to people that probably don't read HN, and doing so with a rather inflammatory tone.
If you want to give feedback to your distro maintainers, please get in touch with them in the appropriate channels (IRC, mailing lists) and ask them their reasons, but please first check if they haven't already stated those, one can just get annoyed if every person in the world keeps asking stuff that has been answered countless times.
As others have pointed out, when your init process faults, that's a major problem. In the real world, for the core elements of an operating system, whatever features they have are less important than how well they fail. In this case, the OP has identified a very very bad failure case.
More importantly, this report makes concrete the theoretical complaint against systemd that its monolithic design is not just a philosophical issue, but a practical one. The init process needs to be rock solid, but a law of software is that bigger software is less stable. So if the sprawling nature of systemd means it will tend to become less stable over time, then we have a major practical issue that needs to be addressed.
This is not a matter of not moving on. Rather, given that systemd has been adopted by several major Linux distributions, it's actually far more important now to criticize its design failures, to draw attention to get things fixed. If "moving on" means we shouldn't critique the software, then how will it ever improve?
I think people would care less about systemd if that was a realistic choice. I strongly believe that the whole systemd debate would go away if systemd wasn't a dependency of GNOME and other stuff.
Personally I have no strong feelings one way or the other, but I do fear that the BSDs are going to be left behind as they to some extend have been piggybacking on Linux in terms of desktop software.
journalctl --no-pager
Problem solved. I mean, you have the command, it's right there in the usage text. Which, awesomely, does invoke a pager when you do journalctl --help so if you're stuck on a 80x25 console (as you might be on a broken system) you can easily actually read the help options.
I can't speak for you not speaking to the first problem, but the second is more of "this is strange and I didn't care to read the text"
> In the grand systemd tradition, there is no option to control this; all you can do is force journalctl to not use a pager or work out how to change things inside the pager to not do this.
(emphasis mine)
> Problem solved. I mean, you have the command, it's right there in the usage text. Which, awesomely, does invoke a pager when you do journalctl --help so if you're stuck on a 80x25 console (as you might be on a broken system) you can easily actually read the help options.
From reading the text, I'm under the impression that the issue is the mangling of $LESS. One would expect that $LESS is set up how the user wants it, if the user wants -S he'll set that.
man journalctl
$SYSTEMD_LESS
Override the default options passed to less ("FRSXMK").
export SYSTEMD_LESS=$LESS
I mean seriously. If you have $LESS defined then you obviously have an expectation that less will behave in a certain way. An expectation that has in all probability built up over years of tweaking your shell.
For some other application to ignore that preference and in fact force it's own preference on you and only allow you to use your own preference for how less should work by reading not the docs for less but the docs for an unrelated application? That breaks so many rules it's just sad.
Yes it's easily fixable with a quick peek at the man page for systemd which might not be the first thing you check given you've been using less for years. Systemd's manpage is the last thing I would check. First would be what is my $LESS set to? Second would be is less still less? At some point it would occur to me that maybe systemd is being aggresively stupid and check their manpage.
But given I have no particular preference I would be inclined to assume it couldn't possibly be systemd's fault. I must have messed something up.
Put this minor annoyance in the middle of attempting to debug a segfaulting init? That's what we call a major annoyance.
I think it makes sense to use a pager by default.
I am not convinced that your less options that were (granted, you referred to the full environment) 'tweaked over the years' necessarily apply. When you defined those options you were thinking of all the cases where you invoke less, or at least where you knew less was being invoked.
Now journalctl defaults to use a pager if you invoke it. I'm not convinced that
journalctl
and
less /var/log/messages
should necessarily be configured the same. Maybe it would make sense to fall back to $LESS if $SYSTEMD_LESS isn't configured - I'm not sure - but I do think that it makes sense to have this environment variable.
>For some other application to ignore that preference and in fact force it's own preference on you and only allow you to use your own preference for how less should work by reading not the docs for less but the docs for an unrelated application? That breaks so many rules it's just sad.
Hmm.. Not in my world. The author was launching journalctl, not journalctl | less (which, as others have pointed out, would probably work as he expects it to). If you launch 'foo' and it doesn't do what you want and aren't an expert in all things 'foo', read man foo.
Which, for foo=journalctl, spells out that
- there is an option to stop truncating very long fields
- there is an option to use no pager
- the default pager is less
- the default less configuration is as quoted above
If you launch journalctl, see output you don't like and _don't_ skim the manual? Now that is what I personally would consider sad.
If there really was a need to override behavior of the pager for just this one tool, the normal solution should be to locally override these cases:
alias journalctl="LESS='...opts...' journalctl"
- less_opts = getenv("SYSTEMD_LESS");
+ less_opts = getenv("SYSTEMD_LESS") || getenv("LESS");
TL&DR: Setting defaults when $LESS isn't present is fine. Overriding behavior when a special, tool-specific request is also fine. Requiring everybody to search documentation and set a new environment variable to restore the standard, expected behavior is lazy and rude.
I set $LESS (and $PAGER, and $EDITOR, $VISUAL, $LESSOPEN, ...) for a reason, and the value(s) change depending on what I'm working on[1]. Overriding these should never be done as default behavior, or it is creating extra work for no reason. Assuming you know WHY I override $LESS (or any other environment variable) is always bad behavior, because you cannot know what all possible requirements that everybody will have.
// yet another reason I don't consider systemd to be usable
pager.c:92 http://cgit.freedesktop.org/systemd/systemd-stable/tree/src/...
pager.c:59 http://cgit.freedesktop.org/systemd/systemd-stable/tree/src/...
[1] I use custom tool that can push/pop environment variable overlays, which setup use/project specific behavior for a lot of stuff... including the pager.
Now, given that we both consider the $SYSTEMD_LESS and $LESS cascadation (? You get what I mean) a decent idea and seeing that you found the right spot to add it, even gave a ~diff~ in your comment here...
Have you considered reporting that as a bug or sending that as a patch?
Wouldn't it make sense to defer judgement ("another reason I don't consider systemd to be usable") until your patch is accepted or rejected?
If you launch journalctl, see output you don't like and
_don't_ skim the manual? Now that is what I personally
would consider sad.
Flouting that convention is bound to be rage inducing when encountered during a particularly nasty debugging session.
The attitude of many of the core systemd devs seems to be the polar opposite. They decree from on high, and if you don't like it thought luck.
In a sense this is the same difference in attitude we see between Apple (systemd) and Microsoft (Linux kernel).
Microsoft has in the past bent over backwards to ensure that people could run older software on newer version of Windows.
Apple in contrast have removed and changed expected product behaviors virtually over night.
So
- nothing to see here, seems okay
- not related to the subthread at all anyway
1: http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01488.html
Which is, in fact, mentioned generally in the very next paragraph that you failed to quote from that message:
> But the problem appears when system services seem to think that they
> *own* those flags, and nothing else matters, and they don't do
> something "sane" any more.
I mean.. Go ahead, flame not just Lennart et al, Linus might be wrong too! Reading the mail I linked to leads to the following
- userspace can and should read the kernel command line, very explicitly including the word 'debug'
- userspace shouldn't crap out and cause a failure to boot. Surprise!
The latter was a bug. The former is what you're trying to stuff into this thread for no reason. You might not like the 'kernel debug command line affects systemd' behavior, but that seems to be a) something the systemd guys want to keep (Ready your pitch fork..) and b) Linus himself seems to consider okay - IF that doesn't cause regressions.
That whole mailing list thread is basically just about personal issues between Linus and Kay and a 'not my problem' attitude towards regressions. The general idea of using the 'debug' kernel command line parameter to mean something specific for systemd? That's considered okay and correct.
So no, your whining is still unrelated to the (journalctl/less) topic at hand, you just felt that your pet peeve should be mentioned here as well. And .. it still makes no sense. There was a bug in systemd. It got fixed. Systemd reading/using kernel command lines without their own namespace? Fine. Correct. Good. Accepted.
"debug" can mean a lot of things and everyone's entitled to use "debug" to mean something system specific. No need to invent a systemd.debug here. Straight from the horse's mouth, we're done.
(For reference: You hijacked the thread with "(they) wouldn't namespace the kernel command line options so systemd's debug output isn't confused with the kernel's debug output" - and that's obviously wrong if you read the mail I linked to - and maybe a bit of the discussion around that)
Correct, I don't like that behavior. Just like I, along with a bunch of other people, don't like this pager options handling. If I like it or not has nothing to do with what Linus says or what the systemd guys want to keep. I also find this namespacing of options to be inconsistent, either inconsistent with the rest of the systemd ecosystem and/or inconsistent with how environment variables are defacto-standard used to do things like this.
I didn't "hijack" this thread. I made one comment, a snarky comment I felt was related. Usually my comments get ignored. But I guess not today. Glad I could keep you busy.
Dunno if dumping the output of every last damn command performed by systemd, and the daemons it starts, into dmesg can be considered sane...
I mean .. you have no / or /var yet, where do you send your log messages?
The whole reason for the bug report this angry GP wanted to share with us is that there was a buggy, wrong assert filling up the logs and causing issues during boot. It was literally just a bug. People can now form a mob and complain that their favorite init system never exposed a bug like that..
That's not related to
- logging to kmsg (that's okay, just don't kill it because you're doing stupid things)
- parsing the 'debug' kernel command line flag, the original complaint of the GP
It looks to me like you either get a pager with broken settings, or you get no pager at all.
> if you're stuck on a 80x25 console
What's wrong with `journalctl | less` in this case? The only command I use regularly which invokes a pager by default is `man`, and that seems like an actual special case.
Edit: FTR, I agree with your assessment in the first paragraph, but not the judgement I infer you passing. IMO, things should avoid being strange.
I very rarely work with a non-virtual terminal, but when I do I take a photo before typing anything. Perhaps that's only practical for me because it's pretty rare.
Git also invokes a pager by default. And it also sets $LESS when doing so. In fact, it wouldn't surprise me if systemd's automatic use of a pager was based on git; IIRC, it used to set $LESS to the exact same value git used (I believe systemd has changed it since then).
-a, --all
Show all fields in full, even if they include unprintable characters
or are very long.
-f, --follow
Show only the most recent journal entries, and continuously print
new entries as they are appended to the journal.
Edit: Although I'm not sure if he's complaining about truncated fields or long lines. Hmm..
He mentioned that. So, yes, it looks like he cared to learn how it works.
What doesn't seem reasonable is the truncation behaviour. When the output isn't piped, it gets truncated to the width of my terminal. When the output is piped, it gets truncated to 80 columns. I'd expect no truncation when the output is not a terminal.
This is an outdated systemd though (I installed it on a Rasberry Pi [Raspbian] in order to play with it), so this may have changed.
This might be less being smart and getting out of the way. From http://en.wikipedia.org/wiki/Less_%28Unix%29#Usage:
>By default, less displays the contents of the file to the standard output (one screen at a time). If the file name argument is omitted, it displays the contents from standard input (usually the output of another command through a pipe). If the output is redirected to anything other than a terminal, for example a pipe to another command, less behaves like cat.
journalctl | cat
journalctl truncates at the width of my terminal or at 80 columns if the output is not to a terminal.
Of course, the recommended thing to do is use a different program for scripts/pipelines rather than yum, so now there are two different interfaces that need to be remembered. yum looks like a terminal-friendly program, but actually isn't.
I suppose this does give some substance to the systemd criticisms that it is too system-critical for a process that is technically outside of the Linux kernel.
If you're saying, there are scenarios where segfaults caused by upstream code are not upstream bugs, then I disagree.
The rest seems mostly nitpicking. I wouldn't work myself up about the pager issue as discussed elsewhere here. I do somewhat agree with a rather useless default for the range of things to show (i.e. I'd prefer -b [1] to be implied unless specified otherwise, for example).
1: -b means 'since the last boot' for people not using journalctl. The default is instead to show the full journal from its beginning, as stated in the article.
# keep systemd happy
0 * * * * /bin/shutdown -r nowOn the one hand, PulseAudio really would have benefitted from a built-in watchdog to restart it its early days. On the other hand, before PulseAudio, I never really had working sound at all, so I consider PulseAudio to have been an improvement.
Most options may be changed while less is running, via the "-" command.
So, just start `journalctl -l` as you would, then type in `-S` and scroll the screen in any direction to force it to re-draw.
Discussing it any further is bikeshedding, and distracts from the much more interesting segfault discussion.
If that's the case, the bug might have even already been fixed in systemd upstream, but the fix wasn't applied to the older systemd in Fedora 20. The solution would be to backport the fix to Fedora 20's systemd package, so users who do a "yum update" before upgrading to Fedora 21 would have it.
An alternative workaround might be to, immediately after unpacking the new systemd, send a SIGTERM to PID 1. According to http://www.freedesktop.org/software/systemd/man/systemd.html, that signal directly tells PID 1 to re-exec itself, without using systemctl in the process. That would make PID 1 be the new systemd before anything tries to use the new systemctl.
(In my grumpy sysadmin view, it is PID 1's fault even if the distribution is doing odd things around PID 1. Init processes need to be absolutely rock solid and extremely defensively coded, precisely because the world basically dies if they ever fall over.)
(I am the author of the original post.)
On the other hand those that embrace it come from web/cloud development, with an eye towards virtualization and containerization.
I think someone somewhere once compared it to pets vs cattle.
Traditional servers are the pets of sysadmins. Groomed and cared for to make sure they never keel over unexpectedly.
To the cloud "admins" (or maybe i should call them devops?) servers are cattle. they have X of them living in some cloud "farm" somewhere, and can order any number of them to be slaughtered and replace at the drop of a pin.
Well, that's what you get by supporting a system built by obnoxious incompetents that think they can replace the most important process in the machine
"(Oh, and journalctl goes out of its way to set up this behavior. Not by passing command line arguments to less, because that would be too obvious (you might spot it in a ps listing, for example); instead it mangles $LESS to effectively add the '-S' option, among other things.)"
Yes, of course.
And not having the init process crash?
Seriously, this is clownshoes. Their developers and evangelists should get hit by a SIGBUS.
>While I'm here, let me mention that journalctl's default behavior of 'show all messages since the beginning of time in forward chronological order' is about the most useless default I can imagine.
On the other hand, "journalctl -b" (show all messages since the machine has been last powered on, in forward chronological order) is quite useful.
I love less's command line interface for some of this stuff.
The command line switch to chop long lines: -S The in-app command to toggle chop lines: -S
The reverse is almost true for things like /search, F (follow) or G (go to end). They can be used from the command line with +/search, +F or +G