https://www.agwa.name/blog/post/how_to_crash_systemd_in_one_...
Which made me re-evaluate systemd. Fundamentally creating a monolithic tool to take over from many small single purpose tools seems counter to the UNIX philosophy, and makes it harder to hack on my (mostly desktop/laptop) systems.
I think it's an elegant and necessary complexity to save us from much uglier and greater complexity.
Red Hat would like SystemD to be the init system of record and have - for years now - been subsuming other utilities, daemons, logging, su, etc. in order to make it the obligate choice.
Any technical gains from the integrations seem secondary to the control this gives them over the direction of Linux as a whole.
This is obvious dirty pool.
If they'd gotten where they were on the merits, I think people would be a lot quieter.
Beyond the political stupidity, systemd has made itself a single massive and opaque point of failure, especially since it ate logging.
Honestly, what kind of complex init system do most /servers/ need?
But the issue is actually that systemd is not just an init system it's steadily increasing in scope to incorporate more and more services.
For example, systemd-timesyncd (controlled with timedatectl) is an excellent NTP system, and after it's been stable for a few years I'll use it with public NTP servers. It takes just one command to get up and running and it keeps your clocks synchronized, what's not to like?
And considering systemd's vs OpenBSD's track record I suppose it's also less likely to get you owned or set your time to Jan 1st 1970 because of some bullshit like "who could have predicted that in 2016 networks are still so bad that UDP can deliver packets out of order".
And, btw, why should an init system come bundled with a time client or a time client bundled with an init system? When can I expect systemd-solitaire?
That's entirely up to distribution packagers. 39k bytes and you get a dead-simple working ntp client daemon (no server) which requires literally no setup. If the distro packagers want to split it out or not include it at all, they can do that and the build system is set up to make that easy.
So it really depends on how you mean "bundled", it's good and there's basically no reason not to include it in the package.
I also figure it's a bit of a straw man to hypothesize a bug which hasn't occurred. NTP, as far as I'm aware, is mostly static structures on the wire. If there's a problem with the client, it's most likely your clock will just be wrong; bad as that is.
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sbin/init/ http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.sbin/ntpd/
Alas there is no OpenBSD-solitaire yet, but who really needs that if you can play "Hunt the Wumpus" instead:
My CentOS 7 OSD nodes will sometimes not complete booting because it's hung somewhere in the boot process. There is no way to tell systemd to continue, Ctrl + C doesn't do anything and there is no way to get into the system because sshd is still not running.
Booting it in debug mode to try and get more information makes the whole thing work.
Also, when a unit file gets stuck, it goes into this waiting loop, which is a giant pain in the behind... I want to try and fix it, not wait 10+ minutes for the unit files to time out.
Ctrl + C should immediately stop what it is trying to do and dump me into some sort of login shell so that I can then go and fix the issue...
I've also had issues on Ubuntu 16.04 because of third party software that doesn't correctly set up it's Requires/Wants/Before/After's and then systemd will happily hang the whole boot process. Worst thing is that these issues become more and more apparent on systems with more cores/CPU's... so the 44 core, 88 thread systems we have show the issue way more than the 4 thread desktop that developers are using...
Systemd in their infinite wisdom decided that journald can choose when to log and when not to log something. Historically, syslog etc have been a very thin barrier between programs and outputting logs. Journald now decides if and as and when it's going to bother putting logging data to disk.
One of the things they've decided is you don't really need logs when you run the reboot command. In fact, if you really want logs you have to dig in to their documentation to find this: https://freedesktop.org/wiki/Software/systemd/Debugging/#ind...
That tells you to adjust kernel parameters and write a file to disk to explicitly re-enable logging.
I lost a whole bunch of time recently troubleshooting something on a server running systemd that was interfering with the reboot process, because systemd logged sweet FA. I had a pretty good idea what the problem was, but lack of logs made it significantly harder to nail down. The actual problem is one I've run into all sorts of times in the past, and in every non-systemd case, there's always been enough detail in the logs from the init system to nail it precisely from the get go.
How many of those services have dependancies on devices that may or may not be online/attached at any point in time?
The preach of UNIX culture in GNU/Linux sounds to me like emigrants keeping long gone traditions alive from something grandparents used to speak about.
[1] https://interviews.slashdot.org/story/04/10/18/1153211/rob-p...
For me, the UNIX philosophy has not been about the tools, but about replaceability. Systemd has made itself nigh on impossible to replace.
The SysV init system stopped being convenient for some number of people, so they replaced it, and it has been a long and horrible process. It will happen again if it needs to.
> But nobody is thinking about that because "init-scripts sucked".
Well, now they're thinking about it, and talking about it every time they bring up problems with systemd. Those problems and discussions are something that will be brought up when any potential future init system is being designed or advocated for.
Instead of learning from that mistake, systemd just doubles-down on it. systemd will be much harder than SysV to replace because it includes more than SysV stuff. It will happen if it needs to, but why not just make it easy to replace if it needs to? (answer, IMO: insecurities about your product make you want to entrench yourself instead of constantly improve it)
> Well, now they're thinking about it, and talking about it every time they bring up problems with systemd. Those problems and discussions are something that will be brought up when any potential future init system is being designed or advocated for.
Sure, but why not do that now? instead of pushing it off for people in the future to figure out?
Well, I was actually talking about future successors. What systemd did right and wrong at a technical level, as well as at a marketing, design and adoption level, will obviously have been internalized to a greater or lesser degree depending on the aspect being considered and the person doing the consideration.
> systemd will be much harder than SysV to replace because it includes more than SysV stuff.
I don't think so. There are a plethora of other init systems that have either been started in response, and people work out how to get those into these different distributions. The market for init systems is more vibrant than it ever has been, so any eventual successor gets to reap the benefits of everyone's collective experience in this area. Systemd may be overtaking most the large distributions, but let's not pretend the init system in linux, what it interacts with, and what's required to replace some or all of one init system with another aren't far better understood than they were a decade ago.
> why not just make it easy to replace if it needs to?
It may not be easy to replace, but it's far from impossible, and it's getting easier and better understood all the time because of people that don't want to run systemd. That's a good thing.
> Sure, but why not do that now? instead of pushing it off for people in the future to figure out?
Because regardless of how much it seems like people dislike systemd, I'm confident the majority of people either didn't notice, don't care, or like the new features enough to outweigh their dislike of other aspects. I would be happy to consider data that points to an alternate conclusion, but there's a reason why it's taking over, even if there are problematic aspects.
The same author also made a followup blog with even more strange arguments. Saying that systemd security features doesn't matter more or less because applications should be secure by themselves / do that on their own.
I'd refrain from forming your opinion solely on blogposts such as these.
Did anyone actually think this was a remote exploit that could be triggered via twitter? I certainly was never under that impression.
a) systemd violates "the UNIX philosophy".
b) "the UNIX philosophy" applies to long-running processes.
UNIX daemons have existed for a long time, and they have never been as single-purposed as the shell utilities; so arguably "the UNIX philosophy" doesn't have anything to say about daemons. Secondly, systemd's PID1 has basically only as much functionality in it as makes sense. If they move that functionality to another daemon, it'll be worse.
> even more strange arguments [...] because applications should be secure by themselves
I'll quote him on this: "good programmers recognize the difficulty of writing bug-free software and understand the importance of designing software in a way that minimizes the likelihood of bugs or at least reduces their impact".
Design and architectural choices including the choice of programming language have a high impact on the code quality and reliability (incl. security) of a system. No matter how good a programmer you are, there's just a lot additional mental overhead when using an unsafe language like C, as you have to always be on the watch-out for type-safety related bugs, and for memory safety (not accessing invalid references), memory leaks (and always free()ing), and bounds-checking stuff.
To quote him further: "Go and Rust are compelling, safe languages for writing the type of systems software that has traditionally been written in C. Systemd is dangerous not only because it is introducing hundreds of thousands of lines of complex C code without any regard to longstanding security practices like privilege separation or fail-safe design"
Forget to check array bounds just once, and you might have just created a new security vulnerability. You have to be perfect. Not make a single mistake. Run Valgrind and every code quality other tool, and you still could have missed something subtle. C is an inherently unsafe language, and a poor choice for new projects when alternatives like Rust and Go exist.
The other problem is that one should try to avoid building giant monolithic systems. (I think) I read somewhere once that Linus Torvalds said that in hindsight not adopting a microkernel architecture was the biggest mistake he'd made.
Quoting the author: "Systemd is far more than an init system: it is becoming a secondary operating system kernel, providing a log server, a device manager, a container manager, a login manager, a DHCP client, a DNS resolver, and an NTP client. These services are largely interdependent and provide non-standard interfaces for other applications to use. This makes any one component of systemd hard to replace, which will prevent more secure alternatives from gaining adoption in the future."
A single PID 1 process shouldn't be doing a million things. Splitting up responsibilities into different specialized programs, running in their own processes would be the way to go. IPC overhead is not sufficiently large to justify creating one giant statically linked executable that does a million things. It's not the Unix philosophy either.
> it's not remotely executable, nor has any relation to being triggered via Twitter
Maybe he shouldn't have have used the word "tweet" in the title, but it's clear what he's talking about a few sentences in. And, what do you mean it's not remotely executable? Did you try running it?
> I'd refrain from forming your opinion solely on blogposts such as these.
The author backs up his claims by linking to specific bug-causing lines in GitHub. He makes several solid beautifully explained arguments. It couldn't be more well-written. How many blog posts claiming security holes link to specific lines of code? Your dismissal of his entire blog isn't justified.
Is that really what you got from that? If so you need to do a much better job of understanding what you read.
My interpretation, for an alternative take, was that Lennart acknowledged it was a problem and should be secured, but noted that there were tools that needed read-write access, so it wasn't as simple as just mounting read-only. He did note how you could mitigate the problem using fstab, but I think a lot of people took his statement that you could do so as the official response and that no fix would be forthcoming, when nothing of the sort was said. In fact, that comment, even though separate from the prior one, had its text encapsulated in parenthesis, which I take as an attempt to indicate it's something he forgot to include in his prior response, and just an addition, so not the definitive answer from the developers on the "solution".
Note, the eventual fix to the problem was at a kernel level, not systemd, because systemd didn't design efivarfs, they just use the accepted mechanism for accessing EFI variables on linux, which happens to be a filesystem. Whether you think that's a good choice for an interface or not, or think it was poorly thought out and implemented even if it is a good choice, none of those things have to do with systemd or Lennart, since they are kernel level issues.
What UNIX philosophy? The biggest UNIX in the world is Linux, which is a monolithic kernel. The UNIX philosophy lives on in apps, but not down that low.
Systemd is not a single binary either. It is many binaries, every one of them does its clearly specified job and together they live in single repository - just like coreutils.
I find these kinds of discussions a bit strange. I don't want it, it's made my system both less stable and less easily to hack around with. So I'm not going to use it. If you like it, go for it!
Good luck trying that with systemd...
The number of binaries isn't relevant. Claiming that systemd follows the unix philosophies[1] because it has "many binaries" suggests you don't understand that philosophy.
The point is limiting complexity. Smaller, self-contained programs are a lot easier to understand and debug than both a single "kitchen sink" app and a collection of tightly coupled programs and libraries.
If systemd followed the unix philosophy, I should be able to use journald in isolation. Or I should be able to use the init tools just as without bringing in the other tools. The point of a modular design is that each module should be as independent as possible so they can be replaced piecemeal and tested/debugged in isolation.
It doesn't matter if you split the project up into different binaries/etc. Busybox, for example, shoves everything into the same binary, but it's a modular design where it's easy to leave out any module and use the coreutils version instead.
[1] http://www.catb.org/esr/writings/taoup/html/ch01s06.html
The same can be said about the Linux kernel.
On some servers I use s6 toolset. Those are nice. OpenRC is nice. It gets the work done, its so small, its easy to do things with it, and it has less problems than systemd - which I experience on the desktop. Example, started using networkd for managing networks, because hey why not, and NetworkManager is shit, I had gotten tired of typing wpa-supplicant and dhcpcd commands manually. But then, systemd apparantly takes control over the networking subsystem not only the network interfaces - it changes sysctl flags and doesnt delete after itself when asked to stop - it leaves any addresses and other configuration changes it had done. What!?
edit: As all things, systemd-networkd has its place, its just not on a developers laptop, maybe it would be great to have in some kind of cloud environment where you have to configure hordes of machines and virtual machines in them, with networking and networking subsystem configured correctly. Then its nice to have a "do-it-all" tool, but then you should also probably swallow the rest of RedHats output, such as Ansible and look at Project Atomic.
On my servers I do what I've done for about 2 decades now: I ditch whatever init crap my distribution came with and write a single sh rc script that mounts stuff, starts whatever services I want started, and starts getty.
My pet peeve about Systemd is the abysmal error logging for services. Usually it will spit out 50-100 lines that offer zero useful information about why something failed. Adding insult to injury, I have noe easy way to add logging of services. If it were a bash script, I could inject "echo $foo > /tmp/service-debug.log" in the script, figure out the problem and fix it. Now I'm stuck with reporting it upstream and likely getting "Unable to reproduce, closing".
Software has bugs. That's just how it is.