I refuse to use systemd to this day. It's unbelievably complex and became established through political power play rather than any sort of merit. Which is without a doubt not what I expected to see in the Linux ecosystem.
All the initial "ooh and aah" factor of SystemD came from using the kernel's cgroups mechanism, which was a dumpster fire by itself--something dropped by google in the kernel for easier maintenance with the understanding that no one should be stupid enough to use it for something important...enter SystemD.
Apparently cgroups is in better nick these days (as is SystemD), but until the likes of Al Viro come around and give it a clean bill of health, I think I'll pass.
For example, you can make sure some poorly written python daemon will never use more than 50pct of the cpu or over 1 Gb of ram - and systemd services will nicely restart the daemon if\when that happens.
By not using systemd you are depriving yourself of essential tools in modern sysadmin (timers alone are worth switching to systems compared to cron anachronistic hacks)
Them being released at similar times does not mean systemd implemented them, they exist completely separately to systemd.
Literally:
mkdir /sys/fs/cgroup/memory/<groupname>;
echo 100000000 > /sys/fs/cgroup/memory/<groupname>/memory.limit_in_bytes
Adjusting to systemd took me a while, but I believe it was worth the effort, and I also believe most people who dislike systemd are just set in their old ways
I've been charring my brain to understand what you mean by this.
You don't need to have cgroups for all applications in order to use them.
And they don't need to be implemented by systemd specifically for them to be in use by other init's or tools.
SystemD existing (and being pushed so heavily) stymies the rise of such init systems.
https://www.cvedetails.com/vulnerability-list/vendor_id-7971...
There are 11 total from 2015: one arbitrary access, one which allows administrators to run a process as root rather than the intended user (CVE-2017-1000082), and some DoS opportunities. Not great and several great examples of why C should not be used but hardly “every other month”.
Now, it's hard to compare that it against other systems because it combines functionality which people used to have to use third-party tools for or roll themselves so you have to count that against every buggy init script or supervisor feature, apps whose developers never got around to adding the hardening features which systemd does use, etc.
Weirdly though, getting a CVE assigned can be a real PITA (and unsuccessful). The people who do the CVE assignment are generally overloaded, so lower impact/priority stuff often seems to get missed or not bothered with.
The alternative is DWF, which would be more popular if this was a more common problem.
Thanks, that's really good news.
* Unfortunately, they aren't alone in their sloppy handling of security issues. Rust is also another project that is known to not file CVEs for serious bugs if they occur in previous releases.
Fast forward to 2018 and i manage around 500 Redhat servers here. About 150 are at RHEL7 (the others need to be upgraded). So learning systemd is a requirement, it is not going away. I like some points of it. I do like the jounrnalctl command and the ease of looking at logs, however being the hypocrite i am i despise the binary logs and non compatibility with most syslogging devices (i have had to explain lost logs during an audit with systemd and it was not pleasant).
From a end user standpoint systemd is great for my desktop. For the servers it is functional, server boot time does not matter to most administrators. On VM's both boot fast enough, on bare metal....some of our large servers take 5 to 7 minutes to post (thanks HP) so the OS taking 20 to 30 seconds longer to boot is not a big deal.
I follow the systemd mailing list to see what is going on, i think if the guys over there were not as abrasive in some of their answers things would be smoother, and more receptive to the needs of some of the users. I see a lot of features that are desktop specific and that seems to be where their development lies for the most part, desktops and servers are two different beasts.
I guess the infighting just sucks, i am not a fan of systemd simply because i like a simple OS with minimal features to accomplish my objectives, however it is here to stay and i am a Linux Admin so it is part of my job and i learned to use it and keep up with development.
I still use FreeBSD for all my home servers (except my docker swarm), old habits die hard.
I don't develop on FreeBSD either, its just for prod.
You could use nanoBSD in the source tree to create a minimal mostly immutable image with your application. It will still be like a VM though and need deployed somewhere (this is how i run my DNS servers at home, they are about 120MB each and can run on pretty much any VM host (or bare metal)). But docker does have that market better (hence why i have a small swarm at home to play on).
There is also docker for FreeBSD, still in testing but works pretty good it you build your own native images. I build a memcached image and was running that for a while.
How did they get lost?
IBM/Lenovo xSeries servers are the same. 5+ minutes to get through the boot process, due to all the hardware init and checking. That's with UEFI too. ;)
Surprisingly, it seems as if different people have different tastes/opinions. :)
What's also kind of funny to me is, it wasn't designed for cloud-based services. We now have to invent whole new systems that look a lot like systemd just to manage distributed decentralized services that run on Linux systems.
Systemd would have been great for everybody if it had been developed by people with different priorities.
WTF? Are you talking about Android? The user does not even interact with the init system on Android at all, what are you even talking about?
It's not "systemd propaganda" that has been effective, it's the anti-systemd propaganda that seems to get dumber every time I see it.
Honestly, none of this crap matters at all. You can get by just fine with init scripts, and you can get by just fine with systemd services. Except for a few people in the world who have applications that are really sensitive to either fast boot times or really complex system dependencies, any init system will work.
I write service files pretty regularly. If you're a sysadmin, developer or an enthusiast on Linux. you definitely interact with the init system.
If you're a regular Android app developer however, you normally don't do that.
It is complex, and there are bugs like any other kind of software[1], however like any other software it is improving.
[1]: For example, I used to hit multiple concurrency bugs in sysVinit used by Arch Linux. The only way to fix them was to find the correct order that system worked. No more problems of this kind thanks to systemd dependency resolving.
And when this question was brought to general election, community decided it did not care: [1]
So I am not sure who those "important people" were, why do you feel they did not represent you, and what would community choose, in your opinion.
I, having worked with upstart, can say it is worse than systemd.
[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=727708#672... [1] https://www.debian.org/vote/2014/vote_003#outcome
Technical committee members are not elected by community, they're appointed by the DPL.
I have used systemd and currently am forced to use it in production. It works great until it doesnt.
There have been multiple times I have no idea what was wrong as it was faster to simply recreate the system from scratch and hope the same bug didnt happen again.
Let me offer you an anecdote from my company. Now, we're a RHEL shop and not a Debian shop, but we've been around since long before systemd.
Most of our hosts now run RHEL 7, but we still have a handful of old boxen still running RHEL 6, and I've been personally involved with quite a few migrations from RHEL 6 to 7 during my time here. Whenever we retire a RHEL 6 host and replace it when a RHEL 7 host, the general consensus from the people who administrate it is "thank god we can now use systemd instead of having to deal with sysvinit". Every time I have to do anything on one of our remaining RHEL 6 hosts, I brace myself for pain, because our RHEL 7 hosts are just that much easier to manage.
But that's literally the exact opposite of a meritocratic selection.
Do not mistake that with any random Joe gets his opinion voiced.
When I'm getting a free ride, I'm not going to talk down the provider how he should be doing the job differently (not necessarily better; that might be my opinion, but not objective truth). I'm not in his shoes, solving problems he is solving, so my view could be diametrically different than his.
What you are arguing about is whether those people selected systemd based on its merits or based on other considerations is an entirely different question and an unsubstantiated accusation.
In 1991, most of the "important people" in technology would have laughed at Linux compared to OSes like Unix System V and DEC/HP Unix. Even as the 90s progressed most serious shops wouldn't adopt Linux and went for the BSDs (FreeBSD/ NetBSD, etc) or Solaris because Linux was considered hobbyist software. I know because I worked in tech at this time and the same attitude even continued into the early 2000s.
The point I'm trying to make is that random users' opinions are exactly how certain software projects, open source or proprietary, become popular or fall out of public use. It starts with Systems Administrators or developers getting fed up with a certain solution and adopting alternatives until it reaches critical mass and the old is rendered irrelevant.
StandardOutput=tty
Systemd logs all process output by default, so things like this cannot happen at all. Just for that, I'd choose it over sysvinit any time.
By the way, I am not a fan of sysvinit: I read all those shell scripts.
(note that the opposite is not true: many /etc/init.d scripts stop process by killing everything with this process name -- no matter if it is the right process, or user-specific one, or one managed by some other supervisor)
That's interesting. Can you provide some more details?
To clarify: I don't care about the systemd argument; I'm interested in why an init script would need expect. I can think of a few pretty edge-casey good reasons and some scary ones. Care to share yours?
Arch didn't really use SysVinit before systemd though, but their own BSD-like rc system, which I rather enjoyed actually.. And I got hit with concurrency issues after the switch to systemd, because seemingly the dependencies for disks and file systems aren't setup correctly by the Arch scripts requiring me to add a few fake units to fix up the dependencies (otherwise the system would just fail to boot at all most of the time).
Few articles to be familiar with regarding systemd:
https://www.dedoimedo.com/computers/software-development-can...
"The same can be said of the init systems in Linux, like systemd. In a nutshell, the system should start quickly and get into a working session. We had this in 2010 or so, with boot times down to mere 10 seconds using init. No flaws, no bugs. Even in the commercial sphere, working with init, I do not recall any major problems.
Then, suddenly, we have this new binary diarrhea with a hundred million modules, and for the past five years, this unstable, half-baked, undebuggable nonsense is the backbone of most Linux distros. The invasive and pervasive nature of the systemd framework has also affected the stability of the user space, the very thing it should never have touched, and pretty much all problems with the quality of the Linux desktop nicely coincide with the introduction of systemd. The development continues, of course, and for no good reason than trying to reach the level of stability, maturity and functionality that we had half a decade ago. Someone landed themselves a lot of monthly pay checks by writing complex code to solve a problem that did not exist."
"None of the things systemd "does right" are at all revolutionary. They've been done many times before. DJB's daemontools, runit, and Supervisor, among others, have solved the "legacy init is broken" problem over and over again (though each with some of their own flaws). Their failure to displace legacy sysvinit in major distributions had nothing to do with whether they solved the problem, and everything to do with marketing. Said differently, there's nothing great and revolutionary about systemd. Its popularity is purely the result of an aggressive, dictatorial marketing strategy including elements such as:
Engulfing other "essential" system components like udev and making them difficult or impossible to use without systemd (but see eudev).
Setting up for API lock-in (having the DBus interfaces provided by systemd become a necessary API that user-level programs depend on).
Dictating policy rather than being scoped such that the user, administrator, or systems integrator (distribution) has to provide glue. This eliminates bikesheds and thereby fast-tracks adoption at the expense of flexibility and diversity."
Solaris already had a better replacement for initv so did some linux distros.
I think we need a better alternative that does not behave as a black hole, ntp, dns resolution, logging etc. do not get sucked into it and finally we need a way better implementation that does not have a remove or local priv escalation every other week[1].
I find this highly unlikely. Here is my take: people were tired with "sysvinit". Of all alternatives, systemd sucked less. So they went with it.
dpkg -L systemd | xargs stat -c '%F %A %n' | grep file.*rwx | wc -l
73
Now, some people complain that "systemd" package contain other binaries, like systemd-networkd, or systemd-journal-gatewayd (the HTTP server). I don't understand their problems. Sure, it is annoying that "systemd" package contains network manager -- but you don't have to use it; in fact, I am not using it at all.
So I do not see the irony. I think 73 binaries, each optional, easily replaceable, and doing only one thing, is very much inline to microservices philosophy.
There are two major things to consider here:
- how easy is it to replace each of these with an independent third-party implementation - if it is possible, why should the design of that component be dictated by the systemd upstream
A huge amount of the value of Linux was the fact that the system was a loosely-coupled collection of parts. Much of that flexibility has been lost, and with it the utility which drove a lot of the adoption of Linux in the first place.
What did you have in mind when you were talking about hard-to-replace implementation?
I find the replacing all those components to be pretty easy. Those things are all loaded via regular unit files, and they need just one symlink to be completely disabled.
In particular, systemd-networkd is entirely optional, and ifupdown is still supported in ubuntu (even if not installed by default).
systemd-timesyncd is a regular service, and can be disabled directly, or via timedatectl. ntpdate still works when it is disabled.
Many other services, like systemd-hostnamed, do not even run by default. You could disabled them by default and things would still work.
The worst offender is probably systemd-logind, but luckily I did not have to deal with it. Looks like it only needed for GUI logins, not ssh one -- so most sysadmins would not have to care. That said, I did try to use it's predecessor, ConsoleKit, and it was pretty horrible as well.
Seems ironic to me is all
The only "merging" systemd does is that it puts multiple binaries into one package, other than that they are completely separate. While it is annoying waste of space, there is no reason to suspect "coupling".
I find the fact that systemd internals are connected via the same mechanisms as non-systemd apps (units, sockets and so on) very nice, especially compared to sysvinit or upstart. Have you tried to patch network detection in upstart? Not an easy job. And upstart's manpages are less than useful, compared to systemd's.
Above in the other threads someone said "journal". This is a good point, except there is --log-target=kmsg option which disables that interface and switches it to good old kmsg which anyone can parse and read.
Any others?
Except for the journal. It is effectively required by systemd (the pid 1 binary). In the vast majority of cases, that means running systemd-journald.
Every attempt I have seen at operating without it either misses early logs or is extremely fragile/unsupported.
That's a point of fact, not a position in the larger argument here. There are good arguments as to why journald should be effectively a required component.
> /lib/systemd/systemd, the PID 1 process, only starts processes, nothing else.
That is incorrect. It also interacts with the journal, the component implicated in the exploit in the article.
man systemd
...
--log-target=
Set log target. Argument must be one of console, journal, kmsg, journal-or-kmsg, null.
> Every attempt I have seen at operating without it either misses early logs or is extremely fragile/unsupported.
I am not surprised -- this was one of many annoying parts of sysvinit, the missing early userspace logs. This was one of the many annoying parts. Gentoo used to have a hacky way that scraped screen buffer (!) but I think it broke at some point[1].
> That is incorrect. It also interacts with the journal, the component implicated in the exploit in the article.
Fair enough, I oversimplified. It also interacts with udev, systemctl, and does cronjob handling and network interface renaming.
[1] I just checked Gentoo's website now and looks like they have a better way now due to OpenRC.
It was the first Linux distro I ever used and I still love it to this day. I've used many different systems for work over the years but I've kept at least one Slackware laptop or desktop going since early 1995 when I first found a Slackware CDROM in the back of a book about learning Linux.
Although it gets new software, bug fixes and minor updates here and there it's remarkably similar today to what it was then. And I find it Just Works, I've had very little trouble with it over the years.
I think I like it too because it always felt like the most traditional Unix like distro I've ever used and also more BSD than System V which I also generally prefer.
Still a great project that I use every day, I'm not knocking it. It's just not a great example of a change-averse type of stability.
I'd take bash scripts without hesitation over systemd any day of the week. I can debug bash with my eyes closed. Every single time I had to debug systemd, I wanted to kill myself.
A bit like claiming to be a car mechanic and bashing an old car because you can't tune the ignition timing, because it doesn't have a board computer doing it for you and you don't have the skill/understanding to do it yourself.
In the past, you often were required to have at least some skill/experience to get Linux to work for you. Today, it's more about choosing the right tools to use Linux with a just a bare minimum of skill/experience.
An understandable development for sure, but it also opened the door for software of questionable quality, which has at least part of its popularity to thank to the ignorance of a significant part of the user base.
Linux userspace was a bunch of crufty, unmaintained tools from xinetd to logind (literally had no maintainer until system came along) to update-rc.d that additionally were not taking advantage of a lot of new kernel features like cgroups. systemd has done a great job moving the base layer of linux userspace forward.
The old world of cobbling together bash, sed and shell metacharacters was always hacky, insecure and broken as shit and we are way better off now with systemd.
Your arguments hold no merit whatsoever. The fact is that all the problems you describe have been solved, properly, multiple times _before_ systemd entered the picture. The reasons behind systemd mass adoption were political and Redhat exerted a lot of pressure at the time and in many ways, still do.
Solaris SMF is vastly simpler than systemd and has a much narrower scope and focus. I don't recall mentioning bash scripts in the post you replied to, so you are simply being disingenuous by presenting a false dichotomy.
The complexity of Sysvinit is orders of magnitude less than that of systemd.
This whole thread is a discussion of the relative complexity between sysvinit and systemd. Sysvinit uses shell scripts to implement much of the logic around starting and stopping services. Any discussion of sysvinit and systemd is going to involve bash. It's disingenuous to pretend otherwise.
"One would ask himself how the BSDs manage to do it (no systemd), Android (no systemd), ChromeOS (no systemd), Solaris/Illumos (no systemd) ... Your arguments hold no merit whatsoever. The fact is that all the problems you describe have been solved, properly, multiple times _before_ systemd entered the picture. The reasons behind systemd mass adoption were political and Redhat exerted a lot of pressure at the time and in many ways, still do."
This is the post you replied to, with arguments that (still) hold no merit. Now you are trying to shift this into something else rather than stick to the points I made _in this thread_. You pick and choose a reply of mine _from a different thread_. Moreover, you write: "This whole thread is a discussion of the relative complexity between sysvinit and systemd". No it is not. As I wrote: "The fact is that all the problems you describe have been solved, properly, multiple times _before_ systemd entered the picture."
I would take you more seriously if you stopped presenting one logical fallacy after another.
But more generally, you didn't restart daemons if they crashed. Firstly, because they used to be written sufficiently well that they never crashed, because competent people wrote them. Secondly, because continually respawning and crashing does no one any good.
If your deployment strategy depends on never having to run substandard software then you've already lost. Also, it just isn't true that older software was necessarily more reliable. It's just that when you found yourself maintaining poorly written software you just dealt with it through whatever means you had available. I remember having to use an IBM HSM implementation on AIX, something you would expect to just work because it was IBM software written for their own system on their own hardware, but in practice the filesystem kept invisibly crashing resulting in apparently corrupt files and I'd have to restart it every few minutes.
If something is crashing and requires a restart, then doing it automatically is at best papering over a problem the admin should be investigating. It's a mitigation, rather than a solution, and might well cause more problems than it solves.
systemd isn't bringing anything particularly new or noteworthy to the table here.
So? That doesn't mean it was the best way to do things. Humanity existed for thousands of years without vaccines and still managed but that doesn't mean that disease wasn't a thing. Misbehaving services have always been a thing and they've always been a problem. I know because I've dealt with them. If you feel that other methods for dealing with misbehaving services are better, fair enough, but don't appeal to a mythical golden age where everybody wrote software without bugs.
Not only that, but automated service restarting was configurable if desired, albeit not the default. If you wanted it, it was doable with ease. This is not a new feature which systemd brought to the table, it was already available.
Maybe the daemons didn't crash so often as now?
I never understood this new approach, which consists of writing a server process in a half assed way so that it crashes frequently, then solving that by putting a restarter service in front of it, done!
There is nothing wrong with making things easier to use. The problems begin when the people you put in charge of that effort (Poettering and co) are extremely short-sighted, do not properly understand what came before and thus make mistakes that could have easily been avoided, are average-at-best engineers with a personal history of bad projects (pulseaudio, avahi), are bad communicators and take valid critique poorly.
When I was still at Google, colleagues at Android and ChromeOS teams laughed at the mere mention of systemd. The general consensus in the teams I mostly interacted with was that systemd resembled a hidden iceberg with enormous problems, the magnitude of which would slowly become apparent. Which is one of the reasons Google has mostly kept away from it.
SystemD turns your bash scripts into C, and then gives you a config file input.
Before this; on RHEL systems there were similar initiatives, you /rarely/ touched the init scripts, you used the /etc/sysconfig/ directory of configuration files[0];
The major difference being that system administrators could actually tell you line-by-line what was happening. In C you would have to find the version of the code in some version control somewhere and hope that there's not patches applied.
The absolute truth is that you cannot know what systemd is doing, you cannot even strace it. You /rely/ on it telling you the truth and doing the right thing.
And honestly I don't trust it to do the right thing when it can't even get encoding right[1].
[0]: https://access.redhat.com/documentation/en-us/red_hat_enterp...
Hum... Symlinks existence is declarative, configuration files are declarative. Running a specific piece of software to change the state of your service isn't declarative at all.
Have you taken the time to actually read the read the bash(1)? I've noticed that a lot of people (myself included) don't learn sh/bash (and shell scripting in general) initially out of necessity, not with the intent to actually learn the language. If someone wanted to learn a "normal" programming language like C++, Java, or Rust, the would probably start by reading tutorials or guides that cover the major features and teach the core concepts[1].
With bash, most people appear to start with trying to fix something like a build/install script, often resulting in cargo culting a few lines from stackoverflow, forums, "cool one line tricks"-style blog posts, or similar community resources. These can be very helpful[2], but they are not really a way to actually learn the fundamentals of the language.
For me, bash programming initially was very confusing. After I finally forced myself to take the time to actually read the docs/etc, I realized I was making a few incorrect assumptions[3] about the language. After fixing that and learning a few important tricks[4], bash became a lot easier to understand. Now it's one of my favorite languages (for simple tasks).
[1] e.g. (C) pointers, (Rust) variable ownership/borrowing
[2] http://mywiki.wooledge.org/BashFAQ
[3] IO had a lot of trouble initially trying to figure out how to return values from a function. This was hard because, bash "functions" are not really functions like you would find in most languages. I had assumed that you would return values form functions, which breaks when you try to return anything besides an integer. (bash "functions" like foo() { echo "bar" ; } defines programs that return data by writing to stdout).
[4] Always quote variables. Always use curly brackets when using variables ("Hello,, ${name}", not "Hello, %name"). Using $* is almost always wrong; Use "$@" (with the double quotes) instead, which correctly handles spaces/bad-chars in filenames.
It's so hard to find a linux distro that actually works on a laptop but doesn't use systemd, so I eventually just gave up on Linux entirely.
That left me with BSD, Windows, and macOS.
I tried FreeBSD and OpenBSD but performance was atrocious and noticeably worse in almost every way than Debian and win7 on the same hardware. I'm sure BSD makes a good server OS but it's just not mature enough for serious work unless I was willing to keep my laptop plugged 99% of the time. But what's the point of a laptop if the OS performs so poorly it won't even last a 4-hour flight?
I similarly seem to be the last guy on earth who still remembers the bad old days of Microsoft's dominance so I won't pay for windows.
The winner by default is therefore apple.
Then again I just bought a new MacBook air and they keyboard gave out after two weeks and it gave me a rash.
Can't win at all.
The problem with systemd isn't necessarily how it handles system startup and daemon supervision (although I don't care for that), it's that it subsumes so many other things, and with no technical excellence.
Some people just want to know exactly what is running on their system, and how it works. That's one of the reasons I don't use Windows.
(I do confess that I never took the time to learn systemd properly, but I highly appreciate it for introducing me to the BSD world).
He tried other open source operating systems and didn't find them acceptable. Sure, there are Linux distributions that have formed to avoid systemd, but it's not clear how long any of them will last.
Mac os isn't exactly unicorns and rainbows either, but there are some benefits of being part of a larger user base, too.
On the Linux front, I've been using Void and it has been pretty fantastic. Runit is the default init. It is a more hands-on distro (think Arch 10 years ago), but it's fast, slim, and doesn't have systemd.
Don't you think that there's a reason for that? That systemd solves the problems that make it possible to react to events and makes laptop-usable distros possible?
Meanwhile, enjoy launchd. Maybe you will find out, that they are conceptually similar.
If systemd had followed a similar philosophy, it would have been accepted without anything like the same amount of criticism. As it is, its scope creep is frankly absurd and dangerous.
The philosophy of systemd exemplified through PRAXIS is one of subsumption and uniformity (== taking away choice) under a singular vision defined by the systemd implementors. In practice, this means that you are penalized in various ways if you don't "buy in all the way". You can take a look at all the distributions that ship systemd by default and see how many have bought in all the way vs not using the "additional functionality".
It is one such solution. But there are others, which from an end-user perspective are much easier to deal with (e.g. runit, shepherd etc.).
So, for instance, my Void-powered laptop shuts down quickly and cleanly when I ask it to, and I don't have to wait 1:30 minutes (or more) on shutdown.
It isn't building in an NTP client, or a hostname resolver, or configuring your network, or taking over the system logging, or generating QR codes. It has a singular focus and design which makes conceptual sense. It's not a dumping ground for a disparate collection of poor quality alternatives to existing tools.
Systemd is a much worse implementation of a Launchd concept, just as Windows is a worse implementation of a Mac like GUI concept...
Were you thinking of journald,... etc. instead?
I’m not sure what your link is supposed to prove.
One of them should really change to avoid this unfortunate confusion.