Linux VS open source UNIX
adminbyaccident.com
adminbyaccident.com
OpenBSD is a fascinating operating system. I'm only a few chapters into Absolute OpenBSD (https://en.wikipedia.org/wiki/Absolute_OpenBSD) by Michael W. Lucas, but from a Linux user/admin perspective it is enthralling. I can't wait to learn more about it.
That mindset has such a strength in that such tools form reliable building blocks for other tools and system components in a way that is really approachable for amateur/volunteer developers. It takes no time to parse through the man page of a simple tool. And only a bit longer to parse the code. That makes it easy to build improvements, or build on top of it.
But trying to wrap your head around systemd in even a few hours isn't reasonable. And binary log files? Wtf. It also seems like relying on certain aspects of it's behavior is a good way to end up breaking your system.
BSD still has that old school Unix mindset that made Linux great in the first place. Especially up against Microsoft's products.
The proof in the pudding for me is to look at the running processes of a default system between BSD and Linux with systemd. Yikes.
Like most, you could have simply moved to a distribution without systemd. Unlike Linux, the BSDs offer less choice when it comes to both init, service supervision and service management options. It is also evidenced in the comparison table in the OP. If anything the debacle resulted in creating more awareness and leaving people open to more choice.
That said, adopting one monolith instead of another sounds more like a knee jerk reaction than anything else.
> The proof in the pudding for me is to look at the running processes of a default system between BSD and Linux with systemd. Yikes.
This only scratches the surface. Daemons running under systemd have their PIDs supervised instead of using unreliable hacks like pidfiles. This is one of the reasons why systemd is preferred in large deployments and professional environments.
systemd had the opposite effect of giving people more choice. Before systemd every major distro ran it's own init service. Often they were variants of init.d but sometimes they might vary quite significantly. Debain, Ubuntu, Suse, Arch, Slackware and Gentoo -- they all had different inits. One of the reasons systemd was push so heavily was to standardise things across the different distros. Now the only distros still offering anything different are either those that are also gradually falling out of favour due to the changes in user trend; or newer distros that are only a minor blip in the Linux community anyway and unlikely to ever grow beyond that.
The exception here being dockerised Linux eg Alpine. But they fill a different use case.
> This only scratches the surface. Daemons running under systemd have their PIDs supervised instead of using unreliable hacks like pidfiles. This is one of the reasons why systemd is preferred in large deployments and professional environments.
Honestly, I'd take pidfiles over binary log files any day of the week. However you're right that systemd does bring some good ideas to the table (if it didn't, nobody would have adopted it). My issue with systemd is that it's needlessly hard to separate the wheat from the chaff. I know systemd isn't the monolith people often describe it as...but even so, it sure feels like it's an all or nothing deal at times.
Also, there's what, Gentoo and GUIX that don't use systemd? Gentoo still prefers building all source on the system which is a non-starter for most, and GUIX I didn't think was ready for a daily-driver role yet.
The most popular non-systemd distro I hear about is Void linux, which is a rolling release style distro that's pretty comparable to(though not based on) Arch.
As a tangent because you brought up distro's and difficulty, I know Arch has a reputation with some as being too difficult for regular skilled users, but as a regular skill level user myself, I find that the Arch/Void style rolling release paradigm of doing stuff makes things easier for noobs like me, not harder.
Once you get past the initial hurdle that is the first hour or two of install and setup(which you can do just by copying youtube tutorial instructions line by line) then everything after that is much simpler.
For me, I find that the majority of everyday linux problems all boil down to trying to install or update some kind of software. Either software you already have has a bugfix or feature you want in the newest update, or you have some new software you want to install. And because developers like new shiny things, often times the newest version of whatever has a dependency of some other new shiny thing.
Installing new software in a fixed release distro often quickly gets too hairy for me, but in Arch I can almost always a simple install directly from the main or user repositories. This has so far eliminated 95% of everyday problems for me defying the usual narrative I hear of it being "harder" to use for normal Joe's like me.
And yet the Linux Desktop community insists on sticking with the package manager/repo model that makes stuff like this such a pain. Worse, it's actively hostile to the concept of portable applications in general.
[1] https://devuan.org/ [2] https://mxlinux.org [3] https://artixlinux.org/ [4] https://voidlinux.org/
If I may take umbrage to one quote; Netflix uses FreeBSD for it's streaming platform.
"...systemd is preferred where it's preferred."
They are Linux though, not BSD.
Makes double sense when you consider the fact that at google they basically wrote their own software defined network appliances.
But Azure, AWS and even Oracle cloud do not have support for ipv6 on instances.
It is the exception that proves the rule.
The expression is highlighting that you need to observe a negative in order to determine whether a rule is true. For example, if I make the statement: "Only birds have wings", then any observation of a non-bird without wings are exceptions to the rule that proves said rule.
The observation of a bat is indeed an exception to the rule, but it does not prove it. In fact, it does the opposite. It disproves it.
Don't get me wrong, systemd actually adds a lot of needed functionality, but I really feel like it is a major philosophical branch away from the conventional Unix thinking of 'make a tool that does one thing really well'.
The person replying to you had an argument based around choice: the practicality of the solution was ignored in favor of an abstract concept of choice, rather than practical benefit.
(I'm saying the person arguing with you was arguing against something you weren't arguing for.)
As far as I can tell (not actually using Debian or Ubuntu myself), there was only a debacle in Debian and Ubuntu, because they took too long to decide, and ended up having to rush their migration.
All the other distros, that had better-planned migrations (Arch, Fedora, SUSE, etc. etc. etc.) had basically no issues migrating to systemd, and have benefited immensely from being able to ditch the maintenance overhead of diverging init systems and init scripts, all re-implementing the same new features with slightly different implementations due to previous slightly differing implementation details.
People will currently call out OpenRC as the one non-systemd approach that works (e.g. if you acknowledge that SysvInit no longer actually works because no-one uses if for all system initialisation, instead leting it delegate lots of control to other bits and pieces), but before systemd won, there were: * SysVinit + RH-derived initscripts * Sysvinit + SUSE-style initscripts * Upstart * OpenRC * prcsys - parallel rc system (Mandrake and derivatives)
All of these had similarities, and small differences, e.g. they all tried to support dependency information in init script headers.
As a sysadmin, it was frustrating to: * Have to read every init script to see where it read its defaults from, was it /etc/sysconfig/pkgname or /etc/default/daemonname . Did it support setting limits from the defaults file, or would you have to hack the script to support your bigger-than-the-maintainer-imagined environment. * Debug where the application was logging to, if it didn't setup logging correctly by default.
As a maintainer of a package used on more than one distro, it was difficult to: * Test your init script to ensure it worked on RH-based distros and SUSE and Debian and Ubuntu. While Upstart said it supports init scripts, it fails in some random way that doesn't occur on RH and SUSE. * Setup logging correctly, as not all distros provided a helper script to setup the right logging daemon (syslogd, vs rsyslog vs syslog-ng etc.)
We won't talk about the more modern features such as cgroups etc., where only about 2 init systems support the feature adequately, and no distro or packager actually managed to support such advanced features on more than one init system, and certainly not more than one non-systemd system.
systemd mainly stays out of my way, but makes it easy to troubleshoot and customise daemon behaviour in a consistent fashion.
> But trying to wrap your head around systemd in even a few hours isn't reasonable.
You don't need to wrap your head around systemd to do things with it that you would really struggle to do with any other init system. But, you've probably been too busy fighting it, or trying to prove why it's worse, rather than trying to do real things with it, or you would be able to point to valid bug reports.
> The proof in the pudding for me is to look at the running processes of a default system between BSD and Linux with systemd. Yikes.
You're comparing apples and oranges here, you should rather be comparing two release of the same distro, before and after systemd became the default, and you would have seen very little difference.
Many things about Linux would be easier if we standardized on only one implementation of each thing rather than multiple competing implementations (e.g. only one desktop environment, only one text editor). But we would also lose everything that makes Linux great.
> You don't need to wrap your head around systemd to do things with it that you would really struggle to do with any other init system. But, you've probably been too busy fighting it, or trying to prove why it's worse, rather than trying to do real things with it, or you would be able to point to valid bug reports.
There are any number of major security vulnerabilities, but that's not really the point. Most of the really obnoxious things systemd does are by design so there are no bug reports. Tight coupling to kernel and udev versions, to a non-standardised binary logging daemon, to a custom DNS implementation...
Sure, but there is nothing inherent about a text editor or desktop environment that results in you only being able to support one at a time.
It is only practical to run one system initialisation daemon on one system image at a time (unless your system initialisation daemon is too broken to support all the requirements, e.g. if you find yourself running daemontools on sysvinit).
Duplication for the sake of duplication that benefits no-one is just a waste of resources that could be spent on improving other parts of the ecosystem (addressing some of the other gripes the author has).
> There are any number of major security vulnerabilities, but that's not really the point.
There are only about 2 in systemd itself.
> There are any number of major security vulnerabilities, but that's not really the point.
systemd has fewer CVEs than the default syslog implementation used on most distros before systemd (rsyslog), but of course you knew I was referring to open bug reports, not resolved ones.
> to a non-standardised binary logging daemon
There are pros and cons to this. I would prefer to be able to dispense with journald in some situations, but in many cases it is very convenient to have, and it solves problems that no Unix system had completely solved before it.
> Most of the really obnoxious things systemd does are by design so there are no bug reports. Tight coupling to kernel and udev versions
When tight coupling is done by the BSDs, it is considered a good thing, when systemd uses features that don't exist on the BSDs, suddenly that's a bad thing ...
> to a custom DNS implementation...
While a custom DNS resolver is provided as part of the systemd source, which has some advantages, it is by no means a requirement, and it a totally separate service/binary. Many distros don't install it by default (yet).
Sure, just as it's only possible to run one desktop environment at a time. But just as it's good form to write programs that will run correctly under any desktop environment (by keeping everything loosely coupled and ensuring any interfaces are stable and documented), it's good form to write programs that will run under any initialisation daemon. I don't mind systemd being what gets installed by default. I mind very much when systemd is the only option because other components have been encouraged to use systemd-only interfaces. (and realistically that's the only reason systemd "won" - many of the competing options were much nicer technically, but that doesn't matter when gnome requires you to be running systemd).
> When tight coupling is done by the BSDs, it is considered a good thing
I wouldn't say that. Tight coupling in and of itself is always a negative. If there's clear value gained (e.g. useful features) then that might be enough to outweigh the negative.
I stopped using Arch when my system became unbootable after systemd was initially pushed out.
I have had a couple of snags over time, and many could argue that it's my fault for not reading "the new way" of how things work; like systemd-resolved not resolving anything (even with allow-downgrade which is the default iirc) which isn't DNSSEC validated, such as my entire corporate DNS environment.
Overall I feel like there's many benefits for a desktop system to use SystemD.
For the record, my largest gripe with systemd is systemd-resolved too (followed closely by binary logs).
That's totally not the case, all the previous solutions were slightly broken (pam_console, ConsoleKit etc.), and unmaintained.
> For the record, my largest gripe with systemd is systemd-resolved too
So, uninstall it. It's not a part of systemd (the daemon), and totally optional.
I was using fedora religiously (I mean, _really_) between Fedora 15-21, and systemd was actually what caused me to get off of the distro, the upgrade path was truly awful, even for Fedora.
The response was "but it's bleeding edge, what do you expect". And to be honest, I expected better.
Also it meant I had to unlearn a lot of my linux knowledge, and it was specific to fedora at the time.
Also, systemd in its early days was astonishingly broken, and many of the gripes you hear of today likely stem from people who were exposed during that period. (such as binary logs getting corrupted and unceremoniously truncated, boot being non-deterministic meaning your machine didn't boot 1 time out of every 10 and huge smatterings of "waiting for 1m 30s" even with very commonly used packages on a base system).
All that to say: I haven't seen a smooth transition from a non-systemd distro to a systemd enabled distro.
Although, truthfully, I haven't upgraded a Debian installation in-place.
[0] https://humblebundle.com/books/linux-bsd-bookshelf-2019-book...
But I can't figure out how to do custom selection of charity any longer. Does anyone know?
It's in a collapsed section titled "Choose where your money goes" which is beneath the price selection tools now.
Complains about a certain guy's benchmarks being affected by using different compilers, but of course that guy switched to using the same compilers already
Than he does the same thing with epoll- complaining about something being broken, but then admits it was fixed in 2016
I wasn't really able to pull anything material out of this article. I guess this is why people like benchmarks- it's hard to write an opinion piece about performance.
Also FreeBSD usage is smaller so there is not as much effort to get software working on it, like Steam or obscure/specialized software. In practice Linux packaging efforts are fragmented among distros so the gap is not huge, but it's still present.
As far as I know, on most BSDs, unless you happen to have a complete source tree snapshot for said system (becoming more difficult to maintain if you build multiple systems without special tricks to retain binary artifacts), you're out of luck, you'll get today's STABLE or RELEASE, not the one from 17 days ago when your current production box was built that works perfectly unlike the one today that is broken.
This is trivial to accomplish on all Linux distros.
(I would say this kind of issue should be listed in the table in the article).
the "preferred" way to do this in a datacenter context with FreeBSD is to run a repo yourself with poudriere; this has the downside of causing you to build all your dependencies (via ports), but the upside of having everything built with your exact specifications in mind and in a way which ensures that they are rolled out in the way you prefer and that they are consistent amongt all instances.
For "old school sysadmins" who like controlling the versions of things in a very well defined way and running their own repositories it is absolutely ideal.
However, it affects developer velocity of course.
A specific example: the part about how epoll() is broken because of its design, and how it causes 'thundering herd' issues when using multiple threads/processes. I had plenty of experience with epoll from over ten years ago, and it wasn't a problem even back then. Multiple processes listening on the same socket did not all wake up when a new connection came in. Sure, why not bring up specific issues with Apache, but there are plenty of other web servers that work very well (nginx, etc) and have existed for over a decade.
This happens right now and “powers” a stupid amount of the economy. And it’s wasteful.
With level you won't miss events, with edge if you have a process crash you might.
It's a trade off.
I would say that is actually true ... and is the root cause of the complexity crisis Linux is experiencing these days. There is a strong financial incentive to add things but no financial incentive to take things out. Every time you add a feature to a complex system you take something away from everything else.
Is there anything similar to firejail[1] for OpenBSD?
[1] Firejail is an easy to use SUID sandbox program that reduces the risk of security breaches by restricting the running environment of untrusted applications using Linux namespaces, seccomp-bpf and Linux capabilities.
Firewalls: Linux, and no mention of bpfilter?