Operating Systems Without Systemd
annihilatormodule.com
annihilatormodule.com
Who hasn't heard of, and even know the elevator pitch for Gentoo? For decades already no less.
It's great, and it obviously qualifies to be in the full list, but it's not interesting or novel to anyone who already uses unix-like os's other than osx.
1 - Can we name our distro Void.
2 - Why?
1 - Since it devoids of systemd
I realize that Gentoo became a meme at some point, but it also seems that it might have transcended to the level of infrastructure. Thus, you rarely ever heard about it for the same reason you don't hear people talk about hammers -- it is doing its job running tens (hundreds?) of millions of chrome books among other things.
No flare, nothing fancy, it has solved a whole bunch of packaging, deployment, and environment control problems and gotten out of the way of its users (mostly ... chuckle), and it did it all with a pile of bash scripts and a little python here and there. Want to use systemd because you have a use case that would benefit from it? Go right ahead! The engineering work is done, no need for fights on the mailing list. Don't want to use systemd? `eudev` to the rescue! There are sound engineering reasons for not using systemd while also wanting to use some other (seemingly) completely unrelated piece of software that just happens to have a dependency on the kitchen sink.
Despite being a rolling release distribution, I find that the UX and workflows are more stable over long periods of time than any of the more popular LTS style distros. When I started fiddling with Ubuntu so that I could help my colleagues debug issues it took me an embarrassingly long time to realize that there isn't really such a thing as Ubuntu, just 14.04, 16.04, 18.04, etc. Each is some different form the other that stack overflow questions from one LTS are often simply wrong for another.
Writing this I wonder whether rolling release distros aren't effectively the original form of chaos engineering -- if everything is going to be changing all the time then it forces the developers to solve a different and perhaps more fundamental set of problems and ultimately leads to greater stability in the long run.
The Nix thesis[0] explicitly cites Gentoo multiple times; it's very much inspired by, and intended to surpass, Gentoo.
0. https://gitweb.gentoo.org/repo/proj/prefix.git/plain/scripts...
I assume you're referring to the following sentence, but that's not what it's talking about.
>As claimed in the introduction, deployment systems tend to have monolithic trust models. Typical Unix package management systems such as RPM [62], Debian APT, or Gentoo Linux [77] allow installation of software by the administrator only; software installed by individual users is not managed by those tools
This is talking about a single instance of the package manager; for any given instance of the package manager (where an "instance" is defined by various things like the list of installed packages, package data, various package management databases, etc.), only the owner of that instance is allowed to install new packages. For a system-wide package manager, only root is (safely) allowed to install new packages.
Even at the time, RPM and dpkg, and presumably portage, had the ability to create non-system-wide instances of the package manager, like Gentoo Prefix does; but even for such instances of the package manager, it's only safe for the owner to install new packages. (You wouldn't want random other users installing packages in your home directory!) Such instances don't share anything with the system-wide instance - it's essentially like creating a chroot or container.
The reason this is being pointed out in the thesis is that Nix does not have this limitation; any user can safely install arbitrary packages into the system package manager, which are visible to the system, share package data, etc.
I'm taking pains to be clear about this because, IMO, this is one of the places where Nix has the clearest advantage over traditional package managers.
edit: just read it again, it is a list of OS he likes, so it was not an omission :)
The controversy mainly surrounds the claim that it is a single package that tries to do way too much by itself.
Calling systemd an init system isn't correct. systemd contains many things, one of which is an init system.
The other claim is that the systemd developers aren't open to much outside opinion, but have power over the community because important parts of basically all desktop Linux systems are now part of systemd and have strong dependencies on it (such as udev).
1. https://www.freedesktop.org/software/systemd/man/systemd-jou...
I otherwise don't care about systemd, except for the more-than-one locations I might find an init script hiding on my system. It's fine.
But don't make me use journalctl just to tail a god damned file like a normal person.
Edit: I may have reached that point on friday night where the bottle of port and my grey beard have finally reached "old man yells at cloud" stage. My apologies.
I absolutely love the idea of journalctl (without commenting on the execution).
As an admin, there was always the chance that $DAEMON_OF_THE_DAY put their logfile in a particularly creative place, thus sending me on some wild goose hunt. Now it's just `journalctl -u $DAEMON` and that works immediately, every single time.
Also, as a user running journald on a notebook, I love that it's just one config switch to send all the logs from all the services to /run instead of /var, to reduce the write load on the SSD. Sure you could fiddle with a bind-mount but that would have to happen very early in the boot process so I don't think it would be as reliable.
As a daemon writer, it's one less option for me to care about. Especially when my daemon is just some bash script, logging to stdout is the easiest way.
And logging to stdout also meshes well with running the same daemon/script in container environments like Docker or Kubernetes where containers usually don't even have syslogd. (In fact, I have containerized a service that requires syslogd and that was kind of a pain. I ended up using a mini-syslogd implementation that just dumps all logs on stdout so Docker can pick it up.)
Unless, of course, some of the binaries are corrupt.
My major justification is I use tail and head constantly, every single day, and it's rarely on system logging. It's java logs, it's python logs, it's rust logs. I'm not going to be giving up tail or head any time soon.
Now I have to type "journalctl --help" every single time I want to look at a system or daemon log, which isn't super common, and when I do need to I'm already frustrated something isn't working.
Put another way, this is the exact WORST time to throw up more roadblocks over something. That's why I assert it should be logged twice, and for those who don't want it to be chewing up diskspace or want to relocate it, well, that's where you can configure it.
I just want my old log files back in their old locations without forcing me to sift through man pages when all I really want to do is figure out why the fucking bluetooth keeps dropping connection. I understand why others see this as a great step forward but it sounds more like the windows registry and event viewer to me than it does the configuration file and log file approach I adore from my unix systems. I'm sure the registry and event viewer has a lot of devoted fans as well; the difference is it's been there for most of my life (or at least, the parts where I was doing more than playing X-Wing vs Tie Fighter), which is why I mostly just wandered away. I'm not a sysadmin, I don't try to be and I don't want to be, and it seems like systemd is hellbent on forcing me to be one whether I want to be or not and I am so not here for it.
> Now I have to type "journalctl --help" every single time I want to look at a system or daemon log, which isn't super common, and when I do need to I'm already frustrated something isn't working.
I don't think that keeping users from ever having to learn anything new is a good justification for doing things a particular way.
> I'm not a sysadmin, I don't try to be and I don't want to be, and it seems like systemd is hellbent on forcing me to be one
Personally I don't understand this sentiment. As a user I have found that systemd has made using Linux systems much simpler for me. Yes, I had to read a manual, but I don't think that is anything like "forcing me to be a sysadmin". Didn't you have to read the manual for tail at some point in the past?
I fundamentally agree with you; learning something new is not necessarily a bad thing. Forcing you to learn something when you're actively trying to figure something utterly unrelated out is where it gets dicey. Not everyone wants to do the "Malcom in the Middle Hal Changing a Lightbulb" routine when there's already a Very Important Thing to Fix.
The problem with systemd is that it took something that already worked perfectly fine for my needs and said "let's overengineer the everloving pants out of this! get in the car grandpa, it's time to roll!" I didn't ask for it, I have WAY more important (to me!) things to do, and it's actively running interference with that. Surely you can see that level of frustration, right?
Put another way, I'm sure it's a LOVELY bikeshed for some. Some people have wanted a bikeshed this fantastic and wonderful for a long time, and they will use this bikeshed with great joy daily.
Meanwhile, every time I'm trying to get something I actually care about done, it's like pulling teeth. Maybe I'm using it wrong, and I'll be the first to admit I have the patience of a toddler when it comes to reading the man pages. What I can DEFINITELY say, though, is that if systemd never existed my quality of life would be doubtless better than it does with it existing. That's really the only metric I need.
Like I said: definitely in old man yells at cloud territory, maybe crossed with a dash of lazy man commits to further laziness and myopic man can't see the forest for the trees. I'm not saying you are bad to like systemd, I'm saying that it made changes I didn't ask for, it didn't do it in a way that was painless and approachable without a time commitment, and I've yet to be satisfactorily convinced that the value proposition is worth it - not for my uses, at any rate.
This means that a dead system can potentially not be read by a live system, as the journal can have undocumented breaking changes.
Also, they couldn't make breaking changes to the format or it would prevent people from reading their own old log files after upgrades
If I get some 3rd party documentation, that says it may or may not be correct, and the implementation is literally the only authorative documentation then it isn't actually documented.
> they couldn't make breaking changes to the format or it would prevent people from reading their own old log files after upgrades
This is the point. They do not guarantee that you can.
Change is good, so let's change away from systemd.
The thing is that systemd solves a lot of problems for distros in a good enough way (frankly, currently in a way that is better than any existing alternative) that they are happy to have systemd deal with those issues while they focus on the stuff the distro makers are interested in.
Not a given, since one of the issues with systemd is its integrations. Replacing init now requires you to replace your system logging and session manager, at a minimum, as well as replace all your service files.
Unlike e.g. upstart in its native mode, systemd can run SysV init scripts.
https://unix.stackexchange.com/questions/233468/how-does-sys...
Systemd favors the distro maintainer and cloud application service provider, and maybe the appliance manufacturer.
It's the exact opposite of a useful tool empowering the the individual.
systemd-resolved is nice if you're using systemd's network configuration system, but it's a pain when you're trying to run your own DNS server.
udev really shouldn't hard depend on systemd. I perfectly understand having some systemd specific integration, but why make it mandatory?
There being said, various people have been working to cut it into separate components (eudev, elogind, etc.)
udev is part of systemd now, because the systemd developers were the only ones willing to maintain it.
(This is a serious question; I'm not trying to start a fight)
So you still thought some updates were supposed to happen but it wouldn't have happened if it wasn't for systemd as people are too lazy to touch any mostly working legacy bits.
It's supposed to enable easier encryption of home directories, but breaks ssh logins.
Therefore it's probably only suitable for multi-user systems (which need to protect home directoried from other users) that are also never accessed remotely - a subset consisting of approximately 0 systems.
This results in certain packages switching to require systemd modules(gnome & udev) and essentially locking systems without systemd out of using said software. Communities supporting non-systemd init systems now have to maintain forks of these modules so that they can keep these packages supported on their distros.
The other issue, which is the issue I care more about is that it does these things in a way that breaks compatibility with existing workflows and exposes it to a number of different issues due to its increased complexity.
Don't get me wrong, systemd is excellent in many ways but like many others, I am not a fan of it overall and choose not to use it where possible.
An example of something very controversal about systemd: journald.
journald is a replacement for the system journaling daemon and is a generally very good journaling daemon. It is in many ways more performant than other journaling daemons and can be easier to use as well. The issues however come largely from the fact that it uses a binary log format. This breaks the ability for other tools access logs nearly as easily unless explicit support for the interaction is added by either journald or the tool in question.
Compare that with random older default debian log line from one of my systems:
Feb 14 01:15:30 shakes kernel: [ 4.014481] [drm] No driver support for vblank timestamp query.
Journald just works for the most part and is much more convenient in most regards nowadays. If I remember correctly, interop was more of an issue before the big distros started to switch to systemd but now things seem to have mostly worked themselves out.
With regard to my comment about wanting logs stored as text, it really just comes down to having had to deal with corrupted binary files one too many times and unless it is absolutely unavoidable I find myself preferring text files for that reason. I'd imagine file system based compression leaves most if any storage benefit from binary logs rendered moot.
I know you can run a secondary syslog daemon from the output of journald or alternatively use a job to backup to a text based format but it would be so much more convenient to be able to just set a config setting. Having easy config and things "just working" is supposed to be one of systemd's strengths anyway.
Final thing to note is that at least from the perspective of someone running Gentoo w/ OpenRC, issues with traditional loggers are typically config mistakes more than anything. As for your debian log example, syslog-ng which I believe Debian had used should default to isodate for the datetime format. At least this is the case after RFC 5424 was adopted in 2009. That should give something along the lines of "1985-04-12T19:20:50.52-04:00". Nowadays unless the defaults are changed, most logging systems shouldn't have that datetime issue.
At least from my experience, the only distro I have used that has offered any facilities in the package manager for handling config updates is Gentoo. Whenever there is a package update, if the default configuration files for the package change during an update, you are notified to run `etc-update`. This command then allows you to diff your current config with the new defaults, choose to use the entire file for the current config or the new default, or patch certain lines from the new default into your existing config.
Personally I find this super convenient, especially because whenever you perform a package update or install it will let you know whether you have any outstanding configuration files or important upgrade notes to address.
I'd be greatly interested to know if other package managers have these types of config and service management utilities. I looked around but I didn't really find anything.
the future is what is all that matters and getting there is what matters. have you heard the phrase "the end justifies the means"?
that's it.
Things like NTP, syslog, DHCPd, cron, iptables, mount, automount, handling /tmp, /dev, DNS, su/sudo, etc.
Letting the init system talk to the network before userspace has the ability to setup firewalls gives me the heebie jeebies.
Often the implementations of systemd don't handle failures well, leave off critical features like encryption, DNSSEC, etc. They are often hard to debug, and generally worse than the features they replace.
Not that Linux didn't need a nice init system, but that systemd is trying to take over everything. The lastest volley on that front is taking over management of /home with homed, but in a way that's incompatible with ssh.
Do you really want to depend on systemd's homed for managing /home? I'd rather not.
> Letting the init system talk to the network before userspace has the ability to setup firewalls gives me the heebie jeebies.
Are these just random claims? The init system is userspace, and you can have any service wait for anything to access the network, including disabling the whole network target and triggering it manually.
And yet, any honest assessment of the pros and cons of the homed proposal has to discuss the impact on SSH users in the cons section.
Here’s one for systemd DHCP: https://blog.erratasec.com/2018/10/systemd-is-bad-parsing-an...
Firewalls wouldn’t help with either of these, in fairness.
You can't. And the systemd answer is that you simply throw away ssh.
The design of classic Linux DNS simply assumes that 100% of Linux systems are university lab machines, and that there is a single DNS server that is 100% reliable.
I've heard people making various arguments to support the classic Linux DNS client's failings as if they were features. These have included: that I should "fix" my telco's DNS servers (wtf!?), set up a HA load-balancing solution on premises (in my home!?), use anycast routing (really!?), reinvent the DNS protocol to not require multiple servers (I'll get right on that), or change third-party software to not require DNS (wat?).
When pressed, all of these people eventually boiled their arguments down to: Fine, yes, it's broken, but it's the way it is, that's what the man page says, that's what was established in the 1980s, and systemd is different so we don't like it!
It's mind boggling how conservative Linux people are...
This was on some Ubuntu LTS. 18.04, maybe? I paved the machine, installed a distro that doesn’t chase the new shiny every six months, and moved on.
What?
Naturally this is very controversial and people complain that it is against the UNIX philosophy.
The problem with Systemd is that every time they introduce new non necessary feature they break something in the system. https://bugs.kde.org/show_bug.cgi?id=417038 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394 Do they run tests ever?
Now when I launch my desktop I have to login in tty console first, start tmux, detach it, logout from tty, then login into desktop session normally. Otherwise logout from desktop session will hang and logging in again will fail.
They suggest using `systemd-run` instead, but it's broken too at this time. https://github.com/systemd/systemd/issues/3388
- Linux distributions without systemd (https://web.archive.org/web/20190208034948/http://without-sy...)
- Operating systems without systemd (https://web.archive.org/web/20190208034948/http://without-sy...)
Then he adds, "Here I’m just going to mention a few operating systems that I personally have found interesting." So these few are just ones where he thought he had something fresh to say. He did not aim to compete with the more objective lists he just mentioned.
Crossplatformness or security or convenience in exchange for bloat are all EZ trades. 2.8/62.8 GB of RAM used right now, btw.
That list is hella weird and should be named "Experimental OS that I'm curious about, but don't want to run myself. Also, none of them has systemd"
Denial of service today. Disk operating system when they reinvent FAT poorly.
That's... one point of view. Another is that Android is very successful at content consumption while falling short at more complex workflows, possessing a remarkably fragmented ecosystem, failing to deliver security updates to most of its users in a reasonable timeframe if at all, and possessing little to no flexibility at the lower layers (ex. supporting BTRFS or ZFS).
No shit, it's a telephone OS. I'm pretty sure 'complex workflows' weren't one of the design goals for Android - nearly everyone has devices much more suited to that task.
NixOS and GuixSD being the next generation of distributions and people are calling it "neat project with some unique features, but it is more fun as an intellectual curiosity than as something practical" while arguing if a distribution should use Systemd or not...
I'm not sure it will ever become mainstream. It might remain small forever, e.g. like the haskell/fp community.
The best part is, there wouldn't be fragmentation of package maintenance between distros.
It’s about an ever expanding take-over of userspace by a single module, which is not the Unix way.
And about a high-handed project leader who is paid by Redhat to do it full time.
* http://jdebp.uk./FGA/run-levels-are-history.html
getty was obsolete in the System 5 world two years earlier than that, in 1988.
* http://jdebp.uk./FGA/inittab-getty-is-history.html
Whilst not peculiar to Linux, it after all having been created for Minix, the van Smoorenburg init+rc system did resurrect some things that were already years gone in the System 5 world. Calling them "sysv" has always been somewhat of a misnomer. They are Linux/Minux clones of old System 5 mechanisms that were out of date at the time.
Another smaller thing: /etc/os-release replacing the various /etc/*-release files.
Fictional Debian dev B: "That's how we've always done it. Plus if we touch it it might break some guy's script."
It may not be exactly cargo culting, but at least an adjacent concept.
I wonder how long it'll be before the contents of that file start resembling a user-agent string, complete with "Ubuntu-16.10/Compatible"
* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=444678
* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=659891
* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=659853
I absolutely _don't_ think that my init system should also be my bootloader.
It also shouldn't manage my network configuration, or be responsible for controlling the system clock.
And it shouldn't assert control of my home folders.
And it shouldn't be spidering its way into the Gnome login manager, which grew a systemd dependency a couple of years ago.
My problem with systemd isn't pid 1. It's all the rest of the shovelware that comes along for the ride. And it's the "you'll use what we tell you to use" attitude of the project.
This could not be further from the truth.
Of course, since this hack solved the gnome bug, it was enabled by default by many, thus breaking applications like screen and tmux.
What should have been done is fix the issue in whatever offending application. Certainly not at systemd level. Because in that case, you need to do some insane things (IMO), like link tmux with libsystemd [1].
The backstory is actually enlightening to me. To me, RemainAfterExit just seemed like the obvious sane decision and I actually wondered why it ever was different. When a user logs out, I absolutely want everything cleaned up after them unless they explicitly want something like screen or tmux to linger.
> you need to do some insane things (IMO), like link tmux with libsystemd
That would probably be the easiest choice for the tmux devs. If they don't want to link libsystemd, they could have accessed logind's DBus interface directly (either by calling dbus-send(1) or via libdbus). Contrary to wide-spread belief, systemd does, in fact, present well-defined and documented interfaces that other daemons can implement as well (and they do, see elogind). libsystemd is not magical.
It's nice to know that libsystemd is not required in all cases, but I still have a problem with software having to be modified because systemd changes how libc's daemon() function behaves.
* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394#221
Good thing it's not?
systemd-boot is an entirely different binary, it's not even installed by default by any large distro I know of. The only commonality between them is that they are developed under the same broad project umbrella, just like "cp" and "ls" and "cat" are all part of the GNU coreutils umbrella.
This is a theme for an entire class of incorrect complaints about systemd -- the misconception that just because a tool is called systemd-$thing means it's part of the systemd init system, rather than "a tool developed by the systemd project".
The best thing about Linux is that you are free to pick and choose even low level components of the system as much as you'd like.
gnome-shell used by the Gnome/Ubuntu login process also depends upon evolution-data-server!! See `apt-cache depends gnome-shell` for other dependencies...
Debian chose systemd instead of upstart because of licensing, not features, according to https://wiki.debian.org/Debate/initsystem/upstart
If your service file is a tad more complicated, then you quickly discover that there's a scripting language masquerading behind key/value entries.
I have a hard time wrapping my head around dbus being the core component used for coordination among systemd components. dbus wasn't built for this use case, and having PID1 depend on a well behaved dbus for proper system operation sounds insane. systemd adds a lot of features, which means additional complexity. My understanding so far is that the tooling is not there yet to manage this complexity.
Another gripe I have with systemd is that it does parallel service start introduces non determinism at boot. Which means your setup could work very well most of time, but fail miserably 1 time out of 10. Not exactly what you'd want if you are rebooting a remote server for example. s6 [1] init doesn't appear to have this problem.
Kind of ironic when one of the oft touted complaints I recall about Linux vs the BSDs or virtually any historical Unix is how the userspace and kernel are not one integrated distribution.
I don't see any of these people making better contributions. It's easier to sit there and insult the project leader for being paid by RedHat - but let's face it. Without commercial sponsorship, Linux would be dead.
So quit your whining. If you all ready don't like systemd that much - why don't you implement a better alternative - as you all seem to have the answers.
You confuse Systemd the init system with Systemd the project.
The Systemd project can be described as "GNU coreutils but for low-level system components". People hear about all the tools that the Systemd project maintains and think that all that functionality is built into the init system, when the reality it's >50 separate binaries, the majority of which (like systemd-boot or systemd-resolved) are completely optional and aren't even installed by most distributions.
And while I understand that some parts, like journald, are not so independent - it's still not "a single module" in a way that prevents it from being unix-y. The Unix philosophy is about separation of concerns, not having a bunch of interchangable options for low-level system components.
And you know what? It’s fantastic. The API is generally well-designed (if not without a few sharp corners) and the whole thing is incredibly well thought out. Sure, plain text logs let you use standard UNIX tools. But doing so sucks. God help you if there’s a newline in your logs or if you have to ensure that you iterate over all the logs even across restarts or if you need to parse logs back out from text into the underlying attributes.
And in 20 years, Red Hat will have finally finished re-inventing Windows NT on top of the Linux kernel.
Yes, I’ve used OpenBSD. It’s great, but not for me atm.
This gets pulled out far too often. Systemd is a family of projects thast are largely modular and independent, not a single binary that takes over everything. Each part's goal is to do one thing, and you're free to take or leave each piece.
By this logic, Bell Labs should have never worked on UNIX or C - they were a telephone company!
Or,
Unlike children, who call other people names, the adult in the room rarely does?
No, this didn't start now.
In fact Guix System 1.1.0 was released a week or so ago.
Most contention is over the latter, not so much the former by itself.