Systemd: The Good Parts
christine.website
christine.website
But the one thing that still really pisses me off about systemd-the-project was the fact that they ate udev-the-project. In my view, that decision was unnecessary and was done for purely anti-competitive reasons.
If you're not familiar, udev is basically the user-facing device manager for Linux. It's what allows you to easily configure rules and permissions for all the USB devices that you plug into your computer. These devices still need kernel drivers, but udev is how you tell your system "Please let user ohazi use this device, and also give it a convenient name like /dev/someDevice0"
By devouring the udev project, the systemd maintainers have guaranteed that dealing with USB devices on non-systemd systems was going to be a giant pain.
Then came the forks -- Gentoo maintains eudev, which is a systemd-free fork of udev. But really, this shouldn't be a fork. Udev should be independent and available on all systems. If systemd wants to do something special with devices, they should use udev APIs like everybody else. In my view, this is what finally allowed systemd to win the init war, despite all the protest. Ideological arguments about how best to start daemons is one thing, but you simply can't use a modern system without a sane approach to USB.
Edit: It seems that some of my concerns were overblown, e.g. you apparently can run udev without systemd as pid 1. Udev has a build-time dependency on systemd, but not a run-time dependency. The eudev fork removes that build-time dependency, and (maybe?) also papers over some other inconsistencies. I'm pleased that this is the case. I'm annoyed that the discourse around systemd/udev has been muddy enough to lead me to incorrect conclusions (and I'm aware that the original version of this comment likely added fuel to this particular fire). Oh well... live and learn.
By doing this, we have made the maintenance and support for this core userspace tool much easier for everyone who was involved in working on it.
If the developers involved in a project do not do what you want with that project, feel free to fork the project as that's the beauty of open source! And hey, that's exactly what the eudev developers did, go use their fork if you want to, no one is forcing you to use the version from systemd, just like no one is forcing you to use any other open source program. It's your choice to do so or not :)
And if you have questions about how this happened, look at the source for when it was merged, it's all there for everyone to review :)
Also, udevd is a program, not a library.
That way udev stays a separate project and you don't have to bring in a big library to use it. Systemd needs udev both ways so it changes nothing for it apart perhaps a bit of maintenance for a separate library for the shared code.
Expanding udev's library would probably end up a decent size library of other functionality, I don't think there is any good way around needing this. This is not code that could be deleted from systemd if it was separated. The way eudev has handled it is by doing what you describe and copying those shared functions out from systemd: https://github.com/gentoo/eudev/tree/34b2037d379e33f1cf79a34...
The result of that being there are two copies of the same shared code floating around, which is what they were trying to avoid.
(That bug was causing udev to give multiple disks the same name in /dev/disk/by-path, which I think is pretty much core udev functionality, given that udev was intended to supersede devfs.)
I think useful action would have been more likely if udev had still been maintained by a team who believed they'd taken on the responsibility to make /dev/disk/by-path work, rather than maintained as a small part of a project whose maintainers are mostly interested in other things.
(Also, IIRC from what I had to do to work around it, they could have fixed that bug by reverting the change that introduced it. It wasn't a case of existing udev code not working with new hardware; it was caused by someone making changes without understanding the consequences.)
# journalctl -u ssh
And I've also noticed that low-level systems projects generally attract relatively few stars (i.e. a javascript library or go library will have more stars than a well used c library or systemd). My theory on that is that the people involved in projects like systemd, gnu, etc, see github as mostly just a git repository host, and don't bother with stars etc.
I guess from my perspective, I put zero weight into the stars on a project, and issue count only matters in the context of how the project deals with issues. What do you see as "worrying" in there? Do you see more value in those numbers than me?
This tells you why "low-level systems projects" attract less stars. It's because well-known, long-lasting projects attract less stars. There's a little point of starring a repo of a project that you recognize and use. You remember its name already. Meanwhile, a random JS library that looks like it could come in handy in the future? These things come and go, and you'll forget its name 5 minutes from now anyway, so you star it.
That's the main issue (not udev specifically but the overall philosophy of trying to make systemd mandatory for as many things as possible).
Any distro adopting systemd is basically forever giving up any hope of moving away from systemd in the future.
I'm still on Debian (Debian user since "Buzz") but I already tried Devuan (Debian fork without systemd) once and may switch to it at some point.
I simply don't like the overall mindset of those pushing systemd.
This is not correct, I don't know where you got this idea. The systemd-udevd daemon still runs without systemd. The code has moved into the systemd repository, and it has a build dependency on libsystemd, but otherwise it has no runtime dependency; your distro should be able to package it separately if it wants.
> > Also, AFAIR the connection between systemd and udev doesn't really go deeper than the fact that they're sharing the same upstream tarball. It is still possible to build and use udev without systemd
> But that's not really the case operationally, and it's why eudev was forked away from it for Gentoo.
I believe the reason for eudev is to avoid some of those other unwanted changes from upstream udev that broke udev scripts. Which to me is a fine technical justification, but not really related to another udev implementation having a build dependency on libsystemd. The udevd package seems maintained so I really don't understand what operational issues that comment is getting at.
I still have concerns that having a shared codebase makes it easier for interdependencies to materialize later, but if they've managed to keep it as decoupled as is claimed here, then I'm pleasantly surprised, and also annoyed that the communication around this issue has been confusing enough to lead me to an apparently incorrect conclusion.
[1] https://wiki.gentoo.org/wiki/Gentoo_Without_systemd#The_udev...
Using the kernel interfaces that udev relies on without using udev itself is also unsupported and the developers consider it within their rights to break those as well.
>The main reason why they haven't yet is most likely that their attempts to integrate dbus into the kernel and systemd more tightly were rejected by the kernel devs; if I remember rightly that was expected to be the point at which you could not use it without systemd.
That seems quite dubious, udev has never depended on dbus, and there really would be no reason for it to ever do that.
>Using the kernel interfaces that udev relies on without using udev itself is also unsupported and the developers consider it within their rights to break those as well.
This I know is true, the netlink stuff varies really wildly from driver to driver. The alternative would be to put udevd and the other userspace bits into the kernel, which I really doubt anyone wants to do.
binutils and gdb were also moved into a common repository and gcc merged some repositories as well.
There is probably a reason why even many binary systems adopted eudev instead. — the maintainers often justify this with “problems” with udev but never go into much detail about what that is but the language does keep suggesting that it restored some functionality that is missing if it run sans systemd.
People often misunderstand what libsystemd does. See the many reactions here, but also e.g. the response by Devuan around libsystemd.
Too often people assume things around systemd, or too heavily rely on incorrect information found via Google.
That things are being done this way by a few projects to me isn't enough. Nowadays people did not thoroughly investigate things before they take a decision. See e.g. the various responses in this thread.
That does seem kind of unfortunate for anyone that wants to bootstrap a systemd-less distibution.
It's just one of those things where it's better to stay away.
This would explain why they imitate or copy many of its features (and misfeatures), and seemingly tend to replace the Unix ways with entirely different approaches. Effectively forcing things on users, as it was with pulseaudio and systemd, is also a very macOS thing to do.
I'm not opposed to replacing Unix with different approaches, as long as they keep playing on its key strengths, and do not remove modularity and composability. That is, Plan 9 is fine with me. Much of what Red Hat does, sadly, no.
I've heard this a lot, but having used Gnome briefly and MacOS fairly extensively, I just don't see it. Also, have you seen KDE's Dolphin? If ever there was a Finder clone, it's Dolphin. With regards to copying MacOS, Dolphin devs have wiped the floor clean with Gnome.
Besides Dolphin specifically, KDE in general is flexible enough that making it look and feel like MacOS is pretty easy. Gnome though? It's not particularly MacOS-y by default, and Gnome devs have naked contempt for customization, so good luck if you want to try anyway.
However, I share your suspicion that most Gnome developers do not actually use Gnome as their daily driver.
Interestingly, classic MacOS, at the times of 6.5 or 7, was somehow more customizable, and seemed to have more power-user features. Just like, well, Gnome back in the day %)
So, my daily driver is Xfce.
I think Dolphin is one of the very best Linux apps out there while I found Finder to be very clunky, often so unintuitive and useless that I resorted to command line instead. To be honest, it would probably have been fine if I took the time to learn the Mac idiosyncrasies like missing cut/paste, but Finder was one of my main gripes when I had to use Mac for work.
Dolphin just has weird omissions; for instance try to sort a directory of symlinks by the time those symlinks were created. As far as I can tell, you can't do this because Dolphin follows the links and sorts by the dates of targets (contrast with `ls -ltr` and Finder, which sort by the dates of the links.)
The apps are a different story, those tend not to go for a lot of customization and will usually just focus on optimizing for one or two use cases. If you have other use cases, the suggestion there would probably be to make a separate app.
The problem is they hopped aboard the same bandwagon Microsoft did with Windows 8, where every started throwing tablet/touch style UI onto desktop interfaces.
For the life of me, I don't understand why there wasn't a happy middle ground of "or we'll render the main menu as a regular menu bar".
But at this point it's not a GNOME criticism so much as an entire UI/UX industry criticism.
Not that I don't like Red Hat. They are the poster child of a successful open-source company, and did and keep doing a ton of great things. I only don't like certain directions of development of certain userspace things which happen to be done under their corporate umbrella. These things, systemd in particular, were quite noticeable on the Linux landscape as of recently, both in the tiny desktop niche, and the huge server sector. Nobody's perfect, you know.
systemd is great though.
I can image someone thinking then "I'm implementing everything twice, both for udev and systemd. I'll just merge them"
However, as a heavy user of both projects, I see what those decisions have led to in practice, and have opinions about what this might mean for the community in the medium/long term. The reality on the ground today is that you either run systemd+udev, or you run OpenRC/something else + eudev. And having observed other projects that went down the "we forked, but merge regularly to try and keep the forks in sync" rabbit hole (e.g. ffmpeg and libav), this almost always ends badly.
The goal may be to keep the two separate projects compatible, but inevitably environment differences, bugs, refactors, etc. cause the two projects to diverge. This sucks for everybody, but it sucks a lot more for people using the less popular fork, and this can eventually kill the fork. I worry about eudev long-term.
This is a strange mindset. Do Star Wars fans not have a right to complain about bad movies just because they weren't part of the cast or crew?
The developers of udev at the time agreed to merge to the project into systemd, as it made a lot of sense to them.
There was no "devouring", no "anti-competitive" bullshit.
I work for a company where every year there is a mandatory refresher on anti competition laws / behaviour. The suggestion that this is done for malicious purposes is highly offensive, yet so easily stated. This without actually going into a lengthy and thorough explanation. I think you're failing to understand how offensive you're being towards these developers. So easily using very harsh words, so easily dismissing what anyone else says, plus what the developers said. Yet not actually proving any good proof for your statements.
Because saying that they were separate groups implies a level of propriety to the decision that doesn't hold when the groups "overlap".
> You're saying it was pushed.
Am I? Where?
> But actually the maintainers agreed.
At the risk of repeating myself, this implies a level of propriety to the decision that doesn't hold when the maintainers are the same in each group.
> I work for a company where every year there is a mandatory refresher on anti competition laws / behaviour.
Didn't take, did it? Just for giggles, see if you can identify the common factor in these classes of anticompetitive behaviour:
- Cartel price-fixing
- Refusal to deal
- Conspiracy to monopolise
- Dividing territories
- Regulatory capture
I don't. I also don't think they need you to defend them.
Such are all the common criticisms on systemd.
It was never a technical objection; it was a political objection to the fact that systemd unnecessarily coupled things that could and should be separated, but were coupled not for technical reasons, but for anticompetitive ones, and this goes beyond systemd.
Many Red Hat projects have a way of finding themselves to be dependent on one another in ways that are often hard to justify from a technical standpoint, and often even use unstable, undocumented interfaces that they make available for one another.
The argument is that it's not done due to incompetence, but commercial strategies.
Saying that some optional code has issues as a reason to not just use something else, but as a reason to not use the project: not logical.
It will fail to unmount most of the filesystems, since they are of course busy, but often it will succeed in unmounting /var, /tmp, /home and others. Then it will continue starting on further services as if nothing happened and even proceed to gdm. But of course without /home I can't even login.
No idea how to even start debugging it.
Despite dropping me to a rescue shell, systemd entered a continuous loop of retrying fscking /var while I was attempting to identify and fsck the problematic filesystem.
Few things are more infuriating than having your system drop you to a rescue shell, while it continues to endlessly change the state of the system - and not just any state, but the very area you're in the rescue shell to investigate and resolve. It just kept making /var busy while I kept attempting to manually fsck it...
If you are not curious, reinstalling everything could be quite fast too, depending on what you have installed.
And this is not a problem that happens on every boot, so it doesn't show up on analysis... not to mention that most of the analysis is designed on trying to reduce boottime, which obviously is no help (thought it is rather fast already).
It is rather easy to guess why a service is starting, but not why it decides to stop 'orderly', much less why a .target decides to stop...
Indeed systemd-analyze focuses on boot times, though I find it is useful to get a list of services that are run.
If you are adventurous you could probably replace the umount (resp. the systemd-umount) binary by a script that runs umount (resp. systemd-umount), and also prints the process tree (and its PID) in some file so you can get insight on what ran it.
You could also replace umount with a wrapper that logs the parent process tree on execution.
And of course you can open an issue on GitHub.
Only do this if you've reproduced what seems to be a bug in one of the latest two releases, otherwise contact your distro.
The systemd github issue tracker is not a support forum, you'll find this stated in the issue submission form.
But on a normative level, perhaps a project that doesn’t want to support users shouldn’t have subsumed a huge swath of user land things?
It's not that the project doesn't want to support users in some absolute sense, it just doesn't want it happening in github issues.
The project provides the systemd-devel mailing list for that, but it's still preferred that users stuck on old versions determined by their distro's release cadence seek support from their distro vendor. It's likely such slow-moving distros are patching their LTS versions anyways, and are best positioned to support what they ship.
You may not like it, I'm just playing messenger here, to try save some bother on both sides of this subject. Use the mailing list for support questions, or contact your distro vendor for support.
I ask because I use /dev/disks/by-label and have never had this problem, and if I ever had to experience it, it would drive me insane.
/dev/disks/by-label is a legacy from the old days of systems with static disks wired in a specific, unchanging configuration.
However, unlike UUIDs they don't necessarily exist, are not as likely to be unique (you should only get duplicate UUIDs by cloning filesystems) and are easier to change. This makes them less suitable for mount configuration.
But this was a long time ago, on a much older systemd (like Fedora 17 or 18).
As someone said by-label is not stable and could be changed, causing problems. We still use it though but not in customer facing servers where these problems arise. Maybe related to lvm/snapshot etc.
[0] https://freedesktop.org/wiki/Software/systemd/Debugging/
It isn't impossible, and OK in a pinch, but why make it hard on yourself?
https://ostechnix.com/how-to-boot-into-rescue-mode-or-emerge...
You can also look at systemctl list-dependencies local-fs.target to see if it has any failed dependencies
You can also use systemctl show local-fs.target to sanity-check it to see if there are any local modifications to the target that are breaking it
I had straight up errors in a network unit file that caused weird behavior (IP address disappearing). This was a long time ago though. Sometimes moving a unit from one target (graphical) to an earlier (multiuser) solves the issue.
You might also have "defaults" in /usr/lib/systemd/ to compare with.
systemctl and friends are really good tools for troubleshooting but takes a little while to get used to. Duck duck go is your friend.
Finally, some volume technologies like VDO requires a x-systemd.requires statement in fstab.
$ systemctl status local-fs.target
Active: inactive (dead) since [...]
21:12:56 ... systemd[1]: Reached target Local File Systems.
21:12:57 ... systemd[1]: Stopped target Local File Systems.
Incredibly enough, local-fs.target is a dependency for graphical.target, so the system should not have continued booting. But not only it has continued booting, systemd even thinks that it finished booting all-OK. State is "running" (not "degraded" as it would be if any service/mount failed), with 0 failed services. Even though both graphical.target and local-fs.target are 'dead'.
`systemctl` is the systemd command line tool for inspecting and manipulating services/units. `systemctl status` is the command for showing the status of a service or unit.
The `--` part is not specific to systemd at all, it's used in many commands to separate flags from positional arguments. This lets you use positional arguments that might happen to start with a dash without them being interpreted as flags. [1].
systemd manages mounts, they're named according to the mount point, with a transformation to turn it into a filename (mostly converting slashes to dashes, plus a .mount suffix). Hence, the root mount ('/') is named '-.mount'.
[1] See, e.g., https://unix.stackexchange.com/questions/11376/what-does-dou...
I wish they had picked ':' for '/' instead of '-'. That one would be pretty uncommon to find in a mount path, and has a bit of precedent on OSX.
(Who would ever mount anything on a path with a colon anyway? That would be as silly as using slashes instead of dashes for options)
https://www.freedesktop.org/software/systemd/man/systemd-esc...
Hint: to specify units starting with a dash, use "--":
systemctl [OPTIONS...] COMMAND -- -.mount ...
systemctl status *.mount
It's the same command you'd use to check any other systemd unit, for example, if you want to check Docker's status:
systemctl status docker.service
find ~/Code -type f -name *.js
You could put quotes around it to make it clearer:
systemctl status "*.mount"
systemctl status -- $(systemd-escape --suffix=mount --path /)
If the system has unmounted all the drives, there won't be any logs. This happens often enough with systemd / journald to be a big source of annoyance.
The only option you get is to tell systemd to boot in debug mode but that fundamentally alters the timing and behaviour of so much that it often stops the weird behaviour from ever happening (similar fun can be had with `dracut` where enabling debug tends to stop any race condition from ever happening)
There's some delicious irony in having to write a shell script to combat the init system that was designed to replace all the shell scripts.
I like alpine because it uses a simpler, lighter approach. It's not for everyone, and I'm pretty sure alpine maintainers don't intend it to be for everyone. It serves its niche really well as it is. For those who like systemd there's many distro choices already.
Edit: As detaro mentioned in another comment ( https://news.ycombinator.com/item?id=27176391 ) they are actually working on something, but they aim to solve the various issues around alpine that don't fit well with alpine's philosophy. It sounds like a good story and I'm open to this idea. The main reasons I don't like systemd are its complexity and its dbus reliance, basically bringing too many desktop paradigms into all Linux systems, even servers and containers.
So about my above point: I'm open to it adopting a service manager but not too similar to systemd :)
Alpine uses OpenRC (https://wiki.gentoo.org/wiki/OpenRC). IMHO Alpine and OpenRC tend to make production deployments more predictable and reliable, because of deliberately simple init ordering, as well as a smaller surface area as compared to systemd.
What they're doing looks like a great alternative. I also hope they will not have any reliance on dbus as systemd does. Reliance on dbus is one of the main issues I have with systemd (besides its size and complexity).
I think it's one of the examples of Systemd pulling in way too wide a scope.
> I have been an Alpine user for almost a decade and it's one of my favorite linux distributions.
> This talk is intended to show how green the grass is on the other side and how Alpine can benefit from these basic ideas.
She's probably familiar with it.
I always ask ""What problem is this trying to solve?" and then "Is this a good solution for that?"
I appreciate the "startup dependency problem" and when I was in the Systems Group at Sun we worked with AT&T on their solution which was the whole "run level" thing.
And I appreciate the whole "shell scripts can do anything" problem that just using a naming scheme on shell scripts to run can lead to unexpected behaviors.
From an architectural taste perspective, I prefer systems which are more comprehensible in parts and thus don't try to abstract out semantics to the point of incomprehension.
So at the end of the day, something with a better model, better visibility into its operation and configuration, with a way to validate and debug dependencies would be good. But I haven't been impressed with systemd.
* It gets rid of a system cobbled together from init, inittab (which people rarely remember exists), monit, cron, and inetd/xinetd.
* It removes a bunch of horrible hacks in init scripts such as calls to sleep.
* It removes the need for pid files.
* It removes the need for a lot of stupid boilerplate, like the whole start-stop-daemon stuff in scripts.
* It allows precisely tracking what belongs to each service. I can easily find out what a random process belongs to.
* It captures all the log output of each process
* It drastically improves the upgrade process -- I never need to look at a 3-way diff of /etc/init.d/apache2 again
* As a developer it means I don't need to write a slightly different script for every distro
* It makes logging a whole lot pleasant
* It allows having user services without having to hack around it with cron
* It allows a whole bunch of settings to be applied to any random service, without having to edit a shell script and figure out how to stick it in there.
* It means services run identically when started by hand, without the confusion of a service inheriting my local environment.
* It makes my VM servers boot faster. And if you're going to pipe up with "I reboot my servers once a year", that doesn't cut it for me. I boot my servers on demand, when they're needed. They adapt to the load.
It is a nicely concise summary of the problems. I'd disagree with the characterizations of the previous system as "cobbled" or "horrible hacks" but there are solid things that need to be solved well by any system initialization architecture/service.
Recently, I started using pipewire, which is designed to rely on a user-level service to start and stop the pipewire process.
To bypass this requirement, the slackware guys came up with an elaborate solution that wraps the pipewire command with a daemonize tool that is supposed to cleanup after itself: https://www.linuxquestions.org/questions/slackware-14/using-...
Meanwhile, the gentoo guys just did the simplest thing possible: https://github.com/gentoo/gentoo/blob/f788f38/media-video/pi...
Of course, with both of these workarounds, the semantic changes from "start this process once the user logs in" to "start this process once the desktop environment starts", but the end results are identical for almost all users.
You mean Slackware users on some random forum. Besides, the solution they came up with uses XDG autostart which has nothing to do with systemd. Not to mention that it's not even doing the exact same thing as the Gentoo solution and running two more commands in addition to pipewire. On top of that, Gentoo also seems to be using XDG autostart for Pipewire[1] so it's basically the same approach.
[1]: https://github.com/gentoo/gentoo/blob/f788f389419678d2f2399e...
Believe it or not, that's actually the official slackware forum. And whatever solution those guys come up with, it will likely become the official solution.
> Besides, the solution they came up with uses XDG autostart which has nothing to do with systemd.
The slackware solution involves a project that nobody has heard of before, just so it can imitate the "user-level service" feature provided by systemd: https://github.com/raforg/daemon
> Not to mention that it's not even doing the exact same thing as the Gentoo solution and running two more commands in addition to pipewire.
The slackware solution requires starting those 3 processes (pipewire, pipewire-media-session, pipewire-pulse) separately from 3 different .desktop files, likely because the daemon tool above can't properly reap the pipewire-pulse process (not sure whose fault is this though).
On the other hand, the gentoo solution can start all 3 processes with just 1 .desktop files, because `pkill` takes care of it. Simple and effective.
I think the key difference, in this case, is that the slackware guys are trying their best to imitate a systemd feature, while the gentoo guys seem to focus more on finding the best way to allow users to enjoy pipewire.
My bad. TIL.
> The slackware solution involves a project that nobody has heard of before, just so it can imitate the "user-level service" feature provided by systemd
Daemonizing processes has been a thing far before systemd even existed. Solutions similar to this "daemon" project are commonly employed in SysV init scripts. If anything, systemd made those things irrelevant by offering a more simple configuration to daemonize programs.
> The slackware solution requires starting those 3 processes
> On the other hand, the gentoo solution can start all 3 processes with just 1 .desktop files
It looks to me Gentoo doesn't start the other two processes. I don't see how "daemonize" or pkill is relevant here.
> slackware guys are trying their best to imitate a systemd feature, while the gentoo guys seem to focus more on finding the best way to allow users to enjoy pipewire.
With respect, this is a unsound take. The explanation is more simple and it's that Gentoo does not currently handle pipewire cleanup.
lol. I had to trash an Arch Linux box because it failed to upgrade systemd from 208 to 211. Everytime I upgraded the system hung trying to mount filesystems so I had to roll back.
On the desktop, I feel like boot time is slightly longer than it was 20 years ago, yet hardware is faster now. I used to shut down my computer every night back then; now I leave it running, because it takes so long to get back to where I left off.
I think it takes my Linux system ~13 seconds to boot? That is definitely less time than it took 20 years ago, but I think that's mostly b/c the storage is no longer slow spinning rust running on IDE cables more than anything else…
It's Monday morning at 10:30, I've already got 29 tabs on 3 windows across 2 virtual desktops (2 monitors each) running on my main desktop. I've trimmed down to just 30 various rxvt windows with ssh or vim open, with files and notes at various stages of completion.
Some tabs may restore next time I load my browser, some won't.
My current server boots in 10 seconds, which isn't bad.
Interesting that you mention Sun. Didn't SMF precisely replace runlevels with SMF milestones[1] in much the same way systemd replaces runlevels with target units?
[1] https://docs.oracle.com/cd/E23824_01/html/821-1451/hbrunleve...
Good to see people down-voting a completely factually correct statement that they don't believe and don't want to personally investigate. Go read the systemd sources or read any article analyzing how systemd does dependency based startup and see for yourself.
The biggest problem with Upstart was that it put the dependency chain upside down and just started a service whenever it’s dependencies were fulfilled, no matter whether it actually made sense to start the service.
The Upstart design was probably easier to implement. With the systemd design, the dependency tree needs to be traversed bottom up (figure out what needs to be started to start the targeted set of services), then top down (start dependees when dependencies are ready). With Upstart, top down was enough. Or so it seems.
And yet, it still tries to use pidfiles (which systemd has a better alternative for), and notes "We really should check if the service is already going" which they don't do because it's an actual non-trivial issue if you're writing a standalone init script.
And yes pidfiles are crap and I don't like them either (a genuine criticism of sysvinit and advantage of systemd) but systemd isn't the first init system to do away with pidfiles so it's hardly a unique strength.
> Does anyone use these init scripts?
By default every popular distro before systemd/upstart did. Many services still use them anyways (systemd just generates a shim to run the start/stop/status operations on them)
Honestly I wouldn't be surprised if much of the negativity surrounding systemd (other than people who don't like the architectural approach) is from not being able to use a clean startup configuration, instead having to use a bolted-together compilation of startup methods under the systemd "brand".
It's been a while since I've confirmed this with a new install, but while my Arch Linux setup is very clean, Ubuntu has always been a mess. It still has compatibility with the old "service" command (from Upstart I think?) for goodness sake. My Ubuntu install still has a ton of scripts running under the systemd shim, while as far as I can tell everything that is running on Arch has a .service file. If you're going to appreciate the value of systemd, the latter is a much better experience.
https://github.com/openbsd/src/blob/master/etc/rc.d/sshd
On an archlinux system running systemd I see that the equivalent of the rc_reload function is "ExecReload=/bin/kill -HUP $MAINPID" (which is actually less safe...)
I'm not sure if you think that's declarative but to me it's basically as declarative as you can make a shell script.
> By default every popular distro before systemd/upstart did. Many services still use them anyways (systemd just generates a shim to run the start/stop/status operations on them)
I mean is anyone CURRENTLY using them? Because as far as I'm aware the linux sysvinit situation was a gigantic mess which nobody wanted to bother fixing despite the fact that OpenBSD and other BSDs had already fixed it by that point.
I think most linux distros can be taken as an example of how not to do things if you want to make your life easier.
Nowadays BSDs don't use these awful scripts. OpenRC based systems don't use these awful scripts. Who is using them?
RedHat and friends, Debian all used convoluted init scripts which somewhat validates the need for systemd.
However, it's easier to deploy and keep track of your own services, sudo and chdir are built in, there's a watch dog, easy journaling. That's nothing to sneeze at.
As far as the watchdog but things go there’s plenty of good enough solutions that don’t really on systemd: daemonize, DJB’s daemontools, etc.
But that’s a completely fine solution, service files are not meant for declaring arbitrary logic.
* Want logs since boot? journalctl -b. The first boot is -b 0, the previous boot is -b -1.
* Want logs for smartd? journalctl -u smartd
* Want warnings and above? journalctl -p warning
* Want logs since yesterday? journalctl -S yesterday
* Want logs on April 22 between 10 and 11? journalctl -S "2021-04-22 10:00" -U "2021-04-22 11:00"
* Want logs with microsecond timestamps? journalctl -o short-precise
* Or in JSON for parsing? journalctl -o json
* Want tail -f? journalctl -f
* Want only kernel logs? journalctl -k or journalctl --dmesg
* Want to filter the output right away? journalctl --grep=some_regex
* You booted a live environment, but want to look at the installed OS's journal: journalctl --root=/path/to/mounted_root or journalctl --directory=/path/to/journal
* Want to look at the logs from a container you booted using systemd-nspawn (and friends)? Just use journalctl --machine=container_id
* What time intervals do the boot indices (for -b) correspond to? Look no further than journalctl --list-boots
I'd say journalctl really has a lot of generally useful flags.
I don't know what "boot indices" are, but your other commands with plaintext are as follows:
dmesg or cat /var/log/whatever
grep <pattern> /var/log/whatever
cat /mnt/host/var/log/whatever
cat /mnt/container/var/log/whatever
No, because it only matches the message field, so you're not grepping anything extraneous like the timestamp or service name. This is also convenient if you want JSON output, which includes a bunch of extra stuff | grep might match.
The performance argument makes no sense, because if it's slow, then --grep will also be. Any slowness is likely before things get to that point.
> Why special flags to load different categories of log? (A: because all the logs are mixed together in a big blob).
And that's a good thing, because it means you can mix everything together and see the interactions between the different parts of the system in the order they happened. And this can be achieved by logging everything only once, rather than having duplicated logs for such an use case.
> (A: because journalctl throws away the logfile metaphor).
That's also a good thing IMO because I don't really care about the details of log storage. Whether a log is named this or that, and is compressed or not, and if the data I need is in .log or .log.1 or .log.4.gz is all a distraction from my actual need -- seeing a particular set of data. The fact that I can just get the log files from Apache from a given date, regardless of what file that happens to be stored in is a good thing.
> I don't know what "boot indices" are, but your other commands with plaintext are as follows:
Per-boot log classification. You can easily ask for the logs since the last time the machine booted, or what happened during the previous boot, and so on. And the list shows timestamps, so you can tell at a glance that the machine last rebooted today, and has been rebooting every 3 hours as of late.
This is very useful if the machine say, crashed and rebooted at some point. So you can just ask for journalctl -b -1, which will give you the log of the previous boot until the crash. You don't have to do any grepping to figure out where is that in the logs, because the system can trivially give you that.
If it is per service, will you grep the service name? It will have many false positive lines. Raw data is not too useful.
[1]: https://www.hashicorp.com/resources/systemd-the-good-parts
Direct quote from the website.
Eh, no. There is already a short list of viable non-systemd candidates. We.. well.. I don't need less. It already feels like I am transgressing some law by typing this on PopOS.
Meanwhile others claim that systemd makes docker redundant anyway[1].
Luckily that won't happen as systemd doesn't support musl which Alpine uses for libc.
I didn’t see this discussed more after it was asked, but I could use some pointers on how to answer this question since I have a service I wrote that I think should start on reboot, but has to be started manually. It would be nice to know when I have fixed it without having to reboot after each change to just to test. Anyone know how I can do that?
Compared to sysv-init this is a regression, because init scripts usually ran sequentially and also didn't involve unreliable systemd-udev events.
Compared to Solaris SMF this is pitiful. SMF takes into account service dependencies from its declarative service descriptions and computes a deterministic sequential startup order. If it has booted once, it will always boot again, in the same order, reliably. Best of all worlds.
I find this really odd because we spent decades with that exact class of problems on SysV init, in part due to the primitive dependency model and the lack of standard mechanisms to launch daemons or react to events meant so many different reinventions of that particular wheel (need Apache to wait for NFS mounts? Here’s another shell kludge to maintain!).
Switching to systemd gave a single, simple answer for all of that: use the init system. I don’t know what services you manage but for the mostly Debian/RHEL environments I’ve worked on your description more fits the before state than after in my experience.
2. You could use systemd’s health check or pre-exec to handle prerequisite checks like this if you can’t use the dependency mechanism for some reason (there’s no reason why you couldn’t have a custom type=oneshot service set as a requirement for Apache).
There’s nothing “evil” about scripting like this - the thing which makes me tired of SysV is the limited standard feature set (daemonization, logging, restarts, dependencies, event handling, etc.) and challenges coordinating across distributions for things like overrides. The wheels fall off shell scripting when you need to deal with that much complexity and do so in a portable manner with a bunch of shell code you don’t control - a health check is far less tedious to deal with, especially since you have a ton of functionality in systemd which avoids the need to handle the hard parts.
It would be nice to have hard errors when dependencies are not explicitly declared. For example, if you don’t declare dependency on the network, your service gets put in a sandbox where it can’t access the network at all. In practice, that kind of sandboxing sounds difficult.
My personal experience is that it is fairly straightforward to debug races in startup. Some service fails, you look in the log and see that it has a dependency that hasn’t started, and then add that missing dependency to the service’s declaration. Just speaking from personal experience—this doesn’t come up especially often and doesn’t require a large amount of time to fix, and the benefits of fast startup save you much more time than the extra debugging costs you.
This goes both ways - compared to sysv-init we finally get a good way to manage services in response to device changes.
Or not if one of them fails suddenly. That’s just a false sense of stability (not saying it could not be stable, but that is not due to having a sequential startup)
Services should have proper dependency chains and that’s not that hard of a problem to overcome. Without them every other part of service management will be incorrect.
> By listing that directory you can see what services load on boot.
Loaded: loaded ... ; enabled ...
I'm looking forward to the successor of systemd, or at a minimum, them rolling out config syntax 2.0 which breaks compatibility in an effort to make stuff more sanely named and documented.
I know that would cause huge churn. I know it will never happen. But I can dream, because I don't think anyone can look at how systemd configs define dependencies and say that they both understand the nuances and that those directives are are named sanely in a way that normal people can understand, and that's indicative of a lot of stuff with systemd.
Even though it was probably designed by some of the most knowledgeable people WRT init systems, it grew organically as they discovered new things they needed, and it needed a lot (which I think just goes to show how little was really understood by anyone about how to tightly couple these things).
Now ask me about pulseaudio and there's a project with nothing but issues, looking forward to the new pipewire overlords.
Don’t forget Bluetooth, WiFi, and graphics (cough Nvidia). Or any peripheral, for that matter.
Until vendors stop forcing us to play games with their broken blobs, Linux will forever be just a grand experiment for frustrated hobbyists and unfortunate professionals around the globe. A digital tragedy of the commons.
Linux is the most widely used kernel in the entire world. If you consider 'Linux' to be the OS, it is #1. If you consider all distributions that use the Linux kernel, it is also #1, collectively and individually (Android.)
If you look at the ranked list of Supercomputers, Linux has the entire top 500.
What the hell are you talking about?
Which, worth noting, does its own thing for init & services ( https://cs.android.com/android/platform/superproject/+/maste... ).
I think it's fair to assume they were talking about Linux as a desktop OS, which for some reason absolutely refuses to learn lessons from the other 99.9% of the market. And is also infatuated with internal fragmentation despite its tiny usage. That makes it really fun as a hobby for sure, and powerfully flexible for datacenter usages (from the mundane to the supercomputers). But also pretty terrible as a "I just want my laptop to work reliably & well" OS.
Which I had for the best part of 2 decades. Until systemd came along, infected pretty much every mainsteram distribution, and broke things that have always just worked like DNS.
Instead people decided "ooh mac is shiny, lets copy that". If I wanted OSX, I'd use a mac. If I wanted windows, I'd use windows.
Actually Apple has its own launchd service management system, and that one doesn't try to handle stuff like logging or dns. Apple changed the licensing of launchd to Apache license 2.0, but no linux distribution has adopted it. Ubuntu checked it out, but didn't use it because launchd used a more restrictive license at the time. Interesting that the license change didn't trigger a wider adoption of launchd. https://en.wikipedia.org/wiki/Launchd#History
in a way this would disprove the assertion that everyone has to use systemd, because of dependency creep.
But recently, when I changed my fstab file to mount a drive at boot, I made a typo. My system wouldn't boot at all and it wouldn't even drop to a shell so I can chroot the file system like you would with init.d.
I thought that there's some arcane command I don't know about. But there wasn't. You're locked off, and expected to boot off something else to fix your system. Horrible.
Having this said, I honestly expected way more on the section about benefits to users.
I don't know exactly what went wrong in your case, but killing Unix systems by messing with fstab has been a rite of passage for more than four decades. Systemd certainly didn't invent that. Hell, I broke a chromebook not 48 hours ago messing with the boot setup.
But FWIW: managing a recovery image is, amusingly, not historically a job systemd has tried to take on. This is what your live image is for.
Otherwise, edit the kernel commandline to add "systemd.unit=emergency.target" to the end, which triggers systemd to straight boot into this console without trying anything that it doesn't need to bring up the console.
I also could not find a way to NOT combine split horizon DNS with network interfaces (like most documents online only include scenarios like resolving corporate domain names using a VPN interface but that wasn't my case). I have multiple VPN connections to the same network (runs BGP basically) and I did not want the split DNS to bind to some specified interfaces.
Moreover, systemd-resolved seems not able to do split DNS without systemd-networkd. It needs systemd-networkd managed interfaces to do that. What? I just want separate DNS servers for separate domains. My WireGuard interfaces are managed by wg-quick. Correct me if it is possible to use systemd-resolved standalone in this case.
The hard systemd-resolved dependency of systemd-networkd also had issues. Sometimes, I just want systemd-networkd to configure resolv.conf using DHCP DNS but it looks like I HAD TO START systemd-resolved AND LINK /run/systemd/stub-resolv.conf to /etc/resolv.conf in order to achieve that (correct me if there's a way to not use resolved completely). Excuse me?
systemd-networkd itself was buggy, too. I once encountered https://github.com/systemd/systemd/issues/18108 and my friends reported they faced this issue as well. I don't mean that software should not have bugs but encounter them is really inconvenient.
In conclusion, I mean, systemd components seem to fit together and help you do a lot of things, but they did't do the things well, compared to separate programs for their specified tasks. Moreover, you have to deal with various components of systemd and figure out their relationships in order to achieve some specified tasks. For split DNS, I'm using dnsmasq now and it's working pretty well.
And the great things is, that declarative format means that systemd can actually be replaced by something else, as one just needs to be able to parse the ini-style configs and execute the ExecStart and ExecPreStart in order.
One really cannot do that to the other init-systems, AFAICT, they are far too scripted and rely a lot on side effects to do anything like that, besides packager choosing different solutions for the same problem. Some KI-paper could possibly take a shot at parsing out some sensible information reusable for another tool ;-)
I can get the dissentient about systemd incorporating so much stuff, but just from POV of it as init-system it is really nicer to work with, speaking as user and as someone which job includes packaging software for a Debian based distro.
it was robust, simple and straightforward.
however it was built around bash. which was large and slow to start up,compared to e.g. dash as scripting shell. and dash was not widely around yet. so each script had the cost of bash startup, the milliseconds of which added up.
and at some point the tipping point was reached.
Yeah, I know, but in practice this did not really work out and was a mess. In fact, every update was scary to do, in that time I frequently just re-installed the distro on my private workstation, simpler than to deal with the rc stuff. A plain setup may have worked, but anything slightly complex was just bonkers to do.
> it was robust, simple and straightforward.
Do you package for a relevant distro or is this just your experience as user? Asking because I just personally do not know any maintainers of software with a slightly complexer system interaction which would agree with that. There surely are some, but FWICT on some ML's anti/pro systemd flames it's <1% of maintainers - and from them it's actually almost never the new unit file format what is the issue, but the way systemd eats up basic system services.
IMO the systemd unit files are exactly what you say "robust, simple and straightforward", the implementation may not be, I never even touched that, but the unit file schema is just better, as it's not bash/dash with some slight optional possibility to use some hacks like comments (urgh) as schema, but actually provides a standardized format for doing that and more.
As someone maintaining dozens of packages and as common user, I'm happy that I do not need to mess around and fight the init-system, and I'm not the only one thinking so[0].
[0]: https://www.reddit.com/r/archlinux/comments/4lzxs3/why_did_a...
Thanks for the constructive downvote though ;-)
the dependency comments were standardized, and the parser would give errors for incorrect comments.
and of course for the RC scripts, great powers came with great responsibility --- for the package maintainer.
I recently had to deal with the limited feature set of journald logrotation, compared to logrotate and syslogd. without going into the details, just compare the man pages, if you hit the limits of the available declarative feature set, you find yourself patching or doing fancy work arounds.
if the features that come with systems are sufficient, it's a happy switch. if some corner case is missing, good luck.
ps: I did not downvote. I don't do that for a healthy and interesting debate. that's not what we do on here, no?
Cool
.---------. .---------.
| SystemD |------>| Service |
'---------' '---------'
Not Cool
.---------. .---------.
| SystemD |<------| Service |
'---------' '---------'It's not great if the service depends on SystemD, because then it can't be used with other init systems (without patching / forking, at least).
Things that are good: simple syntax, easy to use utils
Hidden gems: cgroups, file systems containment, etc. Systemd could completely eliminate the need for stuff like docker by totally isolating a process.
No, writing stuff to a file was not fine. It's only fine if all you do is to 'tail' and 'grep' it, but any time you want to do something more advanced, it's a bloody mess.
Why parse a text date, when it started its life as an unix timestamp? Why write a bunch of regular expressions trying to get basic data such as an HTTP response code, when it could be a field in the log that you could just search for? Why deal with parsing while the log may rotate or truncate under you, when the log system could be making your life easier?
I wrote a whole bunch of log parsing stuff in my day, and so I have no nostalgia whatsoever regarding this "just write stuff to a file" approach.
Amazing for you maybe. If you have a slower spinning HDD journalctl -eu <service> takes 10-30 seconds (not an exaggeration). Meanwhile cat or tail -f of a log file is almost instant somehow.
> Why write a bunch of regular expressions trying to get basic data such as an HTTP response code, when it could be a field in the log
I mean, this is a problem which doesn't require systemd to solve, you just need a better log format. Someone could have standardised a line based logging format with fields and then someone else could have written libraries to write it and libraries to parse it.
> Why deal with parsing while the log may rotate or truncate under you
That's a log daemon problem that I haven't found to exist with any log daemon I have used.
That's because you're asking for a log of a service since logging began, which may well have been months worth of logging. You're saying "give me everything that happened ever to this service in a pager, then scroll down to the bottom".
If you want tail -f, use a -f: 'journalctl -f -u <service>'. That's going to be a lot faster. You can also use '-b' to make it return a log of what happened since boot (worth looking into the manual for the details of that one), or specify a start date with -S, like '-S yesterday'.
> Someone could have standardised a line based logging format
And what were they waiting for? In the end, systemd did it, but it's a pretty recent invention. There was plenty time to get there first.
> That's a log daemon problem that I haven't found to exist with any log daemon I have used.
Because log daemons classically don't care about this at all -- it's a problem for whatever reads the files that the log daemon generates afterwards.
journalctl -fu is just as slow. I don't keep months of logs and the amount of logs I do keep easily should fit within a minimal amount of space. Somehow systemd manages to screw this up completely and I really don't think this has anything to do with user error.
> And what were they waiting for? In the end, systemd did it, but it's a pretty recent invention. There was plenty time to get there first.
I don't know what people were waiting for. But I think what systemd did could have been done in a more transparent and decoupled fashion that didn't involve binary logs. Just because they did something right doesn't mean that the way they went about achieving it isn't inherently flawed. I get that logs suck but fixing that shouldn't require me to change my entire system management stack.
> Because log daemons classically don't care about this at all -- it's a problem for whatever reads the files that the log daemon generates afterwards.
The log daemon I use (svlogd) first closes the current file, renames it and then opens a new empty current file. If it finds that there's now too many old logfiles it deletes the oldest one. I don't see how this could possibly break any tool which is reading the file except for cutting the stream short when tailing during a rotation. Moreover I don't think that's a problem since if you need to do some kind of processing of the logs then you should just put that in front of or after the instance of svlogd in the pipeline (svlogd can be asked to forward logs to stderr which then means you can chain it with something else).
Haha, I wish. Then that file becomes too large. So you want to truncate the start. However, removing bytes from the start of an open file is a bad idea. So you open a new log file every day instead. You really want to compress the old log files too, to save space. So now you can no longer just grep the logs as before. You're now also duplicating code and configuration for every daemon. So you write a daemon called `logrotate` that does this centrally instead. Into which you then copy the log paths configured for every program on your system and how long the logs should get kept. However programs don't much like it when you move the log files out from under them. So you patch every program respond to a signal by checking whether it's log files have been moved. Which then breaks for multi-process daemons. You come up with a whole buffet of ugly hacks to make that work most of the time.
Then you install a program which doesn't come with a logrotate config. Then you decide you'd really want a more structured way to analyze logs than writing fragile regexes. Then you want to run something as a non-root user. Then you have an ephermeral batch job. Then you want to ship off all of your logs to another location and really don't want to duplicate the log paths yet again. Then you want to see what happened just before the server crashed last boot. Then you run two of a daemon and give them different log paths. Then you want to look at your kernel and application logs together. Then you want to delete just last month's logs to free up some space. Also wouldn't it be handy if there was some kind of index to make things faster? ...
A digression, but I keep wondering why filesystems never optimized this scenario. Being able to trim the start of the file is really useful in certain cases, and without support options are pretty bad.
It also seems fairly easy to support at the cost of a few bytes in the file entry (a start-of-data index into first sector), and resizing a file by seeking past end is already a thing.
I must be missing something...
Collection is just the first part of a log system though of course.
> Systemd could completely eliminate the need for stuff like docker by totally isolating a process.
It does: nspawn is much leaner than docker and it's available on all Linux systems with SystemD.
This literally made me laugh out loud.
Over the years I have historically had several production system outages due to journald corrupting its own data store, being unable to log, and services hanging due to being unable to log (process stdout buffer full, journald not reading it, so process stuck trying to write output).
It certainly has improved over the years, and it has been a long time since this issue has happened (on one of the systems I manage at least), but I have accrued some pretty deep seated feelings about it at this point due to historical pain.
Btw, you still can do that nothing is stopping you, but I don't think each and every service needs to decide (and let user configure) where to put its logs, how to separate them (maybe by errors, maybe by users/domains) and how to rotate them.
I'm still not a big fan of systemd but it does make some things easier (but why wouldn't they think for a moment how easy "systemctl restoptart foo" will be to type, tab complete and issue different commands one after another)
Really, though, if I care about isolation and don't want to use docker, I'm gonna reach for FreeBSD + jails anyway, and I don't think systemd's gonna change that for me any time soon. If I want containers with better tooling outside the target OS, I'll use docker. Systemd falls in an awkward middle that I don't have much interest in.
But that's exactly what happens with docker anyway (e.g.: Docker is still running in the Linux VM when running it from a different operating system). And considering the widespread systemd adoption throughout the Linux ecosystem, it's far more likely for systemd to be already installed in the VM than docker anyway.
Systemd docker-replacement on a non-systemd Linux would need to run in a VM, though.
Systemd-docker-replacement: works only on systemd-Linux. Tooling for any other arrangement, including non-systemd linux, left as an exercise to the user.
Docker: works on Linux—systemd or otherwise. Has tooling to making working on macOS or Windows non-terrible and lets you use (mostly) the same commands as you would on Linux.
I'm saying that cross-platform tooling is what makes Docker Docker. Systemd can't replace it without replacing the tooling, even if it can do the same thing only on systemd-Linux.
Would it? There's no reason this hypothetical systemd-docker-replacement couldn't be architected in a way that would not have a runtime dependency on systemd-the-init-system.
In any case, I'm not even sure if systemd wants to be a docker replacement (although it does seem to pick up more and more container features lately). But there's definitely some overlap between the two projects (in particular around process/service management) but podman is a much more direct competitor to docker than systemd.
_FD: I’m not a large supporter of SystemD after using it when it was first introduced in Fedora and watching it take over the ecosystem, I do use it and as a sysadmin/SRE type I’m heavily versed in it and how it functions, so, everything with a pinch of salt._
I gave the video a good shot, half the time is spent going over some systemd-ecosystem basics;
The author then makes the case that Alpine is a less attractive development/production-deployment platform due to lack of this ecosystem (though they make effort to avoid directly saying that systemd would be a good solution itself.)
I couldn’t disagree more honestly. I love the concept of Alpine: as small as possible and no smaller.
The attention to minimise dependencies makes it the distro of choice in containers; I’ve even considered (strongly) running it on some machines where systemd would have been a lot of needless complexity.
I think deviating from this path for alpine is a grave error, there will always be a place for a niche system that is small; especially when it’s commonly paired with some other log managment/service orchestrator like Alpine is.
Additionally, it gives people like me an option to run without init systems that could be considered heavy and intrusive with heavy dependency chains.
So, kindly, Alpine: please don’t.
https://suckless.org/sucks/systemd/
I guess it's a bit of a watershed
You're being highly offensive towards these developers for no good reason. All kinds of grand claims, just because things are done on a way that you don't agree with.
You've stated that systemd developers are wanting to force people to eat things. And their action is to actually force it into someone's mouth, likely requiring the person to be put into restraints to make this possible.
You're being highly offensive. Systemd has existed for loads of years, yet this accusation continuous to be repeated. Aren't you aware of German libel laws? If I'd accuse you of such behaviour, wouldn't you get upset? Why say such things, things that suggest criminal behaviour, so easily?
These things are repeated so often that it isn't even moderated upon. It's odd that this behaviour is tolerated.
Those questions the page raises are based on what systemd tells you, not what I'd be askiny myself. I wouldn't want to know most of those, nor do they actually answer if "DNS is down" or not (what does it even mean).
From my computer: dig www.mysite.com -- checks DNS is available externally. Check it at a few sites (dig www.mysite.com @1.1, @8.8.8.8, @9.9.9.9 etc). My monitoring checks my website every 10 minutes anyway from multiple locations across the world and tells me if something is wrong, so it won't be that.
If I think the problem is that DNS lookups are failing on the server (so it can't see db1.mysite.com or whatever), something I might suspect if it takes a couple of seconds to log in as the reverse dns times out, I'd just log in and type
$ dig www.google.com -- checks DNS lookups are working internally
$ cat /etc/resolv.conf
The easiest way in a systemd world to do that is
$ tcpdump -i any udp port 53 -nn
In systemd that's no longer the case, another daemon is running on my machine, another daemon to go wrong, another daemon to hide what's actually happening, another daemon to wake me up at 4am because it's not working.
I type systemd-resolve --status
and get 82 lines of nonsense (on my desktop), with perhaps 1 line perhaps telling my the 'current' server configured, and a few lines telling me other servers. But then there's different servers on different links, so it's all a bit of a guess, hence tcpdump is far more reliable to see where lookups are actually going.
I have a small monitoring PC which runs a bunch of VLC sessions and webpages. It broke a couple of months ago because of systemd hijacking DNS. That means spending time looking in to why it's broken, which is a waste of time, or just having a script to fix it on bootup. The script is an ugly hack, but systemd is draining. It gets in my way, I never asked for it, but practically every distro forces its use in various (and constantly changing) ways. That's why some people get pissed off by it, it doesn't just add features you can use, it fundamentally changes the core of your machine without even buying you dinner first.
(Yeah, I'm a dinosaur who remembers dead tree IBM manuals. Sue me.)
As such, my opinion on systemd is the old ways weren't all that great and I'm glad the youth are trying to improve things.
For me there are 3 things that SystemD gets right:
- no shell scripts for service configurations
- no forking services (no need for daemon mode anymore)
- logging and service management is tightly coupled (this is arguably good with some bad parts, required CPU for logging with systemd is not great)
And the rest is just pure madness.
- DNS (but whyt?)
- NTP (but why????)
- system limits (seriously, da f???)
- pid 1 should be a very simple service (More: https://ewontfix.com/14/)
I have spent 25 years on Unix, so I guess I understand the concepts.
Other than "sysvinit based init had horrific shell scripts with 100 lines of repeated nonsense", given that almost every alternative to systemd which still uses shell scripts needs around 2 lines of shell per service (including the shebang) what exactly is the issue with shell scripts (if all they do is run a wrapper which sets up the environment of a program or the program itself)?
> - no forking services
When I used archlinux these were extremely common and I used archlinux as recently as last year.
> - logging and service management is tightly coupled (this is arguably good with some bad parts, required CPU for logging with systemd is not great)
So runit handles spawning a log daemon before the service is started to ensure all log entries are kept and comes with a really simple yet extremely powerful log daemon (but obviously you can use almost anything else including cat if you want). It manages to integrate logging tightly into service supervision while being maximally flexible and decoupled. What do you think about this?
If there is only one thing that I could keep from systemd, it would be the use of unit files instead of shell scripts.
The rest is your observations, nothing to disagree or agree with.
... Does systemd?
I still have endless problems with unreliable service startup and I've read all the documentation, have done all the debugging and asked all the people I could find.
> I also spent many years in the land mines of implicit overrides in shell script hells and thank you but no.
I'm not sure what you mean by this.
> Same thing in init scripts. Did you implement reload or restart? Is stop actually doing something or a noop.
So this is linux sysvinit style init scripts, OpenRC doesn't have this issue (as far as I can tell) and definitely I've never had to ask myself any of these questions when using daemontools or runit. Maybe you should familiarize yourself with how some modern non-linux-sysvinit inits work because they're certainly extremely functional, reliable and clear.
> If there is only one thing that I could keep from systemd, it would be the use of unit files instead of shell scripts.
I've actually considered this and I think it would be incredibly trivial to make a unit file parser which you can call from a shebang which would do this kind of thing. You could even have it fork off the parser to run it unprivileged. I think I'll try writing it sometime soon. You could even integrate all the namespacing, cgroups and lots of the other features systemd provides into it.
multilog is the simplest (smallest) and most versatile log rotating daemon I have found.
I'm 31 now.
Usability: journald requires administrators to learn a new command to access their journals - everytime I have to use it I need to look up the syntax again. Compare with logs in /var/log and you can use all the standard unix tooling (text editors, tail, grep etc).
Reliability: in my use journald has made my servers more unreliable. Two CentOS 7 VMs, setup at the same time. On one journald works just fine, on the other approximately once a month journald would stop logging. It wouldn't start logging again until you noticed and rebooted the server. The real issue this exposes is that on a systemd machine, journald doesn't just log for itself, it also supplies the logs to syslog. So on this server when journald broke, there were no logs whatsoever.
This issue was apparently common on the version of systemd shipped with CentOS 7. The fix was to disable log compression in journald. What it highlighted to me was the inherent issues in an all encompassing system controller like systemd, in that if there is a bug somewhere in there you lose not only the added bells-and-whistles its intended to provide (journald in this case) but also the old previously reliable functionality (syslog).
In my mind the fix for this is a redesign of systemd to make it an optional layer on top of the reliable functionality rather than a low level system component that everything else needs to depend on. In the case of logs journald should consume logs from syslog, not provide them to syslog.
A more accurate description would be "Systemd is a set of building blocks that you can use to make Linux more like Windows".
As far as I can tell the only "windows-y" thing is binary logs and even then you got all the tools to get a plaintext stream to process. Corruption of the logs can be a problem but it's nothing compared to...dunno, I assume people are drawing parallels to that and Windows' binary Registry, but then again I don't think I have had a Registry-related failure in decades. Besides, if anything, dconf is much more Windows than anything systemd could do, but I don't see anywhere as much vitriol for it.
I'm thoroughly confused about this small war with systemd. I missed all the early drama so I barely know what it's about, but in practice systemd has never failed me.
What I want is a solid system that does what it's supposed to -- start stuff -- in a maximally boring fashion.
While I prefer Linux in general, Windows does have some good ideas in some places, and I don't see any issue with using them.
systemd was in fact heavily inspired by MacOS's launchd. At worst, it can be said that complex boot managers will begin to converge on a design that actually works ~ MacOS did, Windows did, and now Linux has with systemd. The three have plenty of differences, though systemd is still closer to launchd than not.
And in actuality, it's very tweakable. Did you notice that the source code is open, for a start...?