I understand it’s not everyone’s cup of tea and have nothing against peoples personal decisions. It’s just the ship has sailed regarding systemd regarding that most people use it and it’s the industry standard.
In neither case was there any diagnostic information available (in the journal or daemon status) that would provide pointers to why it behaved as it did, and unlike with an external daemon manager it doesn't seem easily possible to forcibly debug what exactly it tries to do when performing an action (without debugging the entirety of systemd, which probably would require a second machine in physical proximity or at least running the problematic system in a VM?). In the end, I still have no solution to the former problem (which occurs occasionally) other than doing a hard reboot; for the latter, I gave up and made a service (using the hoster's virtual console service) under a different target that performed the necessary ifupdown calls to make the server reachable again upon reboot. Neither of these are things that I remember having to do on Linux before this "enterprise-quality" technology was rammed down my throat; it really feels much closer in UX terms to patching out bugs in system DLLs with a hex editor back in the days.
The claim was that systemd was compatible with sysV init scripts. This is not true, and the breakage might not even be noticed until you are dealing with data corruption. If you have a startup script that does an su to a different user, systemd will start the application, but it will just kill the application processes that were spawned after the su, rather than doing a proper shutdown. E.g., if you use the startup script (only sysV) Oracle provided, systemd will kill your DB's processes in an apparently random order rather than allowing a proper shutdown. The issue is in the way systemd uses cgroups to keep track of which running processes are associated with a particular startup script. Every version of systemd is affected.
Systemd initially claimed compatibility with fstab. But, it broke things. Systemd does not process the entries in fstab sequentially. This breaks fuse filesystem mounts that depend upon a backing store mount. Systemd later added additional mount options to try to hack around the breakage, but, in my experience with glusterfs, they are necessary, but not sufficient, and I had to add overrides for other systemd service units to get things to start reliably.
I also had a fun time cleaning up the mess after systemd made remote systems using full disk encryption, unbootable. The responses by Poettering in the bug report from the Debian systemd maintainer were what really convinced me this systemd thing was going to be a huge mess. TLDR Poettering basically said, he never used a feature like Debian's keyscripts, and wasn't willing to make the existing system work. Years later Debian has a hack that allows keyscripts to unlock disks in the initrd before systemd gets involved in the boot.
Another fun issue early on, on an embedded system that was using ext4 without a journal. The system experienced a hard power down. When it came back up, it prompted for a root password for the emergency (single user) shell. But, it wouldn't auth. It started echoing back parts of what were typed mixed in with garbage characters in its prompts (including the plain-text root password). Messing around in this state, I realized that it was executing (as root) whatever I typed as the password. So, for password, I typed fsck -f ... and was able to get the system bootable again. System was reverted to sysV, and everything worked properly again. Maybe this was a systemd-logind issue?
I use systemd since it "won", but it has not been a good experience (the above is a small subset of issues I've personally seen, but pretty representative of impact).
And then there's systemd-resolved…
They point against systemd would be resolved both being broken and being unable to replace it with another resolver.
They closed this super-critical bug despite not fixing it, claiming they're making the right decision in their implementation when they in fact misread the spec
https://github.com/systemd/systemd/issues/2514
Yeah, that's right, with resolved you cannot connect to other machines on your local network by specifying their simple names, because it blackholes those requests, and there's nothing you can do about it.
Huh? You most certainly can. I mean, somehow "ssh koyomi" works here on my network, no matter if machine I run this on uses system-resolved or unbound or something else. My DHCP server supplies options 119 and 135 and that's all that's necessary to make it work.
What you can't is to have a CNAME with a bare hostname i.e. make "foo.example.org" magically resolve to "foo.local" when you're on that particular LAN with "search local" (and $deity knows what if you're not). To be honest, I'm not even sure what's the point of this setup, unless you do split-horizon (but then why not respond with "foo.local." proper?)
Yes, I do have a "proper" domain name for my LAN and my .local is for mDNS (although I don't think I've ever used it with short names, always foo.local explicit; and I use mDNS very infrequently) - guess which is why I haven't ever seen this behavior. Thank you for clarifying.
Meanwhile, RFC4795 says that LLMNR senders should not send queries for single-label domains to DNS and that no search list should be applied to such. So querying a single-label domain does not seem likely to reach DNS on an LLMNR capable stub resolver.
My advice to one who wants to use DNS to resolve hostnames on the local network: Avoid using a domain reserved for mDNS domain and use another one.
And yes, I'm a little upset about it being broken by default and having to hunt down the issue, which did take quite a bit of time.