Hacker News Reader

Comment by Faaak | Hacker News Reader

I hate systemd-resolved because it uses glib's resolver.

The problem is that this resolver doesn't understand root queries. That means that `dig +trace` doesn't work anymore, and you have to use an external server for the root zone.

This is such a trivial use case not taken into account by systemd (they don't even want to fix it), that I'm saddened by it..

View on HN

1. I think you mean the "glibc" resolver, not the "glib" resolver (I do not believe that GLib contains a resolver).

2. systemd-resolved does not use glibc's resolver.

It is true that systemd-resolved does not understand root queries, and it is unfortunate that the systemd maintainers seem to be unwilling to add support https://github.com/systemd/systemd/issues/5897 . However, this has nothing to do with it using glibc's resolver.

However, as clearly documented in the systemd-resolved(8) man-page: There are 4 supported modes of operation with regard to /etc/resolv.conf, which each have different trade-offs, and are configured by changing /etc/resolve.conf itself:

(1) symlink to /run/systemd/resolve/stub-resolv.conf

(2) symlink to /usr/lib/systemd/resolv.conf

(3) symlink to /run/systemd/resolve/resolv.conf

(4) regular file

`dig +trace` should work properly in modes (3) and (4).

The big difference between those modes is "how do you want programs that don't use the system APIs (and instead do DNS themselves) to behave?". Modes (1) and (2) say "have them send their traffic through systemd-resolved", which as you note has some deficiencies; modes (3) and (4) mean "allow them to talk to the remote DNS servers directly".

Thank you for expressing the use cases so clearly. It is not easy to tell from the documentation.

Thanks for the info; I'll fix that on my machines.

Though I don't know why they would "break userspace" like that.

Probably because the main dev "knows better". Looking from the outside at common complaints across systemd, there seem to be plenty of weird defaults that hurt power users, with that being the primary reason behind not changing it to something more standard.

It's a situation where you can't do the right thing in all places.

Some programs, like statically linked Go programs, don't use the system APIs because they don't want to link against external libraries. So, if the system is configured to use mDNS for .local hostnames (either via systemd-resolved or via nss_mdns), those programs will mysteriously do the wrong thing, because they _only_ speak plain DNS. You can hack around that, and get them to do the right thing, by having a stub DNS server that they talk to, and have that stub server fire off mDNS requests instead of DNS requests as appropriate. Similar story for things like DNSSEC or DNS-over-TLS. That's what modes (1) and (2) do, they make statically-linked-programs that would otherwise use the system APIs do the right thing.

Other programs, like `dig`, don't use the system APIs because the really do care about speaking DNS directly. They just need the list of DNS servers to talk to and search suffixes to use. Giving them that is what modes (3) and (4) do.

It's literally impossible to keep both sets of programs "doing the right thing" at the same time. So the question is: which group of programs do you want to do the right thing? The systemd devs don't know (much less "know better"), so they made it configurable. Which one is default is up to the distro, not systemd.

Just to add, since I didn't speak about the differences between (1) and (2) or (3) and (4):

You have to use (4) if (and only if) you're going to use a resolvconf(8) implementation other than systemd-resolvconf. For example, I use dhcpcd; for kinda silly reasons dhcpcd and systemd-resolvconf don't work together, so I'm stuck using openresolv as my resolvconf(8) implementation. So I have to use (4). If you're fine using systemd-resolvconf, you should use one of the other 3 modes.

I'm honestly not sure why you would ever use (2) instead of (1). It's like (1), except search paths won't work right for programs that don't use the system APIs. (Maybe it existed before (1) did, and still exists for historical compatibility?)

I tried giving resolved a go, but kept seeing issues where it would suddenly start returning an error response (I wanna say servfail), would log a dnssec validation error, and refuse to work until the process was restarted.

I've since switched back to unbound with dnssec-triggerd and life is again glorious.

I had to give it up because it was crashing on every query when I had a local DNS server also configured.

systemd is such complete garbage that I immediately write off any distros that run on it as unusable. For all the fail of SysV init, at least it sticks to its own job and little else.

SysV init was so full of fail (race conditions, no dependency management, slow as hell, stupid metaphors -- run levels? really?) that people wrote Systemd. Anyone can expand on systemd to have it do stupid shit it shouldn't. But you can do that with any technology that is composable. It does not mean that the core of systemd isn't anything short of incredibly useful.

And yet Linux distros like Ubuntu around 9.0 were much more stable than these days.

With 16.04 I get core lockups (Intel microcode) and cannot switch out of X with Ctrl-Alt-F1.

NetworkManager no longer connects the mobile surfstick, so I do it manually now with pppd.

I can't claim that I had a specific problem with systemd, but I never had an issue with SysVinit either.

The overall trend however is that Linux distros accumulate crapware by RedHat's NIH developers. I'm not surprised that NetworkManager is also by RedHat.

I assumed the new network layer in ubuntu 18.04 (which replaced /etc/network/interfaces) was some pan-distro/redhat inspired solution.

Apparently not -- netplan is ubuntu originated. They don't do themselves any favours.

I was really disappointed by this myself. Netplan seems slick, but I don’t want yet another new, distributor-specific way to manage this stuff. Red Hat seems to be quite good at wrangling such tooling, but I don’t know of a solution by them/others as slick as the netplan config.

Ctrl+Alt+F1 switching is not possible unless you expose virtual terminals on the other virtual displays. Arch Linux at least does this by default, except that your first X session will start on the same display as your virtual terminal (which is good for security relative to the earlier practice of moving Xorg to VT7 while leaving an active shell on VT1).

For networking I suggest looking at ConnMan. It has a somewhat unintuitive UI, but it is lightweight, fast, and I have yet to find a wireless network that it will not let me connect to.

> and cannot switch out of X with Ctrl-Alt-F1.

You can't switch users within x either, it's been broken since at least 14.04 (according to an old close bug report I found). It works the first time but on subsequent tries you just get the lock screen.

The bug report was closed after several years of nothing.

Since some time my main Linux box won't continue booting to the graphical login manager unless I hit some random keys (any will do) on the keyboard. I guess there is some service asking interactively for something, but I can't see any prompt (maybe swallowed by systemd, who knows).

Since some time switching VTs resets all hardware accelerated applications (i.e. crashes them).

Since some time unlocking the graphical session is randomly broken and requires loginctl session-unlock from another VT (which then promptly triggers bug 2).

These things keep coming and going... it's not even Linux-specific. Stuff just breaks all the time and you shrug and move along, hoping that maybe someday it's fixed. For most of these I wouldn't even know who should receive a bug report. Maybe it's the graphical login, maybe the compositor, maybe mesa, maybe a kernel driver, maybe systemd, maybe maybe...

I have encountered this since 16.04.

On current Ubuntu releases gdm (login screen) now runs on VT1, which is what you are trying to switch to with Ctrl-Alt-F1. Your current session is likely on VT2 (Ctrl-Alt-F2)

You could try Ctrl-Alt-F3, F4, etc to get a login terminal.

I hated working with SysvInit but to be fair Debian's version of SysvInit had dependency support and parallelism and was therefore reasonably fast. The only real issue was how hard it was to write a correct script resulting in almost all of them having bugs in edge cases.

Not all SysvInit implementations were equal, which was an issue in itself.

So every distro except for Void and Duvian?

Yep.

It's never been a better time to go BSD instead.

MX Linux does a nice job with SysV, and Artix uses OpenRC. I've tried both and liked them, though Artix broke pretty hard once during an upgrade. But yeah, it was hard to find a Linux distro that runs well without systemd.

To name some big ones: Alpine, Gentoo, Slackware.

Gentoo!

It’s a great distro, gives you loads of choice (even lets you go with systemd) and the developers actually care about open source and the act of giving a shit (unlike what I see from the systemd bug tracker).

I installed gentoo in 2012.

Still compiling it, but I'll let you know how it goes

Did you make sure to triple compile the compiler for optimal speed?

If you only care about how fast the system will be, you only need to compile the compiler twice, not thrice. Probably that is his problem and why it's taken him 7 years so far. He should have compiled the compiler thrice...

For anyone wondering, when you install Gentoo, you untar a stage3.tar.gz into a / that you chrooted into. That tar file contains a precompiled environment and toolchain. From there most things get compiled.

For example "emerge x11-base/xorg-x11" would install X and related utilities, but you wouldn't do that. X is a dependency, what you really want is whatever desktop environment you like. So "emerge qtile" and the rest is handled. The compilation time thing isn't a big issue on any reasonable solid state hardware, and there are precompiled packages available for large popular projects (like office suites).

Install instructions: https://wiki.gentoo.org/wiki/Handbook:AMD64#

I wrote scripts that build from scratch to my custom desktop configuration in < 8 hours using Packer on a single i7. I'm also toying with the idea of a buildserver that creates precompiled binaries :)

10-Year Gentoo veteran here. Every time I move away from Gentoo I come back purely because I have some headache caused by systemd or pulseaudio (which seems to depend on / be depended on by systemd). If they didn't get in the way I wouldn't care, because not needing to compile packages would be nice.

Recently I purchased some bluetooth headphones, they don't work well without PulseAudio and so far I haven't gotten anything but raw audio files to playback directly.

Same goes for Firefox. I think there's an intermediate plugin to play directly to ALSA, but for now I'm just playing spotify||netflix via chrome.

Point is, it's slowly becoming harder to avoid Systemd and PulseAudio. Because all the major distros use them, the support, development and documentation for !systemd has died off radically :(

I used to have a poetteringware mask, but I had to let pulseaudio slide (but without udev and systemd (eudev works well)) because I couldn't find support for my non-standard setup anywhere for alsa.

Your point is super valid and it makes me sad, I donate/try to support as much as I can but the amount of developers that are apatheic to niche cases is increasing day by day.

There is apulse this which I used successfully in the past, it’s included in the portage tree. Since I happen to like systemd itself, I don’t have much of an issue with that. I used to avoid pulseaudio and networkmanager, but on my latest installation on a laptop I just included them and they are working fine for now. We’ll see how long they survive this time, esp NetworkManager is starting to bother me again ;)

12 years here. I finally let pulse back on my system a year or two ago when after enough upgrades (notably Firefox) my sound config went from perfectly fine for years to unreliable. At least pulse is better now. And more stable than the version I have to use on work machines (ubuntu). No dependencies on systemd required, though, that project will never be installed. OpenRC is great, and it's simple enough that it doesn't need any team of paid devs figuring out how to cram more features into it.

For Firefox, I recommend using sndio.

I'm surprised to hear of problems with Firefox and ALSA. On all my Gentoo machines I simple compile Firefox with USE="-pulseaudio" which sets --enable-alsa within Firefox's build. Still working fine up to and including Firefox 66 directly to ALSA.

My go to was PCLinuxOS, though I ended up going back to kubuntu later for other reasons. Really liked it, might go back someday.

Alpine is great, too.

Yup, it's nice but slow to boot for some reason (dhcp?). Which isn't that great when running VMs.

Systemd has serious issues that mean personally I don't ever use it, but I feel like most common distros pick it just so gnome behaves.

I'm still not sure about the complete garbage thing but sysV init definitely wasn't good enough.

busybox init if all you want are some ttys and fs and openRC are probably what you want.

Systemd means I never have to write or debug another init script never again. It gets multiple passes from me just on that.

I don’t follow this logic. I’ve written unit files and they can be very easy, but when you start looking at something like postgres on Debian and want to move it to another directory, then suddenly you’ll see all the hacks that have been employed to make it work.

Stuff like running /bin/true for the non-versioned unit, so that it can coerce the activation to take it sequentially when it actually executes the versioned unit.

There’s also the fact that “no boot is exactly the same” which means reproducing a race condition at boot is basically impossible if it’s anything other than the most trivial.

And anyway. The truth here is that you haven’t actually removed the shell script. You’ve just added boilerplate C-code around it.

If your biggest complaint is/was the composability of shell scripts for the common use-case then this is very much a solved problem in the BSDs (where an init script is usually around 5 lines).

EDIT: please come back with some concrete examples of why I'm wrong. because not only would I like to resolve my own issues; it is also the case that you shouldn't be using downvotes as a "I love systemd so I disagree" button.

And Debian has had start-stop-daemon for over 20 years and it deals with most init script faff.

I never had an issue with sysv, but there are several modern init systems such as runit and openrc that don't require selling your soul. Systemd extends tentacles throughout a system in ways that make me question how anyone ever thought it was a good idea.

There is Devuan for Debian based setups. For everyone claiming that SysV is bad, look at the Solaris SMF for something superior to systemd.

>For everyone, look at Solaris for something superior.

I'm still sad that such a good system fell in the hands of Oracle :(

I take it you don't work with linux professionally? I'd enjoy being a fly on the wall during the meeting where you convince your boss to run the company critical infrastructure on void linux.

Systemd is still optional in Debian, right?

Not really. But you can always install Devuan over it at least, which gives you a choice between sysvinit and openrc (and others like runit or sinit).

https://devuan.org/os/init-freedom/

The runit init Void Linux is quite slick. I also installed an Artix runit image in a virtual machine that works nicely. We are using it as a supervisor on openSuSE piggybacked onto systemd to run critical infrastructure. It works flawlessly, it's also included in busybox but almost always never enabled.

Kinda. There are increasing numbers of hard dependencies on systemd components. And systemd components pull in systemd itself more often than not. So while in theory it’s optional, in practice it seems decreasingly so.

I tend to avoid systemd so I’ve taken to running Debian machines without it. But if you’re doing something like ephemeral cloud installations then it’s not “worth” the overhead of swapping out your init (and having the consequential reboot and potential instability) so the default is a required target for automation anyway. And if you’ve already set a target then why have another one?

This is why defaults can be harmful if widely deployed and considered non-optional (as systemd is on Debian). They become enforced by convention because they are the only valid common target without wrangling.

https://lists.debian.org/debian-devel-announce/2014/08/msg00...

  For the record, the TC expects maintainers to continue to support
  the multiple available init systems in Debian.  That includes
  merging reasonable contributions, and not reverting existing
  support without a compelling reason.

CentOS 6 (and increasingly Oracle Unbreakable Linux 6) are alive and well in enterprise.

Reply on news.ycombinator.com