A Caddy Cert Expired Because Systemd-Resolved Was Selectively Broken | Hacker News Reader

A Caddy Cert Expired Because Systemd-Resolved Was Selectively Broken

rant.mvh.dev

41 pointsPaulHoule33 comments

What's the server's hostname? If it's the same as the domain that's not working, perhaps you're running into this bug [0]? If that's the case, then adding

  # /etc/systemd/system/systemd-resolved.service.d/override.conf
  [Service]
  Environment=SYSTEMD_RESOLVED_SYNTHESIZE_HOSTNAME=0

might fix it.

[0]: https://github.com/systemd/systemd/issues/34897#issuecomment...

There's this adage about consuming whatever newspaper reporting, everything seems fine and dandy, until one day they report on something in your field. Say, chemistry, and you're a chemist, and... it's all wrong. Even the basics.

My field is networking, roughly from Ethernet physicals to TCP/UDP.

systemd has no f*cking clue what they're doing on networking. You need to not use systemd-resolved, and not use systemd-networkd or systemd-timesyncd either.

I really hope they are service manager people and know how to write a service manager, and I have no counterindications on that. But don't let them touch your networking, aside from service-managing on that.

(My recommendation for DNS - on servers - would be to install unbound locally and use that. It's not great for clients since it doesn't deal well with frequently changing network connectivity, or rather, it does, but only if you restart it on network attachment change, which implies flushing all cached data.)

> systemd has no f*cking clue what they're doing on networking. You need to not use systemd-resolved, and not use systemd-networkd or systemd-timesyncd either.

Can you give some more details, or do you have some examples of where systemd is bad here? Because I'm quite happy with systemd-resolved and systemd-networkd, but I admittedly know hardly anything about networking.

> [...] But don't let them touch your networking, aside from service-managing on that.

What do you recommend as an alternative then? systemd-resolved and systemd-timesyncd are easy to replace, since there is lots of good DNS/NTP software available, but systemd-networkd seems trickier: I don't like NetworkManager, I'd rather not write a bunch of ad-hoc shell scripts, and I'm not aware of any other alternatives.

> Can you give some more details, or do you have some examples of where systemd is bad here?

I can't, because it's death by a thousand cuts. I remember systemd-timesyncd having security issues, but not what exactly. systemd-resolved was initially missing basic security best practices (source port randomization, if I remember correctly), despite their being well established and well known in the DNS community - AFAIK the systemd people just didn't know of them.

> systemd-networkd seems trickier

Indeed; I was primarily speaking about servers, and there my answer is: either ifupdown-ng or ifupdown2. (The latter if your networking needs are more complex.)

On client devices the choice is between bad and worse. Personally speaking, I consider NetworkManager bad and systemd-networkd worse. So I'm using NetworkManager on my laptop... it still occasionally breaks IPv6 for me.

I will admit: this is burned-in experience for me, learned over several years, and the original reasons have expired from my brain's cache. And I probably didn't give systemd-networkd much of a chance (I don't remember anything on that, for resolved and timesyncd I have at least vague bits), with that being more recent (at least for me) than systemd-resolved and systemd-timesyncd.

Is it possible it has gotten better? Definitely. But TFA indicates at least systemd-resolved hasn't.

A lot of knowledgeable people hate systemd, and I start suspecting that they have good reasons for that. However, it's extremely hard to get a good list of related substantiated facts about systemd in order to convince a larger community. If you encounter something specific, please write some kind of article about it.

> in order to convince a larger community

Even if you would get such a list it would be dismissed anyway, because 'works on my machine'. Eg: see a sibling response to the OP.

You're right in that I should really be able to back my position up better. I don't know if/when a situation where I have to deal with the systemd network components will come up though :/

So, it's 12 days later and we upgraded Debian stable on our production DFZ BGP routers.

Result: systemd-networkd in a restart loop continuously consuming 100% CPU. None of the systemd restart-limiting options are being applied. The only message in the log is "Could not enumerate links: Connection timed out".

It's not a strong data point, but it's a data point. I did say "death by a thousand cuts". There's enough of these to sum up.

To add insult to injury, I'm relatively sure systemd-networkd was disabled before the update, but I can't easily confirm that.

> systemd-resolved was initially missing basic security best practices (source port randomization, if I remember correctly), despite their being well established and well known in the DNS community

https://lists.dns-oarc.net/pipermail/dns-operations/2016-Jun...

It was fixed in 2016. RFC5452 is 2009.

As the first paragraph states it's not a big problem for a local forwarder but all other bullet points are on the case.

hot take "predictable network names" should have been a kernel flag - give us all our eth0 in peace. i shouldn't need to set a flag to get back a default feature.

just what i needed to do: configure ens328452356aflhjdslhfda to get my network going.

ln -s /dev/null /etc/systemd/network/99-default.link Or net.ifnames=0 Take your pick...

net.ifnames=0 should be the default: if someone needs it to be the stupid ENS nonsense, they should be the ones forced to set it to 1 at the command line, not me.

> hot take "predictable network names" should have been a kernel flag - give us all our eth0 in peace. i shouldn't need to set a flag to get back a default feature.

No, because the feature being on by default does nothing but make your interface names longer, but being off can be the difference between a restarted server having or not having network access because of undocumented behaviour with it's firmware causing the network interfaces to come up in different orders.

Personally, I find the predictable network names feature infuriating 100% of the time. Changing the name of network interfaces made nothing better, and actively made things worse for systems with a single NIC (virtually none of the systems I use with a single nic use the same name). Network interface configurations could already be bound to MAC addresses rather than names, and that had been implemented even before the enpXXX style names came into being.

Persistent ethX names were far better to me, and we had those before enpXXX via udev. At least then when I logged into a random system with a single NIC it was called eth0. With 2 NICs they were eth0 and eth1. Simple to predict, no thinking required (unless the system already had bound those names to other MAC addresses).

Usability dies by a thousand cuts. Forcing needlessly complex behaviour onto people with simple use-cases is not an improvement.

< Persistent ethX names were far better to me

They weren't persistent, that was the problem.

eth0 is predictable: a given system will always have the same interface for the same physical hardware

enp0s31f6 is unpredictable

The whole operation is a lie!

If I ever have a networking problem in a personal Linux system, it gets solved by removing the weird networking thing that the distro decided to use and move to full systemd-resolved / networkd. In my experience, that works every time.

I'm not doubting commenter's expertise. It's entirely possible that resolved and networkd are utter horror inside but still work from at least this user's POV.

> There's this adage about consuming whatever newspaper reporting, everything seems fine and dandy, until one day they report on something in your field. Say, chemistry, and you're a chemist, and... it's all wrong. Even the basics.

Gell-Mann amnesia effect: https://en.wiktionary.org/wiki/Gell-Mann_Amnesia_effect

> systemd has no fcking clue what they're doing on networking. You need to not use systemd-resolved, and not use systemd-networkd or systemd-timesyncd either.

I'm not sure how to reconcile that with enjoying each of those:

systemd-resolved is the default on many distributions, and does DNS-over-TLS stub resolution for me. What do you suggest as an alternative that handles changing network connectivity, and DoT?

* systemd-networkd is very well documented/organised. I find it pretty useful on servers.

* systemd-timesyncd doesn't seem to cause issues for me. It being an SNTP client (rather than NTP) seems appropriate for distributions. I would like to move to NTS or Roughtime at some point.

> My recommendation for DNS - on servers - would be to install unbound locally and use that

And now your developers are running around and cursing you because nothing works anymore.

Because Docker silently retargets the interna Docker resolver to 8.8.8.8 if it sees 127.0.0.1 as a resolver address on the host.

Because people who wrote Docker have no fucking clue how the system works.

NB: see https://news.ycombinator.com/item?id=47441785 to solve the Docker issue with a local resolver.

> It's not great for clients since it doesn't deal well with frequently changing network connectivity

This is is something Linux specific I guess, I run Unbound locally on my Windows laptops for years and never had a problem which would require the Unbound restart.

> which implies flushing all cached data

It doesn't really matters in 2026. Just look in your cache and note the default TTLs for like 90% of records.

Some docker issues archaeology says that there was tension between docker team and redhat / systemd

That's probably why podman exists.

It is why Podman exists.

Yes?

RH wanted to control the stack and Docker was the thing what sidelined RH to a mere distro provider. And don't forget about RH dips into OpenStack at that time.

Running on 127.0.0.53 isn't a systemd-resolved exclusive magic trick, you can just configure unbound to do the same thing.

Not 127/8

You need to configure your resolver to anything but 127.0.0.0/8 to not to trigger Docker automatic "help".

Most of the time local resolver is bound to 127.0.0.1 just because it shouldn't be available from the network and because this is how it's always done.

The problem is what Docker actually silently break things because it's devs are too good to bother, the same thing you note about systemd devs.

Hell, it was easier to just provide internal Docker resolver on a some network Docker uses internally by default for the NAT shenanigans - but Docker devs 'knew better' and instead made it to point to a 3rd-party absolutely not related service which brokes local resolution on the containers.

Want to know a really cursed dirty little magic trick?

(as root:)

  ip route add table local local 8.8.8.8/32 dev lo
  echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
  cat <<EOF >/etc/unbound/unbound.conf.d/steal-8.8.8.8.conf
  server:
   interface: 8.8.8.8
  EOF
  service unbound restart

(before anyone asks: you don't just go add 8.8.8.8/32 as an address on lo because then that's a valid source address the system might use to send traffic out. Not having it as an actual address is the point of this trickery. Also this will work with most DNS caches, doesn't need to be unbound.)

And to be clear: I said "cursed dirty" and I mean "cursed dirty".

Yikes. Container resolving is still broken though.

Not if your containers are L3 routed through the host ;)

(Or did you mean resolving the names of the containers? That'd be handled through unbound-control local_data)

> My recommendation for DNS - on servers - would be to install unbound locally and use that.

At least on Ubuntu 26.04, you can't easily bind Unbound on a WireGuard interface because of services dependency order, so the Unbound service errors during system boot because the WireGuard interface doesn't exist yet. And IIRC neither `ip-transparent` nor `interface-automatic` fixed it.

On Alpine Linux all this just works.

There was interesting commentary on https://lobste.rs/s/z0ozbb/caddy_cert_expired_because_system...

Was there? I see 14 comments, of which only 3 have any value at all which are: fanf finding an irrelevant bug, oliverpool explaining the use of LetsEncrypt staging, and hexadecimal suggesting setting up expiration alerts.

The rest is:

  - People bitching about systemd and replies to their bitching totaling 7 posts (literally half the comments) 
  - conversational detritus like "well written blog post" and the whiteboard comment

Reply on news.ycombinator.com