Fedora and Fallback DNS Servers
lwn.net
lwn.net
The fallback is only used when the network connection is otherwise configured and functional but no DNS servers have been defined and the entry isn't in the hosts file. The daemon respects DHCP and manually provided servers and even the hosts file before it tries a fallback.
Modern distros rely heavily on DNS because their package infrastructure is almost all load DNS load balanced groups of servers. So gone are the days where a public package mirror was just an IPv4 address and a base path, i.e. a singular server. A mirror may not have the same IP address day to day.
A distro disabling the fallback is stupid. Fedora should have at least provided some addresses for the fallback even if they didn't want to use Google.
As for the "hurr durr why does my init system have a DNS resolver?" type questions it's just painful to see that level of ignorance in what seem to be smart people. Systemd contains an init system but it's a project providing a number of system daemons (oh shit the name makes sense). It uses a bunch of useful modern Linux features.
I'm partial to the "hurr durr Unix philosophy!" arguments. They're particularly tone deaf on systems loaded with awk, sed, bash, perl, web browsers, m4, ssh, and emacs. The "Unix philosophy" is a decent idea but not a suicide pact.
If you hate systemd go use a distro without it, there's plenty. You can also use one of the BSDs. You've got options. But if you don't "trust" systemd or just hate it at least have an informed position about it. A dumb commit made without understanding or thinking about the consequences is on the distro maintainer, not the upstream project.
[0] https://wiki.archlinux.org/index.php/Systemd-resolved#Fallba...
I think that's unfair they had clearly put a lot of thought into it. I'd argue this is a very clear-cut case of letting perfect be the enemy of good and software engineers thinking they're lawyers. Might want to get some real legal advice on the matter if you're potentially leaving a user without a system that can't research the problem.
It's not even clear what part of data controllers and/or processor means to the providers of software users choose to install in which a system default fallback makes the Fedora org subject to GDPR. AWS might want to worry if they're providing the running Fedora system for me.
That's why the Fedora commit was stupid. It didn't solve any technical issues, it didn't consider negative side effects, and was completely unnecessary.
I have no problem with systemd but no, the OS should absolutely most definitely never use any other DNS than what is supplied to it from either the user or the network. Not only could the server some day become malicious but it is also extremely bad practice to hardcode something like this, even for fallback. It should fail loudly (edit: and if systemd really want a fallback, ask the user if they want to use a fallback on failure before using it).
That's the best thing about it too. It deliberately makes use of Linux features, making it much better than portable lowest common denominator software. Cgroups really does make services more manageable since even badly behaving forking daemons can't escape.
Linux has a long way to go to get on par with those two for the average Joe, and no, having DNS fallback absolutely won't help with that.
Note that the article specifically mentions cloud deployments breaking, so this isn't even something that a novice user will encounter, although it confirms my impression that the cloud is being used a lot by some of the greatest idiots in tech.
There are a lot of good reasons to not want a system to fallback to Google/Cloudflare. For one, I simply don't want it not obeying my own dns server in the middle.
Additionally, resolved isn't "not obeying" your DNS - this is the fallback that is only used if DHCP doesn't provide a server and the user doesn't configure one. DHCP and user configuration always take precedence.
But maybe Linux users are not so technology affine...second irony off.
I don't know about the newest version of macOS but iOS and the versions of macOS I used didn't have any fallback DNS servers.
I wouldn't want a feature like that enabled by default. It can be useful if I have the option to configure it but enabled by default populated with DNS servers chosen by whoever submitted the pull request? No thanks.
"One could even go further: the privacy level using those public DNS servers might actually be higher than using the DHCP-provided ones in many cases"
(I'm a massive systemd fan, but ignoring DHCP provided DNS servers is IMHO a bold move)
> We use the fallback only as ultimate fallback, when the other option is to not work at all.
> we only did that in case no better DNS configuration was available, i.e. as last resort, one step before giving up entirely
edit: my bad, the default servers are hardcoded, but you can also provide your own fallback servers. Not sure if that removes the defaults entirely though. If anything, Linux distros should decide what the defaults are, not systemd.
Overall I think this is a positive change and it's what every distro should do. If a distro wants to define default fallback servers then it should just be in the default /etc/systemd/resolved.conf, not hardcorded into the binary.
Curious: briefly, what was the nature of the failure? Thanks!
I’m not exactly knowledgeable about networking but was able to figure out that DNS wasn’t working. Switched to Google’s DNS and went about my day, but for the premier “professional workstation“ Linux distro it’s very disappointing.
Edit: I don’t want to blame fedora too much, I assume the actual issue was with my router or ISP. Windows and every other Linux distro worked fine out of the box though so I don’t really have a reason to spend more time on it.
first things first: why isn't your network (via dhcp or local configuration) providing/setting a dns server?
second things: it's fedora that messed up. fallback dns servers are supposed to be there as fallback when nothing else is provided.
It's not systemd's fault, it's fedora's.
if your dhcp server isn't providing a dns server and you're not setting it explicitly then having something else other than systemd would have meant facing the same problem.
I don't mean to be condescending, just curious why you didn't figure it out.
For users that struggle to diagnose completely broken DNS, I think diagnosing half working DNS will be extremely difficult to do.
Call me old fashioned, but I prefer completely working or completely broken over the “kind of sort of half assed mostly working in fallback mode” that seems to have become popular.
SystemD is more than just an init system, it also runs some system service, like DNS. If you still think of systemd as an init system, then your knowledge of it is outdated by at least a couple of years.
But, if you want to use it as merely an init system, feel free to disable resolved and install dnsmasq instead like in the Good Old Days.
I'd much rather have my system not working than have it randomly start sending the domain part of my browser history to Google without as much as a notification. When you install Fedora, the last thing I'd expect is that it starts acting like a tracker for Google when my DNS server goes down.
I think they should've implemented this differently, sending out a notification that the DNS configuration is broken, with the option of switching to either Google, Cloudflare or letting me fix the network configuration instead of silently failing.
The less tech savvy users are still running Windows where the exact same thing would've happened, with errors just as obscure. No matter what OS you're running, the problem is with your network config, so it should be fixed by whoever manages that instead of relying on some hard-coded defaults somewhere deep inside the DNS resolver. In most cases, that means calling your ISP or calling your IT service desk for support.
I personally hate it when software goes and does some random shit like pick its own DNS server because it somehow has determined that something was broken. Windows is full of this "smart" behaviour and it's one of the reasons I always feel like I need to anticipate its responses to my actions instead of just using it.
a couple years ? here's the blog post from 2011, a decade ago: http://0pointer.de/blog/projects/why.html
There's been as much time between now and this blog post, than there has been between this blog post and the introduction of windows fucking XP !
> systemd is in the process of becoming a comprehensive, integrated and modular platform providing everything needed to bootstrap and maintain an operating system's userspace. It includes C rewrites of all basic early boot init scripts that are shipped with the various distributions.
> systemd is also a big opportunity for Linux standardization. Since it standardizes many interfaces of the system that previously have been differing on every distribution, on every implementation, adopting it helps to work against the balkanization of the Linux interfaces. Choosing systemd means redefining more closely what the Linux platform is about. This improves the lifes of programmers, users and administrators alike.
being more than an init system has been the goal of the project since pretty much the beginning
And some new find, cp and ls with mostly the same options, but some having the same names of the GNU ones but that overwrite data at random.
And I'm eager to discover what the systemd folks can do in a remake of tar!
I'm not against systemd myself, to be clear. I think putting services like these into modules of the systemd environment makes sense, and as long as you can disable them if you don't want them, that's perfectly fine. I've had to disable one of them on one of my servers because _something_ was going wrong preventing resolved from caching repeated DNS responses, causing a huge load when a script started scraping a particular domain without any clear indication as to why in the logs. As long as I can do that, I'm perfectly fine with whatever the systemd folks want to integrate next!
I'm sad to see this here. Troubleshooting skills are worth their weight in gold.
That being said, I am somewhat disappointed that the general exploration of that solution space, and the creative use of those kernel features, seems to have ended as systemd matured, expanded its' scope and saw widespread adoption. I also think that the extent of systemd's authoritative dominance over the ecosystem has had a negative impact on user freedom in the broader world of open source. The fact that Snap has actively made a policy choice to _not_ give users the ability to completely disable automatic updates[0] and instead force at least weekly updates is I believe an example of the kind of "we know better than you" mentality that systemd has helped to foster and normalize.
I'd encourage everyone to sometime try booting a Linux machine with boot argument `init=/bin/sh` and see what it actually takes to get a system up to a running state with working networking.
[0] https://forum.snapcraft.io/t/disabling-automatic-refresh-for...
Personally, I'd rather it not work, so I can diagnose it and fix it correctly, rather than being blind to the problem, but that's the anti-grandma response not particularly cachet in today's world (which I get, I have a grandma too).
Both companies only collect anonymous analytics. Google does not use the data for personalizing ads. They both do this to improve speed and security of the Internet. For cloudflare it probably also serves as marketing to help get their name out there.
I did not claim that. I said “gain enormous insight into what everybody is doing”, and I chose my words with care. They might only gain insight into what everyone in aggregate is doing, if they are not saving information about each request. But, of course, almost nobody in the world is in the position to know if they are not doing that. It is almost impossible to be in a position to know that Google or Cloudflare is not doing some specific thing.
> They both do this to improve speed and security of the Internet.
Let’s get real. They are doing this to make money. Exactly how this is making Google and Cloudflare money is anyone’s guess, and it might be via some version of what you wrote, and it might be different reasons at different times and at different companies. But again, as I wrote above, almost nobody is in a position to know these things.
> For cloudflare it probably also serves as marketing to help get their name out there.
Yes. Again, money.
Fixing it required issuing this not-at-all intuitive command on the command-line:
sudo nmcli connection modify id CON_NAME \
ipv4.ignore-auto-dns yes ipv6.ignore-auto-dns yes
It's the Network Manager command line tool (nice to know its name, that's why I could never find it). The option you are setting has this description: "Do not use DNS Resolver from DHCP".
I didn't find a GUI switch on my computer with that meaning.
And yeah, I think my bigger irk is that this seems to contradict the network settings in my DE's GUI, where I had explicitly tried setting the DNS lookup servers to 1.1.1.1 instead of using DHCP.
Fallback servers can be explicitly configured via a configuration file. Systemd provided default fallback servers, if none were set in the config file. IIUC the config file having fallback servers set would prevent the default fallbacks from being used, even if the config file servers weren't responding.
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
In the face of failing or missing primary DNS, the system should loudly complain about it, then get authorization from the user to fall back to a default DNS server.
http://fedora.mirror.constant.com
http://mirror.arizona.edu
http://mirror.atl.genesisadaptive.com
http://mirror.genesisadaptive.com
http://mirror.lax.genesisadaptive.com
http://mirror.math.princeton.edu
http://mirror.siena.edu
http://mirrors.syringanetworks.net
http://mirror.us.leaseweb.net
http://mirror.web-ster.com
http://repo.radeon.com
https://codecs.fedoraproject.org
https://d2lzkl7pfhq30w.cloudfront.net
https://download.docker.com
https://ewr.edge.kernel.org
Relatedly, your IP is also disclosed via NTP. NTP hosts can be advertised over DHCP, similar to DNS resolvers. And systemd also has built-in fallbacks for NTP, which has actually caused major headaches working on security compliance for U.S. government services because the instances should never use any NTP server other than time.nist.gov. But if time.nist.gov is unavailable for w'ever reason, systemd by default falls back to non-compliant NTP hosts. Disabling this is tricky if you rely on DHCP-advertised NTP servers (which is preferable to minimize the diff between govcloud and non-govcloud deployment images).
If the installer offered DNS fallback and let you choose from a list of server options or no fallback it would likely be less controversial.
It is, after all, leaking a list of sites you're connecting with to whoever the dns provider is.
I'd rather have it not function, because that way I know something is wrong.
If something breaks somewhere on my network, but my experience stays 'the same', I may miss the fact that something broke because of this 'magic' that occurs behind my back.
If I'm surfing the web, and my browsers hangs with "Looking up news.ycombinator.com..." or returns and error with "Cannot resolve/find server with name example.com.", then I know to start digging into things.
The answer is (I suspect) that the systemd people haven't supplied a resolving proxy DNS server of their own to be run, and don't want to tie themselves to running a particular third-party's software. So they tie themselves to a particular third-party's service instead. systemd-resolved has to have somewhere to forward to by default, it being architecturally a purely forwarding proxy DNS server.
The exposure of the IP address is another aspect. While DNS can expose more info, the complaint also includes IP addresses specifically. I believe this has severe implications: Does any outside communication need to be acked on setup? Should ntp servers be preconfigured on OSes at all? I wonder which other services depend on similar behaviour. Will it end up like consent forms on websites?
I personally think a fallback DNS which masks network errors is a nice thing, as systems should be resilient for users. Errors just need to be discoverable for admins. I usually want things to work, not debug errors of others (or even my own).
They should just have some standard way for a distro to specify the fallback servers, or no fallback.
The the various distros could populate that user-driven, conscious decision somewhere that systemd-resolved can pick it up.
replace the word cloud with computer and the cloud argument for fallback evaporates. Cloud has somehow become a surrogate for competent understanding of basic internetworking? cloud-init cant provide a DNS server? provider DHCP isnt a good enough default? Cloud is in this case some premium hand-waving from the systemd community.
fallback in itself is a disingenuous offering as it tries to sidestep and clean up some seriously fundamental problems a user might face that they should address before even standing up their microservice or app or whatnot. If you dont have a cogent DNS provider you should stop deploying, not complain loudly and continue merrily on. Google and Cloudflare have on numerous occasions proven they can and will arbitrarily terminate any and all service to a user for any or no reason at all.
Cloudflare DoH in particular has open and recurring threads for shadow domain blocks or SRVFAIL they return from their service that are neither announced to users or explained. usbank is a TLD that sees a good deal of quietly planned intervention from the DoH service for some reason.
Could you provide a link or other source for this? I've never heard of this (and can't find anything via google), and I feel like it would be front page news on sites like HN if cloudflare was intentionally providing bad DNS replies.
Edit: Nevermind, I've found it - and found out you're basically trolling with that level of misrepresentation:
https://community.cloudflare.com/t/servfail-with-u-s-bank-do...
The issue was:
1. On usbank's side
2. Correctly relayed to the client by cloudflare's DNS rather than lying about the response they got
3. Corrected after cloudflare reached out to usbank to alert them of the mistake
There is no 1.1.1.1 shadowbanning. This is a bunch of crap.
Exactly. If you can’t deal with basic DNS, that’s a good sign you’re not qualified to manage a bunch of cloud infrastructure.
- no logging policy, iow adheres to GDPR
- global anycast service with Multiple points of presence around the world
- 99.999% uptime since 2016
- not opinionated: supports blocklist and 'no security blocklist' (9.9.9.10)
- supports ipv6
- support for dns over tls
- support for DNSSEC
- works with systemd-resolverDisabling "fallback" to Cloudflare, Google, et al, is what everyone should do. The idea of "fallback" is completely and horribly flawed, and only an idiot or a corporate shill could think that it's a good idea.
Such include Void and Arch.
As an anecdote, it took me a while to tweak my Void installation to be just right, but I've had no surprises like that. The distro offers you the choice of any or no network manager, any DNS resolver, ALSA or Pulse, etc and assumes nothing. The guidebook specifies how to configure your choice of these packages too.
It's unfortunate to see well-known distros go this way, but there are others who keep user choice alive.
>What part of the community do you mean?
The btrfs, epool, lvm, kernel panic disabler, lxc and systemd especially.
[0] I have a strong suspicion that systemd-nspawn is not a mess because it does exactly one thing - sets up cgroups and namespaces to run a container and runs a container.
Serious question: what is vulnerable in my (non systemd-resolved) desktop DNS configuration I've been using for years? This statement is presented as if it were a "everyone knows this" fact about workstation DNS, but I don't. Just your average Linux desktop, nothing fancy.
> Serious question: what is vulnerable in my (non systemd-resolved) desktop DNS configuration I've been using for years? This statement is presented as if it were a "everyone knows this" fact about workstation DNS, but I don't. Just your average Linux desktop, nothing fancy.
venerable, not vulnerable