What happens when you update your DNS
jvns.ca
jvns.ca
When you send a DNS query to a recursive server like your ISP's or something like 1.1.1.1, you make a single DNS query and get back a single response, because the recursive DNS server handles all the different queries that Julia outlines in the post. As the client, we have no idea what steps just happened in the background.
But when you run `dig +trace`, dig is actually pretending to be a recursive name server, and making all those queries itself instead of letting the real recursive name servers do their work. It's a fun hack but that means it's not always 100% accurate to what's going on in the real world [0]
[0] https://serverfault.com/questions/482913/is-dig-trace-always...
For a given application server it might be:
- check local dns caching resolver
- check local network caching resolver
- if internal domain, check local authoritative resolver
- if public domain check isp resolver
- recursively resolve from there
The other possibility is different IP's being returned by a DNS server based on where a query is coming from, eg a CDN. If you're in location A and your ISPs DNS server is in location B, the CDN's DNS server may return a different IP based on if the request is coming from A or B. ECS [0] is supposed to mitigate this, but may or may not be used.
It's disturbing how many clients we'll see hitting an old IP address for 30 days after a change.
Still there can be caches that don’t quite agree as the other comment mentions.
So when you ask for www.amazon.com it ends up making multiple DNS lookups, as www.amazon.com is a CNAME record.
Nothing about this CNAME lookup gets passed back up the stack to your application; you just get that end-result: the IP address.
host www.amazon.com www.amazon.com is an alias for tp.47cf2c8c9-frontier.amazon.com. tp.47cf2c8c9-frontier.amazon.com is an alias for www.amazon.com.edgekey.net. www.amazon.com.edgekey.net is an alias for e15316.e22.akamaiedge.net. e15316.e22.akamaiedge.net has address 23.204.68.114
(My first thought was, wow, surprised they outsource that.)
https://www.theregister.com/2013/02/15/star_wars_traceroute/
The most egregious case I've seen was an Amiga site. The site went down and for several days reported that users would need to wait for the updated records to propagate and lots of loyal fans were insisting anybody who couldn't read the site was just being too impatient.
What was actually wrong? They wrote their new IP address as a DNS name in their DNS configuration rather than as an IP address. Once they fixed that it began working and they acted as though that was just because now it had successfully propagated.
On the other hand propagation is a thing when it comes to distributing modified DNS records to multiple notionally authoritative DNS servers.
This can be a problem for using Let's Encrypt dns-01 challenges for example, especially with a third party DNS provider.
Suppose you write a TXT record to pass dns-01 and get a wildcard certificates for your domain example.com. You submit it to your provider's weird custom API and it says OK. Unfortunately when you do this all it really did was write the updated TXT record to a text file on an SFTP server. Each of the provider's say three authoritative DNS servers (mango, lime, kiwi) check this site every five minutes, download any updated files and begin serving the new answers.
Still they said OK, so you call Let's Encrypt and say you're ready to pass the challenge. Let's Encrypt calls authoritative server kiwi, which has never seen this TXT record and you fail the challenge.
So you check DNS - your cache infrastructure calls lime, which has updated and gives the correct answer, it seems like everything is fine, so you report a bug with Let's Encrypt. But nothing was wrong on their side.
Now, unlike typical "DNS propagation" myths the times for authoritative servers are usually minutes and can be only seconds for a sensible design (SFTP servers is not a sensible design) so you can just add a nice generous allowance of time and it'll usually work. But clearly the Right Thing™ is to have an API that actually confirms the authoritative servers are updated before returning OK.
Been there, done that, got burned. If you're mucking around with DNS records that are going to be verified by someone else, never trust your local lookups, always try to verify through a fresh third party that everything resolves correctly before you submit.
Because oops, something went wrong, the verification hit a wildcard entry with a cache time of days, and now you have to wait that long before trying again, because that entry isn't budging from other resolver's caches...
So you fire up cmd.exe and issue ifconfig /releasedns, ..., ipconfig /?, ipconfig /flushdns and then you go back to pinging the bloody address instead of using nslookup because you learned from another idle/jaded sysadmin to use ping as a shortcut to querying DNS, instead of actually querying what the DNS servers respond with.
Obviously, a better thing to do when checking your DNS entries is dig out ... dig.
DNS changes _do_ propagate: from the one you edited to the others via zone transfers and the like (primary to secondary etc) and thence to caching resolvers.
"Negative caching was an optional part of the DNS specification and deals with the caching of the non-existence of an RRset [RFC2181] or domain name."
Optional.
Now, I know how DNS works and so do you but your average sysadmin wants instant results as they thrash around trying to get something to work. You see the same effect with firewalls. If state tables don't get flushed after a change then the change will seem to be ineffective. Been there and done that. States often die after five minutes and so do Windows DNS fuck ups.
That is why the master says: "wait longer" followed by "turn it off and on again" or "go out and smoke a fag" or "get me a coffee" or whatever. When your arse is on fire in IT, learn when to trust your judgement and take five.
When we get a complaint that our recursive servers are returning an 'old IP' as if we did something wrong... I start by explaining that they likely didn't properly lower their TTL prior to making a change.
Sadly we'll never win this battle; so at some point in the future we'll need to join the likes of google in providing a page to force cache expiration of domains at customers request :/
It's the idea that it spreads out slowly from the centre to the far reaches that annoys me - which is what most people seem to be suggesting when they refer to "DNS Propagating".
It's not one big push, it's millions of tiny pulls.
The timestamps are provided in DJBs TAI64 format, use something like https://github.com/hinnerk/py-tai64 to convert them.
[1] config file spec at https://cr.yp.to/djbdns/tinydns-data.html
[0]: https://www.joshmcguigan.com/blog/run-your-own-dns-servers/
It is baffling to me that inside the datacenter, I can control the balancing strategy for every service-to-service transaction, but for the end user's browser, all I can do is some L3 hacks to make two routers appear as one (for failover purposes). L3 balancing would be completely unnecessary if I could just program the user agent to go to the right host, after all. The end result is unnecessary cost and complexity multiplied over a billion websites.
Some applications use SRV records for load balancing. Many VoIP and video conferencing apps do this. There is a better list on Wikipedia.
_service._proto.name. TTL class SRV priority weight port target.
Right now, you can try resolving common hosts, and you will see that they often provide several hosts in response to a lookup. What the browser does with those IPs is up to the browser, the standard does not define what to do. What the administrator that sets up that record wants is "send to whichever one of these seems healthy", and some browsers do do that. Other browsers just pick one at random and report failure, so your redundancy makes the system more likely to break.
What I want is a way to define what to do in this case. Maybe you want to try them all in parallel and pick the first to respond (at the TCP connection level). Maybe you want to try them sequentially. Maybe you want to open a connection to all of them and send 1/n requests to each. Right now, there is no way to know what the service intends, so the browser has to guess. And each one guesses differently.
(You will notice that people like Google and Cloudflare skillfully respond with only one record with a 5 minute TTL. That is so the behavior of the browser is well defined, but it also eats their entire year of 99.999% uptime with one bad reply. Your systems had better be very reliable if DNS issues can eat a year's worth of error budget.)
In the rrDNS, remove the A record of the hosts that fails tests, or that has a load that's too high
> Maybe you want to try them all in parallel and pick the first to respond (at the TCP connection level).
Something a geoIP at your DNS can do, certainly not as good as doing that in the client, but it should be decent enough.
> Your systems had better be very reliable if DNS issues can eat a year's worth of error budget
Or, if you aren't google or cloudflare, use a 30 to 60s TTL in rrDNS, with health checks to selectively remove IP that fail, on pools splitting your servers by region with geoIP - this way, if 1/10 of your east coast servers fail, nobody from APAC will be impacted, and only 1/10th of your US east users, and only for the TTL (I'm abstracting ISP that cache for too long, but you already mitigate a lot of the problem there)
I can see how it would be easier to handle that in the browser, but you may already be able to do that with some JS to estimate the latency, then store the result in a cookie that causes a reload to www.eastcoast.yoursite.com if the user sticks to www.yoursite.com or after returning home goes to www.apac.yoursite.com while new measurement say "not optimal" and update the cookie
I don't really like it because it still means a minute of downtime when the topology does change. I would prefer telling the browser what strategy to use to try a new node, rather than relying on heuristics and defaults.
This chapter in the Google SRE book explains how our load balancing DNS works:
https://landing.google.com/sre/sre-book/chapters/load-balanc...
Source: my team runs this service
It might be a little more resilient than even a very minimal nginx, but more than that, I think it must give you more control about what happens when a packet is not "answered" after some set amount of time - you write off who should have been the answerer, then resend that same packet to another server. Keep a buffer of packet, scrape them from the buffer when ACK'ed by the answerer, resend them to another answerer if not ACK'ed after some set amount of time.
Am I guessing correctly?
It seems a bit overcomplicated for normal usecases, but adequate for a large scale like google.
The main thing I wanted to respond to in this thread about a single bad server destroying your yearly SLO is described in the first paragraph in the section on load balancing at the virtual IP address.
Overall, virtual IPs are still an interesting solution.
Nameserver records (NS records) used to locate a resource are served by other nameservers. NS records are chosen from among those offered in response to a query (RRs), and should all be tried if necessary to elicit a response. The algorithm isn't strictly specified and some nameservers will shuffle the order in which they return RRs in their answers, some won't assuming the stub resolver or app will do it. The foregoing also applies to A and AAAA records (returning IP addresses for names), and this has long been used as a quick and easy form of load balancing/failover, except that it doesn't really failover very well unless your app is coded to try all of the different answers (and the stub resolver returns them to your app).
Nameservers querying other nameservers (caching/recursive resolvers) are supposed to compile metrics on response times when they make upstream requests and pick the fastest upstreams once they learn them.
Stub resolvers (running on your device) typically query nameservers in the order you specified them in your network config, but not always.
From the foregoing, you can probably see that running a caching/recursive resolver close to your devices is supposed to be desirable, by design.
So far, so far. ;-)
As specified, and it's never been changed, DNS tries UDP first. "Ok", you think "that must mean it will try TCP" but that's not actually true: it only tries TCP if it receives a UDP response with TC=1 (flagged as truncated). But if there's a UDP frag and it doesn't get all the frags or never gets a UDP response at all it /never/ tries TCP.
You're mixing two very different environments above: 1) a datacenter with (let's just assume) VPCs and 2) a web browser.
In case #2 I'll match your ante and raise you an overloaded segment which is dropping UDP packets, in which case stuff may fail to resolve at all. Oh look, I drew a wildcard: traditionally browsers have utilized the devices stub resolver, but since they've pushed ahead with DoH they've had to implement their own. People think I'm a DNS expert (what do they know?) and I guess conventional wisdom amongst myself and my peers is that UDP should perform better than TCP but anecdotally people are claiming that DoH and DoT perform better for them than their stub resolver. "Must be your ISP messing with you" says someone, "yeah right, that's gotta be it". Me: "did you try running your own local resolver?" them: "wuut?"
So here's where I confess that the experts aren't always right, because I run my own local resolver and I have the same problem: when the streaming media devices are running DNS resolution on the wifi-connected laptop sucks and if I run a TCP forwarder it starts working! (https://github.com/m3047/tcp_only_forwarder).
Now to case #1, the datacenter. I hope you're running your own authoritative and caching server, and you should read about views in your server config guide; using EDNS to pass subnet info is a kludge. If you're writing datacenter apps, you should consider doing your own resolution and using TCP (try the forwarder, I dare you), and provisioning accordingly (because DNS servers assume most requests will come in via UDP).
If you want load balancing "you know, like nginx" I've got news for you: BIND comes with instructions for configuring nginx as a reverse TCP proxy. Oh! Looks like I've got a straight in a single suit: nginx provides SSL termination so I've got DoT for free!
For example, consider the case where you resolve ycombinator.com. You get:
ycombinator.com. 59 IN A 13.225.214.21
ycombinator.com. 59 IN A 13.225.214.51
ycombinator.com. 59 IN A 13.225.214.81
ycombinator.com. 59 IN A 13.225.214.73
Don't get me wrong, I think stub resolver logic is stuck in the 1980s!
If your app or device doesn't have such a history, and no way to obtain it, then maybe the server can do it based on what it knows about its history with IP addresses "close" to yours (the EDNS kludge).
> It doesn't respond. What do I do now? Tell the user ycombinator.com is down? Try another one? All of this could be defined by a standard
I would argue the DNS is clear about this from its own behavior: it tries another one.
Although it's not clear from `pydoc3 socket.create_connection`, it's pretty clear from https://docs.python.org/3/library/socket.html#creating-socke... that socket.create_connection() that it will "...try to connect to all possible addresses in turn until a connection succeeds."
So I would say that the correct action would be to try all possible addresses until one succeeds.
Well, there was an RFC (found it, RFC 3484) that told you to pick the one closest to your network (which wouldn't make a difference in this case, unless you were in say 13.225.214.0/27 or so). But that's not actually helpful, because given two destination IPs, one in the same /8 as me, and one not, I don't have any information that would help me determine which is a better choice.
From experience, most browsers will try a couple IPs before showing an error message, but that's not standard. If you have a fancy authoritative server, a lot of traffic, and a bunch of server IPs, you can get OK balancing by telling some clients some IPs and some clients other IPs; but it depends on having enough diversity in recursive servers; if all of your users are coming from one mobile ISP, chances are you won't get a lot of balancing.
(And I'm sure you already know all this :)
Better to have clients with a bit of intelligence. :)
If you're talking about subprotocols/versions of HTTP like HTTP2 then you can define subservices, so you could have _http2._http. But no one has proposed that yet :)
Of course with anycast, multiple A records can be redundant :)
There's actually the germ of an interesting idea in that statement. If I'm going to go to the trouble, let's say, of running a local TCP forwarder (good for the whole device), can I run a packet sniffer at the same time and watch netflows and edit the responses I return to the device based on what I see performance-wise concerning those flows?
Expert me says that web sites are loaded with too much cruft and since the far end terminations are spread far and wide, there's not enough opportunity to apply that learning in any practical sense. But I could be wrong. (https://github.com/m3047/shodohflo)
Typo, meant 853.
[0] https://r-1.ch/r1dns/ (and yes it has a lot of bugs, don't break it :)).
you wouldn't be the R1ch of old something awful forums fame, would you? I used to run a waffleimages mirror :)
That's not true. The vast majority of clients will move to the new IP within the TTL, within 5min (not 15min). Then there will be some stragglers that slowly update over the next hours/days (typically poorly written bots)
Source : my own experience updating a site with 500k hits per month and sniffing and watching network traffic at the 3 endpoints: DNS, old IP, new IP.
If the major internet powers agreed to stop serving responses to DNS servers they detected weren't respecting reasonable TTL's, do you think they could "bully" the industry into tightening things up? (Kind of how Google and others compelled the web toward widespread HTTPS)
I've seen 8.8.8.8 return something other than NXDOMAIN for some domains that do not exist
Cloudflare will not honour dns ANY requests
Knowing how to query the authoritative nameservers is a handy tool for debugging.
Test with:
dig @ns1.google.com www.google.es +subnet=193.8.172.75/24
dig @ns1.google.com www.google.es +subnet=157.88.0.0/16
Note how you get different IPs returned.
Are you talking about the (mathematical) "domain" in the DNS specs, or the popular domain i.e. the web server?
The latter is arguably true, in which case the geoip proposition is moot: there is only one web server. Maybe you mean the web server has multiple addresses instead of being anycast. Ok, yes that happens; and some DNS servers do use geoip to tailor replies to try and hand the closest address. Here is from tbe BIND ARM:
"By default, if a DNS query includes an EDNS Client Subnet (ECS) option which encodes a non-zero address prefix, then GeoIP ACLs will be matched against that address prefix. Otherwise,they are matched against the source address of the query"
Regarding the former, does anyone have info on how many DNS providers use anycast? I think a lot; or maybe I should say that a lot of domains are hosted on anycast, the DNS isn't as distributed as it used to be. If you're using DNS as a distributed key/value store, I hope you're doing a better job thinking about externalities (leakage) than e.g. the antivirus companies in terms of locating authoritatives and how you update them opaquely.
Personally I think stub resolvers are stuck in the 1980s. They could do a lot more by monitoring traffic health and editing DNS replies. Due to peering arrangements you could be in the same IX as someone else but that might not be the best route. Traceroute, SYN exchanges, (IP) TTLs might be better signals for determining the health of a particular path. I'd never thought about it until this thread started, maybe the stub resolver could use netflow analysis to inform editing the responses it returns to the applications.
So my case for top level (TLD) GeoIP: I have many DNS servers for my web-address/domains: Three in EU and two in US. The problem is that when the TLD servers sends the list of DNS servers it's randomized. Instead I want it to return the list in GEO order (and also network health order), so that the recursive resolver ask the best/closest DNS server. So in worst case scenario for a recursive resolver in EU tries a DNS server in USA, and it happens to be down, and then gives up. The best case scenario is that it tries the closest server in EU.
Trying to solve my problem I've tried the top 10 DNS providers (ranked by uptime and query speed), which uses anycast. Only two could be used as secondary/slaves, and both of them took over two days to propagate an update (they did not use TTL). The reason why I need fast updates is because of Letsencrypt which requires DNS challenges for wild-star domain SSL/TSL.
About anycast use, the root server's have been using anycast for a while now. Some TLD's use anycast I think (haven't actually checked). For web-hotels, and ISP's most do not use Anycast. ISP however have the DNS server's very close to the end users and they are good at caching. Which is the second reason why I'm against DNS centralization. Querying from example 8.8.8.8 is often 10x slower then using ISP DNS (assuming the ISP have the query cached).
Anycast, although proven to work nice for the root servers, which makes it harder to DDOS-attack, doesn't actually work that great. I argue they could just list the servers in GEO order instead of configuring BGP routes. When I evaluated the "top 10" Anycast DNS providers, sometimes my amateur setup got lucky (eg test server from US got the US IP first and vice versa) and thus beat the Anycast network in query performance/latency.
Recursive resolving algorithm for caching servers is actually addressed in the RFCs, it /should/ be trying all of them and using its own findings to prefer the best-performing one(s). But it doesn't know about anycast, if a recursive was switching between anycast nodes (with the same address) then that would imply that routes were flapping. :-(
Interesting data points.
I think the elephant in the room is the Universal Terrestrial Radio Access Network (UTRAN) a.k.a. "mobile" and I don't work with provisioning much so all I can say is that I suspect that if mobile is your concern you just prostrate yourself to the UTRAN masters and co-locate wherever they tell you to.
SSL/TLS cert management is a fiasco in my opinion, it's a shame that DNSSEC hasn't achieved market dominance so that if you own a domain you can automatically sign certs for it yourself. (Then we wouldn't need CA lists in browsers and OSes either.)