Problems with low DNS TTLs
00f.net
00f.net
I've never regretted a short TTL.
Edit: Worth noting there's lots of software that seems to only resolve hostnames at first connection, then hangs onto it forever. Lots of java internals for example, unless you poke in specific configuration.
I've seen more problems caused by the JVM, by default on some configurations, caching DNS indefinitely, regardless of TTL, than caused by a short TTL.
I remember having to bounce Java apps every time DNS changed, which never made sense to me. It's literally the point of DNS to not have to do that.
And I do not think it does. DNS without caching is useless traffic overhead. Just like HTTP responses without gzip compression. DNS entries almost never change, therefore it should be cached accordingly.
K8s Ingress and Cloudfront alone will probably make the customer visible IP addresses nearly static forever. We don't live in the old world where we had to take a server down any more. It's all managed.
If it's going to take more than 3 hours to setup the new traffic target anyway, sure. You have time to fiddle with TTLs. If you have servers ready to go elsewhere, 1-5 minute TTLs are nice so you can quickly move things when you notice a problem.
1 minute is a long time, you can do a ton of requests in 60,000 milliseconds.
On the flip side, setting the TTL long can be a disaster. You can't fix it after the fact. If you have a 1 hour TTL then that's potentially 1 hour before the changes needed to fix a service fully take effect. That's 1 hour of helplessness.
I definitely think it's worth the millions or billions of DNS requests.
Use short TTLs, your customers will thank you.
Default TTL: 12 hours;
[<startdate> to <enddate>] TTL: 10 minutes;
Default TTL: 12 hours;
[Thu 0000-1200, repeating]: 5 minutes;
[<startdate> to <enddate>] TTL: 10 minutes;
I dunno, I'm sure there's other downsides I haven't thought of, and proper implementation would require thinking through side effects. But after a long time dealing with it feels like there's some room for something beyond one single TTL ever which must be specifically changed (with a wait for propagation) well ahead of time whenever anything planned with a risk of issues needs to be done. Maybe?
eg.
- Check if there's a maintenance window in the next [max TTL]
- If not, set TTL = [max TTL]
- Otherwise, set TTL = [time until maintenance window]
- Repeat
No one ever said this.
Less than a minute is excessively short. Most long-lived applications will internally cache your IP address for longer than a minute no matter what you do.
You'd also have to do that for SMTP, IMAP, etc. Probably not worth the complexity as low TTLs seem to work well enough with few enough downsides. DNS is already a tricky enough protocol.
For a lot of things changing domain is quite a hassle, and the old domain will still be around, cached, and considered valid (because its ttl hasn't expired yet).
I agree with GP in saying that a short TTL never created too much issue.
Now, to come to the original issue: short ttl.
What do you expect, in this day and age?
With all this cloud stuff going up and down, being created and destroyed again and again, ephemeral stuff that doesn't even last a whole single day...
Of course you need a short ttl.
I've seen enough people complaining about overloaded DNS servers.
I've seen way more people complain about outages that lasted hours because of long TTLs and a deployment mistake.
1. DNS lookups add on the order of 100ms to load time.
2. In both cases (5 min and 15 min) the user is going to do a DNS lookup on the first page, then have cached DNS while they browse for a bit, then have a DNS lookup at some point in the future.
Most web sessions are relatively short, so I doubt most users would even notice the difference where they see an extra 100ms load time at, say, 2 points in their session instead of 1.
In my experience the appropriate use of the preconnect and dns-prefetch hints has a much bigger impact on perceived performance than worrying about DNS TTLs beyond 5 min.
Very much this sentiment.
When migrating a website many years ago, I forgot to lower the TTL of companyname.co.uk. I had lowered the TTL of www. but not the root.
So when the big switchover came, half the traffic stayed where it was. Not only that the new backend fell over.
having that option to roll out/in would have been really useful.
Now with the cloud stuff goes away with, so having a TTL of 3600 means long outages.
As a user with 20+ years of experience, I have never regretted using authoritative DNS as the source for DNS data instead of DNS caches. Instead of using a shared cache, I bind local authoritative servers loaded with zone files containing the DNS data that applications need to the loopback. IME, lookups are faster than with a cache.
IMO as a user, recursive queries to shared caches are overrated. I want to know what lookups the applications I use are making and I want control over what DNS data is available to them. IME, giving applications license to lookup any resource at any time means this permission will be abused to further the interests of the online ad industry and all those who service it instead of 100% serving the interests of the user.
For example, a commercial router's management console does not need to be able to look up the addresses of ad servers. Letting it have access to a remote shared cache, or even a local one, that does recursive queries is unnecessary. The user has already paid for the router.
Do you get faster web pages if you cache for a longer time? If you do, shouldn't web browsers "soft-invalidate" (use the entry, but update it right after) the cache entry when you're just past TTL and "hard-invalidate" (update it before using) after? Do they do that already?
The client needs to wait for the result of a DNS query before it can do anything else. The bandwidth is irrelevant, the problem is the delay.
Usually DNS queries are cached by a server near the user, so they are very fast. But if the authoritative name server has a very short TTL, then those cached results will often be stale, and the name server has to resolve the name recursively, which can be slow.
You're right. But the article seems to claim the number of queries can be reduced with a higher TTL, which is why these should be enforced. Fair enough, but so what? In any case, the TTL is irrelevant for the first query. For the next ones, if you allow yourself to rely on the previous result (the "soft-invalidation" I mentioned, for lack of a better word), it shouldn't have any impact for the user.
With a long TTL, the chances are higher that your router or your ISP has the name cached. Round trip to your ISP is very short.
If you have a short TTL, the ISPs name server may have to query the next server in the chain.
If the authoritative name server is 10000km away, that means at least 60ms extra round trip time (speed of light).
It's just very easy to shoot yourself in the foot with a long DNS TTL, in the worst case taking down your entire site until the TTL expires if you ever misconfigure it. Why risk it for a small gain?
Also, even theoretically it's not clear to me that it would help end user experience. Someone correct me if I'm wrong, but I believe if there's already an open connection from the browser to the server based on the TCP keep-alive settings, the browser will continue to use it rather than kill it and open a new connection starting with a DNS lookup even if the DNS TTL expires. So for an otherwise well-tuned site, each user session should expect to do a DNS lookup at the start of the session but then the rest of the requests in that session won't need to.
If that's true, then the only way DNS TTL will affect the end user experience is if the TTL is long enough such that there are a significant number of instances where the time between sessions for one user on the same device is longer than the DNS TTL. Most sites don't have their users returning every 30 minutes, they're lucky to get someone as a daily active or even weekly active user. So the TTL might have to be ~ a few days long to impact a significant number of users (and even then, probably only by a few tens of milliseconds at the beginning of the session).
Seems to me that short TTLs in connection with the current trend to include scripts from dozens of different domains could easily double the number of average roundtrips per page load here.
I think most people would call a website down if it was just a white screen for 30 seconds, so while it's a nice try on the part of the browsers, you can see why people use short TTLs to get bad backends out of the pool as quickly as possible.
Much longer and you run into all the trouble that operators have with keeping DNS accurate. DNS is hard. It is easy to break. And 15 to 30 minutes of waiting is about as much normal human attention span you can apply to a problem that sounds like, "Ok, we're all done, is DNS ok?"
5 to 10 minute TTLs only benefit operators. Certainly, any TTL less than 5 minutes is an indicator that your operators have no faith whatsoever in their ability to manage DNS.
1. Create instance of service running version n+1
2. Switch public DNS records to point to new servers
3. Wait for TTL to expire
4. Turn off old servers
(Obviously I'm simplifying; if nothing else there should be testing steps in there)
Unless I've missed something, wouldn't the author's suggestion to artificially raise the TTL by ignoring the upstream TTL result in the application breaking for customers if they used a DNS resolver that did this?
But I bet you still ran forwarders on the old hosts for at least an hour after you cut over DNS.
I promise you we did not.
I've definitely lost count of the number of clients that would cache the old IPs, despite valid and low TTLs being in-place well in advance of a migration.
In our case it was worth keeping things working for a few days, but after a week at the outside we'd kill the proxying/forwarding.
For example, if you offer an "entrypoint" that you can guarantee and technically make to be stable, then use longish TTLs. Anycast IPs are an extreme, but inbetween there are many useful modes of exploiting longish but not too long TTLs.
On the other hand, if you implement system failover in a locally redundant system and want to exploit DNS so you don't have to manage additional technology to make an "entrypoint" HA (VRRP, other IP movements, ...), low TTLs are nice. AWS is I think using 5s TTLs on the ElastiCache node's primary DNS names.
Finally, 15m max is what I'm comfortable with. Any longer or much longer, and ANY MISTAKE, and you can easily be in a world of hurt. It's no fun sitting out a DNS mistake propagating around the world and the fix lagging behind.
And this is only a view on "respectable TTL" values. DNS services like Google's public dns probably ignore any or all TTLs for records they pull, and refresh them as fast as possible anyway, at least according to my observation. In that sense, I doubt that most of the internet is still using "respectable" TTLs --- I suspect most systems will RACE to get new data ASAP.
What this points to is a need for an authenticated DNS pushes for refresh/invalidation.
All supporting resolvers could keep a list of supporting clients that were told that "foo is at address 42". If the record changes, the authoritative DNS server sends a DNSSEC signed unsolicited response to all previous requesters to update their records. Obviously the TTL can be extended to keep the cache of requestor IPs reasonably sized.
Will this happen? Well, for UDP DNS it depends on DNSSEC, which is already not well supported, and it fixes something that is broken but not terribly so. One could imagine Google arranging this between its DNS resolvers and Chrome, for instance.
For DNS over HTTPS, this becomes much more feasible.
“Pushing” the message out that the record has changed would also prove tricky to implement I’d say.
Obviously you need one of these bitmaps for every change domain (i.e. 1 per zone, or 1 per A/AAAA/CNAME record set, operator choice), and you need to clear it every (extended) TTL.
So a CDN with 100,000 dynamic IP records might split themselves into 1,000 change domains of 100 records each, have a 1 hour TTL (expiry's staggered), and use 500GB ram to do this.
if so, that's a nice spot for user tracking.
if not, how do you push data to clients behind a nat, without a stateful and persistant tcp connection? for most dns queries udp is sufficient (although sometimes tcp is necessary).
500GB ram for 100k records seems quite a lot btw.
That's just not how this works at all. While you could use RR records for this purpose, I believe the author is suggesting that load balancing will happen automatically when the client simply can't connect to one of the addresses. That's not load balancing. That's failover.
Additionally, most of the use cases for this that I'm aware of are Cname -> A record. This is to say, this method is being used with precision rather than RR.
I agree that running 60 second TTL's regardless of need is inefficient, but at a fast glance, the full argument doesn't hold up for me.
So that would be both load balancing and failover, why doesn’t the argument hold up?
next client comes in, I'll serve three again, possibly different from the three i've served before.
the client doesn't even know that there are two other possible endpoints (unless maybe until the next query).
edit: i just tried running this
watch dig -t A www.amazon.com @8.8.8.8
If a client makes 50 requests before the cache expires, then those will all be based on the cached result.
This is still efficient enough that there's probably no more than a single DNS hit for every web page load, even with a short (say, 5 second) TTL, because most web assets will be loaded within that five second window. (If your web page takes longer than 5 seconds to load, you have far more significant issues than a few UDP DNS requests.)
Whether the list of invalid use cases are straw man arguments are left as an exercise to the reader, but this article seems to be arguing only one side of the perfectly valid trade-off between flexibility (low TTL's) versus latency (high TTL's).
In other words, if high TTL's are so great and there's no compelling reasons to not use them, why not make them one year? Ten years?
On the other hand, many (probably most) applications can probably absorb a five-minute outage without anyone screaming too loudly.
Clearly there is a balance between "long" and "short" (probably somewhere between one second and infinity). It's good to think about these things and optimize for lower latency, but if five-minute or longer TTL's simply don't fit your use case, then don't feel bad about it.
Netlify's "previous" IP was down for ~4 hours.
more dns queries with a lower-ttl (say 10 minutes) means one additional round-trip every 10 minutes. how long can a round-trip be, 200msec worst case scenario?
that looks good.
now on the other hand, assume a 48h ttl and something breaks. now you've got all your users unable to reach your services for up to 48h. or worse, some of your users will go to the old ip, some to the new.
what's worse for the user, a round-trip from time to time, or an extended outage ?
It's typically much more than 1 round trip, given typical amount of crap frameworks and gadgets that each load themselves and dependencies from around the world on a typical webpage. On a page that takes 50 seconds to load all the crap from 50 servers each with a 5 second TTL through a flaky ISP DNS server, you get hit all the time basically.
200ms is not worst case, that's more like median, worst case is DNS being stuck for minutes with responses lost or very slow (say 20 seconds). Often DNS is the only thing that's broken, and if you're in one session (with no new DNS request required beyond refreshes from expired TTL) it makes the difference between the user being stuck or being able to proceed unhindered, until they go to a new site requiring a new request.
It took seconds for the request to time out because the host tried all three records one after another, and gave up only when all three had failed.
But, uh, that's an uncommon situation.
At two months post test, those test servers were still getting some traffic.
No thank you, if there's an outage that needs a DNS update to resolve it, 5 to 15 minutes is much more reasonable.
You can use dnsmasq --min-cache-ttl= to set the minimum.
Unfortunately you have to recompile to have a minimum longer than 1h.
In the case of github's example the author is fixated on DNS where in reality the DNS entry is entry point into fastly's anycast CDN endpoints where the DNS is used to point into the general direction of the correct anycast entrypoint. Fastly's CTO did a great talk a few years ago about load balancing which addressed the DNS issues based on the actual data they have from the edges that service billions of requests.
TL;DR of the DNS portion of that talk is "use as low TTL as you can humanly get away with"
I think that doesn't really affect anything traffic-wise. Just a thought I had in mind reading the article.
Why are DNS records set with such low TTLs?
- Legacy load balancers left with default settings
- The urban legend that DNS-based load balancing depends on TTLs (it doesn’t - since Netscape Navigator, clients pick a random IP from a RR set, and transparently try another one if they can’t connect)
- Administrators wanting their changes to be applied immediately, because it may require less planning work.
- As a DNS or load balancer administator, your duty is to efficiently deploy the configuration people ask, not to make websites and services fast.
- Low TTLs give peace of mind.
- People initially use low TTLs for testing, and forget to crank them up later.
Why is this not valid?
If your DNS hosting provider charged you per query (some do, especially when adding features like health checks & load balancing), then it might make a big difference.
...
Please tell me you're not testing in production.
If a load balancer or DC fails we need to ensure traffic moves away fast. Similarly if you want to take a system out for maintenance or perform migrations.
Well, if that's the case, you better have your redundant systems on your normal DNS entries, because there is no chance you will distribute new entries over the internet in 5 minutes, whatever value you specify at the TTL.
Easily 90% of new connections will move following the TTL. Of course, some traffic got a DNS result once in 2003 and is going to use that forever. If it's important traffic, you can trace it and follow up with them. If not, you do the best you can and let the rest go.
Then you were supposed to update it to a longer TTL when your change had propagated.
So, I guess, after understanding things better... there is no use case really since if it's a new record your change will always propagate, and if it's an old record, lowering the TTL on update doesn't really matter since the old TTL will still be in effect.
It's still no guarantee all changes will propagate within 5 minutes. But it gives some ease of mind to know the bulk of change won't take a day.
Also a lot of people forget the negative caching of NXDOMAIN records which is set by the TTL of the SOA record. Which means that it will take a while for your new record to be resolved if you started querying before you set the record.
Especially it gives you peace of mind that should stuff go badly wrong you can easily revert the change.
Does it actually make a difference? I dunno but it just feels right.
Eg. Five days from now, I'm going to make an infrastructure change that will affect public DNS. Today, I lower my TTL on affected records. Five days later I make the public changes, still using the lower TTL. Once I am satisfied my change will stay in production, I modify TTLs to be more sane.
One would probably be OK if it was 5 or 10 mins, but it depends on what's behind that dns entry and how often ISP can change the IP.
In our case, primary is AWS with a protection service in front of it, and secondary is our own servers at a data center. So something like VRRP wouldn't work.
That’s what is missing from this discussion.
Indeed the root DNS servers are not a single server but pools of geographically distributed servers via anycast.
Every request that comes from the same network is going to be routed the same way. Anycast works great for regional load balancing in general, but it doesn't work for subdividing individual networks.
[1] Google Cloud networking in depth: Cloud Load Balancing desconstructed - https://cloud.google.com/blog/products/networking/google-clo...
[2] What is AWS Global Accelerator: https://docs.aws.amazon.com/global-accelerator/latest/dg/wha...
[3] Tumblr: Hashing Your Way To Handling 23,000 Blog Requests Per Second: http://highscalability.com/blog/2014/8/4/tumblr-hashing-your...
[4] Load Balancing without Load Balancers: https://blog.cloudflare.com/cloudflares-architecture-elimina...
The issue is that you can't do percentage based routing with anycast... in fact, you can ONLY do shortest hop routing with anycast (at least for WAN anycast). That means that, while different edge networks can go to a different datacenter, every individual edge network will hit only a single datacenter.
The key issue is that anycast is a very blunt tool. You are relying on your BGP announcements to route traffic, but you aren't actually in control of where a particular request goes.
Determinism isn't necessarily required either. Probabilistic shifting works fine mostly.
Honestly, the only way I would see this resolved is if Google demoted sites with low TTLs in SERP ranking, but they're no saints either (I can see a 5 minute TTL for google.com over here).
But if I understand it correctly, you can point a CNAME with a long TTL to the appropriate cloudfront.net record, and then you only pay for the CNAME one. The cloudfront.net lookup will not cost you anything. But the latency for your users will be worse because it adds a lookup (because an ALIAS record gets resolved without a lookup).
https://aws.amazon.com/route53/pricing/
"DNS queries are free when both of the following are true:
The domain or subdomain name (example.com or acme.example.com) and the record type (A) in the query match an alias record.
The alias target is an AWS resource other than another Route 53 record."
Sure but if it can connect but then pukes out on something like a bad ssl or broken app, it’s not going back and trying another host.
So, when using dns for load balancing, it’s preferable to have a low ttl with a dns record tied to a host health check. If a host goes unhealthy it takes itself out of rotation, auto scaling brings a new one in, and it’s fully warmed up in a minute.
Another usage is to load balance out a lot of users to different web nodes for instance.
Edit: spelling
This is probably the cheapest and best solution available for improving DNS related UX issues, and is likely to be something where a commercial DNS provider might do well.
It's everyone below that that don't operate very large scale services that will see the benefits from longer TTLs.