Hacker News Reader

Hacker News

Comment by JimWestergren | Hacker News Reader

Full thread

JimWestergren

What is the use cases for having the TTL shorter than 5 minutes?

View on HN

The article posits why:

Why are DNS records set with such low TTLs?

- Legacy load balancers left with default settings

- The urban legend that DNS-based load balancing depends on TTLs (it doesn’t - since Netscape Navigator, clients pick a random IP from a RR set, and transparently try another one if they can’t connect)

- Administrators wanting their changes to be applied immediately, because it may require less planning work.

- As a DNS or load balancer administator, your duty is to efficiently deploy the configuration people ask, not to make websites and services fast.

- Low TTLs give peace of mind.

- People initially use low TTLs for testing, and forget to crank them up later.

But those are no valid use cases which was my question. So there are no valid use cases at all?

> Administrators wanting their changes to be applied immediately, because it may require less planning work.

Why is this not valid?

But 5 minutes should be fine? It surprised me that so many has 1 second or 20 second TTL.

If I need to get a web service up and I can save 4 minutes by setting a low TTL when I configure my DNS record why wouldn't I?

Because you're pushing the cost on to someone else.

If your DNS hosting provider charged you per query (some do, especially when adding features like health checks & load balancing), then it might make a big difference.

Valid migration plan should be able to handle it without very short TTL.

A low DNS TTL for testing purposes is a valid use case.

This is about production systems of large enterprises.

...

Please tell me you're not testing in production.

Found an earlier HN discussion about this question from November 2019: https://news.ycombinator.com/item?id=21437160

The major cases revolve around failure recovery, and traffic distribution. A 5 minute outage is not acceptable in many industries or at scale.

If a load balancer or DC fails we need to ensure traffic moves away fast. Similarly if you want to take a system out for maintenance or perform migrations.

> A 5 minute outage is not acceptable in many industries or at scale.

Well, if that's the case, you better have your redundant systems on your normal DNS entries, because there is no chance you will distribute new entries over the internet in 5 minutes, whatever value you specify at the TTL.

There are tons of things that don't follow TTLs, but a large majority of normal people traffic does.

Easily 90% of new connections will move following the TTL. Of course, some traffic got a DNS result once in 2003 and is going to use that forever. If it's important traffic, you can trace it and follow up with them. If not, you do the best you can and let the rest go.

The DNS based load balancing isn’t a myth if you want to do any kind of load balancing that isn’t round robin. If you want to, say, send 10% of traffic to data center A and 90% to data center B, and you don’t want to use up 10 IPs to do that.

AWS application load balancers have records with a TTL of 60s. Presumably they are doing it because they want the flexibility to change the IP addresses or the number of IP addresses dynamically. Seems like a reasonable use case.

Maybe for those cases where one can get random IP from their ISP and has a e.g. hopto.org configured?

One would probably be OK if it was 5 or 10 mins, but it depends on what's behind that dns entry and how often ISP can change the IP.

Do any ISPs generally change dynamic IPs more often than modem/routers reboot?

Some ISPs do it on a daily schedule. I know that the German Telekom rotates customer IPs every night at 01:45 AM, because a friend of mine is with them and that's the time when he drops from the video conference for a minute (if we stick around that long).

DNS based network load balancing. If you have two data centers, and you want to be able to dynamically and deterministically shift load between them, you want a short TTL so you can control the percentage of traffic going to each data center.

If you want to deterministically shift load, you use routing, not DNS, to manage your load.

That’s what is missing from this discussion.

How would you use routing to balance load at that granularity?

Rather easily. There are routing protocols designed for such things. Far more reliable than trying to hijack DNS for load balancing.

Indeed the root DNS servers are not a single server but pools of geographically distributed servers via anycast.

Anycast doesn't support percentage based load balancing unless you control all the hops between client and server, which is almost never the case if you are serving the public.

Every request that comes from the same network is going to be routed the same way. Anycast works great for regional load balancing in general, but it doesn't work for subdividing individual networks.

Rather than get into a lot of details, here's some excellent starting points:

[1] Google Cloud networking in depth: Cloud Load Balancing desconstructed - https://cloud.google.com/blog/products/networking/google-clo...

[2] What is AWS Global Accelerator: https://docs.aws.amazon.com/global-accelerator/latest/dg/wha...

[3] Tumblr: Hashing Your Way To Handling 23,000 Blog Requests Per Second: http://highscalability.com/blog/2014/8/4/tumblr-hashing-your...

[4] Load Balancing without Load Balancers: https://blog.cloudflare.com/cloudflares-architecture-elimina...

Yes, I understand how anycast works (I work for an anycast based CDN).

The issue is that you can't do percentage based routing with anycast... in fact, you can ONLY do shortest hop routing with anycast (at least for WAN anycast). That means that, while different edge networks can go to a different datacenter, every individual edge network will hit only a single datacenter.

The key issue is that anycast is a very blunt tool. You are relying on your BGP announcements to route traffic, but you aren't actually in control of where a particular request goes.

Anycast/BGP traffic engineering is not nearly as accessible as DNS based load balancing. There's several DNS providers you can use to add load balancing and health checks on top of your existing hosting, wherever that is (and you can do it yourself too). Using anycast for this means a specialty hoster, a third party in the data path between your clients and servers, or running your own ASN. I'm sure that gets more precise results than DNS, but it's also a lot more work, and it's harder to replace if the providers involved stop being a good fit.

Determinism isn't necessarily required either. Probabilistic shifting works fine mostly.

I always was under the impression people lowered the TTL when updating records... which doesn't even really make sense since the change won't propagate until the previous TTL is overrun anyways.

Then you were supposed to update it to a longer TTL when your change had propagated.

So, I guess, after understanding things better... there is no use case really since if it's a new record your change will always propagate, and if it's an old record, lowering the TTL on update doesn't really matter since the old TTL will still be in effect.

You could lower TTL if you know a change is coming.

Exactly. The TTL is lowered in advance of the maintenance window, sufficiently far out to allow any entries with the old TTL to expire from most resolvers. Once the maintenance has been completed and validated, and sufficient time has elapsed to decide there are no issues requiring roll back, the TTL is raised back up to its stable value.

It could help to lower your TTL way before you plan to update. If your current TTL was 24h, you just update it to 5m 24 hours before you plan the actual change of the record itself. The new record can them be set to 24h directly (unless you want a quick turnaround for rollback).

It's still no guarantee all changes will propagate within 5 minutes. But it gives some ease of mind to know the bulk of change won't take a day.

Also a lot of people forget the negative caching of NXDOMAIN records which is set by the TTL of the SOA record. Which means that it will take a while for your new record to be resolved if you started querying before you set the record.

> It's still no guarantee all changes will propagate within 5 minutes. But it gives some ease of mind to know the bulk of change won't take a day.

Especially it gives you peace of mind that should stuff go badly wrong you can easily revert the change.

Yes - it’s called planning. Lowering TTL well in advanced is not a new concept and works very well if you have enough advanced notice from the application owners ahead of time :p

Unfortunately, data centers rarely give 24 hour notice about catastrophic fires or provide advance notice about utility outages when the automatic transfer switch is going to fail.

I lower the try a few days before I’m going to make a big change. This is also how we did it in the past at a large corp I worked at.

Does it actually make a difference? I dunno but it just feels right.

You got the use case right, but I think you're still thinking about it wrong. In-use public DNS changes aren't something that just crop up and need to be done immediately.

Eg. Five days from now, I'm going to make an infrastructure change that will affect public DNS. Today, I lower my TTL on affected records. Five days later I make the public changes, still using the lower TTL. Once I am satisfied my change will stay in production, I modify TTLs to be more sane.

That makes a lot of sense, I've updated a lot of DNS records, but not generally in a planned manner.

In practice though other things come up and the admin doesn't get round to increasing the TTL back to sanity later because nothing is broken if they don't.

Sure but in a department where people are sysadminning reactively, they’re going to have that problem crop up everywhere. This type of thing wouldn’t be neglected in a well thought-out change process.

Windows Update. To reboot a server, you need to take it out of production. With a TTL of 5 minutes, it can take an hour for (nearly) all users to stop using that server.

Sorry - why would a 5m TTL take an hour to stop using? Shouldn't it be 5 minutes?

If things behaved nicely, yes. There's all sorts of weird DNS caching behaviour out there. It's not unusual to find folks with DNS servers / clients that are caching records for 1 hour+, and then of course there's people running super old versions of Java that used to cache DNS forever by default (before JDK 6). There's a very clear set of user that seem to cache for 10-15 minutes, regardless of any DNS TTL.

You can't fix systems that ignore your TTL by specifying lower TTL values.

Sure. My general approach is to use lower TTL values (~ 5 minutes) and just accept that if people do dumb things, they just have to put up with things randomly breaking unexpected.

Good grief- you do not need to reboot the server; just flush the cache https://www.dnsstuff.com/clear-flush-dns-server-cache-window...

They mean they're rebooting the server having the IP that's entered in DNS, not rebooting the client consuming that service.

We have a service that uses AWS Route53 health checks, and set the records to 60s TTL because if there is a problem at the primary service fails healthcheck, we want it to get the updated DNS records, which point to another data center, fairly quickly.

In our case, primary is AWS with a protection service in front of it, and secondary is our own servers at a data center. So something like VRRP wouldn't work.

Reply on news.ycombinator.com