Hacker News Reader

We should do the right thing even if other people do the wrong thing.

Traffic signs shouldn't be taken down just because some people speed regardless of the posted speed limit.

Fundamentally however, giving clients control over when failover occurs means giving up control over failover.

Literally just a couple of hours ago, I was patiently explaining this to some Azure techs from Microsoft.

The conversation went like this:

> "Just use Traffic Manager for active-passive fail over!" they piped up, helpfully trying to solve my problem.

> "Client software often ignores DNS changes."

> "We designed Azure so that Traffic Manager does failover and load balancers are always active-active."

> "That's nice, but client software ignores DNS changes."

> "Umm..."

Fundamentally, the DNS protocol is just too old and crusty for the modern Internet, much like SMTP and FTP. It's best avoided to the extent possible.

The big providers seem to be converging on Anycast IP routing, where a single address can be routed to multiple data centres dynamically. You're all probably familiar with 8.8.8.8 and 1.1.1.1, both of which are Anycast IPs, but anyone can create a global static IP like that in minutes via a service such as Azure's Cross-region load balancer: https://learn.microsoft.com/en-us/azure/load-balancer/cross-...

> Fundamentally however, giving clients control over when failover occurs means giving up control over failover.

Fundamentally the problem is that you want to direct a client to particular datacenter and it is not always shortest part (shortest != lowest latency) so anycast DNS isn't helping here.

There is reason biggest CDNs use it to direct to particular DC instead of " just anycasting everything" like you're proposing, it's suboptimal.

> You're all probably familiar with 8.8.8.8 and 1.1.1.1,

And both companies behind that still direct you to particular datacenter, not anycasted IP, for the actual content.

Anycast IP are also "expensive", you need to use entire /24 route to get one, and it takes routing table space in every router's memory that is limited, and requires costly replacement if it runs out everywhere, not just where you distribute it.

Aren’t routing tables also cached (not by clients, but by ISPs)? What is their typical TTL?

It varies; for direct peering/connections where a link goes down (and fast failover protocol is used), usually 5 seconds or so. Global propagation takes in the order of 5 minutes or less: https://labs.ripe.net/author/vastur/the-shape-of-a-bgp-updat...

They aren't really cached. Routes are kept in the routing table as long as peers are responsive, if they stop responding to hello's or a link goes down to directly connected peer the routes are removed (or replaced with a less preferred one). While its possible to configure a router to keep routes in the table for a peer that is down for some time its typically not used unless you have a good reason for it.

They are added and withdrawn on demand, or dropped all at once if router loses connectivity with another.

So really it's processing time of every node in the chain. Few seconds to few minutes, not really that different from DNS.

The advantage is that there is no updates if there is no changes, instead of update every time TTL expires but that's about it.

routing tables usually aren't "cached" in the same sense that DNS caches entries.

however. Inside ISP;s routes are usually distributed by an IBGP with a couple of route reflectors, and route reflectors will drop routes if no router within the IBGP domain is advertising the route.

Routes between ISP's are usually fairly static thanks to mechanisms like graceful restart, which makes sure routes do not get dropped, even if BGP has a restart between routers. THere are various other mechanism in which to achieve this. (look up route flap dampening if you want to learn more).

The service I worked for in AWS faced this as a routine pain. No matter what the TTL, stuff completely ignored it. In the end the only sensible thing we could do was accept that we were going to impact customers and that we just can't realistically account for misbehaving DNS infrastructure. We had something like a 5 minute TTL on our public record, and would wait 15 minutes after removing the IP from the DNS records before we would take action that would potentially break connectivity.

By setting the DNS TTL, we had established the terms by which we offered the service. Failure to accept those terms is not something we can control.

What we generally found was that clients disconnected would reconnect almost immediately to one of the other IP addresses. Any disruption to the client appeared to be almost negligible.

It doesn’t help that Java’s default configuration is to cache DNS entries forever for “security”[0]. Any long running server that makes external requests will eventually hit an issue where one their dependencies no longer works because DNS has changed and the TTL is ignored. I’ve seen this bite teams for 15 years and it will probably keep doing so well into the future.

0 - https://docs.oracle.com/javase/7/docs/technotes/guides/net/p...

Only if you have a security manager enabled (=are running in an applet/webstart). The entire security manager concept has been deprecated, so things are looking better

.NET HTTP clients used to suffer from this, too, but they no longer do as of .Net Core 3, as long as they're used correctly.

I seem to recall that HAProxy and Squid used to do this too.

I'd say about 80% of traffic moves within 2, maybe 3 TTLs. That 20% of traffic though... I've never really seen traffic on old servers go to zero real looking traffic, even after waiting months. Sometimes there's a dropoff when you send RSTs to SYNs, but sometimes not.

If you're using a DNS service that charges per query, set the TTLs high! Otherwise, 60s is probably good enough for most failover. I wouldn't go too far under 30s, unless you have a good reason; some recursives do stupid things with low values, and why tempt fate. Otoh, a lot of high profile sites and services do have < 10s ttl, so it probably works?

From the CDN & authoritative name server side Ive mostly dealt with eyeballs and a smattering of devices. With ~1min TTLs itd be 80% in 5min, 95% in 20-30min, 99% at an hour, 99.9% 1day, the remainder entirely variable. Literally stopped announcing BGP prefixes for years, turn them back up, get immediate client traffic to 80 & 443. Have done down to 10s and lower and its fine (as far as we were concerned).

That's my experience exactly. About 80% move within an hour, and then the remaining 20% trickle in after that.

> "When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds"

Likely because the clients just weren't doing DNS lookups. I would often run into clients written to resolve a hostname and then query the resolved IP for the duration of the process (often this is done as part of a library or runtime). Our customers wouldn't restart their apps for weeks or months so there was always a long tail of clients which never picked up dns changes.

DNS wasn't designed for failovers. We can use it for that, but it's never going to be perfect by design.

Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way.

> Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way.

But you can't just suddenly make one IP address refers to another datacenter with another ISP in another country, can you?

You sure can. There are many details and caveats and it takes a lot more infrastructure and technical sophistication. Many small sites don't control their own routing at all, for example.

DNS is very simple, easy, and mostly good enough. And DNS failover is now well-enough known that we can mostly expect clients to accept fault if they don't engineer to expect it.

Yes, with BGP and anycast.

What you can't do with the above is more fine grain load balancing. If your LAX node is getting overloaded, you can use a combination of geo and DNS to start splitting some amount of traffic away.

via the magic of Anycast :)

>It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake.

IMO that is a little misguided. 60-second and higher TTL will be honored most of the time. And I don't really get what that ISP in Alaska was doing: DNS is about 0.1% of Internet traffic (according to some quick googling), so decreasing that will save you only a tiny tiny bit of bandwidth at disproportionate inconvenience for your customers.

Not sure about what kind of connection Alaska has but I imagine this is more about reducing latency for new connections than bandwidth.

When it comes to TTL I try to imagine the perspective of the person on the other side that implemented the resolver at a university (which copies might be anywhere) or isp a couple decades ago. What odd behavior did they have to manage and what quick hacks might they have done to fix them?

From this I have chosen a minimum of 5 minutes of TTL. Anything that might have caused a large amount of dns traffic in the early day of the internet to go off every minute, or worse ever few seconds, might killed their server and taught the young programmer to sanitize inputs.

>Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching.

That doesn't really matter. Once a server becomes saturated you just remove it from being returned. Having multiple records being returned just slows down how fast a server can become saturated. You don't need the multiple records to be saturated at the same rate.

I think you're misunderstanding. If I have a lookup for foo.com configured to return 1.2.3.4 half the time and 5.6.7.8 the other half of the time, I want half the traffic to go to each load balancer. Maybe I'm running different software on each, or behind each. Maybe I'm using different networking gear. There are a ton of cases where getting close to 50/50 is what I need to get good experimental data.

Why would you think that it doesn't matter that the traffic doesn't match the ratios I picked? I picked them for a reason.

I think charcircuit suggests returning the inverse of the actual measured traffic: if the target is 50/50 and backend A gets 70 while backend B gets 30, one solution could be to return A in only 30% of the requests and B in 70% of them, leading to an actual 50/50 distribution.

I am suggesting that the returned servers are the servers with the least load. Yes, there may be uneven load being assigned to these servers, but clients are being given the choice of selecting a server with low load.

I'm suggesting to use more horizontal scaling. So if you have 1.2.3.0/24 and you advertise 2 servers at a time you can start with 1.2.3.1 and 1.2.3.2 and then you would cycle out servers as they increase in load. If one server is getting less traffic then it just takes longer for it to cycle out.

People without cached records will always get the returned servers with the lowest load.

I was referring to load balancing and not splitting traffic for an experiment.

You're going to tend to have unevenness there because you're returning results to recursive DNS servers, not clients; so maybe you get unlucky and return A to more large ISPs; their recursive servers serve more clients per lookup and you've got lumpy results.

Then, if you return both IPs, you're very likely to eventually come across some people who sort multiple returned A/AAAA records and will prefer one that's "closer" to their current IP. Because if you're in 1.0.0.0/8, it's certainly going to be better to connect to 1.2.3.4 instead of 5.6.7.8. Same thing happens in v6 land, of course.

If get unlucky A's load will rise quickly and it will no longer have a record for it. As the TTL expires the resolvers will get a new set of servers with low load.

Lumpy results just means the times for an IP to be rotated out will be lumpy, and not that the server load will be lumpy.

You're suggesting that the DNS server(s) get realtime load feedback and adjust appropriately. That's possible, but not always available.

Yes, that is how you scale DNS load balancing. There is a pretty low limit for the number of records you are allowed to return in practice. Putting 100 A records works with 1.1.1.1, but will break other's resolvers on the internet.

>That's possible, but not always available.

If you build it, then it's always available.

Sure, but a lot of people use other people's DNS services. The number of times I've been allowed to build a DNS service for my employer is zero. I've used static DNS, where you just put in your X number of A records and hope; dynamic services where you put in X and have them serve only Y of them in any request and hope. I've used services with load feedback, but those are always a much higher tier.

>Sure, but a lot of people use other people's DNS services

There are many of DNS services which offer an API for querying and updating records.

It is very easy to write a service / script that just fines the X least loaded servers and then call an API to set those as the available records. In practice you will also want to implement some monitoring that it is actually working.

> At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high.

DigitalOcean does this, or at least some of the collocation/hosting providers they are using for their infra. They appear to redirect :53 connections to their own resolvers that cache NXDOMAINs.

It's an ugly situation. DNS seems like the wrong way to do load balancing. Meanwhile web browsers seem to have decided that it's a great idea to send out DNS requests for every domain on a web page (of which there are dozens, thanks to adtech, CDNs and other stuff.)

If you're in a corporate environment with control over the infrastructure, low DNS TTLs allow you to mostly guarantee a failover. There will always be clients that ignore TTLs, but most of them are fixed by IT rebooting the box.

As the poster says, when it comes to the public internet DNS TTLs are more like guidelines.

Luckily most client DNS caches are cleared on a reboot.

FYI, if I remember correctly Windows (and MacOS?) used to cache negative responses for "a while".

> It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake.

That's a weird conclusion. My conclusion would be that you can't force low TTLs for everyone or even most users to you don't can't use them to allow for quick changes and may as well let everyone else benefit from caching too.