Hacker News Reader

Comment by Kudos | Hacker News Reader

> I’m not including “for failover” in that list. In today’s architectures, DNS is not used for failover any more.

I mean, my company does this for certain failure scenarios involving our CDNs. Can anyone tell me why we're idiots, or is this just hyperbole?

View on HN

This is very common (Dynect, NS1, AWS, GCP, etc all depend on this for monitoring and failover). The author is an incorrect.

amazon.com, reddit.com, facebook.com, and others use low TTLs on their domains for this reason. Anyone who can't maintain an anycast infrastructure around the world and doesn't want to depend on Cloudflare will use this method.

For literally my entire career in SRE, well over a decade now, I’ve only interacted with systems that use DNS for this purpose, from small shops to parts of every HN reader’s life. That sentence in your quote, as well as the assertive nature of the post on such a weak foundation, are sufficient to disqualify a hiring candidate on account of lack of experience despite the authored software presented. It simply does not align with reality when presented with two logically separate networks and a required mechanism to transition between them.

The only other alternative for that scenario is using anycast addressing, and that has a colorful bag of limitations that are quite different from those of low-TTL DNS (including being out of reach for most).

DNS failover is used extensively, especially in the cloud world.

I see nothing wrong with using low DNS TTLs for failover - really don't understand the author's objections here, and them claiming that "DNS is not used for failover any more" significantly discredits them, IMO.

> I mean, my company does this for certain failure scenarios involving our CDNs. Can anyone tell me why we're idiots, or is this just hyperbole?

I came here to say exactly that. Our company uses DNS entries with low TTL for failover and load balancing purposes as well -- it's a very common approach. Services like AWS Route 53 and CloudFlare make it very easy to setup and low cost. I was surprised that the author didn't give much acknowledgement to this type of usage.

Don't worry many companies use DNS for failover (including Amazon).

You aren't idiots if you're using it where there are no better alternatives - it's preferable to use load balancers etc where available, but there are places where it's very much "DNS or nothing".

How would I use a load balancer to fail traffic between, say, London and Amsterdam with no fiber in place between them? Where would the load balancer physically exist in that scenario and how would it fail to the other when power is lost in one location? Would I make a third PoP to isolate it? What would then be my redundancy story for that PoP? How would I relocate traffic to my backup load balancer PoP number four?

Within a single network, sure, load balance all you want. That’s not the scenario low TTLs go after.

Some folks are using anycast to accomplish this, but that involves a different set of problems.

> How would I use a load balancer to fail traffic between, say, London and Amsterdam with no fiber in place between them?

What people use in those sitations is Anycast.

Of course, DNS itself, or e-mail, don’t need this kind of redundancy, since the NS (or MX) records themselves provides a list of failover servers. The corresponding alternative for HTTP, SRV records, has been consistently stonewalled by standard writers for HTTP/2, QUIC, etc.

There is an interesting draft RFC which I am keeping an eye on, but I don’t want to get my hopes up:

https://tools.ietf.org/html/draft-nygren-dnsop-svcb-httpssvc...

> What people use in those sitations is Anycast.

This requires that you blow a publicly-advertisable prefix for every unique combination of services you would want to fail-over.

E.g., if you wanted to be able to have independent fail-over between your customer-facing self-service portal and your webmail interface (each relying on specific state that you can't replicate synchronously, and can't guarantee replicate consistently with each other), you would need to /24s, one dedicated to anycast for the webmail interface, one for the self-service portal, and separate from any services which are active-active.

Whereas using DNS, you could use your other existing public /24s that you are already using for your active-active services.

In the last days of IPv4, an extra 2 /24s just for this is quite an expense.

Within a region, Anycast is how many big companies move things around mostly seamlessly. Why inside a region? If RIPE or ARIN catch you advertising their IP in the other's territory, they will send nasty emails and threaten to take back your CIDR blocks. I have no idea if they will follow through as we always stopped violating their rules when told to.

Outside of a region, they use DNS and sometimes a combination of WAN accelerators and VPN's, dark fiber.

Reply on news.ycombinator.com