Mammatus: Use DNS NS failover to create HA Web Service. | Hacker News Reader

Mammatus: Use DNS NS failover to create HA Web Service.

mammatus.thimbl.net

26 pointstricknik20 comments

The main issue with using DNS for failover is that the name servers of many major ISPs don't respect TTL values, particularly those under 3600 seconds.

As such, when a failover occurs a site will be unavailable for many viewers for several hours and possibly even days, making the solution anything but highly available.

Agreed. A better solution is to have all nameservers always return 2 A records in response to queries for a specific domain name. That way the browser will itself be responsible for trying to connect to each of the IPs to determine which one is alive (then with DNS pinning it will stick to the working one). And it does not matter if an intermediate DNS caching resolver ignores the TTL.

Of course the ultimate solution for HA is to implement anycast with BGP...

Hi mrb, this is just the first release. Mammatus will return 2 A records if the endpoint domain does, so that will be built in. Of course anycast is a great option if the organisation has the option of getting an ASN, running BGP, etc. If that is not an option, Mammatus is a reasonable alternative.

One ASN is not enough. Think of the "routing mistakes" caused by censorshop etc. Or if your IGP routing blows up….

So if you really need to archive 99.999999% you'll have to invest A LOT of money, time, people.

Oh good. I read the Mammatus overview and from the resolve=self option I was under the impression it was returning only 1 A record.

Right, there is also resolve=endpoint, i.e., the "Running Dog Lackey" configuration. And others will also be possible.

Hi forkqueue, DNS pinning is indeed an issue, the scope of which this experiment will help identify. However, this would only affect those with an existing record cached for an unavailable node, no new requests would be directed that node. Futher, a Mammatus cloud can also be used as a back-end service for another front-end service which can evade the cache by prepending a random string or timestamp to the subdomain, i.e. 123.mammatus.thimbl.net. Mammatus ignores subdomains of it's subdomain, so this works. Also, information from research into DNS rebinding attacks shows that browser based pinning drops it's cache when it gets a failed requests, so a re-request would also work after failing once. Mammatus is not meant to replace all other HA techiniques. Certainly for organizations with the appropriate budget, there are potentially more robust options for front-end systems. For many situations, this is a good technique, considering how inexpensive it is. Also, this same issue exists for systems like dynamic DNS, which remain quite popular in many use-cases.

The main issue with using DNS for failover is that the name servers of many major ISPs don't respect TTL values, particularly those under 3600 seconds.

I've seen this repeated in many places but never with any concrete examples. This leads me to wonder just how widespread the practice is — can you substantiate the claim?

I don't know much about ISP DNS configurations, but the problem extends to clients as well. For example, until Java 1.6, the default TTL for DNS lookups was forever.

That's beyond the scope of my question — my concern is solely resolvers extending TTLs.

Two examples I've seen recently are Road Runner in the US and Virgin Media in the UK. There are many, many more.

Just to be totally clear — you've explicitly done a lookup against their resolvers for a record that is not in flux and received a TTL higher than what the authoritative servers give out? Any chance you could post their name server addresses?

No, I've altered DNS and seen many thousands of connections continuing from these providers even after low value (300) TTL records have been changed several hours previously. I've observed this on a number of occasions.

In the case of Virgin Media, they have transparent proxy caching on some parts of the network I believe, so it's possible this caching was happening there rather than on the DNS servers themselves.

Well, not much that can be done about caching proxies. Regarding DNS, have you been able to look at whether the IP's from failed requests show up on the new IP shortly after? I.E perhaps browser-pinning from open sessions?

maybe it's about the defintion of high availability. If you need 99.999999% a DNS based solution will not work.

But even Google, Yahoo and others use a short TTL to switch between their datacenters… So this should be "reliable enough" for nearly everything out there.

$ dig www.yahoo.com

;; QUESTION SECTION:

;www.yahoo.com. IN A

;; ANSWER SECTION:

www.yahoo.com. 48 IN CNAME fp.wg1.b.yahoo.com.

fp.wg1.b.yahoo.com. 2711 IN CNAME eu- fp.wa1.b.yahoo.com.

eu-fp.wa1.b.yahoo.com. 33 IN A 87.248.122.122

also:

$ dig www.google.com

$ dig www.twitter.com

$ dig www.cnn.com

Here is a recent blog post we wrote: http://visualwebsiteoptimizer.com/split-testing-blog/maximum...

In a nutshell, IE doesn't respect TTL and caches DNS for 30 minutes.

Hi paraschopra. What does it do when the request using the cached DNS fails? In conversations with security experts researching DNS re-binding attacks, I've heard that pinning is dropped after one failed request, as techniques like sending multiple A records for a domain depend on this. Also see my comments to forkqueue.

In my experience it doesn't request DNS again if request fails. Waits till TTL expires. Multiple A records are useful if you have all servers exactly synchronized (as request can go to any server) plus on the moment of server failing it will still doesn't help as IE will keep reconnecting to same IP.

Though I haven't tested multiple A records.

Thanks, I'll look into this more. In the case of Mammatus response, the TTL would already be expired, so the only problem would be if it pinned for longer and didn't request again on failure.

highly interesting !!

Reply on news.ycombinator.com