Let's Make DNS Outage Suck Less | Hacker News Reader

Let's Make DNS Outage Suck Less

kvz.io

37 pointskvz14 comments

dnsmasq --interface=lo --all-servers --server=172.16.0.23 --server=8.8.8.8 --server=4.2.2.2

echo "nameserver 127.0.0.1" > /etc/resolv.conf

http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html

--all-servers : By default, when dnsmasq has more than one upstream server available, it will send queries to just one server. Setting this flag forces dnsmasq to send all queries to all available servers. The reply from the server which answers first will be returned to the original requester.

Dnsmasq also has avoidance of unresponsive servers built in, this is a bit more of a blunt instrument.

Thanks for the link! I did find dnrd (http://dnrd.sourceforge.net/), but I really wasn't looking into proxying DNS requests and introducing another daemon / dependency, that could at some point fail or get confused, requiring even more tech to keep it alive, or provide insight in it's workings/reasoning.

Really wanted to solve this using as transparent & archaic tech as possible in the hope this still stands when big player's resolving servers fall.

Granted, it is blunt; archaic tools often are ; )

The example you gave won't stick for most people:

    echo "nameserver 127.0.0.1" > /etc/resolv.conf

If you use dhcp3-client (chances are you do), add this line to /etc/dhcp/dhcplient.conf to try your local DNS cache first:

    prepend domain-name-servers 127.0.0.1;

Note that this will slow down stuff like captive portals at airports that want to push you to their "accept terms" page first.

I use djb's dnscache instead of dnsmasq, but dnsmasq works fine too.

http://cr.yp.to/djbdns/run-cache.html

You know, sometimes the simpler solutions work best. I like this one.

I'm sitting here contrasting this against my normal approach* and the approach here is:

    1) easier to explain to others
    2) self documenting
    3) just as effective as running a caching nameserver

* i have vm hosts configured in CM to have a bunch of promises applied, one of them is to run a caching nameserver, these hosts are the only ones allowed to do zone transfers. The vm instances running on top of these have a promise applied which has them use their underlying dom0 for dns queries.

If you use the rotate option with timeout you'll quickly hop to the next working resolver. http://edwin.io/optimized-resolv-conf

Do you have data you can share regarding "Amazon EC2 resolving nameserver (172.16.0.23) is unreachable too often" ?

You don't need to do rotation, either -- suppose the first nameserver, when up, is consistently faster than the others. Just setting a timeout of 1 is appropriate, then.

A timeout of 1 can result in false positives. 3 seconds is recommended according to the manpage. Should the occasional false positive be none of your concern, then you still add at least a second to every request. If you make many, that's many seconds.

This is more of a performance optimization and less about a HA solution. With rotation you also avoid being throttled. If that's not a concern based on the # of request you make then you can simply timeout and go to the next resolver on the list.

From the manpage: "[rotate] causes round robin selection of nameservers from among those listed". This means in your configuration, the broken nameserver will be hit once every 3 times, resulting in a 1second timeout before trying the other 2. So in every 9 requests your setup results in 3 seconds delay for as long as the nameserver is down.

In case of Amazon, you actually prefer their nameserver as it resolves names to LAN IPs where possible. With your strategy, ~66% of my instance hostnames would be resolved to the less efficient WAN IPs, that's with all 3 nameservers being reachable.

As for data, there's papertrail and Premium Support tickets with Amazon admitting to their downtimes. In most cases (e.g. the one on Jan 14, 2013 11:27 PM PST) "there was a problem with the underlying host".

Remember it doesn't even need to be the service itself going down, there's weaker links in this chain like misconfiguration of the virtual host. Cables can get tripped over. Anycast is not possible here.

I'm not blaming them. This article was about mitigating suckyness in case of outages, not proving that such a thing even exists : )

So there are 2 different things being addressed. There's a HA solution (handle failovers) and then there's operational performance. Introducing cron to manage this solution, means cron needs to be running and monitored at all times. Also worse case scenario if cron dies and you have a now down nameserver configured.

Agreed on aws I would use their resolver to request the internal ip. Another option is setting up your environment in VPC. With VPC you'll be able to assign a static internal ip to your instance.

btw I did find many reports on the 172.16.0.23 lookup issues.

We notice the outages because customers tell us to upload x to y.com. If that fails because of 'unable to resolve y.com', we get a support ticket. I guess for your average web 2.0 the effects are less harsh. Also as Amazon said, these outages can be local due to bad iron. It could have been we've been unlucky with that. At any rate, I was pressed to do something about it.

Good suggestions otherwise. I'm not sure what will happen with an RDS Multi-AZ failover if you address it by it's static VPC IP?

Also about your worst case scenario. It's true that I rely on cron to be working. While other proposed solutions rely on more obscure daemons to be up & running, there still is a risk, and it could be mitigated by just writing all the nameservers to the resolv.conf. I've created an issue for that: https://github.com/kvz/nsfailover/issues/1

They're using DNS for DB communication. That's a major faux pas.

Using IPs for communication to Amazon RDS instances is inadvisable. You'd have to use WAN addresses outside of EC2, and LAN addresses inside of EC2. Addressing them by domain resolves this. Additionally I don't think Amazon guarantees your box will be accessible on the same IP addresses after Multi-AZ failovers.

You're definitely doing it wrong.

Reply on news.ycombinator.com