Ask HN: DNS redundancy, how to do it right? | Hacker News Reader

Ask HN: DNS redundancy, how to do it right?

42 pointsleo829 comments

I hope I'm doing this right, since it's my first submission. It is a question directed to sysadmins of HN:

How do I reach nameserver redundancy?

Right now our provider is getting DdoS'ed, so my employer is not reachable by mail, web etc. If I do a whois on the affected domain, I'll get multiple nameservers (which the provider owns).

Looks like this:

nserver ns01.provider.tld nserver ns02.provider.tld nserver ns03.provider.tld nserver ns04.provider.tld nserver ns05.provider.tld

Actually two questions arise from this:

- Is it a good idea to setup my own nameserver which basically just "copies" the entries from my current provider and specify it (wherever that may be). By doing this I won't have to maintain 2 different NS, only the one from the provider since the 'secondary' will simply be a copy of the primary?

- Is it a good idea to simply increase the TTL of the important A/MX-Records? Will for example, 1.1.1.1 still resolve my domain correctly, even if my providers nameserver is down for an hour? (assumed I have a TTL of 3 hours for example)

Thankfully, I'm not the CTO, but since he mentioned to me that this happens regularly to the provider (being DdoSed), it got me really curious what the right mitigation to being unreachable is.

View on HN

My $dayjob is at a domain registry operator.

Find out if your provider will allow you to add your own nameservers and allow zone transfers to them. Most will but find out because you REALLY don't want to synchronize changes manually.

You don't need anything fancy, add one or two of your own nameservers. Something at Hetner, OVH, DO, AWS etc are all fine. You only need small basic Linux box with Knot, NSD, or Bind installed and a gig or 2 of memory.

Don't worry about Powerdns if you only have a couple domains. Its not worth the extra setup for a secondary then.

Make sure you do not allow recursive queries to the rest of the world (Bind) and make sure you turn on the rate limiting to be safe. As a first step that will really help. Obviously longer term you want move the nameservers away from your current provider and either outsource the management or set up the primary yourself. The general rule we recommend is at least 3 nameservers preferably on multiple continents.

This setup is very robust and will stop everything short of a serious DDOS attack. If that's a real concern then you need to outsource to a specialist. I like and have used netnod.se and Packet Clearing house (https://www.pch.net/) but they are very much not free.

If you are going to do all the DNS yourself then Powerdns is great, but get someone with DNS skills locally to give you real advice.

I gave this a +1, although I have some reservations. As long as they take care of the basics, it's a grind but they can "learn on the job" without putting themselves or others in harm's way.

Being an open recursor isn't as bad as being an open (email) relay. They'll learn about Response Rate Limiting the first time they get used in a reflection attack. And so on.

If they install a web interface to manage DNS all bets are off (what manages the management interface?).

The Basics:

* Having a strong allergic reaction to anyone using the words "simply" or "just".

* Setting up SSH with at least two of a password, a key, IP address restrictions.

* Not allowing dynamic updates until you know what you're doing with security and access controls.

* Keeping patches / software up-to-date.

* Editing zone files / configs by hand.

If they can do those things, then the machine will only be listening on two ports (53 and whatever they put SSH on).

The default for BIND is to only allow recursion on local networks, not just for anyone. I'd be very surprised if other popular DNS software simply openly recurses for anyone.

Unless it is your core business, you pay companies like Cloudflare to do it for you.

If it is your core business, but you are also big enough: you also pay Cloudflare or companies like them to do it for you.

If you are in between those two: as long as your name servers can be found, they can also be DoS'ed. But they can't be found, they also can't resolve anything. And now you're getting into the true problem: the bigger pipe tends to win, and if the bigger pipe has more origin ASNs and IPs to bug you with, individually black holing them won't be feasible either. So now you need to have a 'bigger pipe' and that's not something most companies want or can invest in.

However there is value in outsourcing to two companies, and putting nameservers from both in your NS record. That way if one of them suffers a service outage for whatever reason, your users won't even notice.

Yep, we also have two. Not a DNS expert but wouldn't half the clients (or their resolvers) stop working if one of your two providers are down? But you can recover by removing the provider that is down from your NS, assuming that the TTL is not too long.

Generally, recursive resolvers will retry queries multiple times and use all the NS records as needed.

You shouldn't have a broken NS record if you can avoid it, and it may add a little bit of delay, but things will still work.

Some recursive resolvers also keep statistics about authoritative servers, and when a domain had multiple NS records, the recursive resolver will prefer the option that has had the best response time in the recent past. That helps hide broken servers.

> Generally, recursive resolvers will retry queries multiple times and use all the NS records as needed.

IIRC some recursive resolvers ask all of them simultaneously and then just pass the first request received.

At $oldjob, we once got an extortion email of dubious credibility from an unidentifiable party, claiming they would DDoS our (a lucrative commercial web enterprise with lots of daily users) infrastructure, if we refused to pay 100 BTC (the BTC<->USD rate was a lot lower back then than what it is now ;)). As the infrastructure lead, I used this as an opportunity (albeit in a bit of a hurry ;)) to strengthen our resilience against this kind of threat, also on the DNS level.

The pair of authoritative nameservers, which we were self-hosting at the colo space we rented, was based on PowerDNS, with a replicated PostgreSQL database behind it. A "shadow master" postgresql instance was were control over zone data could be exercised, and that postgres instance used streaming replication to shuttle its dataset to read-only secondaries over a purpose-specific SSH tunnel (nowadays, we'd probably be using wireguard instead). There, powerdns authoritative DNS server instances picked up the zone data from the host-local postgres databases, and served that up by means of DNS.

This setup proved very easily extensible (spinning up a new, additional secondary was a matter of a few minutes via a simple ansible playbook that set up a new SSH tunnel, a postgres hot standby, and a new pdns instance that drew its zone data from the local postgres instance), and we chose to deploy two additional nameservers at dedicated server providers in nearby Europe to host our tertiary and quaternary authoritative DNS servers. The only remaining, but tedious task left was updating all the glue records for the domains we handled on these nameservers.

In the end, the entire threat proved hollow, as the deadline passed with zero impact on any of our infra. We never learnt if it was just empty to begin with, or if the adversary decided not to bother attacking a visibly well-prepared site. But the resilience-improved DNS infrastructure was a nice thing to be able to rely on in the coming years, and I think pretty much the same architecture/setup is still in operation to this day.

We have similar setup (MySQL because version upgrades are less disruptive) and I can confirm it is basically bulletproof.

Use a large provider's DNS. I can't suggest supporting CloudFlare because of their dubious selective enforcement of ToS regarding sites that actively organize and encourage harm to people, but there are much lighter weight alternatives.

AWS Route53 is basically free and doesn't require you to use any other AWS services, the same is probably true for any other cloud provider. Smaller providers like DigitalOcean / Linode / whatever should also be fine - I use DO for personal stuff, but would happily use it for larger capacity projects. Many cloud provider DNS APIs are supported in terraform, so you don't have to worry about what the UI is like.

Your DNS registrar also probably offers this service.

I will say this: if your provider is unable to protect their own DNS service, you should find a better provider. While CloudFlare and other similar services have incredibly resilient DNS, most folks don't need that. Anyone who is in the business of hosting online services should be capable of running a resilient DNS service. If not, you have to ask yourself how resilient anything else they offer you is.

While there are lots of ways to screw it up, DNS is incredibly simple compared to basically any other service, reliability-wise. BIND on any reasonable hardware can handle katrillions of queries.

If all of your records are traditional static records, DNS redundancy is pretty simple. DNS includes AXFR for secondary servers to pull the zone data from the primary. You can set up your own primary and use commercial services as secondary, or use one service as primary and another as secondary, etc. If the primary is ddosed, you might not be able to make updates if the secondaries can't connect for axfr, but often that happens on different ips than the public service ips, so there's a chance it still works.

But, it gets a lot trickier if you use any sort of DNS based load balancing or other trickery. Then you've got to set up both services as close to the same as you can and cross your fingers; there's tools out there for that, I think terraform can do it, but there's more focused tools as well. This is a good practice, but it's hard, so it's usually not done.

Top tier DNS services rarely get (succesfully) DDoSed or have other service outages, but it does happen.

Hi. I'm with Packet Clearing House, which @elp mentioned. I would second all of their technical advice, but note that PCH is a public-benefit non-profit, so exists to provide service at no cost to governments (ccTLDs) and critical infrastructure operators (mostly IXPs and CERTs) but, as required by the IRS, charges market rate to for-profit private-benefit organizations.

A few additional notes:

- You should keep what's un-politically-correctly generally referred to as a "hidden master" for your zone data on a machine that's somewhere that won't be targeted by a DDoS that's aimed at you or your ISP, and have an ACL that only permits zone transfers to your authorized secondary authoritative servers.

- You should probably get a few other organizations to act as public-facing authoritative servers for you, so all your authoritatives don't share any avoidable common failure modes. Different people administering them, different technology stacks on different hardware in different places.

- For servers you run, consider running DNSdist in front of them. It's a DNS load balancer which has very efficient internal caching, and which will allow you to answer a lot more queries per core than a full-fledged nameserver would. Run it in front, even on the same machine, to get more bang for your buck.

- A high TTL will indeed help a lot with DDoS against your nameservers (since everyone will cache answers rather than being dependent on getting a live connection to your nameservers. But it will also make you less nimble in responding to a DDoS against your actual content servers, since you won't be able to move them quickly to a different provider. I tend to favor high TTLs, but reasonable people support both sides of that argument.

Are you getting "DDoS'ed" from the DNS Server or from the HTTP Server?

Usually the later is more common since DNS tends to be a quite robust software to handle some levels of heavy the traffic (and the protocol being _lighter_ than HTTP).

Anyway you should separate the DNS Server from the Web Server, for this particular case personally i recommend "DNS Cloud providers" like AWS Route 53, they give you by default 4 different geo-located points and provides an API if you want to fight back the DDoS'ers (by changing your DNS records to 127.0.0.1 or 255.255.255.255 for a short period of time), usually these solve the email issue.

As for the web server, this is tough a well placed DDoS won't be stopped even Cloudflare have been hit with huge attacks that they couldn't handle (despite what their PR department says, the fact that even a bigger network than Cloudflare like Akamai couldn't protect Brian Kerbs tells you a lot about how tough these space is), best way is null routing the bad actors and spreading different ways to access your services like asking customers to go to frontendXYZ.mydomain.tld.

"Right now our provider is getting DdoS'ed, so my employer is not reachable by mail, web etc."

"our" here might suggest that your provider is also your employer's provider, and that your employer is not reachable in general by anyone by mail, web, et cetera. But reading the rest of your message makes me think that perhaps you're saying that your personal provider is being attacked, not your employer's, and therefore you can't reach your employer's mail, web, et cetera. Is that the case?

Is the attack just taking out your provider's DNS servers? If so, then just run your own recursive resolver. It's literally as easy as setting up BIND on any machine on your network with a default configuration file that does the barest minimum. Clients on the same subnet will be able to query it without problems.

There's no reason, nor advantage, to running anything using your provider's DNS servers when you have your own, particularly when their DNS servers can be taken down so easily.

The general advice is that you should have more than 1 name server and they should be on different networks/servers, to prevent the exact issue you're suffering, that a DDoS on one network doesn't cause an entire namespace outage. You are right to think about increasing the TTL of anything that's static and constant, like MX records. That TTL value buys time when there are issues before it is propagated to hosts that need to query your records as it's not cached at their end. Another trick, is to have the authoritative server where you perform the updates not referenced in your domains NS records, and instead only list the secondary (replica) servers in your NS records. So you maintain control and any attack based on NS records is on the replicas, and they refetch the record from the authoritative server periodically (based on SOA record settings).

You want multiple NS records containing hostnames of nameservers from two or more providers on independent infrastructure (i.e. not two providers that are both hosted on, for example, AWS' compute).

I would advise against running your own nameserver unless you have confidence in your ability to operate it correctly.

You can increase the TTLs if you don't anticipate record data changing frequently, or are able to tolerate delays in your DNS record changes being served (until the cached answers expire).

Choice of resolver (e.g. 1.1.1.1, 8.8.4.4) is out of your control (except of course, on your own devices and machines). Increasing the TTLs may improve robustness, assuming that your clients' resolvers are well-behaved and respect TTLs [0].

[0] https://www.ctrl.blog/entry/dns-client-ttl.html

You can have multiple secondary servers, what is not a good idea is to have just one, because whatever it happens with it things may fail elsewhere (i.e. mails lost, that may be worse than losing connectivity with your website for a while). And once the zone is setup and the secondary working, you only need to modify things in the primary.

Regarding TTLs, how frequently you modify those records, or how probable is that you have to modify them in a near future? Those times will say to resolvers for much time they could cache that information, and give a time for propagation of a change in that information. With current bandwidths and having slaves you can have relatively frequent updates, but have some margin to have meaningful caches in remote resolvers to speed up access to your sites.

> Will for example, 1.1.1.1 still resolve my domain correctly, even if my providers nameserver is down for an hour?

1.1.1.1 is virtually impossible to DDoS, because it’s anycasted in _a lot_ of places, and Cloudflare has the capacity to mitigate the largest of DDoS attacks.

> - Is it a good idea to setup my own nameserver which basically just "copies" the entries from my current provider and specify it (wherever that may be). By doing this I won't have to maintain 2 different NS, only the one from the provider since the 'secondary' will simply be a copy of the primary?

This would be a caching NS. It's not a bad idea, especially if it can automatically forward to a different service provided one is down, but you might as well just use 1.1.1.1. I've never seen 1.1.1.1 down.

Actually this is a much better idea than hosting my own namserver.

Yes. The most important thing here is that 1.1.1.1 is announced in seriously a lot of places. I just went to Hurricane Electric's looking glass, several times picked 3 routers at random and put in 1.1.1.1 as the destination. All of them returned it as the 2nd hop, meaning it's most likely announced in the same exact datacenter as where HEs router is located.

That gives you all the redundancy you'll ever need. Of course, you need to rely on Cloudflare to not mess it up. But you're not going to do better than CF.

I've seen some do this "stealth secondary" architecture before and it's helpful for local zones. Sometimes they make the local recursive resolver be a secondary of the local zone, but don't include the recursive resolver in the NS set of the zone. That way the recursive resolver has a full copy of local zones for responses and doesn't have to ask an authoritative server (technically it is one, but only known to hosts that use it as their local recursive resolver). This only works if you don't use public resolvers.

In this case, the vendor probably needs to be replaced. That's the easy fix. The higher TTL will help as it won't be querying them, but then making changes will be a pain in the future, until unless you pre-plan.

These are some things I am have though about for managing DNS. Any other recommendations would be great.

1) Separate Registrar from the source of the DNS records

2) Multiple NS records on the registrar

3) Multiple vendors for DNS records with Terraform sync between the two vendors

4) Limit A records or point A Records to CNAMES

5) Low TTL times for NS records, risky if you are hacked though or make a bad change

Simple answer:

1) Spend money.

2) Get a different provider.

Beyond that, get free help or advice. I guess that's what you're attempting now.

Let's start with spending money. You need to convince somebody that it's worth it. (Know your assets, know your risks, etc.) If it's not worth it or it's killing the company, and they won't do anything about it except to send you out to beg for help: get another job. Seriously.

As for the provider and "DOS": put their balls in a vice. What kind of DOS, exactly? What mitigations do they employ, exactly? Are they visibly seeking advice and assistance? Where's their outage page? I (charitably) assume you're being coy because you don't know... not because you're just being coy. Get the facts; cache la poudre; name and shame them.

Who else is affected? Band together, share notes and intelligence. Openly. Fully. Go read some of the DNS server mailing lists and dns-ops. If you can't swim in those waters, go home. Hide, and hope they go away.

Most of the answers here are akin to poking a dead beached whale: "smells bad!" "look, there's its liver!" "that's the blowhole, that's how it breathes" "looks like a propellor strike": factual, but not gonna help the whale.

I'm baffled by the premise of your question: exactly how does this lead to needing to do "redundancy" correctly? Is the provider not doing it correctly? No evidence is provided to support the assertion.

Reachability and services: there are a lot of tactical measures depending on how services are hosted. Mirroring the domain is a tool in the arsenal, depending on your line of business and communications needs (monitoring a SCADA system for emails to make sure the nuke doesn't melt down is different from some rando wanting to return a party dress).

"multiple nameservers": anycast is a thing.

Ummm... mirroring the domain ("just 'copies'": WTF?) IS maintaining an NS.

I'm not going to assume anything about your TTL. Name the domain. Tell me the TTL. Let me confirm it. ("simply increase the TTL": WTF? I think "simple" is the important word there. None of this is simple.)

m3047 | FWM6, internet plumber

Pay someone.

But if you really want. PowerDNS + MySQL replication for the "slaves" then put slaves in few different places with good bandwidth.

Instant updates, very bulletproof.

Is it a good idea to setup my own nameserver which basically just "copies" the entries from my current provider and specify it (wherever that may be). By doing this I won't have to maintain 2 different NS, only the one from the provider since the 'secondary' will simply be a copy of the primary?

Do you mean authoritative secondary replicas? If so, that is not uncommon. If your DNS provider is being targeted and your company is not then your DNS servers will still respond and a percentage of clients will try them. While root servers allow 10 records, anything beyond 4 will become less useful as different resolvers cap the number of NS records they will try. You can look up which OS/resolver has what behavior and then make a decision based on what OS most of your customers use. Amazon for example uses batches of 5 anycast records. If your commercial DNS provider is also Anycast, theny one could use 2 of those records and 2 of your own company hosted DNS just fine.

Look into how your commercial DNS provider handles zone transfers then set up a couple decent servers that uptake all the zones you want redundancy for. Just know there is no concept of priority meaning what order they are listed in the root servers does not matter. Whatever servers you set up will need to take a percentage of the traffic your commercial provider is absorbing. If doing this on a VPS provider I would suggest Vultr as they support Anycast meaning you can spin up many VM's to handle the load and still only have a couple public IP addresses without any load-balancer bottlenecks.

Is it a good idea to simply increase the TTL of the important A/MX-Records? Will for example, 1.1.1.1 still resolve my domain correctly, even if my providers nameserver is down for an hour? (assumed I have a TTL of 3 hours for example)

There are pros and cons to high TTL's depending on how your organization handles changes, failovers, etc... There are some discussions on the web about these pros and cons, too many to name here. It is also important to understand how clients actually cache high TTL's. For example, some clients will cap NS TTL to 86400 seconds regardless of how high they are and some clients will cap A TTL to 1 or 3 days. Then there is the factor of recursive server memory and end-users. ISP caches will expire records much faster regardless of TTL due to memory pressure. Each ISP and public DNS server handles this a little differently. So a high TTL can sometimes help assuming your infrastructure does not depend on being able to fail over things fast and that you are not planning on changing MX end-points. This requires some foresight into how one architects their infrastructure to fully recognize the benefits from higher TTL without incurring operational risk.

I am testing 1.1.1.1 right now and it took many requests to finally get my records cached on all their nodes, so if your domain is popular enough they may be useful.

I suppose that was a long-winded way of saying, "It depends". You should meet with your infrastructure team and think through what systems depend on having a low TTL and keep those low. For anything else a higher TTL is probably fine.

[Edit] It sounds like maybe you were just asking about recursive servers so most of this doesn't even apply.

Some places to browse for more detailed answers would be StackExchange [1] ServerFault [2] SuperUser [3] Just be sure to lurk a long time before asking questions. They are particular about how questions are formatted, how on-topic they are for the particular forum and if one has done an exhaustive search for existing answers.

[1] - https://unix.stackexchange.com/

[2] - https://serverfault.com/

[3] - https://superuser.com/

Thanks a lot for the detailed answer. I think having a higher TTL + using 1.1.1.1 as nameserver would be a good idea. I'll let the 'infrastructure team' know about this suggestion

If using the public DNS servers to resolve things then certainly have a few of them. 1.1.1.1 and 8.8.4.4 in case CF has problems.

[Edit] Testing 1.1.1.1 it seems I have finally built cache on all their nodes. It took about 25 to 30 requests. Second test took only a few requests for a different record, same domain. Now I am more curious about their back-end.

If you have your own recursive servers in your datacenter then you can entirely control how many things are cached and how that cache behaves. Unbound is a really good option for this as it is fast, has controls around memory, threads, min/max TTL and you could even push your authoritative zones to the edge Unbound nodes if desired so that those records never expire. Some people take this a step further in their datacenter and have Unbound running on every instance to keep response latency low and handle upstream recursive fail-over better than the OS resolver does.

Another benefit to running your own caching servers is that you can purge records that you know are out of date during outages.

> Right now our provider is getting DdoS'ed, so my employer is not reachable.

> It got me really curious what the right mitigation to being unreachable is.

Sometimes what can be done technically and the best solution are not always the same.

Is there a reason that you can’t simply move to a better provider without resorting to more advanced DNS infrastructure?

Cloudflare for example is sort of the gold standard for DDOS protection and they also offer excellent DNS services.

There are other major DNS service providers with excellent reputations as well, aws, dnsmadeeasy comes to mind.

Reply on news.ycombinator.com