Hacker News Reader

Comment by JohnTHaller | Hacker News Reader

All our eggs in that basket got crushed. Let's take this basket over here and put all our eggs in that instead. Facepalm

View on HN

It's probably temporary? Move to R53, then figure out what's needed to manage records on two providers (if only for internal processes). Top engineering teams aren't going to knee-jerk this, right? Or did Dyn show some unfixable incompetence?

Is it? I thought GitHub, at least, was split Dyn/Route53 shortly after the Dyn outage started as a means of getting back online. Now, they've removed their Dyn, and are now exclusively Route53.

I don't think Dyn showed any incompetence; the parent-poster was merely remarking on relying entirely on a single provider, who, if they get DDoS'd, causes your site goes down. (There was some previous discussion about splitting between providers, but some commenters noted that it was difficult, or at least non-trivial, to replicate records between two providers.)

The problem is that you need to find a DNS provider that allows master and slave configurations of your DNS information. For example, Dyn can act as a master and UltraDNS can act as a slave, however, Route53, you can't be either. With Route53, you are all in.

Lucky for Route53 users, Route53 DNS surface is really large and there is a really good chance that not even is attack could hurt it.

AXFR isn't the only way to sync records between providers. You just need a tool that speaks to the APIs of each provider and can sync between them that way. Heck, I had syncing in place at a startup between Route 53, DNS Made Easy, a pair of TinyDNS servers, and a git repo (which was our historical backup of changes) years ago. It was 300 lines of Python and 100 lines of shell. Albeit, we only had a few dozen or so records to manage, but this isn't rocket science.

Aside: I came out of college as a sys admin with a CS degree and writing tools like this was par for the course. If devops folks aren't writing tools like this today, what are they doing?

Honestly: I think they are spending most of their time moving existing working infrastructure into containerized infrastructure and figuring out how to deploy their blog on k8s. They are working on learning libraries that abstract abstractions.

To be fair Route 53 will split over other providers so you should be good in theory. But yeah if something more specifically targets Route 53 then that could be the same problem.

At least that's my understanding anyway.

So we now run a split view with DYN and AWS. My biggest issue with AWS is again they are a large attack surface, but also they don’t really play super nice with others and no DNSSec.

We are currently evaluating the Netflix denominator tools to spread DNS and sync our alternate providers.

My biggest problem during this outage was that I could not login to my registrar and make change to DNS directly - I had to login to DYN and ADD Route 53, it was impossible to remove DYN completely. And that's how we landed up with a split view.

NOW if anyone can tell me of a competitor for the Traffic Director product that works on Port 25 I'll be happy to consider a migration. Cloudflare has something in the works, but I’d really just like a DNS provider with a virtual load balancer that can handle my 250qps at a reasonable price.

You could build your own DNS set up for Traffic Director. Based upon the recursive server IP hitting you, send back responses that are closest to the user.

There is an RFC to pass information about the client subnet: https://tools.ietf.org/html/draft-ietf-dnsop-edns-client-sub...

Which Google DNS uses to tell your name servers more information about where the client is located. This allows you then direct them to the nearest server.

Reply on news.ycombinator.com