Hacker News Reader

Comment by 6d6b73 | Hacker News Reader

All this talk about redundancy, real-time apps, scalable architecture and and a "simple" DDOS against DNS architecture brings half of the internet down. Honestly did nobody think about having a spare dns at some other company? or even backup dns server exactly for a situation like that?

View on HN

If ONLY they had 10X developers ...

EDIT: joking aside, the issue with multiple DNS providers is primarily (in my experience at the company I'm at having investigated this in the past) intelligent DNS entries. Example, 'return these A records, in this order, based on the number of requests, roughly balanced'.

There's no universal standard, just common aspects. DNS Provider A has one set of features, names for returning A records in some way, based on some weighted averaging, provider B has a different mechanism. So as an infrastructure person you have to:

1. Investigate provider A and B features for intelligent DNS (it's not even universally called intelligent DNS!), and mentally parse the commonalities and differences

2. Create a custom mechanism to keep them in sync internally. So you hope that A and B have an API for maintaining the records.

3. Ensure that when someone in your org wants to make an update, A and B update at the same time in the same way

4. APIs don't change.

In my experience Dyn pushes this functionality very hard during customer on boarding, effectively as a lock in for their platform.

Well I really think those features are useful, though. They let you get some powerful de-facto load balancing and HA and other complex 'features' without any new hardware or much complexity. They give you some easy creative freedom in designing a redundant infrastructure, give you some geo-aware DNS the likes of which I remember paying Akamai a lot of money for years go, etc. But the lock-in problem is absolutely right.

I'm also familiar with the aggressiveness of that sales team. I prefer another provider and they were trying to solicit our business by specifically calling out our provider as amenable to a DDOS attack which had occurred.

Someone with a simpler setup with standard DNS features will find it much easier to use multiple providers, of course.

From where I'm looking at the internet (central Europe), I don't notice anything.

Maybe your internet on the other side of the Atlantic is broken, ours seems to be working fine. ;-)

Edit: Looks like the eastern part of the USA is affected: https://cloudharmony.com/status-for-dyn

Not really, I can't access our production servers which are in US east. Can't access Intercom with which we provide customer support. Our clients are mailing us that payment provider doesn't work either. So we're losing money while being in central EU.

> Maybe your internet on the other side of the Atlantic is broken, ours seems to be working fine. ;-)

Works on my continent.

I'm in Europe now and worked fine something like 30 min ago.

Now facing similar issues reported in original post.

Things are down here on the west coast too...

I'm in Las Vegas and I haven't had access to several sites all day. I don't think this is limited to eastern U.S.

Yep same issue in Denver

Same Im in Illinios and I can't play destiny which is made by bungie which is down for some fucking reason

The TTL for the glue records of a .com domain is 48 hours, so even if you have Route53 set up and ready to go, it takes a long time to switch the zone away from Dyn.

We switched from Dyn to Rout53 a few weeks ago. It took about 12 hours before half of the traffic had shifted over.

That's the reason to have your DNS at at least two different companies, working in tandem. In a case where one is down, your Unicorn Corp doesn't go down with it.

How does that work in practice? Even if I set NS records pointing to two different DNS providers, I don't think a DNS client would automatically switch and retry if one is too slow to respond/times out.

Why not? That's the whole idea behind having more than one NS server isn't it?

Most DNS resolvers will automatically try each NS record until they get a response. That might be your ISP rather than your iPhone but that's an old practice going back to when the internet was even less reliable because some random Sun box under someone's desk failed.

Modern web browsers will also do the same thing if a query returns multiple A records and they get a connection error.

Exactly: there's nothing wrong with only using one provider if you're not willing to pay for two services but if you can't afford downtime you really need active diversity all the way down.

Route53 uses a bunch of different top-level domains for the same reason – if someone does manage to take the .com servers offline you'll be glad .co.uk is run by a separate organization.

N.b., unless things have changed, I don't think Dyn (for one) allows secondary DNS. Maybe that has changed. {?}

agreed 100%. This kind of thing shouldn't happen and this widespread. It's like no one is preparing for worse case scenarios until AFTER something tragic happens.

On the West Coast and I just lost twitter/soundcloud/github - 9:40 am PST

loss of twitter, soundcloud, and github is what passes for tragic these days?

Try PayPal. That's gonna be a big issue for everyone who uses their payment processing.

As a programmer, the loss of GitHub is a pretty big deal in terms of work productivity, and as a maintainer of production systems that rely on assets from GitHub, it's a pretty big availability issue.

I mean, i get it, its a big deal to us. But tragic? Syria is tragic. Github is just above annoying.

Let's say that it's tragic for the people whose job it is to maintain those services.

Yeah, as if a "spare DNS" has anything to do with solving the problem. You could get into your own infrastructure, but your customers are still screwed.

..and they thought war happens only with guns and tanks. don't the know, digital is the new platform to kill each other?

Maybe it is time to widely adopt namecoin ?

Yes!! For sure, after such an attack it will force the internet into rethinking their DNS strategies. Why commit to having a single point of weakness?

For those interested: https://namecoin.org/

they need a round robin dns that is geographically dispersed

Reply on news.ycombinator.com