UltraDNS Server Problem Pulls Down Websites, Including Netflix, for 90 Minutes | Hacker News Reader

UltraDNS Server Problem Pulls Down Websites, Including Netflix, for 90 Minutes

nytimes.com

48 pointsrdl29 comments

Here are the sequence of emails we as a customer received:

1:44PM PST - The Neustar UltraDNS service is currently experiencing DDoS traffic in the U.S. East Region. The Security Operations Team is currently working on mitigating attack traffic and further updates will be provided as soon as possible.

3:07PM PST - The Neustar UltraDNS service is currently experiencing a major spike in traffic across several regions. The Security Operations Team is conducting routing changes and working to reroute traffic at this time. Customer will experience DNS timeouts and lack of resolution. Further updates will be sent as soon as possible.

4:50PM PST - At 4:18 EDT time, a service outage affected Neustar’s UltraDNS service. The impact was predominately on the East Coast of the U.S., however some impacts could be seen in other U.S regions. Neustar UltraDNS Service was restored at 6:34 EDT time. We are currently investigating the root cause of the outage and will provide an update as more information is available.

The data from the outage is all here:

https://blog.thousandeyes.com/ultradns-outage-october-2015/

Check it out.

Serious question here: Do people really spend $90/month for 2M DNS queries?

https://www.neustar.biz/services/dns-services/managed-dns-pa...

And that's even without 'DNSSEC and IPv6 support', 'Regional Routing' and 'DNS Load Balancing (where available)', that's only available for 'UltraDNS Enterprise'... (I'm running my own (Power)DNS, after EveryDNS was bought by DYN and they didn't grandfathered old plans, charging per domain, which made it extremely pricey)

Yeah - i looked at DYN and they are expensive. The "Internet Intelligence" stuff looks interesting but that is an add on.

I need some dynamic DNS for my VMs, I'm getting away with Avahi/mDNS right now but I need to set up an Active Directory lab. Do you have any recommendations for dynamic DNS? Bonus points if it's BIND9 compatible.

Put AD-joined hosts in a subdomain, e.g. subdomain.example.com. AD DCs are authoritative for the subdomain. Your public facing / authoritative nameservers for example.com should be listed (publicly) as the only NS's for subdomain.example.com and should slave it from your DCs.

More money and more queries-per-month then that at nsone.net. They are very good. The api is especially nice for setting up failover and geo pools.

9/10 times it wasn't technical people who made the decision to use services like these.

NSONE is cheaper and has way better geofiltering

I suppose this was also the cause of the 90 minute Bitbucket outage today?

Surprisingly, no. Or at least they're not claiming so. They say it was a problem with a database server. http://status.bitbucket.org/

That was a different "Git and Mercurial over HTTPS slow or not responding" outage from yesterday (or "02:00 and 03:00 UTC on 15 October 2015")

The more recent one started @ Oct 15, 20:19 UTC, they don't seem to have root cause on the status page for it yet.

I am surprised that Netflix uses UltraDNS when they use AWS and they could be using Route53.

I would imagine they don't use it for exactly that case. Imagine a case where most of AWS is having critical problems, you would want (at the very least) control of your DNS to attempt a partial recovery of service for some portion of your customers that have providers who respect DNS TTL's.

Running my own DNS servers. But having issues with clients using Rackspace Apps - hosted email (still down due to UltraDNS).

Would there be anything against using multiple large anycast DNS providers?

Netflix is 100% dependent on AWS for infrastructure. If AWS is having problems, there is nowhere to redirect to.

of course there is, they could redirect to a page explaining the problem instead of a generic error page...

UltraDNS is more advanced / pro than route53.

Could you provide some details about how and why it's better?

Amazon.com uses UltraDNS and Dyn

Confirmed, this issue more or less took down Amazon's Merchant Web Services API today.

It's because Ultra lets you do geo based routing by state and province and Route53 doesn't (at least not yet)

How can unicast be DDoS?

The article says it was an internal issue, not DDoS or hacking.

Interesting. We use UltraDNS and didn't go down at all.

Titular hyperbole, much?

Original (current?) title on the page was UltraDNS Server Problem Pulls Down Websites, Including Netflix, for 90 Minutes.

We changed the title from "UltraDNS takes down the Internet for 90min" to the article's title.

Reply on news.ycombinator.com