DNSimple DDOS Attack | Hacker News Reader

DNSimple DDOS Attack

dnsimplestatus.com

125 pointsdedene115 comments

We're kinda tied into DNSimple since we use an ALIAS record for our bare/naked/root domain. Amazon's Route53 supports aliases, but via a 301 redirect, which doesn't work in an SSL context (without browser warnings).

Nonetheless, we just spun up a Route53 zone, exported our zone from DNSimple, imported to Route53, and hand-migrated our ALIAS records to static A records in the new zone.

Not perfect or permanent, but we've gotten around the outage. Also, I just learned that pointhq has (seemingly-undocumented) support for ALIAS records in the same style as DNSimple, so this could be another avenue to explore.

A few people do ALIAS style records, DNSMadeEasy and EasyDNS do "ANAME" records which are pretty much the same.

Will these services not have the same thin pipe issue that's currently affecting DNSimple?

Not that I have any reliable info, but what I've heard, DNS Made Easy is a pretty stable and established DNS provider.

They brag about "99.9999% uptime history" at http://www.dnsmadeeasy.com/technology/.

Though they doesn't seem as innovative and nice as DNSimple. Really hope things work out for DNSimple (really like the idea of their beta feature GitHub sync).

How does their beta-feature with github work? I can't find any obvious link and it seems frustratingly close to something I offer over at https://dns-api.com/ ..

(I wrap Amazon's route53 with DNS entries read from github/gitbucket/similar.)

And link to their Git support is here: http://support.dnsimple.com/articles/github-sync/ (if only DNS wasn't down..)

Interesting thanks.

An interesting/custom choice to use JSON, and real github integration rather than using hooks as I did.

Can you use custom nameservers with your service?

AWS Route 53 has added vanity DNS in the last few weeks, so you can make your DNS servers appear to be ns0.yourdomain.com rather than ns153.awsdomain.com (or whatever)

However, making your DNS servers to be responsible for serving their own DNS is a bit of an extra complexity and risk that no customer will ever care about..

Nice. Got a link for this one?

It was oddly snuck into the private DNS announcement

http://aws.amazon.com/blogs/aws/route-53-update-private-dns-...

"You can create generic "white label" name servers such as ns1.example.com and ns2.example.com, use them in your delegation set, and point them to your actual Route 53 name servers."

When you upload a new zone it will be assigned a set of nameservers - four. You can't choose what they are in advance, and you can't specify the TLDs. So you might end up with "ns-1933.awsdns-49.co.uk.", "ns-1109.awsdns-10.org.", or similar.

Does that answer the question? I'm a little hazy on what you're actually asking.

Old DnsMadeEasy customer here (used them for several years before moving to AWS). They were rock solid. We used them for hosting DNS for major cruise lines (fancy, very fancy ones), as well as other large Fortune 500 clients.

Their interface is pretty bad, but the backend is hardy.

It's unlikely, DNSimple appear to have relied on a single network provider and a limited number of name servers.

If I do a traceroute to the 5 DNSMadeEasy name server records (they actually run many more hosts) I go via 3 different networks - GTT, NTT, and Tata.

FYI you can use ALIAS records without a 301 redirect if the ALIAS is to an ELB.

How are you exporting the zone with things as they are?

Oof, really good point – I suppose I got lucky in that I was at dnsimple.com just a few hours ago for other purposes. Their DNS was still cached for me, and it all worked flawlessly. I didn't even suspect their outage extended to their web servers, though that makes perfect sense.

FWIW, the IP I have cached is 50.31.213.210.

How were you able to find out what IP was cached for a particular domain?

same question here - how did you do this while dnsimple.com is inaccessible?

You can force your hosts file locally. Set "dnsimple.com" to "50.31.213.210". On Mac use "sudo pico /etc/hosts" and follow the formatting of the other entries.

We can watch this happen live @ http://map.ipviking.com/

Fascinating traffic floods from various locations, but the attack is not continuous.

How does this link show this specific attack?

As far as I understand, ipviking simply hosts honeypots around the world and uses those to graph "attacks" against IP blocks, etc.

I would very much like someone to correct me if this assumption is incorrect, because it'd be neat to actually watch targeted DDoS attacks, but I don't think that's what ipviking is offering.

Noticed a lot of hits to St. Louis, where they're based. Do you think this has anything to do with the attacks?

Are these live attacks on any servers?

Free solution that worked for me: Set up a free account on cloudflare.com, duplicate all dns records (thankfully I have a simple setup)... but next time I will keep a backup zone file!

FYI - Instead of an Alias record on DNSimple, CloudFlare will allow a CNAME record for the root domain using "CNAME flattening".

You can now set CloudFlare's DNS service to "bypass Cloudfare" on all records by clicking the icon so you don't get any of their magic (unless you want it).

Then add CloudFlare's 2 nameserves to your domain as your first 2 name servers. No need to remove dnsimple's name servers.

Now you have 2 DNS providers in case one fails, just make sure the records are the same across them both!

Hey thanks for the post, but how do I add CloudFlare's nameservers? DNSimple won't let me add NS records on the root domain: "You may only delegate subdomains". So in theory, I can fix the www subdomain, but my naked redirect to www won't happen if DNSimple is still down.

Any ideas?

You would set your name servers with your registrar, such as godaddy. If dnsimple is your registrar as well as your DNS provider you may be out of luck until they come back online. The help page is here though: http://support.dnsimple.com/articles/setting-name-servers/

Anybody can shed some light on the difference between "CNAME flattening" and ALIAS?

they are essentially the same thing -- both are "proprietary" names for the same feature, which is behind-the-scenes recursive CNAME chain lookups by the authoritative nameserver, to return A records directly.

DNS is so straightforward, so easily distributed, and so fundamental, that I'm always astounded when it's a single point of failure for so many operations.

I wonder how many of the affected companies do have redundant appservers and load balancers, but missed this piece of the puzzle...

dns is less easily distributed when fancy features like ALIAS (which dnsimple is widely known for) are in the mix. and wide distribution isn't enough to win vs truly volumetric attacks. it takes a lot of ports and compute to absorb 100Gbps+ attacks which are not uncommon against major providers.

DNSimple is widely know for the ALIAS pseudo-"record" because they invented it[1].

Small wonder that a proprietary syntactical sugar leaves you at the mercy of select vendors?

As for volumetric attacks: your point is correct, but is irrelevant if you're using multiple vendors, and a specific, single vendor is the target, like it appears here. Your other authoritative servers would be unaffected.

1 http://support.dnsimple.com/articles/alias-record/, or http://webcache.googleusercontent.com/search?q=cache:ST1BABj...

good luck finding any major online property or infrastructure that isn't making use of some kind of proprietary syntactical dns sugar. it doesn't mean you can't span providers, but it does mean it takes a lot more work to do so.

anyway, you're not wrong, the best approach to mitigate this kind of thing is to leverage multiple dns networks. but doing so is not easy unless the application is still using dns like it was in 1995, and that is increasingly rarely the case.

Using a WWW subdomain with CNAMEs accomplishes effectively the same thing as using ALIAS on an apex domain name, and doesn't rely on anything out-of-spec or proprietary, making it easier to serve redundantly. (Did you ever wonder why google.com and facebook.com redirect to www?)

(Or is there more to ALIAS than that, which wasn't on the page in GP? Happy to be corrected if so)

you're correct about ALIAS (although practically, it doesn't matter: people are going to use the apex whether it's proper or not at this point). i'm more referring to other complex usually-proprietary capabilities of big dns providers, especially traffic routing features. routing semantics are generally not translatable across providers, and if you're using dns based routing (as most cdns, major web properties, etc are) then doing multi-network dns gets a lot harder. if you're amazon, you write and maintain a bunch of code to span providers. if you're not, the barrier to multi-network is high if you're doing more than static dns.

Yeah, that's a fair point. I'm not sure of a good fix for that, either.

You're right, but people want to get fancy with hosting at the apex (domain.com), even though it kills important functionality (CNAMES) forcing the adoption of hacks (ALIAS and ANAME records).

I'm surprised more places don't run their own DNS. It's not that difficult to do and it means you don't have to rely on another third party for service.

Well if this is an attack to get at one of DNSSimple's customers, running your own DNS would be a much easier target. Which is to say that if you were the target, you would already be hard dead by now rather than struggling as DNSSimple deploys defenses.

I agree though that it is a pretty simple service to run for a small domain.

If you're being targetted directly, then all of your services need to be DDOS proof, not just DNS. The more third parties you add, the more likely you are to be taken out by accident. If you have your own web server, you should dump Bind or PowerDNS on it and write a zone file. Problem solved.

DNSSimple isn't mutually exclusive with running other servers... you can even pay rival companies to host the same records redundantly.

But agreed, if you are the target, you're going to be hosed either way.

Not that difficult for whom? Great if you have the in-house resources to devote to managing your own DNS (and can't put them to better use elsewhere), but that is not the case for the vast majority of us. The fact that so many of us use PaaS companies like Heroku should be a pretty big indicator that most platform-related engineering is not going to happen in-house under a certain scale.

If you can figure out DNS on any hosting service's page, you can do it in multiple places.

You could pay GoDaddy, Amazon Route 53, and DNSSimple to all host your records, for example... Management would be slower and manual, but people without resources for "managing your own DNS" won't be changing records that frequently anyways.

The odds of all three going down at once should help your uptime, yes?

Not that difficult for somebody who is a sysadmin or developer. I would hope that services like Heroku offer DNS too, but I wouldn't know. If your site is already hosted at Heroku, it would be better for them to deal with your DNS so that you don't introduce additional third parties. That is my point.

Ah, I entirely misunderstood your point. I thought the point was that the end users of DNS should be hosting. My mistake.

exactly my thought. the year is 2014 and DNS is least of my problems, since 1999 or something...

Can someone help me understand what happens to email sent to a domain hosted by DNSimple while it's down?

I'm hoping it will get queued by the sending server, and make it's way back when DNSimple is up and running. Is that correct?

It will depend on the configuration of the sending server. Some will retry for a while, some will return failures.

Thanks!

I moved from Zerigo to DNSimple, and it's been awesome until now!

What can you do to prevent this in future? Can you run multiple DNS providers simultaneously? So, ns1/ns2 go to DNSimple, and ns3/ns4 go to another provider?

Yes - that should work. I'm about to make that change to bring back our site. 2 on one name server, 2 on another.

Cool. Do you need a provider that supports 'zone transfers'? Or is that only to keep things in sync automatically?

> Do you need a provider that supports 'zone transfers'?

No.

> Or is that only to keep things in sync automatically?

Yes.

I replicated our DNS on DNSMadeEasy, which I've used in the past on a very high-traffic website. They have an incredible track record for uptime. And they're really cheap.

We manually exported our DNS configuration from DNSimple as a single file and imported it to DNSMadeEasy. Pretty easy transition, although we have to make future changes in two places.

I set 2 nameservers on the old host and 2 on the new, and am keeping that configuration for automatic failover in the future. The DNS propagated fairly quickly and our site was up and running for many users within an hour. Glad that's over with.

DNSimple does not currently support zone transfers (see http://support.dnsimple.com/articles/master-slave-support/), but you should write an e-mail to the support (address is on the linked page) requesting it :) With dozens of domains, managing multiple DNS servers is a pain. Zone transfers make this very easy.

Do you know of any providers that will work nicely/automatically with DNSimple?

I don't unfortunately.

If you have an active DNSimple web UI session (or API key) you can change your root nameservers by hitting their web tier directly at 50.31.213.210.

We've successfully switched our domains over to nsone.net.

Even without an active session, I was able to get this working by adding an entry to my /etc/hosts file with that IP and dnsimple.com.

Thanks for the tip! Switched to NSONE.

This worked for us - moved the DNS to Cloudflare, kept the domain at DNSimple

For anyone else who needs to mitigate this in a hurry:

Set up a new account on another host that does ALIAS records (I used pointDNS)

Create your new record without much in it

Change your nameservers on your domain now - they'll take time to propagate

Fill in the records on your domain. If you can't remember them, print out most of your existing records with

dig yourdomain.com ANY

Add the rest of the records to pointDNS

Wait for the new Nameservers to propagate (0-24 hours - it took 15-30 min for us on a small-medium traffic domain today during sales crunch)

Publishing an incomplete copy of your zone seems like a bad idea, an ANY dig at the zone apex is going to retrieve less than 30% of the records (on average) for real-world zones. Negative responses in DNS are cacheable - so you can easily poison yourself.

If you really really want to do it anyway, most caches use either the TTL on your SOA record, or the final field in the SOA record as the negative cache TTL; so lower both of those values to something like 60 seconds.

For those with a more deterministic bent: "propagation" time has a maximum bound of your TTL, which will show with any dig queries.

Propagation is bound at the DNS TTL plus whatever time it takes your DNS provider/setup to relay records to all of its authoritative servers.

Hmm, never considered that. Is it a slow, static process for ISPs to do that? I just assumed that they ran more-or-less stock DNS resolvers with in-memory caches.

Here's where you can request your cached SERVFAILs be flushed from Google's public DNS (i.e. 8.8.8.8): https://developers.google.com/speed/public-dns/cache

Thank you, I wish I could buy you a beer for that

"30 minute ETA from our network provider to begin scrubbing traffic in a location with capacity."

https://twitter.com/dnsimplestatus/status/539551209452232705

It's surprising that they're appear to not be multihomed...

http://bgp.he.net/AS32771

Unlike Dyn or CloudFlare:

http://bgp.he.net/AS33517 http://bgp.he.net/AS13335

They're in ServerCentral's datacenter and ServerCentral is very much multi homed. They wouldn't gain anything by doing native BGP to all these peers in the exact same datacenter when SC's backbone will handle this stuff for them.

Doing native BGP would allow them to anycast, which would increase their reliability and allow them to sink traffic much more easily. DDOS traffic sink starts announcing your AS and anycast IP block close to the traffic source, sinking that traffic and allowing real traffic through.

I'm pretty sure they're only in one of Server Central's datacenters. Anycast won't help. That's why I said they don't gain anything by directly peering.

"New ETA is 30 minutes from now, trying to get systems wired up in the data center."

https://twitter.com/dnsimple/status/539560631863877632

Does anyone have a simple explanation or link to an article / blog that explains the naked domain / ALIAS "problem" that DNSSimple solves? I recently set up DNS with DNSimple (due to nudging by Heroku) and am affected by this DDoS. I am still struggling to understand the exact nature of this issue. All of Heroku's documentation is pretty cryptic (to me):

"Some DNS hosts provide a way to get CNAME-like functionality at the zone apex using a custom record type. " .. and then on to suggest DNSimple as their first suggestion.

http://blog.cloudflare.com/introducing-cname-flattening-rfc-... is a reasonable explanation. fundamentally a CNAME says "when you get queries for this name, go look at this other name instead". among other things, doing a CNAME at the zone apex means resolvers can't then find your NS, MX, or other records at the apex, which is problematic.

Basically, naked/apex domains generally require an A RECORD pointing to an IP address.

Heroku prefers you didn't use A RECORDS at all because the IP addresses in their underlying architecture might change. [1]

[1] https://devcenter.heroku.com/articles/apex-domains

For those wondering about alternatives to ALIAS: if you use a www subdomain, then you can simply use CNAMEs. (Though the appearance is a matter of taste...)

Google, Facebook, etc, all use this approach.

Yes, but if someone visits your apex domain directly, you want to redirect him/her to the www version... And if you use cloud based hosting where the IP can change quite often, that's a pain to maintain manually.

Anyone switching from DNSimple? I really don't want to, but we've been down for almost 3 hours. I've seen chatter about Cloudfare and it looks pretty good, reviews?

We switched periscope.io from DNSimple to Amazon Route 53. DNSimple doesn't have an exporter so it took about an hour, including having one engineer review the other engineer's work.

Many customers were able to resolve the domain in the minutes immediately following the switch, and the rest seem to be trickling in.

Are you talking about a zone file exporter? DNSimple does have one, we just used it to migrate to Route53. http://support.dnsimple.com/articles/zone-export/

EDIT: right, must have been able to log in during a brief period where dnsimple was not down.

That page doesn't load for me. :)

We went by this:

https://twitter.com/dnsimple/status/539521794802483202

https://twitter.com/dnsimple/status/539520808599957505

link doesnt work

Amazon's route53 is an awesome service :)

Recent outages were reported earlier in the week.

(ObRandom: I run a service that wraps route53 with git integration, at https://dns-api.com/ )

We switched baremetrics.io over to AWS/Route53 as well. It didn't take long (NS switch might still be propagating) but not having ALIAS support is a bit of a drag.

We are having Cloudflare issues right now... https://www.cloudflarestatus.com/

DNSimple is my registrar and (was my only) DNS provider. Now that they're back up I've exported the zone file and imported it to route 53 for redundancy in case this happens again. I also I updated the name servers in DNSimple to be 2 route 53, and 2 DNSimple, in that order. Is that the right way to do it? Does the order of the NS records matter? I set them up so that they're in the same order in both places.

That should be right. I'm doing the same thing with CloudFlare. However, it appears that DNSimple won't keep your secondary name servers in the order entered. They are showing for me sorted alphabetically. Apparently, they are working on providing this (http://blog.dnsimple.com/2014/12/incident-report-ddos/) for failover in case of an event like this again.

DNSimple says it was not a direct attack on them but rather domains being brought over by new customers. Does anyone know the actual target?

I wrote a follow-up article about what we at Canopy.co learned from this incident. Check it out (this covers and expands on some of the ideas talked about here):

https://medium.com/@brianarmstrong/youre-probably-doing-dns-...

Unfortunately, it's not the first time it happens, my app is down and customers unhappy.

I always wonder, why is it that someone wants to attack a small company like DNSimple ? Is it that they were blackmailed and did not surrender to the criminals? If so, why would anyone be interested in blackmailing such a small company?

More likely, I'd guess based on past experience, it's that someone wanted to take down one of their customers, and decided (or found) that the weakest link was the DNS provider.

It could also be an attack against one of their customers, again for the same reasons. DNS is a very weak point of the overall system so a very good place to target.

My guess is one of their customers was being attacked–and this was the easiest route to get to them.

You can use my cross-platform cli for dnsimple to export your zone files easily to txt or json format: https://www.npmjs.org/package/dnsimple-cli

dnsimple domain record list example.com > example.txt

dnsimple domain record list example.com --json > example.json

RubyGems.org and Travis-ci are down as a result of this! Not helping with my productivity this morning.

What are the recommended practices to prevent too much down time when your DNS provider goes down?

My site is currently offline from this attack. I am considering that providers like DNSimple simply cannot provide the networking availability to mitigate these kinds of attacks. This is because the solution to these kinds of attacks, often, is that you need a larger pipe than the attackers. Very few people are good at both <some service> and having a huge pipe. For web apps, you can use CloudFlare, which does have a bigger pipe and is designed to mitigate this. But DNS is not a web app, so you can't just put DNSimple behind CloudFlare.

Hence, I'm going to try CloudFlare (assuming they take over DNS hosting, I need to check) and Google Cloud DNS, because then all parts of my site (from DNS to CSS hosting) will be with providers with bigger pipes than attackers can create. Hopefully that will prevent this kind of attack from taking my site down.

Yes, CloudFlare has a full-featured DNS offering, even if you don't use their proxying services.

Which has had a lot of problems this past week:

https://news.ycombinator.com/item?id=8665367

CloudFlare != CloudFront

My apologies, cloudfront had issues, cloudflare did not.

You can use two or more DNS providers, but if you do try not to list more than two name servers from any one provider when you delegate. You should also take care to do the same in your own zones "NS" record set. Some providers don't let you change the NS record set, so avoid those providers.

The reason for this is that resolvers will generally try at least three different name servers before giving up, so if you have three or more from a single provider that may not help.

There is also a big caveat to consider: once you use two more providers, whenever you need to make rapid DNS changes for your own availability reasons you will need to wait for the slowest-to-update provider.

have multiple providers? VPSs aren't expensive and setting up bind9 or bind10 or whatever authoritative name serving daemon isn't rocket science (well, you CAN do it wrong).

my experience shows, that at least 3 servers with 3 different providers is good enough. and "providers" I mean different company, city, datacenter, transit provider...

trusting single entity with anything (even if they say that they have many servers blabla... geologically blabla...)... well, you have situation like this right now :)

My god no. Don't even waste your time with VPSs and BIND.

* Get accounts with AWS (Route 53), dnsmadeeasy, and cloudflare.

* Monitor resolution at all of their name servers

* Either proactively spread your authoritative nameservers across providers, or update your root NS records based on your monitoring.

Can anyone expand on what this means: "This attack is volumetric in nature." (?)

According to http://www.arbornetworks.com/attack-ddos:

"DDoS attacks...will generally fall into one of three broad categories:

Volumetric Attacks: Attempt to consume the bandwidth either within the target network/service, or between the target network/service and the rest of the Internet. These attacks are simply about causing congestion."

Pretty sure it means the traffic is overwhelming them.

Deleted comment

One of the most valuable things I learned in my career was to never kick your competitors when they are down. It upsets the karma gods, and makes you look like a total douchebag.

The best thing you can do right now is to reach out and offer your help, privately. Even from a selfish perspective, you'll learn a lot about the attack that is taking them down now which will help you out when the targeted customer inevitably signs up for your service.

EDIT: The parent comment was spam from Kris Beevers at NSone.

hi. i did not mean to spam or kick dnsimple, we know them and they are a great company and service. we are actively receiving inbound queries about this from folks asking for help, so thought it made sense to chime in publicly here. but you're right, i should have kept it on topic to the discussion at hand instead of offering anything else up.

you're not wrong: in this industry you never kick your competitors when they're down, everyone is subject to the same constraints, attacks, and complications. that wasn't my intention and i said so in the post.

dsl (1402 days old, 4664 karma) - beevek (123 days old, 4 karma). beevek you just lost our business.

That seems a little extreme; perhaps what he did was in poor taste, but to use karma and account age as an barometer for your business decisions seems crazy (or an arbiter in an internet catfight).

Reply on news.ycombinator.com