Incident Report – DDoS Attack | Hacker News Reader

Incident Report – DDoS Attack

blog.dnsimple.com

146 pointsalainmeier38 comments

I need to learn to let things go, but: https://news.ycombinator.com/item?id=4280515

I've been a DnsMadeEasy customer for a while (they had an outage ~4 years ago from a 50Gbps attack), but once my year is up, I'm switching to Route53. The addition of the Geo DNS Queries was key for me. It isn't clear to me why I shouldn't pick Route53. DnsSimple's unlimited queries seems nice, but I kinda like having actual scaling costs forwarded to customers.

I've had a similar thought RE using Route53 for Neocities. Here's the problem with Route53 though. If you get a DDoS attack using it, it's quite plausible that you would be charged for resources used in the DDoS attack. A recent Vice article discussed this: http://motherboard.vice.com/read/inside-the-unending-cyber-s...

DDoS is a nasty problem. We've received a DDoS attack that shut the entire site down for days. We can't use Cloudflare because they don't support wildcard domains without their very expensive plan. I've also heard stories from people using Cloudflare that have still not been able to resolve DDoS issues (I'm not knocking Cloudflare, they're a great company that does a really good job fighting this very hard problem, but sometimes even they have trouble with it).

I'll be completely honest and say that I have no idea how to solve this problem. It's really, really, really hard. Switching to different service providers won't get you very far against the monster DDoS attacks that some people can execute.

If you're going to go the Amazon route then you absolutely need to keep an eye on billing, and set up alerts so that any DDoS which caused a spike in your costs would be caught as soon as possible.

I was burnt by this in the first 48 hours of using Amazon DNS. Very unlucky I guess... I'm amazed they still bill for DDOS traffic, or even traffic from black-listed IPs. It seems many of their competitors don't.

"and even then you can still be screwed if your bandwidth is saturated"

Which is exactly what happened in this case. It sucks to be on the receiving end of this. We couldn't defend against it and let our customers down, and that hurts me deeply. We choose one approach to defense, which was internal, and that was a mistake. We're going to work on rectifying that now.

I really don't understand why some of these low-grade DNS hosting services are so popular when Route53 is available. With Route53 you get a top-grade DNS service that is equivalent, if not better, than the enterprise hosted DNS solutions but at the price of the low-end consumer style services.

I swear by Route53, it is the only service I use on AWS and I have moved a lot of my clients over to it.

I agree, but there are some low-end DNS providers which have good services that give you more "domains" for less than what it costs with Route53. I use Route53 for a lot of my sites, but for tiny client sites (and personal stuff which has lots of domains), it's hard to beat $60 a year for 25 domains at DNSMadeEasy. That's less than half of what it costs to use AWS for the same number of zones. Granted, the price drops after those 25 on Route53, so if you have thousands of zones in one account, best to use Route53. Just as an example of an edge case.

> A new customer signed up for our service and brought in multiple domains that were already facing a DDoS attack. The customer had already tried at least 2 other providers before DNSimple. Once the domains were delegated to us, we began receiving the traffic from the DDoS.

I'm curious did they know this in advance or discovered it after the fact?

I often wonder about business models where the core expense is "unlimited and free". The reality is there is nothing unlimited or free for the service provider. It seems with a business model like this you open yourself to people abusing your service either by accident or by choice. Imagine poor Mr. Customer here who most likely was having horrible problems thinking to themselves "These guys can do it and for free, if I go to X service they'll cost me a lot of money".

I'm a big believer in business models that incentivize both parties properly. I'm sure in general this service provider is arbitraging the 99.9% of domains that barely need any services. That said it only takes a couple of "opps" customers to drive your operational costs through the roof.

Anthony from DNSimple here. We discovered it after the fact, via a tip from other DNS providers.

As someone who has been down this road many times before - I can't stress this enough: DDoS mitigation solutions don't solve the problem of an app-specific layer7 attack and it is important to do some testing of how well your mitigation service responds (and that it isn't a silver bullet.) Additionally, you need to make sure your team has tested and proven procedures for engaging the service, respond to attacks, etc. Services like NimbusDDoS (www.nimbusddos.com) are good because you can do some real scenario testing and make sure your team and infrastructure is prepared. There are other services out there too that I am less familiar with, but either way really good stuff to do.

"unlimited" plans are subsidised by low utilization users who are getting less than what they paid for.

to pull it off properly as a service provider, you really need to have a solid understanding of user usage patterns.

one of the big problems that tips the low/high utilization ratio unfavorably is that unlimited plans that are primarily marketed for being unlimited tend to attract users in the high utilization bracket.

so the challenge for service providers is not just understanding users and understanding that ratio but figuring out how you are going to market to, and signup, those users who will be in the low utilization bracket and will essentially be paying for something they won't be user (which is hard to do)

it isn't hard to find case studies of companies that launch optimistically with one pricing plan around unlimited, to then only go back and revise their pricing and break promises because they didn't understand their users and were unable to market to and signup low utilization users.

one recent example is Bitcasa

The solution here is one for customers, not providers.

Manage your DNS at one location on "master" (potentially a "private" server with IP restricted access and zone transfer ACLs).

Setup 2+ accounts with "DNS providers" that support incoming zone transfers - that is, they can operate as "slave" DNS servers, pulling records automatically from your "master" (once access rules are set of course) and returning results directly to clients making DNS queries.

Most "Secondary DNS" packages are < $50 year, so use a few, and don't worry about individual DNS networks being burnt to the ground.

It seems like inbound and outbound zone transfers aren't offered by a number of providers (like AWS). Do you know of a list of DNS providers that support either option?

A search for "secondary DNS service" should give you several results.

My research into it is from a "manage your DNS records internally, then use a couple of providers for all public facing responders". In that situation all you need them to support is inbound transfers, which several do.

I used to use these two services together do this:

  https://puck.nether.net/dns
  https://acc.rollernet.us/

They're both free to sign up, provide free secondary DNS, zone transfers and fully support IPv6.

I only stopped using them because I wanted to run my own DNS service.

EasyDNS provides integration with AWS: http://easyroute53.com/

They have an interesting blog post about setting up secondary DNS: http://blog.easydns.org/2013/09/10/what-we-are-doing-about-c...

I have no affiliation with them, just a happy customer.

So who do you think the "well-known third-party service that provides external DDoS protection using reverse DNS proxies" is they're going to use now?

CloudFlare?

Hopefully not. CloudFlare is remarkably unreliable for a service that claims to improve uptime.

Curious to hear more about this.

+1 never had any problems so far.

[citation needed]

Last I checked CloudFlare routinely handles[1] 10Gbps to 65Gbps attacks, and has successfully handled attacks as large as 300Gbps and 400Gbps. According to this report DNSSimple crumbled under 25Gbps.

[1]: https://support.cloudflare.com/hc/en-us/articles/200170216-H...

Their last significant outage was only 2 months ago: https://blog.cloudflare.com/route-leak-incident-on-october-2...

As the blog post outlines, the outage was related to an upstream network provider leaking routes. Note exactly something we can prevent for them.

We moved off of CloudFlare because of repeated outages and bugs. Our uptime improved significantly without CloudFlare.

Granted we are probably more vulnerable to DoS, but our general uptime is far better now.

I would assume Prolexic or Incapsula, assuming they're using a high end provider (which they should, DDOS attacks against smaller DNS providers being so easy to carry out).

Out of curiosity, what are the follow ups of an attack like that? The perpetrators are probably using their own servers or compromised clients or servers. Would DNS Simple follow up on this with the abuse/complaint dept of the ISP of the attackers? Are ISP typically responsive to abuse and complaints? If they are not is there any way to black list blocks of IPs assigned to ISP who do not care about being the source of DDoS attacks?

Investing in anti DDoS devices is important but even more important is for the perpetrators to face the consequences of their acts (or anyone who lets his machine being used by pirates - terminating or suspending their contract would be a fair response).

I was looking at http://map.ipviking.com earlier and it was apparent it was a botnet, most likely innocent home users with a virus.

Remember to keep any machine under your control up to date! I'm looking at you, XP die-hards. If you're able to, monitor your network traffic periodically as well.

It's easy to toss out statements like "monitor your network traffic"; do you have any good suggestions for how an average developer with relatively little understanding of networking can go about doing so?

Glasswire for Windows or Little Snitch for Mac?

Well, my router with dd-wrt just gives me a traffic diagram. I don't check it super often, admittedly. I wonder if it could be modified to signal a warning somehow?

It'd be nice if IPs involved in botnet DDoS's could go into a public registry, then get a banner from Google saying, "Hey, you might have a virus, someone reported you to this list."

Abuse would be tricky, you might be able to limit it by letting only a few DDoS mitigation providers populate the list.

A lot of ISPs for example in Germany reuse IP addresses and force a reconnect every 24 hours. I don't think showing me banners because the previous "owner" of the IP had a virus is going to improve the situation.

Other people share a network behind a NATed IP which is also a problem. They'd all receive a banner, check their computer and a test would come up negative.

Google wouldn't know but the ISP would know who was behind a particular IP at a specific time. They are the ones who should police their network when there are abuses.

The original proposal was that google delivers the ads. So google would have to contact my ISP who would then have to return whether or not I was using any of the given "spammy" IPs at the time that they were spammy - or my ISP would have to deliver the banner.

No thanks.

This is actually one of the main uses for the ISP/telco product appliance sold by my employer, Damballa. The appliance reports client IPs which appear to be infected with malware to the ISP, who then reports this their affected customers by whatever mechanism the ISP prefers.

This particular DDoS I actually believe is _not_ due to a botnet, or at least believe there is insufficient evidence either way. The attack appears to be using a technique/infrastructure I’ve been passively tracking for nearly a year, wherein the attack DNS requests are spoofed to appear from seemingly-random clients and sent to open recursive DNS servers across the Internet. This makes the attack look like a botnet to superficial analysis on the target side, but this isn’t necessarily the case. In the small amount of time I’ve so-far invested in trying to track down the origin, I have yet to observe generation of the initial query packets.

> banner from Google saying, "Hey, you might have a virus, someone reported you to this list."

Unfortunately this is already in use with some malicious ads as well as phone scams to get people to give remote access to overseas tech centers that then scam them into paying good money for nothing.

To date the only tech line about this is, "nobody legitimate will ever contact you to tell you you're infected with a virus."

So I don't know how you could develop trust in that environment.

What was the overall makeup of the attack traffic? For example, 50% tcp syn, etc.

Reply on news.ycombinator.com