DDoS Attacks: Best Practices for Prevention and Response
insights.sei.cmu.edu
insights.sei.cmu.edu
>Locate servers in different data centers.
>Ensure that data centers are located on different networks.
>Ensure that data centers have diverse paths.
>Ensure that the data centers, or the networks that the data centers are connected to, have no notable bottlenecks or single points of failure.
Getting Robbed: Best practices for Prevention and Response
* Don't carry all of your money with you at all times.
* Don't advertise that you're carrying large sums of money.
* Have enough money that getting robbed doesn't really affect you.
* Pay someone else to do your errands so you're not at risk.
- buy a ton of network capacity in multiple locations around the world
- route incoming traffic to different locations at BGP level
On top of that we:
- scrub the traffic using using some expensive fancy appliances(*)
- employ a bunch of very skilled people to manage it all and provide support
* which we get a very good deal on by virtue of making said appliances :)
Is active mitigation mainly about sifting through incoming traffic, trying to find signatures and update filters in realtime? Or more replacing dynamic with static content as its targeted? Or are there other ways resources get surged during a crisis?
ie, Is it heavy on manual analysis, or are we at the point yet where we can automate mitigation?
Just curious. Would recommend an AMA, but that's not this site.
The larger the attack, the simpler the vector.
For the Flood (TCP, UDP, DNS, NTP, etc) attacks, creating accurate firewall rules within your cloud-scrubbing provider handles a large portion of this, the remainder can be mitigated by connection rate-limiting or TCP connection mitigations (check to make sure it’s a valid 3WHS before allowing connections to the origin).
Complex L7 attacks require more effort and usually shift around in what they are attacking, this takes more analysis to pin down, though the L7 Bot Defenses, intelligent rate limiting and automatic traffic analysis help with this.
DDoS mitigation is not explicitly a science, there is an art to it as well that comes from experience and learning day-to-day as attacks evolve. As attacks start up, mitigations may be too strict or too loose, that is the benefit of an expert SOC staff to monitor the situation and adjust as needed. Relying on automation for this will likely leave the customer frustrated with the outcome.
There are estimates that the Dyn attack was 1.2Tbps. What provider do you work for that can absorb that?
>"scrub the traffic using using some expensive fancy appliances(*)"
Does scrubbing work with a bunch of randomized source ports and source address? How do you find the "signature" in that to scrub? It was my understanding that the traffic in the Dyn attacks was indistinguishable from legitimate traffic. Can you explain how traffic scrubbing works in such scenarios? Or do you just drop stuff on the floor?
A lot of smart people have spent a lot of time thinking about this problem.
If they only scrub half of that traffic, you're still getting a massive DDoS attack against your origin servers anyway. And the frustrating thing about buying DDoS mitigation is that you have absolutely no way of knowing how much traffic any DDoS mitigation service can actually scrub of a given attack before it hits you...
If the DDoS mitigation provider is doing their job, they will ingest all the traffic, scrub out the bad and return the good. While obvious, there are things a customer should know when entering into that arrangement:
- Does the provider have the ingress capacity to absorb all the attack traffic?
- Are their scrubbing centres peered with Tier 1 transit providers to reduce carrier congestion?
- Do they have a policy on dropping traffic at certain volumes?
- Do they charge you based on attack volume or clean traffic?
- Do they have rate-limiting in place towards the customer to protect them from high-volume attacks while mitigations are optimized to catch all the attack traffic?
I am curious if the traffic analysis at the scrubbing center is anything more than coarse grain I am assuming it is.
I work for F5 Silverline.
It would be naive to brush off that size of attack as it would stress any DDoS mitigation provider.
We have the capacity to take it (and are adding a lot more in the coming months precisely because of attacks like this and the previous big one that hit Brian Kerbs) but you would have to talk to Sales about what sort of guarantees we would be willing to provide.
That said, at least that would be our problem, since we would have to pay for the bandwidth consumed - not the customer! ;)
> Does scrubbing work with a bunch of randomized source ports and source address? How do you find the "signature" in that to scrub? It was my understanding that the traffic in the Dyn attacks was indistinguishable from legitimate traffic. Can you explain how traffic scrubbing works in such scenarios? Or do you just drop stuff on the floor?
(Answer from one of my more knowledgeable colleagues since I work on supporting the infrastructure rather than DDoS mitigation itself):
Most attack traffic has a signature of its behaviour beyond source IP and port.
For simple flood attacks, connection rate-limiting per source IP is very effective. Typically there are some simple firewall rules that will work too, especially with attacks generated against targets that don’t support the attack vector (e.g. block all GRE traffic coming to your website which is only TCP 80/443).
If the attack is operating at L7, there are additional fingerprinting methods we can use around HTTP headers, intelligent rate-limiters, JavaScript injection to prove an actual human, etc.
">For simple flood attacks, connection rate-limiting per source IP is very effective."
But if a each of the 8 million bots in a hypothetical botnet sends exactly one syn packet then you can't really rate limit per source IP. Or the case of DNS a single UDP datagram which might be indistinguishable from a "legitimate" DNS query datagram.
Also rate limiting is different that what I would think of as "scrubbing" unless I am thinking about scrubbing wrong. Back to my example of the botnet where the command and control tells each node to send a single packet or a small handful of packets towards a destination, theres very limit match a rate limit filter on in that case no?
The single SYN will fail the 3WHS test, so we would drop it. For DNS we can usually fingerprint the attacker and block traffic. Because we are a full proxy we can do packet inspection as well. ;)
There's a fair amount of intelligence already built into our products which we of course make use of (BIG-IP LTM/AFM/ASM, "IP Intelligence").
Eventually it may well come down to the SOC monitoring and adjusting the filters in real time as required - that's life.
This isn't global this is per region/per AS. 1.2Tbs would be 12 100 Gig ethernet ports on a router or 120 10 Gig ethernet ports. There are not many people that have this capacity in a POP or even even in multiple POPs in a region such as North America or Europe. Just the money involved in being able to connect that much transit to your edge is insane. Because not only do you need enough transit provider diversity but you also need multiple chassises populated with enough switching fabric. I doubt there is a CDN or hosting provider that has 1.2T of capacity at any kind of a region level. A Tier 1 provider yes but not a CDN or hosting provider.
With botnets like Mirai, innocent networks need to protect themselves not to avoid getting attacked themselves but rather to ensure that they don't participate in an attack against someone else.
I would love a link to some resources about how to do such analysis. Maybe someone here knows a good one?
In general, this has been my perception of the majority of "software engineering" research which I've encountered.
Out of curiosity, what would you recommend differently? Carrying a weapon and engaging in combat?
Granted, at some point, you're going to have to spend money to mitigate the attack no matter what, but if mitigation of DDoSs becomes entirely focused on "Go with a big centralized provider" or "Spend lots of money to mitigate the attacks," we end up in a much different Internet.
Mitigating DDos is one of the main selling points nowadays. That's why OVH wrote so much about the huge attack they defeated (and they didn't name the impacted clients), Cloudflare offered Krebs to host him (he refused) and other providers add scrubbing centers.
I actually think it's rather cheap to defend against DDos if you're a small company. Large companies will have it harder as they have typically more complex requirements and cannot just shift everything behind cloudflare or similar services.
> Deploy appropriate hardware that can handle known attack types and use the options that are in the hardware that would protect network resources...
> If affordable, scale up network bandwidth...
> There are several large providers that specialize in scaling infrastructure to respond to attacks...
This advice all seems rather obvious.
Now: wow DDoS attacks are still a thing. ISPs need to implement egress filtering.
A consumer internet user who visits baidu doesn't need to be flooding an anti-censorship project's github page.
A consumer internet user with an internet-of-shit camera doesn't need to be hitting dyn that frequently.
ISPs need to coordinate and rate limit customers towards something that looks like reasonable traffic.
Because null-routing malformed packets, for example, is completely different from null-routing the port of a service the ISP don't want. What is again, completely different from null-routing bittorrent.
If you only want to protect your website then go for cloudflare pro, it will be enough for 95% of ddosers. If cloudflare is not enough then you need thousands or ten of thousands of dollars to get protection.
I'd guess that they're one of the largest companies providing DDos defenses. I'd guess if they can't handle it there aren't many more that could. But don't know if that has ever happened.
Same can often be said if you replace "university" with "company".
Internal political problems are beyond NP-hard.
The common saying is "the cobbler's children have no shoes".
Not to mention that IPv6 doesn't really help solve the "ISPs aren't dropping invalid packets as they are received" problem.
if you dont want to waste money on the server, just use incapsula