Cloudflare Adaptive DDoS Protection | Hacker News Reader

Cloudflare Adaptive DDoS Protection

blog.cloudflare.com

61 pointslanakei26 comments

Open articleView on HN

FYI - this is Cloudflare "GA week" [0]. So we're going to see a bunch of beta/early-adopter service made GA this week.

Other services made GA today, at the time of me writing this post:

- Cloudflare One [1]

- Cloudflare WAF [2]

- Domain Scoped Roles [3]

[0] https://blog.cloudflare.com/welcome-to-ga-week/

[1] https://blog.cloudflare.com/cloudforce-one-is-now-ga/

[2] https://blog.cloudflare.com/account-waf/

[3] https://blog.cloudflare.com/domain-scoped-roles-ga/

This is very similar to Google’s Cloud Armour Adaptive Protection[1] which recently fended off a 46 million RPS DDoS attack[2].

[1] https://cloud.google.com/armor/docs/adaptive-protection-over...

[2] https://cloud.google.com/blog/products/identity-security/how...

> Although terminating the encryption was necessary to inspect the traffic and effectively mitigate the attack, the use of HTTP Pipelining required Google to complete relatively few TLS handshakes.

Ouch. I mean, congrats for allowing attack traffic to reach the servers, but you know what... once a request is recognized as "bad", then there are hight chances that other requests on the same http connection will also be bad perhaps?

This is exactly, why a more appropriate mitigation is to issue connection-close (in http1.1 world) or GOAWAY frame (in http2 world).

With that in hand, it's possible to bounce the connection back to the bot. Sure, the bot will retry which will put pressure on TLS. But then, if a source IP is trying very hard to establish many connections it's another strong sign of malicious traffic. In TCP you can rate limit SYN packets and _not_ allow more than X new connections per second, quite easily. Another mitigation is to allow the bot to send http requests, but just not answer them. Allow them hang on the connection, burning the bot's resources (memory, concurrent connetions).

Anyway, the point is: requests-per-second is not a perfect metric for L7 / application attacks. A smart application shoudl be albe to push mitigation up to lower levels (like syn rate limiting, or TLS client-hello signature rate limiting), and _not_ intake large amounts of traffic. A better metric is bot count (or number of unique IP's seen), which was shared in the article and is in mid-range.

To recap, a well designed system should _never_ need to handle 47MRPS. It should be able to packpressure it.

Google doesn't terminate TLS on "the servers". It gets terminated on the frontend load balancers (GFE), which is presumably where the attack mitigation happened.

I used CF Magic Transit via one of their enterprise hosts to secure a game server from high 10-40 gbit/s attacks- worked great... however, it caused random dropped/slow UDP packets for legitimate users for a second every once in a while, and we can never figure out how to solve (my guess was something with Argo). Switched away and the problem went away. This adaptive program sounds similar to Corero SmartWall rules, but for their network vs an appliance.

Did you work with someone at Cloudflare on this?

Sadly no, would have loved to, but had to go through the server host which makes sense as they are the direct customer (albeit maybe not as well invested in the specifics or diligent). I've used MT with 2 different server hosts (one an early partner of yours that moved to Path, and more recently one that still uses MT). Early days, MT had a few leaks (IoT device attacks), which I'm happy to say seems to be patched up so kudos to your team. This intermittent random UDP issue is more recent while I was trying the second host, but couldn't notice a pattern or replicate, so I ended up rolling my own with tc, port knocking, and the largest pipe we could find.

Edit: For others, our problem was probably an edge case, and our use is probably not common. MT seems good compared to the alternative solutions I know so looking forward for an opp to try this if I have ever a need.

Pity. I love debugging weird network things.

These days instead of just spamming data at a site to ddos it, you just spam the twitter account of cloudflare and they drop your site.

What? I don't follow, how do you spam the CloudFlare Twitter account?

The current playbook for when you want to take a site down is to sign up to it and post something illegal like a bomb threat. Then immediately screenshot it and post to Twitter. Have all of your followers retweet the screenshot to the cloudflare twitter account and the company will take down the site quickly.

Doesn’t matter how quickly the sites moderators take your post down. Small site owners are expected to react within seconds while big tech is allowed days.

Useful contribution, defending a cesspit of hate like Kiwifarm. Or Stormfront. Cloudflares stance is extremely admirable but there should always be a limit. Things like Kiwifarm don't need protection.

Infrastructure platforms shouldn't be the deciders of what gets hosted online. Sure, those sites aren't nice places. But I'd much rather have governments take action on them if they are illegal than have CEOs in mega corps decide in combination with the court of twitter.

Services like domain name providers and ddos protection are so essential for putting content online that they should be regulated as utilities. It would be absurd for a power or water company to cut service because they decided something legal but undesirable was happening at a building just because enough people tweeted at them.

There are some services that should be allowed to pick and choose what to provide. A forum like HN or reddit for example is perfectly justified removing whatever they want. If you don't like it, go somewhere else. But once you get to the point where your domain name and phone number are getting canceled, its no longer possible.

Oh, I just figured Adaptive meant that they could adapt to not protecting a site faster, in case of backlash.

Oh good, a new way to break things; sure hope nobody using this expands into new markets or gets a surge of traffic outside of their usual viewers!

Sigh. You don't have to use this. Large companies that want super tailored DDoS protection can. Everyone else can just use our standard DDoS protection which is included with every plan and works: https://blog.cloudflare.com/26m-rps-ddos/

Isn't it frustrating when users complain about new features that they aren't expected to use?

Why is it like that? Is it like this in other industries? Do people that use Coach bags complain when Coach introduces a wallet that they won't use?

I simply do not understand it.

Sort of, at least in fashion. This store https://www.bottegaveneta.com/en-ca/large-pillow-pouch-dark-... caused a lot of mocking for their multi-thousand dollar normal looking shirts for example

Don’t think it’s deserved in Cloudflare’s case though here.

Because when they are personally detected due to a false positive it is a poor experience.

Not commenting on this particular case, but I know in smaller dev shops that time spent working on one feature is directly taken away from time spent on other things, and so if the new feature isn't useful to you, it has also taken fixing and dev time away from ones that might be. I also suspect that many people have seen the trajectory where apps get more and more features and don't focus on their core competency.

Again, I don't believe this is the case here, but it might generally explain the phenomena/misplaced snark.

Of course I'm not using it myself. But the other half of that problem is that this will inevitably be another way that cloudflare breaks my experience as a user and I can't opt out of that.

I am running quite a different browser setup from most regular web users.

I am using Firefox with multi-account containers and uBlock Origin enabled, and I also have an OpenVPN client running, the amount of captchas and distrust I receive from Cloudflare (and their customers, unknowingly I suppose) feels disproportionate.

It's funny to me that these companies most likely spends a lot of time and energy on how to optimize their websites and purchase flows, and they might not know how often Cloudflare puts up barriers and destroy their hard work.

CAPTCHAs are going away: https://blog.cloudflare.com/end-cloudflare-captcha/

I see a lot of sites (including those behind Cloudflare) being blocked when using Cloudfare's Warp VPN. 403s, captchas, etc.

It would be nice if Cloudflare could put abuse control measures on the front-side, and stop the abuse before allowing it to exit their VPN. E.g., if Cloudflare detects abuse with their existing anti-DDoS, anti-scraping, etc. measures, and the client is on Warp, instead of creating a poor experience for every other Warp user sharing that exit IP, just block/aggressively throttle the bad actor's traffic at the VPN terminator.

They have struggled with credential stuffing, captchas were the only way to stop our login pages from being bombarded at one time. They are getting better with it though but I would rather not lose a tool of last resort.

That's an awful lot and weird choice of words just to say that you are not the target audience.

Reply on news.ycombinator.com