How Cloudflare blocked a monumental 7.3 Tbps DDoS attack
blog.cloudflare.com
blog.cloudflare.com
> How it works: Abuses the Quote of the Day (QOTD) Protocol, which listens on UDP port 17 and responds with a short quote or message.
Does any reasonable operating system those days support this protocol? Sounds like "IP over Avian Carriers" to me.
Of course nearly all of them are a long paragraph or two of legal jargon that more or less boils down to "fuck off."
_______ __ __ __
|_ _|.-----.----.| |--.-----.|__|.----.-----.| |.-----.----.
| | | -__| __|| | || || __| _ || || _ | _|
|___| |_____|____||__|__|__|__||__||____|_____||__||_____|__|
N E X T G E N E R A T I O N G A T E W A Y
--------------------------------------------------------------------
NG GATEWAY SIGNATURE DRINK
--------------------------------------------------------------------
* 1 oz Vodka Pour all ingredients into mixing
* 1 oz Triple Sec tin with ice, strain into glass.
* 1 oz Orange juice
--------------------------------------------------------------------If you take their random source ports (21,925), ~0.004% come from any single port, which lines up with what they said was "Other" traffic. The numbers don't quite work out right, but it seems like its within a factor of 2, so I wouldn't be surprised if it was something like udp source/dest port = 17 => QOTD.
Also worth noting that the article specifically mentions how owners of ASNs can subscribe to CloudFlare at no charge to get notified when devices on their network are participating in attacks like this.
Not a new problem per https://krebsonsecurity.com/2014/02/the-new-normal-200-400-g...
a funny thing happens when you decide to operate a DDoS-for-hire Web service: Your service becomes the target of attacks from competing DDoS-for-hire services. Hence, a majority of these services have chosen to avail themselves of Cloudflare’s free content distribution service
Cool artifact of the internet!
A: Cloudflare is feeding the trolls because they think that they are invincible. Or: These post-mortems don't establish any proof that the attack was successful, especially if they are covering DDoSes that were barely even noticed by the public until CF publishes a blog post 1 month later -- so it's actually embarrassing for them and hurts their ability to market botnets for rent, at least once they no longer have the literal world record.
B: Cloudflare is feeding the trolls for free testing scenarios to improve the mitigation
C: The trolls don't really care if you feed them, large DDoS is something that's happening all the time anyways
Isn't the obvious point that this is Cloudflare advertising their anti-DDoS services?
Also it makes no sense - why throw 7.3tbps at a server, when half of that or less would have the same effect and not exposed your botnet.
Since this is Cloudflare, my headcanon is that it was a rival DDOS service, after a wild flamewar on some .ru hacker forum.
I think an increasing amount of them are state actors or groups offering the botnet as a service.
Living their best "Im a retail Asus router/iot from Amazon" life.
I see no issue in handing out similar punishments in the digital space. The Internet is a shared medium, everyone who connects to it has a responsibility to not be a nuisance to others.
The main ingredient of crime is intent, whatever you say. A smaller ingredient can be recklessness, but maybe it's the ISPs sending all those millions of empty packets to a single server that should start feeling some heat ?
Yeah, not kill, but participating in a DDoS against a heavily frequented commercial site that makes hundreds of thousands of dollars of revenue a minute, that's still some substantial damage.
In the end it should boil down to the ability of holding the seller of the product with security issues accountable for the damages, and the seller in turn can hold the manufacturer accountable. Maybe that will lead to some substantial change.
A person who buy all reason has absolutely no idea how any of this shit works. If you wanna go after somebody go after the manufacturer of said baby monitor.
Look at the moment, the owner of devices that participate in DDoSes are not head liable, and neither are manufacturers who don't secure their shit.
This needs to change.
At the end of the day it is very difficult to impose security management across consumers. You cannot expect the average consumer to pen test their home network and have active vulnerability scanning software to mitigate potential vulnerabilities that result in Botnets.
It is difficult to hold people liable when someone else misappropriates their assets in a way that was not its original intended purpose. When its difficult to capture the perpetrator people start to blame everything else, that doesn't mean we should just shift liability to the buyer who is just simply an easier target to place the blame on than a random unidentified person in another country.
That may sound like a solution but its not the right one. Now someone has the ability to misappropriate your assets from the other side of the world and you become charged with the crime, when all you did was buy a new Samsung TV. Heck knowing that, maybe someone would target you knowing full well you'd be in trouble for it.
It sounds like that hypothetical site has an interest in not going down if a random baby monitor sends traffic their way.
Also, to underline how silly and poorly thought through your idea is, are you aware that there are nearly 200 countries out there, each of them with many people with their own internet connections? Or are you expecting DDoS botnets to be comprised exclusively of devices in your jurisdiction?
If someone broke into my car and drove it into a wall, I highly doubt I'd be found at fault. If someone broke into my IoT device and used it in an attack I highly doubt I should be found at fault.
At the end of the day it is very difficult to impose security management across consumers. You cannot expect the average consumer to pen test their home network and have active vulnerability scanning software to mitigate potential vulnerabilities that result in Botnets.
It is difficult to hold people liable when someone else misappropriates their assets in a way that was not its original intended purpose. When its difficult to capture the perpetrator people start to blame everything else, that doesn't mean we should just shift liability to the buyer who is just simply an easier target to place the blame on than a random unidentified person in another country.
That may sound like a solution but its not the right one. Now someone has the ability to misappropriate your assets from the other side of the world and you become charged with the crime, when all you did was buy a new Samsung TV. Heck knowing that, maybe someone would target you knowing full well you'd be in trouble for it.
What leads you to believe this is a tech illiteracy issue? Do you believe that only consumer IoT devices are unwitting participants in DDoS attacks?
If laws like the Cyber Recillience Act have been put into effect long enough for us to see a significant improvement in device security, then we can softly begin to rule out that cause and focus on the user.
We are not there yet, because (imo) devices are still not properly secured enough for us to go straight to blaming the user.
You're talking as if people can't just get another account or change ISPs.
Also, it seems you're mainly interested in gratuitously punish people who are powerless about issues instead of thinking about very basic approaches such as rate-limiting policies.
And increasingly, 2gbps, 4gbps and 8gbps.
It’s great, mostly.
As the internet gets more users and more devices connected, the ratio of DDoS volume to a single connections volume will only get larger.
Is there any kind of solution?
1) performed egress filtering to prevent spoofing arbitrary source addresses
2) temporarily shut off customers that are sending a large volume of malicious traffic
How would an ISP determine egress is malicious? Genuinely curious.
For example, one method has the attacked IP get completely null-routed, and the subsequent route is advertised. Upstream routers will pick up the null-route advertisement and drop the traffic ever closer to the source(s). The effect of the null route is that the attacked IP is unreachable by anyone until the null-route is lifted... so the aim of the DDoS isn't averted, but at least the flood of traffic won't pummel any network paths except for (ideally) the paths between the attacker(s) and the first router respecting the null-route. In my experience the DDoS tends to stop more quickly and shift away to other targets if the folks directing the attack can no longer reach the target (because: null-route) and then the null-route can be lifted sooner relative to a long-running DDoS that hasn't shifted away to other targets.
I wonder if this would work in reverse, having a standardised, automated protocol that allow providers like Cloudflare to notify upstream networks of attacks in real time, so malicious traffic can be blocked closer to the source.
Genuinely curious, I'm not an expert in low-level networking ops.
We pay internet providers healthy amounts of money each month. Surely they can afford to hire some staff to monitor the abuse mailbox and react on it - we know they can when the MAFIAA comes knocking for copyright violations, because if they don't comply they might end up getting held liable for infractions.
- ISP has terms of service preventing abuse,
- ISP provides an email address to receive complains about abuse
- once a ISP receives a complain, their check if a customer abused their terms of service
- once a ISP spots a customer abusing terms of service, they act upon it.
ISPs have been doing this since the time ISPs exist.
(1. I know this, because I used to work for a company that made them, and the majority of worldwide ISPs were our customers.)
Note that spoofing source IPs is only needed by the attacker in an amplification attack, not for the amplyfing devices and not for a "direct" botnet DDOS.
Nowadays my ISP just uses dhcp to assign the router an address so you can plug any box into it which talks ethernet and respects dhcp leases to be a router which is nice, albiet 99.9% of people probably leave the router alone.
20 years ago Cisco (probably much longer) routers were able to do this without noticeable performance overhead (ip verify unicast reverse-path). I don't think modern routers are worse. Generally filtering is expensive if you need a lot of rules which is not needed here.
E.g., customer does something stupid with addresses but the “wrong address” is something they control on another network, so it works. Egress filtering breaks it, support call and crying.
More advanced attacks are more tricky to detect, but plain dumb UDP flood should be easily detectable.
You mean my legitimate QUIC file transfer?
And the aggregate across the ISP's network could in theory be monitored - so if you were uploading 1Gbps, yes, it could be legitimate. If you and 582 others were all uploading 1Gbps to the same IP at the same time, much less likely legitimate.
Yes actually --- migration between cloud bulk storage providers.
Edit: I misread Gbps as Mbps above.
I.e. no traffic beyond my legitimate saturation can reach the ISP
I have saturated my link with quic or wireguard (logical or) plenty of times.
The lack of any response on high data rates would be an indicator I've only tried that once and it failed gloriously due to congestion. I don't think there's many real protocols that are unidirectional without even ACKs
I upload files to a single location, and I expect to use the max bandwidth I can whenever I do it. What's your point?
IoT devices (speculated to be used here) would have to have a solution upstream. Things like MUD (RFC 8520) have been proposed, but have problems too - developers need to be able to list all communications of their device and make that available somehow (MUD profile server). Some consumers will never do it on their own, and may want to prevent alerting a device manufacturer they have a device (think connected adult toy...).
Also given that IoT devices may never be updated by their owners, expect to see IoT botnet DoS attacks for years.
BCP 38 is applicable in the DC environment, especially between an operator (hosting/cloud provider) and the customer. Where it is from hard to not practical to use is the network backbone and link between different ISPs. But that's would be a minor problem if BCP 38 will be applied to all stub networks.
Sorry for the worst and most hated possible solution, but I thought I'd at least mention it.
Maybe too many failed capachas causes you to not connect to the IP for an hour.
The default is to allow all available bandwidth, which presumably should be the case from ISP to consumer (most likely a paid-for service), but why should that be the default at consumer router <-> IoT? What need has your printer for 500Mbps outgoing? Or my fancy toothbrush?
If it was automatically accepted, the malware would just change the advertisement.
Heuristic based systems would probably work in most homes, where devices are limited by their historical bandwidth. New devices are unthrottled, existing devices are limited by their historical bandwidth usage with some bursting.
I think most ISPs have apps to control your router now, you could have it trigger a push notification like "Device X is using more bandwidth than normal, and we're throttling it. Press SCARY BUTTON to unthrottle."
I'm not sure if that's the case. Large volumetric DDoS records have been increasing, but connection bandwidths have also been increasing.
7 tbps is a lot of traffic, but it only takes 7,000 nodes with 1G symetric connections to do it. Botnet sizes don't seem to be getting that much bigger.
The basic solution to volumetric DDoS is to get a bigger pipe; this works, kind of, but it's hard to get 7 Tbps of downstream capacity, and you need to be careful that you don't become a 7 Tbps reflector.
The more scalable way is using BGP to drop traffic before it gets to you. Depending on your relationship with your hosting facility and their ISPs or your ISPs, it's often pretty easy to get packet to a given IP dropped one network before yours. Ocassionally, those blocks could propagate, and things like BGP Flow Spec promise more specific filtering... dropping all packets to an attacked IP mitigages the attack for the rest of the IPs on the path, but dropping all UDP to an attacked IP might get all the attack traffic and let most non-attack traffic through... More specific rules are possible if you wanted to try to let DNS and HTTP/3 survive while being attacked.
To work against a 45 second attack, BGP based measures need a lot of automation.
This represents like 75% of surveillance camera systems out there btw.
"Your network is generating an extraordinary amout of traffic, which is likely the result of a virus-infected device. As a result, we have lowered your speed to 100/20. Please read the steps to check your devices and unlock your connection here: ____"
Lowering the speed means "good" traffic is also impacted, resulting in higher timeouts.
count the number of events isn't cheap either.
Most ISPs are already a pain in the ass to deal with. (Fuck you Charter/Spectrum). I don’t trust them to do their due diligence and implement this correctly. Or worse, abuse it.
“hey you pay for 1000/300 package. We detected abnormal traffic. Now you get throttled to 100/100. But still pay 1000/30”. Then they will drag on the resolution process until you give up.
Blocking DDoS is like trying to separate the shit from the bread in a shit sandwich.
It's a completely different problem.
App level DOS use Cloudflare evasion techniques and directly DOS the destination server, while keeping itself undetected by Cloudflare's systems.
Do not assume that Cloudflare will protect you from all attacks, if your app is dogshit python/js/php then even cloudflare wont protect you from L7 DDOS
- testing in preparation for a future attack
- proof of capability ("Nice network you have there. It'd be a shame if something happened to it")
- misfire ('What happens when I push this button that says "don't push"?')
IMHO, ISPs caught in that act should get yanked off the internet.
One-Punch Man is a reluctant mentor, is often broke, loves ramen and cares about others.
They are not the same.
This is a bit misleading; according to Wikipedia[1], the first DDoS is said to have occurred less than three decades ago.
[1] "Panix, the third-oldest ISP in the world, was the target of what is thought to be the first DoS attack. On September 6, 1996, Panix was subject to a SYN flood attack, which brought down its services for several days while hardware vendors, notably Cisco, figured out a proper defense.", source: https://en.wikipedia.org/wiki/Denial-of-service_attack
1) I'm not sure what your problem with the reasonable rounding of 29 years ago to 3 decades is... but the one that comes across is "extra pedantry for no reason"
2) According to wikipedia the "first dos" attack was in 1996. There are other sources most of which attribute that 1996 panix attack as "one of the first" or "the first major" ddos attack. Before that there were other DoS attacks using udp and/or syn floods, and some of them likely involved several computers (and possibly people) working in coordination. Those several computers were probably not compromised machines that had malware responding to a cnc server, so the squishiness has to do in part with how exactly one defines DDoS - some definitions include a botnet requirement, others just need multiple computers working in coordination. It's claimed that Kevin Mitnick was targeting his prosecutor with syn floods in 1994 (over 30 years ago), but its not fully verified and the details are unknown from my research... likely though >1 computers were involved in that flood if it happened.
In the early 90s there were all sorts of fun and games where people would knock over IRC servers by triggering bugs/behaviors in a lot of connected clients. It's primitive but it seems to have a huge number of elements of DDoS. Similar for attacks on various telecomms infrastructure as the soviet union/eastern bloc fell apart in that time period.
Trying to put a hard "29 years ago" line in the sand is difficult to do... techniques evolve from previous ones and there are shared elements that make the line necessarily fuzzy.
So yeah... theres no reason to quibble about "three decades" since theres 35+ years of history around "things that look like DDoS attacks but don't fit a strict definition that requires botnets"
In my defense, reading that for the first time gave me an impression that DDoS attacks themselves were older; I was disappointed and wanted to share so that others wouldn’t get similar hopes. Next time I’ll round more decimals.