The novel HTTP/2 'Rapid Reset' DDoS attack
cloud.google.com
cloud.google.com
The largest DDoS attack to date, peaking above 398M rps - https://news.ycombinator.com/item?id=37831062
HTTP/2 Zero-Day Vulnerability Results in Record-Breaking DDoS Attacks - https://news.ycombinator.com/item?id=37830998
https://www.nginx.com/blog/http-2-rapid-reset-attack-impacti...
By relying on the default keepalive limit, NGINX prevents this type of attack. Creating additional connections to circumvent this limit exposes bad actors via standard layer 4 monitoring and alerting tools.
However, if NGINX is configured with a keepalive that is substantially higher than the default and recommended setting, the attack may deplete system resources.
I'm surprised this wasn't foreseen when HTTP/2 was designed. Amplification attacks were already well known from other protocols.
I'm similarly similarly surprised it took this long for this attack to surface, but maybe HTTP/2 wasn't widely enough deployed to be a worthwhile target till recently?
I tried looking it up and couldn’t find an authoritative answer. Can you recommend a resource that you like for this subject?
That is technically any HTTP request that requires processing to satisfy. For example if I find a page on your site that executes an expensive database query.
Amplification attacks are generally defined as packets that can be sent with a spoofed source address that result in a larger number of packets being returned to the spoofed victim.
One of the 2019 vulns https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-9514 even sounds extremely similar to current attack.
I mean, who do you consider authoritative? Googling "amplification attack" will give you plenty of descriptions from tons of sources. Take your pick. Though most will talk about DNS amplication attacks because that's the simplest example.
As with most things like this, probably many hundreds of unimportant people saw it and tried it out.
Trying to do it on Google, with a serious effort, that's the wacky part.
If I were the FBI, I'd be looking at people with recently bought Google puts expiring soon. I can't imagine anyone taking a swing at Google infra "for the lulz". Also in contention: nation-states doing a practice run.
This is exactly the kind of things that a smart kid who's still just a foolish highschool student would do. I wouldn't be surprised if this attack already exists in the wild, it's not hard to write
Also the subsequent attacks were less effective, that's exactly what some kid would be doing.
You don't even need an expansive botnot. A rich kid whose parents are in neighborhoods with residential fiber with a bunch of friends could probably coordinate it through a discord server
Most of us really don't interact with teenagers regularly so we forget they're out there (they also tend to dislike adults so they make themselves especially invisible around us). When it comes to things like this, that's my first assumption until further evidence.
Merely teenaged graffiti for the digital age.
[1] Add up the "Open int" numbers here https://www.nasdaq.com/market-activity/stocks/goog/option-ch...
On the other hand, SYN cookies were introduced in 1996, so there's definitely some historic precedent for attacks in the (victim pays Y, attacker pays X, X<<Y) class.
It seems that they're more upset by the state of web development today than they are by HTTP2 or anything else that this thread pertains to.
It is _much_ faster, cheaper, and easier to build a bloated website than an optimized one. Similarly, it is much easier to enable HTTP2 than it is to fix the root of the problem.
I'm not saying that it's right -- anyone without a fast connection or who cares about their privacy isn't getting a great deal here.
After all, you don’t need bloat to suffer from head-of-line blocking. You just need a few images.
(Though, personally I’m a much bigger fan of HTTP/3 than HTTP/2. With a more principled solution to head-of-line blocking and proper 0-RTT, HTTP/3 makes a stronger case for why we need a new protocol than HTTP/2 did. I don’t know why HTTP/2 had to exist at all, really, when QUIC already existed by the time HTTP/2 was being standardized. Oh well.)
But it is in the context of the 3-way tradeoff we're talking about here. complexity of the site vs. load time vs. protocol complexity
> You just need a few images.
On the HTTP level those can be deferred after the html/styles/js. Then you already have the content. What on your site would be "blocked" at that point? It's just images holding up each other.
On the TCP level SACK and FRTO should resolve most instances of HOL after 1 RTT. It's not perfect but I suspect a lot of people experience "slowness" not because the underlying protocols are bad but because they're on old implementations. Or because they're on networks with bufferbloat. Upgrade those and we don't need those complex workarounds.
As for HTTP/3... it's a mixed bag. The basic idea is great. The execution is another googleism. They didn't have the patience to get it into OSes, so now every client has to implement its own network stack which multiplies the things that need patching if something goes wrong. And it runs over UDP instead of being a different transport on the IP level like SCTP. And TLS is a good default but the whole CA-thing shouldn't have been mandatory. And header compression also seems like a cure for a disease of their own making, compare which the number of headers you need for HTTP 1.0.
Also, they tried prioritization, but it was too unwieldy in practice, the browser vendors didn't agree, and it was deprecated in the latest RFC 9113.
Unfortunately, most computers only pass TCP and UDP (Windows and middleboxes). So, protocol evolution is a dead end.
Thus you have to piggyback on what computers will let through--so you're stuck with creating an HTTP flavor of TCP.
Luckily, HTTP/1.1 still works. You can always enable it in your browser configuration and in your web servers if you don't like the protocol.
HTTP/3 does that in my experience (lots of train rides with spotty onboard Wi-Fi) quite a bit better though. As HTTP/2 is still affected by head-of-line blocking and a single packet loss can block all other streams, even if the lost packet didn't hold data for them.
Avoiding designing in DDoS relay/amplication vectors requires luck or intention, not just making the protocol small.
DNS was created for a different environment, at a time when security wasn't at forefront so it's not a good example of the opposite.
Abuse and abuse vectors vary wildly in complexity, some complexity is certainly required exactly to avoid dumb bottlenecks if not vulnerabilities. So based on what are you saying something simple will inherently resist abuse better?
That feels intuitive in the "less code is less bugs is less security issues" sense but implies that "secure" and "can't be abused" are the same thing.
Related? Sure. Same? No.
Oddly enough, we probably could have prevented the replay/amplification dos attacks that use DNS by making DNS more complex / adding mutual authentication so it's not possible for A to request something that is then sent to B.
In practice though the only way to "fix" DNS that would've worked in the 80s would've probably been to require the request be padded to larger than the response...
It's not as complex as some "mutual authentication" scheme though lmao
Like the others on that list (off by one, cache invalidation etc) it isn’t actually hard-hard, but rather underestimated and overlooked.
I think if we took half the time we spend on creation, constructors, initialization, and spent that design time thinking about destruction, cleanup, teardown, cancelation etc, we’d have a lot fewer bugs, in particular resource exhaustion bugs.
Now they are telling us a yarn about how they are heroically saving us from the problem they created, but without mentioning the part that they created it.
The nerve of these tech companies! Microsoft has been doing this for decades, too.
With HTTP/1.1 you could send one request per RTT [0]. With HTTP/2 multiplexing you could send 100 requests per RTT. With this attack you can send an indefinite number of requests per RTT.
I'd hope the diagram in this article (disclaimer: I'm a co-author) shows the difference, but maybe you mean yet another form of attack than the above?
[0] Modulo HTTP/1.1 pipelining which can cut out one RTT component, but basically no real clients use HTTP/1.1 pipelining, so its use would be a very crisp signal that it's abusive traffic.
* HTTP/1.1: 1 request per RTT per connection
* HTTP/2 multiplexing: 100 requests per RTT per connection
* HTTP/2 rapid reset: indefinite requests per connection
In each case attackers are grinding down a performance limitation they had with previous generations of the attack over HTTP. It is a request flood; the thing people need to keep in mind is that HTTP made these floods annoying to generate.
https://developer.mozilla.org/en-US/docs/Glossary/Round_Trip...
Servers should switch to closing the connection if clients exceed the stream limit too often, not just ignoring the bogus streams.
Is it just that the kernel doesn't really communicate TCP RST all that well to the application, so the HTTP server continues to count the connection against the "open connection limit" even though it isn't open anymore?
You're right, with HTTP/1.1 we have single request in-flight (or none in keep-alive state) at any moment. But that doesn't limit number of simultaneous connections from a single IP address. An attacker could use the whole port space of TCP to create 65535 (theoretically) connections to the server and to send requests to them in parallel. This is a lot, too. In pre-HTTP/2 era this could be mitigated by limiting number of connections per IP address.
In HTTP/2 however, we could have multiple parallel connections with multiple parallel requests at any moment, this is by many orders higher than possible with HTTP/1.x. But the preceeding mitigation could be implemented by applying to the number of requests over all connections per IP address.
I guess, this was overlooked in the implementations or in the protocol itself? Or rather, it is more difficult to apply restrictions because of L7 protocol multiplexing because it's entirely in the userspace?
Added: The diagram in the article ("HTTP/2 Rapid Reset attack" figure) doesn't really explain why this is an attack. In my thinking, as soon as the request is reset, the server resources are expected to be freed, thus not causing exhaustion of them. I think this should be possible in modern async servers.
Opening new connections is relatively expensive compared to sending data on an existing connection.
> In my thinking, as soon as the request is reset, the server resources are expected to be freed,
You can't claw back the CPU resources that have already been spent on processing the request before it was cancelled.
> By request flood I mean, request flood, as in sending insanely high number of requests per unit of time (second) to the target server to cause exhaustion of its resources.
Right. And how do you send an insanely high number of requests? What if you could send more?
Imagine the largest attack you could do by "sending an insanely high number requests" with HTTP/1.1 with a given set of machine and network resources. With H/2 multiplexing you could do 100x that. With this attack, another 10x on top of that.
This is harder for the client than it is for the server. As a server, it's kind of not great that I'm wasting 64k of my connections on one client, but it's harder for you to make them than it is for me to receive them, so not a huge deal with today's servers.
On this attack, I think the problem becomes if you've got a reverse proxy h2 frontend, and you don't limit backend connections because you were limiting frontend requests. Sounds like HAProxy won't start a new backend request until the pending backend requests is under the session limit; but google's server must not have been limiting based on that. So cancel the frontend request, try to cancel the backend request, but before you confirm the backend request is canceled, start another one. (Plus what the sibling mentioned... backend may spend a lot of resources handling the requests that will be canceled immediately)
I can send ~1 Million UDP packets per second from one machine. So thats 100 million HTTP requests per second you have to deal with. And when I bring in my 20,000 friends, you need to deal with 2 trillion requests per second.
I'd say that's still a problem.
(I'm fudging things a bit. You can probably build attacks that look kind of similar, but we don't think you you could build anything that is actually scalable. But we could be wrong about that! Hence the recommendation to apply similar mitigations to HTTP/3 as well, even if it isn't immediately vulnerable.)
It seems HTTP2 is TCP on TCP for HTTP messages specifically. This must be why HTTP3 is over a UDP based protocol.
HTTP3 is using UDP for different reasons, although it effectively re-implements TCP from the application POV (it's still HTTP under the hood after all). Basically with plain old TCP your bandwidth is limited by latency, because every transmitted frame has to be acknowledged - sequentially. Some industries/applications (like transferring raw video files over the pond) have been using specialized, UDP-based transfer protocols for a while for this reason. You only need to re-transmit those frames you know didn't make it, in any order it suits you.
And the throttling seems even simple: give each IP address an initial allowance of A requests, then increase the allowance every T time up to a maximum of B. Perhaps A=B=10, T=150ms.
You can't simply blacklist weird connections entirely, since legitimate clients can use those features.
If you are making one or few requests per IP you don't need this attack, and also aren't likely to have any effect on a Google-sized entity.
Are you tracking per connection? Per stream? Isn't it normal for multiple requests to happen quite quickly? I load a single page with 50 external assets, those get multiplexed over the current stream - is that okay? Is that abusive? The other stream is handling a video player and its requesting (http2) frames of video data - too much? Too fast?
Dont forget about nghttp2