Slack account takeovers using HTTP Request Smuggling
hackerone.com
hackerone.com
- If you don't have a proxy fronting traffic, no action required
- If you're behind Fastly/Cloudflare [1] or Akamai [2], no action required / they protect against this attack
- If you're behind AWS Cloudfront, no action required / they protect against this attack
- If you're behind AWS ALB, you're vulnerable by default but can opt-in to protection by enabling the "routing.http.drop_invalid_header_fields.enabled" attribute [3]. They initially had it on by default but it broke customers
- If you have a different proxy (e.g. some other provider or your own nginx, haproxy before 2.0.6 [2], etc), you might be vulnerable
[1]: https://portswigger.net/research/http-desync-attacks-request...
[2]: https://portswigger.net/research/http-desync-attacks-what-ha...
[3]: https://docs.aws.amazon.com/elasticloadbalancing/latest/APIR...
0: https://support.cloudflare.com/hc/en-us/articles/204899617-A...
The trick in Request Smuggling is that you're trusting an intermediary (in this case a frontend reverse proxy) to mingle everybody's requests into a single pile for you to process and they don't agree with you about how to do this. Chuck thus gets to submit a request which is mingled with Alice's and you end up letting Chuck modify Alice's request. Oops.
But Chuck sending requests directly to your backend doesn't allow him to do this. You're definitely not going to think Chuck's weird garbled nonsense is part of Alice's request when it isn't even on the same TLS connection.
So I think I'm vulnerable because my setup is nginx->customhttp and my customhttp doesn't understand transfer-encoding-chunked (if there is such a request I just return an error).
As far as I understand this problem only arises if there is both: content-length and transfer-encoding, in which case content-length will be ignored.
Shouldn't this attack be very easily avoided if nginx just discards the content-length-header in such cases? Why should nginx ever send both to the backend?
If your customhttp isn't smart enough to handle Chunked encoding it might also just always Connection: Close every request or even just act like an HTTP/0.9 server, in which case it isn't vulnerable.
Request Smuggling requires that both the intermediary (for you nginx) and backend (your customhttp) believe it is possible for one TLS connection to contain multiple HTTP requests, they just disagree on where the boundaries are between those requests.
If either of them insists that no, one TLS connection = one HTTP request, that's legal (but has poor performance which may or may not matter to you) and immunises against Request Smuggling.
I haven't testet it yet but I'm pretty sure its vulnerable: my server just looks for content-length and if it doesn't find it, it returns an error. So if nginx sends content-length but then continues to send chunked content it should be possible that the next request won't come from nginx but from an attacker (probably not really a problem at the time but nevertheless not the expected behaviour).
My failure was to always expect "valid" HTTP-input from nginx whereas "valid" means my limited knowledge of HTTP.
But the question remains: Is there a reason why nginx should send both headers to the backend?
Valid means what the standard says it means. What you don't know can hurt you.
However, if modifying your backend code is hard, you can apparently tell nginx that you don't do chunked encoding and it will sort out the 501 on your behalf. I have not tried this and YMMV.
And if this is a bug in the specification, shouldn't nginx fix it to help us backend-fools?
EDIT: Here is the link to my blog post https://medium.com/@emilefugulin/http-desync-attacks-with-py...
The first in-band signalling attack I came across was the blue box [1], invented in the late 1960s. It still occasionally worked in the 1980s on older phone systems. It amazes me that we're still creating new systems vulnerable to in-band attacks.
Most of the books I've read basically paint Jobs as a goody-two-shoes type who cringed at the illegality of phreaking - while saying he probably was thinking ahead, knowing he already wanted to start his own company and didn't want stuff like this to come back and haunt him.
It was an interesting dichotomy to me.
If the back-end server is already proxying whatever request it receives from the front-end proxy server, why was it necessary to first get it to redirect an HTTP 1.1 header to an https request? The only thing I could think of was that maybe it has something to do with getting the unencrypted cookie, but according to their description, the forwarded request with the cookie doesn't happen until after the redirect to https anyway.
I know I'm missing something obvious here.
I understand that the backend is parsing different requests than the frontend, but I don't see how the cookies ever get sent to the attacker's domain.
I understand three requests as moving like this:
1. attacker -> frontend -> backend (partially) -> frontend -> attacker
2. victim -> frontend -> backend (including a bit of 1) -> frontend -> victim
3. (after 301) victim -> malicious server
What am I misunderstanding? When do the cookies get sent to the malicious server?
Skip down to the "Explanation of malicious request" section.
If you take all of this together, it's all leading to Slack's back-end server returning to the user's client a 301 response that tells the user's browser to send a new request to the hacker's domain. However, when the browser does this, it's to a different domain, so typically the slack cookies would not get sent with that request, unless I'm missing something.
The "Explanation of malicious request" section doesn't explain this, either. All it explains is that all this effort was to get Slack's server to send a 301 response to the user's browser pointing to the hacker's server, but then it glosses over any information about how the cookies are getting sent with a single sentence, "and all cookies (including 'd') get redirected there too.... :("
My question is, how?
Is something going on where Slack's back-end servers are explicitly setting response headers that allow their cookies to be sent to the hacker's domain? Is there some additional vulnerability that's allowing the browser to send the cookie across domains on a 301 redirect?
Edit: looks like terom above me had the same idea before I did
Edit: this seems likely to not be what he was saying and also not the case.
The TL;DR is that you can make the front-end see something like GET /my/supplied/url\nX:X at the end of your request, but the back-end see it as the beginning of the next request, (and the X:X turns the real GET/POST whatever info a custom header of X:XGET /real/request/url), and Slack returns that request with cookies for that person, but to a domain controlled by you when it comes back. The diagrams included in the post and the accompanying text do a better job of explaining it than I am, I think.
This is my understanding of the bug:
* Attacker sends malformed request to slack servers. Request gets split in two. First part results in something sent back to attacker, next part of request is treated as a prefix to the next legit (victim's) http request made
* Slack backend server responds to victim's request treating it as having part of the attacker's request prefixed. The merged request results in a 301 http redirect response, redirecting the victim to an attacker controlled domain.
* Victim's browser gets the 301, and follows the redirect. When following the redirect, the cookie header is somehow sent to the attacker's site <-- Part i don't understand here
I don't understand why the victim's browser would send the cookie header when following a redirect to a different domain. I don't understand how causing the victim to follow a cross-domain redirect would allow the attacker to extract the victim's cookie.
That's the part I'm stuck on. Is there some browser behavior I'm not remembering where it sends sensitive cookies across domains just because one redirected the browser to the other?
requests.request("GET", "http://localhost/redirect-to?url=http%3A%2F%2Fhttpbin.org%2Fget", headers={"x-foo": "bar", "Cookies": "abc=dcv;"}).json()
{
"args": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Cookies": "abc=dcv;",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.22.0",
"X-Amzn-Trace-Id": "Root=1-5e6bebe1-cd6e71729a818015a06aa7cb",
"X-Foo": "bar"
},
"origin": "91.58.8.128",
"url": "http://httpbin.org/get"
}
> • when forwarding sensitive headers like "Authorization", "WWW-Authenticate", and "Cookie" to untrusted targets. These headers will be ignored when following a redirect to a domain that is not a subdomain match or exact match of the initial domain. For example, a redirect from "foo.com" to either "foo.com" or "sub.foo.com" will forward the sensitive headers, but a redirect to "bar.com" will not.
https://golang.org/pkg/net/http/#Client
Though this might also cause problems if you are using sensitive non standard headers such as X-Token for token authentication, etc.
So while you could probably mitigate this vulnerability on some clients, you are trusting the server to only redirect you to trusted URLs which is not the case here.
Another way would be to make it so the next request is the content of a param which is displayed in some manner. For example, if it's the content to a message post to a slack forum.
Those different than what they outline in the article, but neither seems unlikely to be found on the back-end in some manner. Redirects are often locked down, but not always, and finding something that sends a variable back to you (such as a form port target that you know will error "helpfully") doesn't seem like it would be hard to me.
> I imagine one way to do it is to make the attacking request to a redirect. Since the back-end conflates the attacking request and the next request (or a portion of the next request), you can get that next request, including the headers (i.e. cookies) as the body/payload redirected to that external site. If the attacker controls that external site, they then can see the cookie data.
Are you suggesting that if the backend server was setup as an HTTP proxy, you could get the backend server to proxy the merged request somewhere else?
That's true, but it seems pretty unlikey that the backend servers would be setup that way.
Although i think i might just be misunderstanding what you mean.
> Another way would be to make it so the next request is the content of a param which is displayed in some manner. For example, if it's the content to a message post to a slack forum.
I.e. you're suggesting basically using this as a CSRF attack to post part of the victims request (including secret cookie) somewhere public to retrieve later?
I think this would work in certain situations but would be difficult to pull off. Its a bit more complicated because if the site was using csrf tokens you would need to make sure the victim request was a POST with a token,but that is easy enough - retry until it happens. The bigger issue is the request body would be kind of malformed. If the endpoint you are posting to accepts application/x-www-urlencoded (and not json), i imagine its interpreted super liberally. I guess the biggest issue is if ; is considered a form field separator (since user-agent usually has ; in it). I think some standards tried to promote that, but i dont know if webservers actually implement it. If they do,it would be very difficult for the attacker to control the name of the form field, and thus actually get the part they wanted publicly posted
I'm suggesting that since the back-end server may see the requests as a single request with the second request as part of teh payload of your crafted request, the question then becomes one of "In what ways can I get this site to show one of my params". That's similar to a CSRF, but simpler in that that you don't care that it's unescaped, you just care that it's in the source in some way, since it's what should be displayed back to you for your request.
My form example is straightforward. Imagine you are requesting http://slack.com/help/search. If you reorder the params slightly, such that the "query" param is last, you might get a good chunk of the next request to be sent to the back-end as your query param, which slack will helpfully display for you both on the page and in in the search input on the resulting page. It will clean our newlines, and likely some other characters, but that hardly seems a problem.
> I.e. you're suggesting basically using this as a CSRF attack to post part of the victims request (including secret cookie) somewhere public to retrieve later?
Yep.
> I think this would work in certain situations but would be difficult to pull off. Its a bit more complicated because if the site was using csrf tokens...
I just found one and used it in an example above. Took me less than 5 minutes (I tried a different form that did have a csrf token first).
> The bigger issue is the request body would be kind of malformed.
That's a problem for if you want it to look legit to someone else. If you just have to note that it has the equivalent of /\sCookie:\s(\S+)\s/ in it... well that's not too bad. Removing the newlines makes it all one string, but in the example above they helpfully replace them with spaces. Even if they didn't, it's probably not hard to see where the next header begins.
> I guess the biggest issue is if ; is considered a form field separator (since user-agent usually has ; in it).
Depends on how you do it. My example above uses GET, so any ampersand (which is probably likely to exist in many queries) will throw it off. A POST might offer more options. Even with the GET, you might be able to run it enough times to find a post as the included second request, and that's less likely to have an ampersand in the URL, and you would probably scoop up all the headers at least. If you post with multipart mime, depending on how lenient it is on a final terminator, you might not have any problem at all if you find the right page to post to.
I did just confirm that that same search page doesn't seem to honor params passed in a POST, so that's good (for Slack).
I think the bottom line is that once you can make part or all of someone else's HTTP request show as part of your request's payload, it's a drastically lower bar for exploiting, as there are many possible ways to exfiltrate the data at that point, and they only have to screw up by allowing one of them to work.
But the response to the mangled request goes to the victim not the attacker. You need someway to get it from the victim to the attacker (hence use of redirects in the original bug bounty).
> My example above uses GET, so any ampersand (which is probably likely to exist in many queries) will throw it off.
The point i was trying to make is that various specs suggest that semicolons should be treated as an alternative to amparsands, and they are much more common in headers (especially in user-agent). However i imagine much server software doesnt follow that reccomendation so maybe its a moot thing to worry about.
That's the route they took, but it's unclear to me if it wouldn't work the other way as well (or if that's more likely to work with TE.CL setup than the CL.TE one in place.
If you can use the difference in what is specced to make the back-end think your request is smaller than it is, can you reverse that to make it think it's bigger than it is, so some of the next request leaks in? If that makes the next request invalid, is that gracefully handled by failing that one (victim) request, or does it cause the whole set of requests to fail? I don't know the answer to these questions, but it certainly seems like there's a lot to explore here.
I find it hard to believe that any browser would keep the original `Cookie: ...` headers in a redirected request to a different origin.
In that case, I'd expect all cookies etc. to be forwarded.
I still wouldn't expect an Electron app to subvert basic browser sandboxing by default, particularly where they wouldn't have expected to need to redirect users to other domains with cookies intact. It seems like they'd need to go out of their way to enable that.
I wonder if it has to do with the sign-in tokens they send or otherwise allowing the user to move between the browser and the app within their account. For example, when you're in the app and click "Manage Users" and it sends you to a management dashboard in the browser. or when you click a link with an auth token in the browser and it launches you into the app.
Another approach is reliable encapsulation. Ethernet and TCP/IP, for example, are technically all in one channel. But packet content won't be mistaken for protocol information. You could also look at things like SSH and how it has multiple channels: https://tools.ietf.org/html/rfc4254
I'm sure there are other approaches, too, and I hope people will mention them.
Edit: Found it here: https://github.com/gwen001/pentest-tools/blob/master/smuggle...
I didn't expect that either. I am very pleasantly surprised. I hope my own company would do as well as Slack did here, but I am not certain whether we would.
How do people learn this stuff? Are there any resources that anyone here can recommend?
Realizing that different systems resolve the question of "Well, what should I (the system) do when I get both headers?" and you realize you can break the atomicity of the HTTP request-- which opens up a really interesting (and severe) class of exploits.
I'm not very good at finding vulnerabilities, beyond the obvious sql-injection attack types that we are generally trained to avoid. I imagine, however, getting good at finding these is a skill that can be learned with time. Alternatively, finding vulnerabilities might be more akin to pharmaceutical developments: with a 1,000 swings, you're missing at least 950 of them, or more. A dash of luck in there.
Black Hat talks are eventually published online for free; here's this one:
It also reminds me of the issue of character encoding in (x)html, where you have different ways to specify it (http headers, xml declaration, meta tags...) and if they don’t agree you can get corruptions downstream.
Does this mean they wouldn't want to disclose it if the same exploit was used against users by another hacker?
Wait, 6500 * 4 * 12 = 312,000/yr
Might be a bit high for a week :P
EDIT: Okay turns out that if you live in the Bay area this isn't unheard of - the rest of us make 4x to 5x less then that (Saying this as a mid-level software dev from West Michigan).
software dev. salary / average country salary
ratio of all countries on planet.
If only other companies would realize that that's one of the reasons why SAP is Germany's no. 1 software company.
Some people build 10 different static website generators, others do bug bounties. It doesn't mean they'd go on to sell these exploits and risk going to jail.
When it comes to vulnerabilities with a large enough impact it isn't enough to learn about most of them, because all it takes is one financially motivated actor to weaponize things.
This is reasonable if you look at it as "just another job" -- you're being paid to build Good Software, so just another day at work. Or you're doing a Good Thing by helping a lot of people not get pwned.
This is unreasonable if you look at is as value-creation: "how much is this worth on the black market" or "what is this worth to Slack as a company".
Other people can get into the socioeconomic or means-of-production or entrepreneuring implications of all this, but I just think whether you downvoted or upvoted this provides a useful mirror into how one values one's own professional work.
Leads to the conclusion that they should pay for the results and not for the time it took to find it. Also, there might have been weeks of failed attempts before finding this.
I'm looking at it from the perspective of the market economics. Reward programs are about incentivizing people to do responsible disclosure. If the market (i.e. the black market here) pays significantly higher for an exploit then a reasonable company will try to reflect their payout to match what the "market" has valued that exploit to be worth. This way someone who would otherwise have sold on the black market may be incentivized to do responsible disclosure instead (significant payout, maybe not as high but 100% legal & no legal risk). It's all about shifting the incentives and structure before people even make any decision. I think it's silly that companies get dinged that responsible disclosure programs don't pay out at the same rate as the black market. There's a legal risk element their not factoring into the math. But it should be roughly comparable (my uneducated gut check is within ~20%).
Think of it like drugs. Marijuana on the black market is cheaper. People still opt to buy legal marijuana even though it's slightly more expensive because it's safer, legal, & vendors are accountable to their customers & community. If the cost grows too large then the black market starts to grow again (e.g. cigarettes are a notorious example of this due to taxation as an attempted lever to kill it).
https://i.blackhat.com/USA-19/Wednesday/us-19-Shwartz-Sellin...
And you can always look at things like the Zerodium Price List. I don't know that it's taken very seriously in its particulars, but the general structure of it mirrors what I've heard from other sources.
You'll notice on the Zerodium list that they will pay for serverside RCE in web apps --- but only a particular kind of web app: the kind that is deployed in lots of places. National IC agencies will, for instance, pay for phpBB RCE, because they have targets that use phpBB, and there are lots of phpBB's (when you hear people talking about how valuable a web bug is, ask yourself whether that person has mentioned the weird market for phpBB bugs --- something I've had firsthand [refused!] experience with). What you won't see are bugs in SAAS applications. Again, the reason is that a phpBB vulnerability has a half-life: everyone has to install the patch once it's burned.
I have no evidence for what I'm about to say here, so take it with a grain of salt:
I assume you can get paid for a vulnerability even as esoteric as this Slack ATO bug. But you'll get paid for people who are buying Slack accounts, not Slack bugs. That is: you'll have to be the one exploiting it, and you'll be making one-off deals to use it to get targeted accounts. People sell all kinds of accounts; it would not surprise me even a little to hear that there was a market for company Slack accounts.
But to participate in that market, you'd almost certainly have to directly enter a criminal conspiracy. You wouldn't be selling a bug to a market; you'd be participating.
For me the plausible deniability stops at something like the firm we hired (years ago, they were subsequently bought by CSID which was in turn purchased by another of my employers because it's a small world and apparently some people have unbounded appetite for risk) which steals credentials from people who steal credentials.
I can just about imagine sleeping after doing that all day. Because at least some of the time, indirectly, and after somebody gets paid, you're helping. It'd be like working in insurance sales. Your employer creams a profit off misery, but arguably the misery would be worse without them, maybe.
But any of these "Well I could get more money from bad guys" arguments miss in my opinion the most important constraint which is that most people don't want to work for the bad guys. Peanut farmer or crook is an easy question for most of us.
There is also the cost of the likelihood of being caught while selling vulnerabilities on the black markets. If fascists have some personal stake in the company, black hatting would likely involve more careful and high stakes anonymity measures. Remember that US federal law enforcement agencies operate tor exit nodes!
Bug bounties are the uberification of security research.
I don't understand unsolicited complaining for other people.
They're suggesting that the severity of such a bug warrants a larger payout, because not doing so creates possible incentives for future explorers to consider selling these sorts of things on the black market.
This person may be satisfied, the next person that finds something similar may think twice.
If you observe someone getting paid a much lower-than-market amount for something you can complain that the company is being cheap and likely driving away a lot of potential sellers (security researchers in this case) regardless of how happy the one person is.
Would you feel any differently about the value of this bug if it affected, e.g. Google or Facebook?
People on HN seem generally to believe that for any malicious activity you could do with a bug, there's a bidding group of willing buyers somewhere on some darknet site. That's not the case. Random bugs like this may get passed around, but the bugs that command a price all fit a couple specific molds: they're things you can drop into someone's existing operational process.
A Firefox drive-by RCE has some value: many organizations are set up to actively exploit Firefox browsers. So does an iOS jailbreak: lots of people stockpile iOS jailbreaks, for malware implants and for other purposes.
An important common thread among the bugs with liquid markets is that they have a meaningful half-life: once they're burned, it still takes time to eradicate the vulnerable installations. Serverside bugs are fixed worldwide instantaneously. You can see this dynamic in how grey-market payments are tranched.
Assuming he has done this for several other companies, I would say he has effectively earned more than what is put in