Hijacking Email with Cloudflare Email Routing
albertpedersen.com
albertpedersen.com
Once this issue was fixed we investigated all prior email routing configurations to ensure that this had only been found as part of Albert's responsible disclosure to us.
Since some comments are addressing that this happened 7 months ago. Our disclosure policy is to allow researchers to write about us once the issue is fixed, but give us a week heads up before they publish so we aren't surprised, can coordinate any public comms we want to make, FAQs that need to be written for inbound questions from customers, and can tailor our response to the issue at hand. Can answer other questions if you have any.
We disclosed the issue here earlier this week once Albert told us he was writing a blog: https://hackerone.com/reports/1419341
Ideally you want both but it doesn't always work out that way.
After all, it got them this responsible disclosure.
It wasn't part of the vulnerability, it just allowed OP (who happened not to be legitimately in the beta) to find it.
Do we really think Cloudflare Email Routing private beta was private to somehow trusted parties only though? Presumably 'N-mutual trusted parties' too, for regulatory compliance. I assume not; not least because the product security lead is here in the comments saying they vetted logs etc. after the fact to ensure that only OP took advantage of this.
this isn't some kid writing their first dynamic webpage, it's a public multinational that proxies a large percentage of the internet as security product
the fact this wasn't caught internally is extremely worrying and makes me wonder what other sort of basic quality issues they have lurking
But client managing to opt itself in to a beta, as my GP comment was about, is no big deal. Worst case you're getting something free that you're supposed to pay for.
Beta security shouldn't be worse than the rest of prod, private beta members shouldn't be trusted more than non-members; so if the odd non-member finds a way in it's fine.
Not at all. How often do people complain of temporary solutions becoming permanent? Doing it wrong out of the gate is a surefire way to ensure it makes it to production if there's no further review.
The beta feature had a very bad bug that allowed hijacking other people's email. That is entirely independent of controlling access to the beta feature.
It's only mentioned in the write-up because OP wouldn't have been able to explore bugs in the beta feature without finding access to it first, which he didn't otherwise have.
:-(
I've been in this industry since 1996. Must every programmer make this mistake for themselves before they learn? I see this over, and over, and over again, at company after company after company. I'm really surprised a company as seemingly competent as Cloudflare would make this mistake, to say nothing of the larger error of allowing forwarding to be setup on an unverified domain.
I weep.
Edit: I'd appreciate a response from Cloudflare on how this slipped through the cracks and what changes they are making to prevent such mistakes in the future. Not trusting user input is among the most basic thing I'd expect a programmer, especially one working at CF, to know in 2022, so I'm assuming this was some sort of miscommunication between teams.
Does CF have a launch process for new features? Does that process include reviewing the special document and testing the new feature against the various bug classes?
Bypassing a beta feature toggle isn't really a vulnerability IMO, it just allowed OP to find one in this case.
So nobody here can think of any possible scenario where bypassing a server-side check for 'Account X can access Feature Y' could directly lead to a security issue?
This is absolutely 100% a vulnerability - insomuch as that CloudFlare should have an explicit policy that ALL account features are enabled / verified server-side, not client side.
Think of all the spam that would have happened, had this been discovered on underground black-hat forums.
> Think of all the spam that would have happened, had this been discovered on underground black-hat forums.
What spam would have happened as a result of early access to a new Cloudflare feature, that's independent of any (other) bugs/security flaws in that feature?
(Also, even with the actual vulnerability here, what 'spam' would have happened? This hijacks recieving. Worse, yes, but I don't see how it helps spammers.)
Not really, it was mentioned as part of a report of the main, much more critical issue of 'hijacking email with Cloudflare Email Routing' - note that's the title itself, not 'accessing a cloudflare beta feature'...
I agree it 'has zero bearing on the overall issue'. That is pretty much my entire point.
Sounds like manual testing that should be automated.
It's so common to write tests that say, in one block of code, "these are the inputs, a function or endpoint is called, and these are the expected outputs." But in theory, every single one of those tests should have variations like "if this exact same input was provided, but with a user with different permissions, as well as a non-logged-in user, the endpoint should fail with a permission error."
But then you get into a quandary: if you want to be able to customize your test code granularly, or have your inputs and outputs generated at runtime, you need to be able to write your test setup code as code, not [{ input, endpoint_name, output }] configurations. But if you need code, then you can accidentally run code that calls things directly without the abstractions that would run it with the extra variations.
At a certain scale, this speaks to needing either static analysis of your testing code, or extremely diligent review processes, beyond simple code coverage.
Does anyone have go-to libraries that deal with this at scale?
The point is that software is developed by humans. Humans make mistakes. Find me a single Fortune 500 company that hasn’t had an embarrassing security issue due to human error. As I said, shit happens.
The response time for fixing this and everything else entailed in a big bounty process looks good to me. I’m sure whatever team worked on this feature will learn from it.
At this point I'm nearly an old man yelling at a cloud, but anytime a junior engineer starts I try to go over a handful of basic security things. Seems that most people coming out of school are aware of SQL injection and XSS, but the concept of client-side security is no security is not sinking in, even after it's been explained.
I've spent a lot of time trying to figure out why, and the best I can come up with is that they just think something like the following myths:
* Nobody would ever do that.
* Nobody would ever hit the API directly.
* It would be too hard to do that for somebody who isn't the developer.
* The need for an auth token would make it impossible to hit it with curl or another client.
* Nobody can change the client code because it's obfuscated and unreadable for humans.
Doing it on the server side is harder and a pain in the ass, so there's a lot of incentive to do it on the client and not worry about the server. It particularly doesn't help that as an industry now we're obsessed with speed and quality be damned. We can always fix it/do it right later (spoiler alert: you never do and you never have time because there will always be important new features that are high priority).
In early July I asked if the report could be disclosed, seeing as things had changed since the bug was originally reported. Cloudflare agreed and the report was then moved to the public program. As to why it was disclosed now rather than in February when the public program launched, that was my fault for not asking earlier.
https://hackerone.com/albertspedersen?type=user
You've obviously got a strong career in Security in the future. Have you looked at any Crypto projects? Seems like there are some massive bounties on https://immunefi.com and similar sites.
Bug bounty programs are a bad deal for researchers. The payout for this bug is absurdly low.
Payouts are a joke and progress is slow. It wasn't that long ago people were overwhelmingly just arrested or threatened for reporting these kinds of things but thankfully that's becoming rarer.
The amounts for these bounties though seem to be a token gesture and not much else, especially considering the damage someone could have caused with this.
It was also comprised of two separate $3,000 rewards. Maybe they treated it as two vulnerabilities?
Am new to this kinda stuff but would love to play with it. Always love reading up on responsible disclosures.
I work with it to do reverse engineering of APIs for apps on Android icw Frida.
Seems wild that they can deny you the opportunity to publish your findings for 7 months after the fix for a beta product went live. What is that about? Was that a requirement to get the bug bounty? Seems like a great way to encourage finding another buyer instead.
> Since some comments are addressing that this happened 7 months ago. Our disclosure policy is to allow researchers to write about us once the issue is fixed, but give us a week heads up before they publish so we aren't surprised, can coordinate any public comms we want to make, FAQs that need to be written for inbound questions from customers, and can tailor our response to the issue at hand. Can answer other questions if you have any.
Would you trust CloudFlare handling your important email forwards going forward? I recently switched to them for my forwards a few weeks ago, I would feel better if your answer is a yes .... :)
Once he reported the issue we investigated all prior email routing configurations to ensure that this had only been found as part of Albert's responsible disclosure to us.
We disclosed the issue here earlier this week: https://hackerone.com/reports/1419341