Hacker News Reader

Comment by ejcx | Hacker News Reader

I lead Product Security at Cloudflare, thanks for the writeup Albert and the fantastic security research throughout the past year.

Once this issue was fixed we investigated all prior email routing configurations to ensure that this had only been found as part of Albert's responsible disclosure to us.

Since some comments are addressing that this happened 7 months ago. Our disclosure policy is to allow researchers to write about us once the issue is fixed, but give us a week heads up before they publish so we aren't surprised, can coordinate any public comms we want to make, FAQs that need to be written for inbound questions from customers, and can tailor our response to the issue at hand. Can answer other questions if you have any.

We disclosed the issue here earlier this week once Albert told us he was writing a blog: https://hackerone.com/reports/1419341

View on HN

What's missing from your statement is how you plan to prevent this kind of thing (a programming oversight so simple it shouldn't have landed in production) in the future. Something like mandatory security reviews.

Any info about your devs doing only client side verification of the beta flag? Seems like a pretty bad practice.

Usually rollouts like that are slow to prevent a flood of people using it at once, and to slowly open the flood gates. Not many people will manually change the flag, either. I don't think the fact that the feature flag was just client side is a "smoking gun".

Agreed. Feature flags can be complicated to implement both client side and server side.

Ideally you want both but it doesn't always work out that way.

What was missing was the server side ownership check. We decide which customer owns the real "example.com" which is very battle tested logic, but had missed the check in this new service. The client side validation is expected too, though

I'm curious, if you'll disclose it, what the exclusionary policy was. Enterprise zones?

Tracking, thank you.

It's obviously not good in general, it's 'obscurity' (as in 'is not security') really, but it seems pretty harmless for a gradual roll-out feature toggle? If someone cares enough and knows enough to find they can get past it, let them play with the beta feature?

After all, it got them this responsible disclosure.

It wasn't part of the vulnerability, it just allowed OP (who happened not to be legitimately in the beta) to find it.

Well, the whole point of Beta program is to limit the participation and thus the exposure of attack vectors, and reduce the impact during the testing phase.

I suppose I wouldn't personally think of it as limiting exposure like that - prod is prod, beta feature or not - but if one does for a given product then yes sure it's a bad bug.

Do we really think Cloudflare Email Routing private beta was private to somehow trusted parties only though? Presumably 'N-mutual trusted parties' too, for regulatory compliance. I assume not; not least because the product security lead is here in the comments saying they vetted logs etc. after the fact to ensure that only OP took advantage of this.

I wouldn't rely on any companies logs or audit trail.

Fine, not the point, they did it - implies the invitees weren't trusted 'they won't/doesn't matter if they do pwn each other' customers.

I've built numerous systems that could be called "beta programs" - and the goal of none of them was security related - but rather product feedback related.

I hope you also consider security problems as feedback.

No, why are you shipping it to production if you need feedback on security? Beta feature security should be best effort, as good as anything else in production, because it is production, beta or not.

it should have never hit production in the first place

this isn't some kid writing their first dynamic webpage, it's a public multinational that proxies a large percentage of the internet as security product

the fact this wasn't caught internally is extremely worrying and makes me wonder what other sort of basic quality issues they have lurking

I agree. As an architect at a listed company, trusting user supplied data like this would usually get caught in either planning or at CR before hitting QA.

100% agree. This reflects EXTREMELY BADLY on Cloudflare. This changed my perception almost 180 degrees of what I thought of Cloudflare before. This is not a thing you'd expect devs to do at a moderately important company, let alsone slip through QA, code review and many other checkpoints. The fact that this happened at a company which routes half the internet's traffic through their servers is SUPER worrying. Glad I'm not a CF customer anymore. Never likes them MITM everything.

The client being able to opt itself in to the beta? That's what I'm talking about, not routing anyone's domain - that shouldn't have been allowed in production, beta or not, agreed, and clearly so does CF.

But client managing to opt itself in to a beta, as my GP comment was about, is no big deal. Worst case you're getting something free that you're supposed to pay for.

Beta security shouldn't be worse than the rest of prod, private beta members shouldn't be trusted more than non-members; so if the odd non-member finds a way in it's fine.

> It's obviously not good in general, it's 'obscurity' (as in 'is not security') really, but it seems pretty harmless for a gradual roll-out feature toggle?

Not at all. How often do people complain of temporary solutions becoming permanent? Doing it wrong out of the gate is a surefire way to ensure it makes it to production if there's no further review.

I don't understand your comment at all. I'm not saying anything is 'ok for now while it's a beta'. I'm saying 'client-side validation of beta feature toggles is pretty much fine'.

The beta feature had a very bad bug that allowed hijacking other people's email. That is entirely independent of controlling access to the beta feature.

It's only mentioned in the write-up because OP wouldn't have been able to explore bugs in the beta feature without finding access to it first, which he didn't otherwise have.

Thank you for awarding this a bounty of $6,000.

Reply on news.ycombinator.com