Hacker News Reader

I think you're right on most points, but after reading the write up and response I do think Alex reached out to the employer first instead of the researcher as an intended act of intimidation. That was a mistake.

If it was not done for the purpose of intimidation, then Alex simply would have asked the CEO if the researcher was acting on the company's behalf and after hearing "no" would have ended the call and contacted the researcher directly.

Seems simple doesn't it? Perhaps you are not seeing it due to your friendship, but it seems like a dirty move and only serves to call into question how Alex handled other aspects of the situation.

Yeah, totally. "I did not threaten legal action against Synack or Wes" Who the f do you think you're kidding, Alex?

> If it was not done for the purpose of intimidation, then Alex simply would have asked the CEO if the researcher was acting on the company's behalf and after hearing "no" would have ended the call and contacted the researcher directly.

Then the CEO is going to contact the researcher and he's screwed either way. God knows what the CEO would have say to the researcher privately. Having a middle man to translate is a bad idea in an emergency.

Let's face it, when you used your work email and made another company paranoid, you are putting people on the spot. Employer needs to know (they have legal responsibility), and given the prior research they did and the researcher's claim, I think the reach out is absolutely correct.

Instgram's infrastructure has flaw. That's bad but everyone's infrastructure has flaw. Shit has to be fixed. Doing more than what was needed is bad. If I am told to stop dumping data, I would stop.

Coming from a pentesting background (and now working as a CISO), I can see both sides to this. tptacek is almost certainly correct in his characterization of the events, and I agree wholeheartedly with what he's said. It's important to note that this researcher didn't just chain several exploits together, but sat on sensitive data unbeknownst to Facebook in order to exploit other vulnerabilities later. Those vulnerabilities could not have been exploited without the initial (fixed) compromise.

Think about it a different way. If this researcher had found SQL injection in a webapp, dumped the usernames and passwords, and reported the vulnerability for a bug bounty, he should get paid. If he kept each of those credentials, and then logged into other systems using higher-privilege accounts that he'd compromised even after the SQLi is fixed, he is basically continuing the exploitation of an already-fixed bug. Those don't deserve payouts. Similarly, if he'd established some sort of persistence (such as a reverse shell, etc) on compromised assets, he can't keep coming in and getting more and more bounty payoffs. Fruit of the forbidden tree, in this case.

Where I disagree with tptacek is with regard to the benefit of bug bounty programs. Although I'm not currently running one, I find the idea fascinating and helpful for two primary reasons: first, you're almost definitely going to see generally better results in a well-managed bug bounty program (not necessarily something like Facebook's White Hat program) than traditional pentests or application security assessments. More eyes are almost always better when searching for tricky problems. Secondly, if you're a large enterprise, there are already people "testing" your security. I'd much rather be able to pay out a researcher than drive them to more nefarious buyers. You will probably encourage many people to test your security (which screws up metrics) but if finding security problems is the ultimate goal, it's worth it.

Even in this case in point, Facebook did discover an RCE that could have been (and kind of was) extensively exploited due to the fact that they held the bounty. If an actual malicious hacker had found that problem first, they would have been in significantly worse shape.

Whilst I'd agree that bug bounty programmes can be a good idea for Internet facing assets, I thought that this story actually neatly illustrated their limitations.

With a bug bounty programme you don't generally authorise the kind of post-exploitation activities which we see here as leading to the really serious exposures, and that's not surprising as you can't easily authorise a set of unknown people to be processing your customer data.

This differs from an engaged penetration testing firm, with whom you have a contract which covers things like handling of data gained during a test.

So I don't really see bug bounties ever replacing penetration testing companies for internal work or anything that requires accessing customer data as part of the exploit...

I'd agree, but technically speaking, the bug is not fixed if credentials don't get reissued. Someone might already have access to them.

Also, you can't just expect that "oh, just delete your data pls" will work, can you? You can't trust anyone that literally hacks your system.

> If he kept each of those credentials, and then logged into other systems using higher-privilege accounts that he'd compromised even after the SQLi is fixed, he is basically continuing the exploitation of an already-fixed bug.

Why did those credentials still work post-report?

What if those credentials were accessed from a public dump?

The outcome of this entire clusterfuck of a bounty is one of the reasons there are still very well paid blackhats. There are no rules or terms to follow.

If their terms aren't clear (the terms he's citing certainly weren't intended for keys, rather Facebook user accounts/information), pay out and fix them.

Thanks for the writeup. Based on what you've written, it sounds like you would have been surprised if Facebook had paid $1 million for the original report (and no further nefarious behavior by OP) since it was probably due to a simple oversight, even though it was a RCE that obviously could have been turned into total ownage of instagram. Is that accurate? If so, what class of vulnerability would make you say "Yep that's totally worth $1 million".

Or do you think he should have just stopped and Facebook should have realized how bad it was and paid him a lot more than $2500?

There isn't a parallel universe in which this finding is worth $1,000,000. It it was, every pentester in the country is getting way underpaid, because this is not an uncommon pentest finding.

Makes sense, I'm just trying to get a sense of what sort of thing would be worth that much. Obviously only Facebook can answer that for sure. Heartbleed?

It's really dependent on the company. Ruby RCE would have the same affect heartbleed would to an entirely Ruby stack company.

I don't believe any company would pay $1M for a bounty on their own systems. Only people who intend to use the vuln, or to fix it as they are the vendor.

Fr a vuln to go for $1M requires "discovering SQL injection"-levels of vuln. MS paid $100K for an entire vuln class for ASLR/DEP bypass discovery, and promptly patched the shit out of it. For a remote vuln class, I could see them paying $1M quite happily to not have all of their products re-owned.

What about the parallel universe in which bug bounty hunters are blackhats who directly profit from the exploit? It seems like someone with that level of access could run up, among other things, a decent AWS bill.

I don't know about you, but I value the certainty of not losing a few years of my life to court proceedings/jail time at significantly above $50M.

Well, obviously we're talking about the mirror universe where nerds get away with things instead of scapegoated. Also goatees everywhere.

> It it was, every pentester in the country is getting way underpaid, because this is not an uncommon pentest finding.

No wonder there's a flourishing (and well paying) blackmarket for vulnerabilities. I wonder how much this keys-to-the-kingdom vuln would be worth (Mitm Instagram, bootstrap a botnet, steal celebrity pics, ... the possibilities are endless)

This is no market for these kinds of vulnerabilities at all.

> † (and, to be clear, a friend, though a pretty distant one; I am biased here.)

Alex is good friend of mine and I've known him since college. He's definitely a good guy and understands the ins and outs of security vulnerability research, having done it himself for many years. I'm sure he didn't take the action of calling the researcher's employer lightly, and probably had a really good reason to do so.

There has to be a side of this story we aren't hearing, and probably never will.

> I'm sure he didn't take the action of calling the researcher's employer lightly

He's the CSO, and this occurred under his watch. The exploit was 2 years old, and well known. It highlights an internal security problem at Facebook et al, of-which Alex sits at the top.

In this situation, his years of "doing it himself" is unlikely to have factored in - rather, he felt like he dropped the ball and could be facing some consequences, or at the very least felt embarrassment.

This would have led to a rash thought process, and perhaps Alex jumped to the conclusion of some sort of sabotage by another company.

> I assume the AWS resources have been rekeyed by now

It doesn't look like the SSL cert on instagram.com has changed recently, and the pentester specifically claims to have obtained its private key.

a private key. It's not uncommon to have multiple simultaneously-valid certificates for the same domain. I'd argue that it's actually sort of irresponsible and therefore surprising for a site at the scale of Instagram not to, for backup purposes.

but using that private key can still grant him access to someone's traffic to their machines. isn't revocation necessary to imply security in that domain ever again?

every network has old crufty bug-ridden stuff laying around

"stuff" was the keys to the kingdom, do you think this is acceptable for a company like facebook? So instead of them making an apology, the CSO is trashing the guy who gave them the wake up call?

I do think you are heavily biased ;)

They should have just paid him the money, told him not to do it again, fixed the architecture bug, updated the rules, and moved on.

Alex just went the drama route.

If there's a grey area in your ToS, and a security researcher/hacker type is in the middle of it - the smart route is to appease them and fix the grey area. FB has a lot of resources, and it wouldn't have to deal with the blowback from this.

Why make such a bad situation worse, if you don't have to?

FB messed up. The researcher partly messed up too. Fix it and move on.

If you're biased, you should do the ethical thing and stay out of it, honestly. There is a ton of asymmetry here, and you and your Facebook CSO friend are being bullies. This is pretty grey, you don't have first hand knowledge, and obviously Alex can do no wrong in your eyes.

Everyone is biased. Presenting your arguments and declaring your biases so others can take them into account is the ethical thing to do.

This reminds me of the illusion of objectivity in journalism. If you pretend to be perfectly objective and unbiased, you're lying.

Please address where in your story calling the employer by your good distant friend would be justified.

Sounds like a jerk to me.

https://news.ycombinator.com/edit?id=10754697

As mentioned in Alex Stamos' response, he believed Wes was working on behalf of Synack, and contacted the CEO directly.

Escalating issues with a company to the CEO of that company doesn't seem like jerk behavior.

Wes counters that, "[Alex] never for a second believed I was operating on behalf of Synack"

I'm not sure how Wes knows what is going through the mind of Alex, so I'm inclined to take Alex's word on this.

> Wes counters that, "[Alex] never for a second believed I was operating on behalf of Synack" I'm not sure how Wes knows what is going through the mind of Alex

As blazespin[1] mentioned in this thread, Facebook's own terms states that they only pay individuals. That's how Wes knows - because Facebook's bounty program never deals with companies. The only other explanation would be Alex is ill-informed about the terms of Facebook's bounty program.

1. https://news.ycombinator.com/item?id=10755746

There is no reason for the researcher not to retain those keys, IMO - Once those keys were found to be compromised by the company, they should have been revoked immediately, and considered 'in the wild'. The fact that they didn't revoke these keys is basically a security violation itself.

Dumping the users table on an 'internal' (heh) dashboard -- any company that is doing these bounty programs needs to clarify what a 'user' is. Is it someone using their application, or all employee information as well. It's an important distinction.

Your characterization of the AWS keys being sat on for over a month does make sense, now that you frame it in that light.

That said, Alex Stamos and the rest of the security team should have tried to figure out what vulnerabilities existed from this server instead of just nuking it and thinking that the problem was solved. That was lazy and stupid.

If the researcher didn't try to find what he could do with those AWS keys, they would likely be still valid. It's conceivable that some other people have found them too and did the same the researcher did, only kept everything to themselves. Thus, if the researcher didn't do the thing you consider bad, users of instagram would currently be more vulnerable. Why is then the thing the researcher did bad?

> 5. Fb nukes the server, confirms the RCE, pays out $2500 for it, declines to pay for the second finding, and asks the tester not to use RCEs to explore their systems.

The issue here is that, in hindsight, FB failed at this step.

They nuked the server, but they didn't determine what sensitive information was available on that server, and take steps to mitigate those risks.

I think that's an understandable mistake - cleaning up after a server intrusion is hard. Knowing how much to do after a possible intrusion is even harder. But it is still a mistake and it happened on Alex's watch.

If the purpose of the bounty program is to find out about your security mistakes, then the program did its job here, and Alex should be pleased that the problem was reported so that they could fix it.

That the researcher found the mistake by overstepping what is considered ethical (and I have no doubt that they did overstep) creates a very difficult situation - you don't want to reward that behaviour, but you do want to know about security problems and this one was only discovered/reported because of that bad behaviour.

In that difficult situation it is all the more important to tread carefully. The easy cases where you're paying out a $10k bounty typically don't require much finesse. It's the tricky cases where you need to make sure your actions are well considered and above-board at every step.

From Alex's own summary it's evident that he didn't handle it as well as he could have.

Two of the longest paragraphs in Alex's write up cover what he said to the CEO of Synack, even though Synack had nothing to do with this. Even if we accept that Alex thought it likely the Wes was acting on behalf of Synack (personally, I don't think that was a reasonable conclusion to draw, thought I assume Alex is sincere in his view that it was), he should have determined that up front, and then, once he knew it was not work related, he should have avoided:

- making accusations about Wes's ethics to his boss ("Wes ... had acted unethically")

- suggesting that his external behaviour has implications for his employment ("Wes's behavior reflected poorly ... on Synack")

- bringing in the threat of lawyers ("keep this out of the hands of the lawyers")

When faced with the difficult situation of legitimate security research that has (well) overstepped the ethical boundaries, all the evidence is that Alex jumped to the position of protect yourself, protect the company, intimidate and control the researcher and though that is a common and understandable reaction, it's not the way you turn a bad situation like this into a good one.