Comment by daveloyall | Hacker News Reader

Comment by daveloyall | Hacker News Reader

In my opinion, the author is feigning shock...

He claims to have downloaded the content listed below. And he is surprised that Facebook responds coldly? Note the string "private keys" in this list... Doesn't the author know how long it will take them to recover from this breech? How much it will cost them?

On the other hand, it does sort of re-enforce the idea that he should be paid handsomely, doesn't it? :)

    * Static content for Instagram.com websites. Write access was not tested, but seemed likely.
    * Source code for fairly recent versions of the Instagram server backend, covering all API endpoints, some image processing libraries, etc.
    * SSL certificates and private keys, including both instagram.com and *.instagram.com
    * Secret keys used to sign authentication cookies for Instagram
    * OAuth and other Instagram API keys
    * Email server credentials
    * iOS and Android app signing keys
    * iOS Push Notifications keys
    * Twitter API keys
    * Facebook API keys
    * Flickr API keys
    * Tumblr API keys
    * Foursquare API keys
    * Recaptcha key-pair

Is this not the point of a Whitehat bounty program? To entice someone to discover and disclose a bug in a trustworthy manner?

If they react this way, and can't trust people to attempt to find exploitable security holes on their system (even those that yield private keys), then what is the point at all? The only people that find them then, are not going to be as cooperative about it.

> Doesn't the author know how long it will take them to recover from this breech? How much it will cost them?

This is not the author's fault. He did nothing but disclose bugs that Facebook themselves set in place, and seemed to be very open with them about it, at that.

No, this is not the point of bug bounties. The point of a bug bounty is to find and fix bugs. That's why they're called "bug bounties".

This person took a bug bounty and ran it as a penetration test.

Facebook fixed the one bug he found and paid him for it.

What is the protocol for assuming that a bug might have previously been exploited and keys already compromised? Is that just not worried about unless they see evidence in logs?

I don't know, but that's the security team's job; it is emphatically not the job of a bug bounty researcher to do that.

I don't know much about this which is why I asked.

It seems that severity-based payouts have created incentives that do not match the program rules? Maybe all rce bugs should be paid out on an assumption that if used they'll lead to access to a shell or to user data.

Severity on a vulnerability assessment is based on the bug itself; it's the severity of the RCE.

Yeah - but it's 100% clear from this that FB wanted to brush the RCE under the carpet with a "not at all severe $2500" classification - without ever admitting to losing their private ssl keys or auth token seeds.

He clearly _did_ have a "security vulnerability" that gave him the keys to the kingdom. He knew it, and Facebook know it - and they wanted to pretend it was no big deal.

Any bets on how many months till there's a large-scale breach of Facebook user data? The reality of the balance between responsible disclosure and selling an exploit is much easier to evaluate now.

That certainly is the fun and exciting way to read this story.

Especially considering Alex Stamos apparently requested reassurance that he _hadn't_ accessed particular classes of data - instead of looking in their own presumably non-existent audit logging of people who've had access to the private keys ssl of instgram.com and *.instagram.com!!!

(Seriously??? That's some world-class enterprise-grade "moving fast and breaking things"...)

Why wouldn't it be considered a bug that accessing one low-permission S3 bucket allowed him to access all the other buckets, including user data and keys?

It is a bug. But I think the point Facebook is making is that it is impolite to exploit the RCE bug and then access other systems.

Both tptacek here and Facebook claim that he found one bug. He found at least two, depending on how you classify things: even if Facebook would not like to admit that their security architecture around token amanagement was/is deficient, and the fuzziness of internal security boundaries makes "bug" somewhat hard to define, it was deficient by industry standards (especially for such a large and tech-focused company), and he got way more access than that RCE should have given him. Whether or not he was supposed to go looking for such additional bug(s), it's discourteous not to at least acknowledge that he found them, and thereby provided Facebook additional value over just finding the RCE.

If he had told Facebook that at the same time as he reported the credentials he harvested from the database --- which his timeline suggests he could have --- I'd agree with you.

But he didn't. He put the credentials in his back pocket so he could pull them out when they suggested he hadn't found his "million dollar bug". And so for a month after they fixed the bug, some fucking rando is walking around with credentials to all of Instagram's AWS assets, totally unbeknownst to anyone at Facebook. They turn down his bid for his "million dollars", and he busts the credentials out on them. You think they're going to thank him?

He's lucky it was Stamos and not Mary Ann Davidson.

I think the point is that, after the first bug report those credentials SHOULD NOT WORK because their job should have included revoking ANYTHING that system have access to. How did they know Wes was the first person to find that bug and the linked credentials?

So, the fact that those credentials still worked a month later is a HUGE FUCKING DEAL! Alex, the consummate professional, didn't do his job and instead had a knee jerk reaction to someone slapping that fact in his face.

Exactly.

Notwithstanding the fact that AWS credentials should be very narrow in scope.

It has been incredibly interesting reading through those threads. People are arguing two completely different arguments. tptacek is saying that the dude keeping AWS keys without disclosing this was bad and guy is lucky to not get a early morning wake-up call from men with guns. slewis, comex et al are saying that Facebook not locking down and later disabling AWS keys was bad and Facebook was lucky they didn't get sold on black market. Both sides are correct but it's informative who makes which arguments.

That's not what I said. I took issue with tptacek's statement that there was only one bug.

Bug bounty appears to be a misnomer in this instance. Facebook is specifically asking for reports of security vulnerabilities in their policy:

> If you believe you have found a security vulnerability on Facebook, we encourage you to let us know right away.[1]

Which then begs the question to me: how do you differentiate an acceptable and unacceptable probing of security vulnerabilities when you can't capture the full impact of an issue without attempting to exploit it to its fullest? Because it is certainly not outlined in their policy.

And when you're asking for any whitehat to attempt to discover and disclose security vulnerabilities in your system with only the limpest of guidelines around how to do so, I don't feel that it is warranted to react such as Facebook has here.

[1]: https://www.facebook.com/whitehat

I don't know. I feel bad for Alex but if we want to suggest that Facebook's vulnerability disclosure policy was poorly written, I will ruefully agree.

When you stand up a bug bounty program, you are giving strangers permission to do something that they would otherwise be prosecuted for doing. You should be extraordinarily careful when you do that, and your rules of engagement should be crystal clear. These weren't.

EDIT: Having read the CSO's explanation that the guy was using his company work email, it makes more sense why the CSO would contact the company (and explains away the pettiness my comment was referring to)

One thing I notice: if the CSO felt like this person did something grossly illegal and irresponsible, why not go straight to the police? Why instead go to the man's employer and speak passively aggressively?

Paradoxically, contacting the authorities could have helped facebook's argument. It would communicated to the community at large: "Hey Facebook believes it has clear standing to pursue this guy. Maybe, he really did do something wrong."

Instead, what I'm reading is: "Facebook doesn't actually believe what the guy did was illegal per se... but they wanted to spite the guy anyway."

For me, it seems petty.

Zero is the number of people on HN who would feel better about this situation if Alex Stamos had referred this person to the police to be prosecuted under CFAA.

The researcher has already updated his post regarding the use of his company email. Apparently your original point still stands:

> I never contacted Facebook or Alex using my work email account. It was only after Alex contacted my employer via email that I sent a reply from my work account. Alex indirectly contacted me at work, not the other way around.

Also, why would he be doing this work at the behest of his employer when (IIRC) Facebook's bounty program only pays out to individuals? It would automatically make him ineligible to claim the bounty.

To me it seems like Alex Stamos tried to use some good old threaten-your-livelihood intimidation tactics and failed miserably.

I commented earlier to sort of the same effect, and was thinking a little more about this.

I don't think the goal, or desire, is to be told the full extent or impact of a problem. The goal is to be alerted to spots that may lead to a large problem, or re in and of themselves a large problem.

This seems like it has a few facets to it. You end up reducing the space of things to mostly "ways to get in the front door." Thinking about it, I would probably be frustrated, in general, if I knew someone had important keys to the kingdom I was in charge of. It doesn't change the fact that others may or may not have also gotten the same access, now it's 1-* instead of 0-* people who have it and shouldn't.

I'm still slightly skeptical on the bounty reward itself. This was a simple exploit that got pivoted into some major shit, so do you reward the exploit of the logical conclusion of the exploit? I lean towards the latter, but again, as you said... how do you figure out the impact without... actually trying to figure out the impact?

Bug bounties are an interesting concept, to be sure. -

I'll rephrase the question. Is the broader vulnerability apparent based on the first discovery OR does it only become clear the further down the rabbit hole you get?

I don't know. If we're going to speculate, I'll say: the Facebook security team didn't know this system existed (it's a 3rd party admin console on a public IP address!), and their immediate reaction to it was "nuke it from orbit, pay out the bounty for finding it, and forget about it".

My guess is that they discovered the AWS credential thing on December 1.

If they discovered the AWS credential thing on December 1 after the security researcher reported it, and wouldn't have discovered it otherwise, and it could be the case that someone else found the exact same attack path first, shouldn't they reward him for making them aware of a problem they would not have otherwise noticed? That they wouldn't have fixed? That others that discovered the same attack path might otherwise still openly exploit to MITM all the traffic, to do arbitrary things with arbitrary user accounts?

In your experience, are there other, more careful organizations who would have taken the host offline but saved a disk dump for later investigation?

Which is fine. But threatening to call the cops was really bad.

You don't know that's what happened, even the researcher didn't say that. You're extrapolating.

A much more reasonable and likely explanation of the same set of things we've been told:

Alex Stamos called Synack and said that the AWS credentials, which, by the researchers own admission, he'd chosen to retain long after the vulnerability he reported was fixed, had to be deleted, and that if they weren't and the researcher continued to use them, the situation would be out of Stamos' hands and into Facebook legal's, at which point he couldn't keep him from being prosecuted.

In that interpretation, Alex isn't threatening the researcher; he's (very reasonably) saying "you cannot use these credentials you've taken from the server, and if you keep doing that, I can't take responsibility for how Facebook will handle this, so you should stop right away before you harm yourself."

it's utterly trivial to revoke and reissue aws access keys. trying to paint this as a necessary security measure is incredibly dishonest. the only plausible reasons to loop in his employer and mention legal remedies are intimidation and incompetence and as you've assured us incompetence is off the table...

blazespin > > But threatening to call the cops was really bad.

tptacek > You don't know that's what happened, even the researcher didn't say that. You're extrapolating.

From Wes' blog (presumably based on his boss' oral description of the call): "Alex then stated that he did not want to have to get Facebook's legal team involved, but that he wasn't sure if this was something he needed to go to law enforcement over."

Your bias is apparently badly incapacitating your reading comprehension, because "stated that [...] he wasn't sure if this was something he needed to go to law enforcement over" is exactly threatening to call the cops. Not even your friend mr Stamos, who has presumably read Wes' blog post, is claiming that he didn't. So whom are you saying is lying; Wes, or his boss?

Oh, and "(very reasonably) saying 'you cannot use these credentials you've taken from the server, and if you keep doing that, I can't take responsibility for how Facebook will handle this, so you should stop right away before you harm yourself.'" really, really, really sounds like Vito The Baseball Bat "very reasonably" saying "You cannot use this testimony you got off Loanshark Louie, and if you keep doing that, I can't take responsibility for how the boys will handle this, so you should stop right away before you harm yourself."

Seeing that as SERIOUSLY (as opposed to sarcastically) "very reasonable"... Well, hello, friendship-bias Bizarro World.

Holy christ that is SO wrong. The system should not be so easy to pivot in that way. That was definitely the real bug. If getting the keys to the kingdom is easy as exploiting a trivial bug than Instagram is really really screwed.

As I'm sure it's not the only trivial bug!

Instagram should be thanking Wes for the wakeup call instead of making him the enemy.

I would tend to agree.

Facebook's point is that he found a vulnerability, and exploited it instead of stopping there. I kind of understand their point of view though. "See you have a vulnerability there, and then I can get access to this, and then this, and see now I have the password of your user, and then I'm just one click away from accessing all the instagram pictures I want."

Although Facebook's handling of the problem is poor (why didn't the CSO call the author directly to get things squared out? He does not talk to people who are not C*O?), they do have a point.

I think the author acted in good faith, but got carried away by his findings unfortunately.

"See you have a vulnerability there, and then I can get access to this, and then this, and see now I have the password of your user"

However, these weak passwords could have been exploited separately as part of an attack. It is fair to call it a new vulnerability, even though it was discovered by exploiting the first vulnerability.

Exploiting the bug would have been downloading the actual contents of the S3 bucket (the instagram source and other things). He specifically says he did not do that.

He clearly made a big effort not to violate privacy. The problem is that he made their security look like a joke by getting the keys to the kingdom without anyone noticing. Did that big expensive IDS catch him? Nope. Did any of the log watchers babysitting the AWS logs? Nope. One researcher made the CSO look incompetent in the matter of minutes.

If he had found a bug with something a developer wrote that would be a different story. What he found was layer after layer of Operations (particularly Security Operations) failures. This is something you hire a CSO to think about (or at least hire/manage others to think about).

Are we reading the same article?

> [...] I queued up several buckets to download, and went to bed for the night.

> The next day, I began to go through some of what I'd downloaded, [...]

Key quote:

"Since the Faceboook Whitehat rules state that researchers need to "make a good faith effort to avoid privacy violations", I avoided downloading any content from those buckets"

Listing the contents of the bucket is very different from fetching them. Without listing the contents, he wouldn't know the severity of the vulnerability. There's nothing wrong with that.

While I'm on his side, his wording seems to indicate he did download data from SOME of the buckets, just not those specifically containing user sensitive data.

I'm of the opinion that not downloading user data, but grabbing source code, backups, and secret keys - is a perfectly reasonable interpretation of "making a good faith effort to avoid privacy violations".

What if he hadn't found out about it, but someone else had already taken the files?

Facebook might've never known

That's the Catch-22 of whitehat, isn't it? The whole idea is to find breaches and report them, but any worthwhile breach is going to expose sensitive data. How could you possibly know you've found a hole unless you've peered through it?

> Doesn't the author know how long it will take them to recover from this breech?

I assume Facebook would need to regenerate API keys anyways. Simply showing that author could have accessed the API keys is reason enough to think that he may have even if he claims to not have, or that someone else may have access the API keys.

>>Doesn't the author know how long it will take them to recover from this breech? How much it will cost them?

Doctor diagnoses, patient has cancer.

Doesn't the doctor know how long the patient will take to recover from this disclosure? How much it will cost the patient?

Reply on news.ycombinator.com