BrowserStack Post Mortem (Shell shocked) | Hacker News Reader

BrowserStack Post Mortem (Shell shocked)

browserstack.com

113 pointsdmak33 comments

I cannot really give a good explanation, but I feel like this isn't the whole story. They make it sound like it isn't a big deal, but in the post detailing the "breach" there was some talk about people stating that they have seen other peoples session in progress. If their story on this page is true that really should have been impossible.

I don't think the problem seeing other peoples' sessions has anything to do with this or a breach at all. It's more likely a problem with their software.

EDIT: Sorry, completely misunderstood you but leaving this here anyway.

Nice reminder to patch all your servers, not just your 'active' ones (can't believe they didn't do that).

This totally misses out on 'motive', that mail was far to cleverly crafted to be just some drive-by hacker that targeted browserstack for the fun of it.

Good they did a write-up though, so points for them. It must be very hard to draw the line on transparency when you write a document like this, I tend to think that it is better to spill all the beans, even the uncomfortable parts but for business reasons they may decide to keep some of the information to themselves.

"Both the passwords mentioned, ‘nakula’ and ‘c0stac0ff33’, were indeed in use a couple of years ago during our prototyping phase, and thus were present in the old prototype machine that was hacked. ‘nakula’ was previously our VNC password, and was hashed. However, unlike the hash used for the user passwords, this hash is much weaker. This was due to a limitation in VNC protocol, and we had overcome this liability by regenerating a new password for every session, and thus ‘nakula’ has not been in use for years. ‘c0stac0ff33’ was one of our system user passwords on the prototype machine, before we moved to key-based authentication."

Still points very strongly to a bad leaver. No outsider would have that info, I'd be digging through my older personnel records for suspects at this stage.

Still waiting for that updated postmortem from http://www.codespaces.com/

> All user passwords are salted, and hashed with the powerful bcrypt algorithm, which creates an irreversible hash which cannot be cracked

While using bcrypt is great, it is not uncrackable.

It is still vulnerable to brute force attacks. If you have a strong password it wont be cracked, but most people don't

Can you explain this in more detail? As a crypto noob I have always used bcrypt to hash the password concatenated with a unique salt for that password and stored the hash in my db. Is this no longer sound advice?

I think the point is that if someone choose "12345" as their password, the best hash in the world can't protect it, since it's trivial to try a few combinations and hit on the right password.

Strong hashes prevent bruteforcing millions of combinations, but that actually depends on the attacker needing to try millions of combinations.

So look at the passwords they admit to being compromised. One of them is "nakula". Look at who signed the postmortem as "founders". Google Nakul and find out his surname is "Aggarwal". If you were guessing Browserstack admin passwords hoping to find a weak one - how long would it take you to try "nakula"? No choice of bcrypt/scrypt/pbkdf2 hashing is going to protect against trivially guessable passwords...

No, it's still best practice.

The problem is it's impossible to protect weak passwords, because attackers can just run a password database through your hashing algorithm and see if there are any matches.

i.e. Just because you can't reverse a hash to get the password, doesn't mean you can't hash a bunch of passwords to figure out which leads to which hash

Thanks for making that point - that's been going around my head for a few weeks whilst pondering on password encryption. Rainbow table attacks are still more than possible with all of these algorithms when used on their own?

So salts are certainly the way to go, as, although they're "known" (appended to hash, or stored elsewhere in DB), the hash can't be compared to a generic table, it'd need to be a full dictionary table of hashes already encrypted with that salt (which theoretically/hopefully is used only once)?

Bcrypt question --

I thought that part of the strength of bcrypt was that it hashed password many, many times, and that number of times through the hash is based roughly on the strength of the CPU doing the hashing. If this is true, wouldn't that make it all but impossible to brute force? You'd have to know how many times the passwd was hashed in addition to everything else.

bcrypt is a key derivation function based on the Blowfish cipher. The iteration count can be increased to make it slower, so it remains resistant to brute-force search attacks even with increasing computation power. The hash string includes the cost parameter, a salt, and the hash value. [1] This allows you to change the number of iterations and length of salt at any time, and all previously generated hashes will still be valid. It does not, however, adjust itself automatically depending upon the computational power of the computer on which it is running.

[1] https://en.wikipedia.org/wiki/Bcrypt

> If this is true, wouldn't that make it all but impossible to brute force?

The "key stretching" factor is configurable, but in the end you need to be able to check a login attempt in a reasonable amount of time (say, 1 second). So you have to be able to compute at least 1 hash/s on your production machine (probably much, much more to avoid performance problems).

If the attacker has access to 100 machines as powerful as your production box for 6 months, they can calculate 100 * 86400 (seconds per day) * 180 (days per month) = 1.5 billion hashes. Depending on the value and strength of the passwords, this could be worth doing. It's almost certainly worth calculating the hashes of the top 100000 most common passwords in any case

edit: > You'd have to know how many times the passwd was hashed in addition to everything else

As chill1 said, that's usually stored along with the hash. Even if it wasn't, you could see how long login attempts took you to figure out a rough number, then just check the hash of every iteration up to say that number * 2

If you've slowed somebody down to brute forcing only 1.5 billion hashes in 6 months by using bcrypt, you have been overwhelmingly successfull.

This system here can brute force 180 billion MD5 hashes per second:

http://www.zdnet.com/25-gpus-devour-password-hashes-at-up-to...

[edit] If you only allow passwords to contain upper/lower case letters and numbers, at 1.5 billion MD5 hashes per 6 months, it would take 19 years to check up to and including 6 character long passwords. And because it's bcrypt, and each password has a different salt, you need to do that for each user; you can't build up a raintable as you go.

Hmm. Another one for your security audit. I was able to change the registered email address on our account

1) Without re-entering my creds following login 2) With no notification to the old email address that this had been done.

One of the most detailed, and candid, post-mortems I've read. Kudos.

"All user passwords are salted, and encrypted with the powerful bcrypt algorithm, which creates an irreversible hash which cannot be cracked."

I assume you mean to point out that it is disingenous to say that any hashing algorithm "cannot be cracked"?

It's good that this was published, some tough lessons in there. It's missing a timeline though, which is important to learn from - when was it realised that user data was accessed, how long did it take to figure out that keys were stolen, exactly when & what actions were taken to protect customer information, etc.

Other interesting details not covered was whether an incident response plan existed or was followed, how many unpatched Internet-facing servers were sitting around and whether any existing security measures did actually work in slowing down the attacker.

This is probably(/definitely!) sensitive information, but I'm always interested to know what impact this sort of thing has on a company's customer base. Customers/potential customers being put off by the malicious email/downtime vs. content of post-mortem and how professionally the incident is handled.

Even a negative incident like this has put BrowserStack in the spotlight for a short time ("no such thing as bad publicity" and all that).

Another thing to mention here - we've received no direct contact from BS at all following this - if we hadn't been following this on HN (and we were a little naive) we might still not know this email was not genuine. The blog post mortem is great, but I think it's a poor show not to reach out to affected customers directly.

They have reached out. We received our emails a little before you wrote this, but they send in batches so yours probably came or will come later.

I just got an email from them containing the text of this blog post, you'll presumably get one too, it's just that it takes time to email hundreds of thousands of people.

Is anyone from BrowserStack monitoring this thread? We received the email so we can assume our account was accessed by the hacker. Passwords crypted and salted (ok - changed) but what about the BrowserStack automate username and token, generated by BS themselves, that we are unable to edit or regen?

that is in a different table which was not compromised.

OK, I want to make sure though. How can these creds be regenerated? What happens if we had leaked them accidentally, there seems no way to regen them?

Please have a look at https://www.browserstack.com/automate/rest-api#rest-api-key

Thanks (can't reply direct to the reply for some reason)

There's some kind of algorithm that prevents two users ping ponging comments too quickly. Not sure exactly how it works, but I've had experienced that before.

> All user passwords are salted, and hashed with the powerful bcrypt algorithm

It is so refreshing when you come across a company that takes user password handling seriously.

If only everyone who is hacked provided such detail, we might learn something to avoid being the next provider of such detail.

Me again.

"he was only able to reach less than 1% (our estimate is 5,000 users)"

So you have 500k registered customers then? Surely not.

http://www.browserstack.com/growth

"475,000 registered developers"

The Last-Modified header for the image is from 18 July 2014 so they could certainly have passed 500K over the last few months.

Reply on news.ycombinator.com