Slack was hacked | Hacker News Reader

Slack was hacked

slackhq.com

857 pointstrustfundbaby497 comments

I hate to be the negative guy, and they were hashing passwords better than 90% of the sites, but it would be SO easy to completely neutralize password leakage when the attacker only has access to the database.

https://blog.filippo.io/salt-and-pepper/

tl;dr: Hardcode a second salt in your application code or in an environment variable. Then a database dump is not enough anymore to do any kind of bruteforce.

It's simple, free and you can retroactively apply it.

EDIT: I addressed some of the points raised in this thread here https://blog.filippo.io/salt-and-pepper/#editedtoaddanoteonr...

I don't see that as a negative suggestion: that's a fantastic idea, and for all we know, a Slack employee will read your post, and make their hashing even better. :-)

By not implementing the suggestion, presumably.

This is rolling your own crypto, which is universally bad. To paraphrase Bruce Schneier, anyone can write a crypto algorithm they themselves can't break. Peppering a password hash destroys any future maintainability.

In a general sense, concatenating a pepper with your salt is NOT reinventing anything at all. (You can rephrase that as "Store pieces of your salt in separate places" which doesn't change or violate any traditional, rigorously tested application of salt in cryptography)

There are issues with using it as implemented in some posts here, with nested bcrypts, to be sure, but I think the concept is still fairly sound, though there are certainly implementation pros and cons.

----

As for maintainability:

I'm also familiar with crypt(3)-style password hashes, where a prefix uniquely specifies the algorithm (and subvariant) used.[1]

Why wouldn't this be fitting here? You can then easily detect, and deal with, passwords that have been tagged with "previous" peppers, such as forcing returning users to change password if a previous pepper was compromised, etc.

1 https://www.freebsd.org/cgi/man.cgi?query=crypt%283%29 or http://man7.org/linux/man-pages/man3/crypt.3.html, for some reason I can't find a link to a more comprehensive list at the moment

By that logic, every extra character you concat onto a salt is also "rolling your own crypto."

For example if my salt was CrytoRandom(10), and you increase it to CrytoRandom(15) you've just "rolled your own crypto" according to you.

If that is not the case then explain the difference between CryptoRandom(15) and CrytoRandom(10) + CryptoRandom(5) (longer salt Vs. salt+pepper).

There's a lot of people spreading FUD ("it is unknown!!!") and nonsense (concat two strings is literally rolling your own crypto!) in this thread.

I don't know if peppers are worth the dev' time, deployment issues, and additional maintenance (e.g. rotation). However I do know that the people arguing against it here aren't making rational counter-arguments that hold up under basic scrutiny.

PBKDFs like bcrypt assume the salt is a random value that is not re-used and is not required to be secret. You are re-using part of it, and require that it is secret. That's the worrying part, and it's one of the reasons that several concrete proposals posted here have actual security flaws.

Maybe. But that only makes sense if you reduce the length of the salt to add the pepper, which nobody is suggesting.

If the salt remains the same length and you add the pepper on top, it won't make the final hash less secure/strong, due to the way hashing algorithms are folded.

At worst case scenario you've literally added no security at all with the pepper. There's no rational scenario where it reduces the security when all other things remain equal (i.e. you aren't replacing the salt with a pepper, or reducing the salt's length/complexity for the pepper, etc).

> At worst case scenario you've literally added no security at all with the pepper.

Yes, I think this is the most likely failure mode (though not necessarily the only one - crypto can fail in very suprising ways!).

But even this is harmful, since you are potentially making changes to security-critical code for no benefit. At best you get more complexity and more chances to introduce bugs, plus a false sense of security.

I'm no security expert, but there are some obvious problems with concatting salt + pepper. For example, most hash libraries include the salt in the output so that you only have to store a single string (also encoded in the output would be the hash algorithm and a few input parameters). So now you've likely revealed your pepper in the DB. Oops! So much for extra security.

But that's not all! Hash algorithms are written assuming the salt is random, and now I have millions of hash outputs in which the last X bytes of the salt are shared. Have you proven that this doesn't increase the attack surface? It certainly sounds like it might. This is exactly the type of side-channel attack that tends to break crypto, and you're giving it away for free.

I generally disagree with hardcoded salts, you should assume everything is compromised in a successful attack. But I'm actually commenting here because I don't see how you can retroactively apply the second salt to a hashed string. Could you please elaborate or share a link?

Later edit: I'm referring to your example in your link:

    salt = urandom(16)
    pepper = "oFMLjbFr2Bb3XR)aKKst@kBF}tHD9q"  # or,
    getenv('PEPPER')
    hashed_password = scrypt(password, salt + pepper)
    store(hashed_password, salt)

How do you retroactively apply this?

You append the salt to the hash, and then re-hash it. Not exactly pretty, but it works.

His example doesn't rehash. His example adds the pepper to the initial salt. This is an honest question, how does one apply it retroactively?

I think you are spot on - you can't apply it retroactively in this case. If you set it up as H(H(pwd, salt), pepper), you could apply pepper retroactively, but that's not what's proposed here nor in the SO question linked above. Also if you do H(H(pwd, salt), pepper), you can successively apply new peppers and make it a bit more maintainable, but I am unclear that it is secure to do so.

I don't understand your issue with hardcoded salts. It essentially works the same way as an HMAC. It is some secret material that further complicates the attackers job...It doesn't mean they react to a successful attack in a different way.

After hashing the password he is storing the hash along with the users random salt, not retroactively applying the salt a second time.

In incident response you assume the worst, but in system design you try to minimize impact of common attacks like SQL injection.

There's nothing wrong with nesting algorithms (see the Facebook hash onion), so you can use the following scheme:

    bcrypt(bcrypt(password, salt), pepper)

And do a pass on all your database entries like

    bcrypt(old_hash, pepper)

In your website's example, you have

    bcrypt(password, salt+pepper)

Observe the difference between that and the rehash you just posted

    bcrypt(bcrypt(password, salt), pepper)

> bcrypt(password, salt+pepper)

I hope it's obvious that no one should never do this, since the output would contain the "salt+pepper" bits in cleartext alongside the hash, defeating the entire point of the "pepper":

https://www.usenix.org/legacy/event/usenix99/provos/provos_h...

In fact, this is a perfect illustration of why it's bad to put secret bits into a crypto function in a place that's not designed to take secret bits. Bcrypt does not treat the salt parameter as a cryptographic secret, and other algorithms might not either. And they might leak it in more subtle ways.

One would think it's obvious, yet this is what the person in the linked article suggests doing :/

> There's nothing wrong with nesting algorithms

This is really vague. What kind of algorithm? With itself, or just anything inside anything else? Passing the raw output of any common cryptographic hash (SHA-x) to bcrypt, for example, completely destroys its security, as bcrypt input is null-terminated.

(What happens when you nest DES in A*, anyway?)

Obviously you have to use type safety. You can't cast a binary string to a null-terminated string, you have to convert. That's a problem unrelated to hashing.

I would avoid using any particular symmetric algorithm twice. Otherwise if you have an example of algorithm chaining that can weaken security beyond the weakest link, I would love to see it. (Not that I think nesting is a great idea.)

> That's a problem unrelated to hashing.

It (poor implementation) is definitely related to implementing pepper on top of a secure password hash, though, which everybody is already doing differently.

> I would avoid using any particular symmetric algorithm twice. Otherwise if you have an example of algorithm chaining that can weaken security beyond the weakest link, I would love to see it. (Not that I think nesting is a great idea.)

“algorithm” is, again, really vague. (So is “nesting”.) But for something contrived and not snarky, here:

  h = sha512_hex(password)
  sha512_hex(bcrypt(h, gen_salt()) +
             bcrypt(h, gen_salt()) +
             bcrypt(h, gen_salt()) +
             bcrypt(h, gen_salt()))

The weakest link here is 374 bits (4 bcrypts), but the output is 288.

Oh sorry I missed this post for a while.

>“algorithm” is, again, really vague

Something that you can use to hash passwords. What you gave works if you assume gen_salt is seeded per user.

>The weakest link here is 374 bits (4 bcrypts), but the output is 288.

I'm afraid I don't follow. Your bit numbers confuse me, and I don't see how this results in an algorithm that is weaker than either sha512 or bcrypt.

> I'm afraid I don't follow. Your bit numbers confuse me

They’re bits of entropy (not counting the password itself) (I think). SHA-512(M): 512 bits; SHA-512(SHA-256(M)): 256 bits, for example.

> I don't see how this results in an algorithm that is weaker than either sha512 or bcrypt.

That’s my point. It’s not easy to get this kind of thing right, so just don’t bother with pepper.

Bits of entropy I understand, how you got "374" and "288" I don't.

>That’s my point. It’s not easy to get this kind of thing right, so just don’t bother with pepper.

What? Your point is that you haven't demonstrated that it's weaker than the weakest link, therefore you win?

Edit: Okay I figured out where you got 288. Still confused by the 374. Anyway you need to make truncations explicit. You didn't pass all of the sha output to bcrypt. You're taking advantage of an implementation API bug.

I'm not asking for evidence that shoving together functions from google without understanding them can go wrong. That's trivially true.

I want an example where combining hash algorithms is inherently wrong. Like using a block cypher twice can pop out your plaintext, but probably not as extreme.

Edit 2: Oh, 384!

    passwordHash = bcrypt(salt + password)
    encryptedHash = encrypt(passwordHash, pepper)

This way you can rotate your pepper by doing:

    decryptedHash = decrypt(encryptedHash , oldpepper)
    encryptedHash = encrypt(decryptedHash , newpepper)

Excuse my ignorance, but you probably shouldn't be able to reverse an irreversible hash.

It's reversing the encryption, not the hash.

Where in the original example was encryption involved? The salt and pepper were only ever used in a hashing algorithm.

The pepper is the key to the encryption, not a hashing algorithm.

Thanks for just ignoring my question - THERE IS NO ENCRYPTION IN THE ORIGINAL ARTICLE, so WHY are you restating that encryption is involved at all?

Straight from the original artical:

  hashed_password = scrypt(password, salt + pepper)

  hashed_password = scrypt(scrypt(password, salt),pepper)

Absolutely zero encryption going on here. Therefore, no ability to "decrypt" the pepper result. The pepper IS NOT a key for encryption. Period.

The pepper is being used as an encryption key in that example rather than using hashing.

In pacofvf's example, you are correct.

But, the original example is hashing the password. No encryption involved. So what makes anyone think that they can reverse a hash?

The problem is that pacofvf's answer assumes you don't need to do this retroactively.

This would work if you were building a new system today, but if you had a DB full of one way hashes you're not going to be able to retroactively modify the pepper.

And more importantly, slack straight up stated they salt the password and use bcrypt. It's all one way hashes, no encrypting/decrypting going on.

The point of hashing passwords is that the true password is not revealable.

The point of salting password hashes is to prevent identical cleartext passwords from being stored as identical hashes in the database. Salts are often stored in the database, as well.

The point of peppering keeps a database dump from being at all useful for recovering passwords. It make sure that a component of the process of cleartext -> DB entry is not even in the database, requiring something from the app as well.

Why does encryption work here? Because you've already done a one way function on the cleartext -> salted hash. At that point, there is still no way to reverse the process all the way to get the cleartext. By using a two-way encryption function for the pepper portion, you keep the ability to rotate 'peppers' periodically, in case it is leaked, for example.

Thanks for the informative post

Your question depends on your definition of retroactively.

I think the article's context is if you don't currently use a pepper, you can easily add one and update all of your password hashes in the database.

Update the hash when the user logs in because you have the password. You can tell if it's been applied by having another column, or prepending/suffixing the output with something that scrypt can't output.

Ciao Filippo!

That's a nice trick, I've read about it elsewhere but never used, will do for sure in the future!

Are you sure that the intruder did not had server access? I mean the info: "We were recently able to confirm that there was unauthorized access to a Slack database storing user profile information." is not enough to deduce that this was an SQL injection (although might very well be).

Stay strong :-)

Please do not do it:

http://stackoverflow.com/questions/16891729/best-practices-s...

That's a rather comprehensive answer from @ircmaxell over there, hm. Interesting, thanks for sharing.

so the downsides are "it's not maintanable" and "don't roll your own crypto". I think they are negligible compared to the upsides.

Not at all. "don't roll your own crypto" is a downside that can lead to things completely falling apart or weakening the system overall.

The real downside is that there's a better, proven way to do the same effective thing, which is make a database-only compromise require additional work, without rolling your own crypto. It also supports doing things retroactively for real (not some of the hacks being discussed in this thread) and key-rotation. All the upsides, with none of the downsides.

The fact that the pepper can't be changed/rotated far outweighs any upsides

You can change your pepper by double-peppering your existing password database:

  scrypt(scrypt(scrypt(scrypt(password, salt), pepper2013), pepper2014), pepper2015)

https://blog.filippo.io/salt-and-pepper/

Sounds like a maintenance nightmare. Need to ensure you keep your tongue straight when partitioning or restoring databases, migrating/splitting to new apps etc.

> "don't roll your own crypto". I think they are negligible

Please do not ever consider "rolling your own crypto" a walk in the park. Unless you have a serious security background and some actual cryptography education and research never, never, NEVER do this. It is not negligible, and it is not safe.

You consider "not maintainable" to be an acceptable downside to _anything_ you're doing with your software? What?

I wouldn't say negligible, but rather not slam-dunk convincing

That sounds like fuzzy scare-mongering to me.

1) You should not invent your own algorithm. That's a given. That's why you use bcrypt/scrypt.

2) It's not abusing the algorithm, it's using a longer salt (in the concatenation case).

3) There's nothing wrong with nesting algorithms (just remember to use hex/base64 encodings, not binary). For example Facebook passes passwords through half a dozen algorithms. They call it the "onion". And it includes a pepper.

4) As for being effective, I think the SQL injection case speaks for itself.

5) As for rotation - just don't do it. You pepper gets compromised? Who cares, add a new one on top of the old one.

Also, I'm confused at how the proposed alternative would be harder to get wrong:

> Encrypt The Output Hash Prior To Storage

Your second point might be dangerous - your salt values are no longer random but heavily biased and knowing that all salt values share some common bits might provide a new attack vector.

Is this true? I would think that static bits are no more dangerous than not having the bits at all.

Here is for example an attack recovering a 384 bit ECDSA key [1] by knowing the five least significant bits of the nonce (obtained by a side channel attack) for 4000 signatures. Now hashes and signatures are obviously very different things but I would not bet on the fact that a bias in the salt does not matter.

[1] https://eprint.iacr.org/2013/346.pdf

Interesting. Thanks for the link!

ECDSA is a completely different beast. I'm not aware of a modern password hashing function that would be broken if it was given a non-uniformly random salt. If such function were to be submitted to PHC (https://password-hashing.net/), I would consider it to be a disqualifying factor.

For password hashing purposes, salt doesn't need to be uniformly random, the only requirement for salt is to be unique and unpredictable to the attacker (see http://crypto.stackexchange.com/questions/6119/hashing-passw...). Most password hashing functions use a cryptographic hash on salt.

This particular function, scrypt, uses one-round PBKDF2-HMAC-SHA256 to mix password and salt:

https://github.com/golang/crypto/blob/master/scrypt/scrypt.g...

PBKDF2 feeds salt, basically, into SHA256:

https://github.com/golang/crypto/blob/master/pbkdf2/pbkdf2.g...

If there was a known attack it would obviously be a non-starter. But why take the risk that someone will come up with something comparable - in the broadest sense - to say the differential attack on MD5 that allows exploiting known bits in the salt if there is no need to?

Correct me if I'm wrong, but an algorithm is a set of steps, which this looks like to me

    salt = urandom(16)  
    pepper = "oFMLjbFr2Bb3XR)aKKst@kBF}tHD9q"  
      # or,        getenv('PEPPER')  
    hashed_password = scrypt(password, salt + pepper)  
    store(hashed_password, salt)

That is an algorithm, which composes bcrypt with pepper.

The idea of not using key-rotation alone is insane, but lets just focus on your last point

    Also, I'm confused at how the proposed alternative would be harder to get wrong

Really? AES literally has hardware support, and can be done in a single call, and has been studied for years. How can that reasonably be considered "harder" to get wrong than something proposed by some random guy on the interwebs?

Outside of Peer-Review, what reason would anyone have to use the pepper scheme? As others have posted, there are several community members who's opinions do matter due to extensive research and body of work

> 2) It's not abusing the algorithm, it's using a longer salt (in the concatenation case).

But PBKDFs like bcrypt and scrypt are not designed to keep the salt parameter secret; in fact they assume the attacker knows the salt. And so if they happen to reveal the salt to the attacker, this is not considered a bug in the algorithm and won't have been flagged or fixed by cryptographers.

(And more importantly in practice, the implementations of these algorithms aren't designed to keep the salts secret.)

The "concatenation case" completely defeats the purpose of using a pepper and leads me to believe that you're not qualified to be giving this kind of advice.

In what way it defeats the purpose of using a pepper?

Why combine hashes? Just use XOR. If bcrypt can protect "not_my_password", it can certainly protect "KKQ{H]zTDWVSJVA", which is the former XOR'd by "%$". XOR doesn't decrease the keyspace (or change it in any interesting way), so any attack on XOR is an attack on bcrypt; XOR is fast enough to evade any sort of timing attacks, too.

Careful. If an attacker is ever able to observe the output for a very short password - one byte, say - then only one byte of your secret salt is used, and the attacker could begin brute-forcing the secret salt starting with the first byte.

XOR could also result in NULL bytes anywhere in the hash input, which could drastically weaken passwords,. For example, bcrypt ignores any password characters after the first NULL byte. This is especially bad if the attacker can supply their own passwords, doubly so if they can observe the output, since they can then easily brute-force individual bytes of the secret and use that knowledge to intentionally create NULL bytes in the hash input.

> XOR doesn't decrease the keyspace (or change it in any interesting way), so any attack on XOR is an attack on bcrypt

I wouldn't make any statement like this unless you've actually gone through the steps to prove it.

>Careful. If an attacker is ever able to observe the output for a very short password - one byte, say - then only one byte of your secret salt is used, and the attacker could begin brute-forcing the secret salt starting with the first byte.

Can you be kind enough to explain with a very simple example?

I would appreciate understanding your point - Thank you

Using the scheme above, a password of "a" would result in a hash H('a' ^ x) where x is the first byte of the secret salt. The attacker can simply test all possible values of x (there are only 256) to determine the value of that byte. Knowing the first byte, the attacker can then look at a two-byte password and repeat the process to brute-force the second byte.

(It's also possible to brute-force more than one byte at a time; it would just take longer. For example if the shortest password the attacker can observe is 6 bytes then they would need to try 2^48 possibilities.)

ircmaxell makes a good point in that comment -- not that the concept of a pepper doesn't work, but that a two-way encryption function is a better choice than a hash function for applying the pepper.

Your pepper will be a long, random key that is known to your app server but not your database server. If you store passwords as:

     bcrypt(bcrypt(password, salt), pepper)

then you spend a lot of cycles bcrypting your long, random key, which is pointless (it should already be long enough to be un-brute-forceable), and you lose the ability to rotate keys or ever stop using this scheme in the future. There's also (in theory) some risk that nested bcrypt() doesn't work predictably.

If you store passwords instead as:

    encrypt(bcrypt(password, salt), pepper)

then you can routinely rotate peppers through a simple one-time decrypt-and-encrypt step on your app server, instead of endlessly nesting peppers as suggested in the filippo.io blog post.

The rest of it is more of a qualitative question -- what's the risk that someone gains access to your DB but not your app server, vs. the risk that in implementing the pepper, you somehow screw up and store something easily crackable?

> what's the risk that someone gains access to your DB but not your app server, vs. the risk that in implementing the pepper, you somehow screw up and store something easily crackable?

The question is irrelevant anyway, because if someone gets access to your app server, they get the secret in both cases (pepper and encryption). So the first risk there is present in both cases, which just makes this a static "what's the risk that you screw up in implementing the pepper?"

The question is not irrelevant. It's blindingly obvious that salt+pepper protects against DB-only attacks. Injection that does a DB dump, or vulnerabilities in the DB server that don't exist in the app.

So yes, S+P protection won't save you if the app server is completely compromised, but it does protect you in case DB only is.

And the ability to S+P seems pretty simple to implement and document. Why is everyone panicking that it's hard to do?

I read your question as comparing the originally-proposed pepper approach vs the encryption approach, and I said the question was irrelevant because both approaches have the identical risk that the app server is compromised, but only the originally-proposed pepper approach has the risk that it's not actually secure (e.g. because naïvely composing hash operations may be bad, and adding static bits to passwords may also be bad).

But upon reflection I believe you were instead using the term "pepper" to include the encryption approach as well and merely trying to question whether the added security of requiring an app server compromise is worth the risk that you still screw it up somehow. And to that I'd say that it's not difficult to apply an existing block cipher algorithm when storing/retrieving password hashes so I think the risk there is low.

> Why is everyone panicking that it's hard to do?

Everybody's panicking because the originally proposed pepper implementation is a really bad idea. That approach has not been researched for security implications, and there are many reasons to believe that composing hash operations without using a specially-defined operation like HMAC is bad, and adding static bits to the salt or password (e.g. if you use 'pepper.salt' for the scrypt salt or 'pepper.password' for the password) is also bad.

However, I believe the approach of using a block cipher to encrypt your hashes with an app-wide password is reasonable. It's not composing operations badly (encrypting a 256-bit string, or whatever scrypt emits, is perfectly reasonable given a secure key) or otherwise providing an attack vector on the hash algorithm.

The biggest risk I can see with this approach is you have to make sure the pepper is stored securely on your app server, never visible to the database server (and never accidentally committed to an open-source repo, if you're using an open-source server). Not just that, but you have to make sure that you don't accidentally lose it either (or you'll have instantly lost all your accounts). But this is a solvable problem.

Because it's harder to implement and easier to mess up than using standard encryption, and has unknown characteristics due to the fact that it's rolling your own crypto. Why would you spend the time and effort to implement it when you can do something that is objectively better in every way?

I've already seen one person in this thread taking "S+P" literally, and suggesting `bcrypt(cost, salt + pepper, password)`, which has a very obvious and critical problem. Other "peppering" schemes in the comments here also have significant weaknesses. So yes, even a simple idea like this is easy to botch in implementation -- even if the implementation is just a one-line change!

What is the problem? Does bcrypt truncate "salt+pepper"?

Because the salt is in the hash!

$2a$12$GhvMmNVjRW29ulnudl.LbuAnUtN/LRfe1JsBm1Xu6LE3059z5Tr8m

$2a means it's using bcrypt version 2a

$12 means 12 rounds

GhvMmNVjRW29ulnudl.Lbu is the salt

AnUtN/LRfe1JsBm1Xu6LE3059z5Tr8m is the checksum

You never know. Some developers might not be smart enough to use an SSL connection from their app server to their DB server. The influx of newb devs out there is growing and security is usually not high on the learning list or the new devs priority list. If so, hackers could possibly get access to the DB without even needing the app server in the first place. Get your certs folks!

I don't see how this is relevant. There is no reason to believe that the hacker needs to get at your app server in order to access your DB server in the first place.

Once you use Encryption, it's no longer pepper, it's encryption with weak key storage (hard-coded). At that point, genuinely, why not just follow best practices and store the key securely (e.g. using an HSM)?

Because nobody knows how to do that and it is likely extremely expensive by every metric.

And, no, I am not being facetious by saying nobody knows how to do that. I am being quite literal. Have you ever done that? Do you know how? Do you even know what you would google to figure out how?

I'm yet to see my favorite library of course's documentation on a HSM. How do you do that in e.g. PHP with MySQL? MVC with MS SQL? Java with Oracle?

It's worth it. And yes, I have done that, it's not a mystery, they all provide a API/ABI for interacting with them[1], such as OpenSSL's Engine API or PCKS#11.

Edit: There's even a project to literally do this directly from PHP[3]

[1] http://stackoverflow.com/questions/10796485/interfacing-with...

[2] http://en.wikipedia.org/wiki/PKCS_11

[3] http://stackoverflow.com/questions/3231293/how-to-interface-...

Eh, I had a Luna CA3 sitting on my desk for a while. HSMs aren't that exotic. I'd be more concerned about the HA aspects which could require extra code. I suppose you could just shove a USB HSM in several of your servers and encrypt every password with at least 2 of them.

I think AWS provides both key management services as well as cloud hsm.

I have not used either myself, but I would imagine the documentation is quite good.

Christ you weren't kidding about it being expensive. Amazons Cloud HSM is $5k upfront and $1.8/h thereafter, which is completely infeasible for the agencies I've worked at. A shame, I'd love to do the right thing there :(

That's why AWS also started offering Key Management Service: https://aws.amazon.com/kms/

You don't get your own HSM but it's MUCH cheaper ($1/key/month) and more scalable and available than an HSM.

That's brilliant. I'm definitely going to use that, thanks for posting it!

Joining the chorus; I've done it. Expensive on some levels, but very cheap on others. Dealing with keys as physical objects is a lot less stressful.

I've done it using a $100 Cryptostick (Now Nitrokey). Here's a guide from Mozilla: https://blog.mozilla.org/security/2013/02/13/using-cryptosti...

https://www.nitrokey.com/

Certainly storing your key with an HSM is superior and protects against a slightly large class of attacks, but in terms of getting the same protection as salt+pepper was designed for (limiting the damage of a DB-only compromise), encryption, even with weak key storage, is superior. Adding an HSM is just ways we can improve that even further (which aren't really even feasible with the salt+pepper idea).

Is there any significant evidence that peppering passwords helps? I've seen arguments for and against peppering out on the big bad internet. Everyone has opinions but there are few people's opinions about crypto that I actually trust.

The best article I've seen against this technique is by ircmaxell [0]. Nicely summed up in this sentence "It is far better to use standard, proven algorithms then to create your own to incorporate a pepper."

Anyone have source material (academic paper, Bruce "The Crypto God" Schneier blog post) that shreds some light on peppering passwords?

I'd be much more interested in how many iterations of bcryprt Slack were using. That has a much bigger bearing on events for me. Anyone at Slack know/want to answer that question?

[0] http://blog.ircmaxell.com/2012/04/properly-salting-passwords...

A properly implemented, simple pepper can only help password security and can't hurt it. Obviously you first must be using a good, slow algorithm (bcrypt, scrypt, or PBKDF2 with high work factor), but a pepper will only help you. (Let's assume the pepper is an AES key which all hashes are encrypted with.)

Yes, many times a dedicated attacker who has read access to your database will also have read access to your source code or config files, but many times they won't. And if they don't, then they won't be able to crack a single one of your passwords, while even with a modern and proper hashing algorithm they still may be able to crack passwords.

Take the scenario of a relatively intelligent hacking or hacktivist group, of which there've been several in the past 5 years. Let's say they're targeting someone they dislike for whatever reason, and find out that person is registered on some forum and decide to compromise the forum. (This tactic of lifting a whole haystack to find a single needle is very common for motivated attackers.) They don't care about any of the other users, they just want to try and crack the hash of one single member and have a full GPU cluster with which to do it. They're also willing to spend weeks trying to crack that one hash.

If the user's password isn't particularly strong, it's going to fall no matter what algorithm they used.

But if the forum is peppering all of their hashes, and those same attackers can only manage to gain access to the forum's database and not its local filesystem, then their chance of cracking that password goes to 0.

This scenario is a bit contrived because odds are motivated and intelligent attackers like these will end up gaining access to the filesystem and reading the pepper with enough time and effort, but the pepper is still an additional defense and means SQL injection alone won't be enough to crack passwords.

Hey thanks for the long response. I totally get the premise of peppering I think my problem is with this sentiment "A properly implemented, simple pepper can only help password security and can't hurt it".

From all the advice I've read security and crypto they don't work like that. The assumption is the other way around. A properly implemented, simple pepper can only hurt password security until proven otherwise by rigours testing and analysis.

Time and time again we read stories of a tiny implementation detail that created a sly and subtle vulnerability that simehow leaks information about original plain text by interrogating the cipher text.

bcyrpt with a large work factor and a per user salt is a PROVEN method to prevent attackers learning the plain text. Until I see evidence from a trusted cryptanalyst I'm not going to roll my own by adding in pepper they didn't plan on being there.

EDIT: sorry let me make my point a little clearer. In the event that the hacker can access the filesystem or memory -- whereever you store your pepper -- could the hacker use the pepper and an implementation detail in the peppering technique to learn information about the plaintext or the salt? This question is what needs to be answered by qualified cryptanalysts before developers start using peppers wide-spread in my opinion.

I would argue that while crypto most certainly works that way (use only when definitively proven), security in general is a bit more lax in terms of requirements.

Despite hashes being cryptographic primitives, user password hashing is less about cryptographic principles (preventing first and second preimage attacks) and more about increasing the amount of work an attacker must muster to find an input which hashes to the hash value.

Attributes like collision resistance generally mean almost nothing when on the scale of strings under 100 characters in length, which most user passwords are. Practically, you are never going to run into collision issues if you're using MD5 or later. Your goal is merely to increase the amount of CPU time it takes to find a hash's original input.

Because of this, even if AES encrypting a hash with a random pepper somehow reduced the collision resistance of a hash (I'm 99.8% sure it doesn't), it wouldn't at all affect the speed at which the hash is cracked.

>a sly and subtle vulnerability that simehow leaks information about original plain text by interrogating the cipher text.

Hashes are not ciphertext. For all intents and purposes they can be viewed as CSPRNG output. There is nothing you could do to them to leak info about the plaintext as long as your hash algorithm isn't pathological. There are things you could do to reduce collision resistance, but I addressed that above.

Password protection in web apps is not about encryption or decryption.

> A properly implemented, simple pepper can only help password security and can't hurt it.

Well, yes. But what is the definition of "properly"? There are definitely constructions of "pepper" that look simple, but drastically hurt overall security:

    bcrypt(hmac(password, key), salt)

If hmac returns raw bytes, you're in real trouble: http://blog.ircmaxell.com/2015/03/security-issue-combining-b...

It's sort of like the difference between birth control and counting based contraceptive methods (Standard Days Method). Executed perfectly, they are equally as effective. But with a slight error, one stays roughly as effective (losing maybe 5 to 10% effectiveness overall) while the other drops drastically (down to 10 to 20% effectiveness).

Considering using encryption is as effective as using a pepper, and it's less prone to weakening the core password hash, I suggest using encryption instead of peppers.

I am well aware of that misuse, as I've exploited it during a CTF before. :)

I would consider using the raw byte-output version of a function a very blatant example of "improper implementation".

Also, I agree regarding encryption. In my example I was actually referring to the random AES key as a pepper, even though it'd probably be better called an "application secret".

It can help in the following scenarios:

1. Hacker steals db but does not compromise web servers (because the hmac pepper key lives on the web servers and not in the db)

2. Hacker can run SQL Injection via web server, but cannot otherwise access web server memory/process

3. HMAC key is stored in a hardware security module and hacker cannot gain physical access

All of the above cases are also helped by just by regular symmetric encryption. Why make things more complicated than necessary?

How is HMAC more complicated than encryption?

I still wonder what value does password leak have. I changed my password, my old password was: DV1wn3yHk6W-8m9lZNo_ now you all know it, so what? I don't care, I believe you don't care either. On the other hand, if they were after valuable data, they had access to database and the got what they wanted. So the password is much less valuable than the other stuff they might have wanted. Like chat logs which might contain credentials to other services.

Right, but many people use the same password for their email account as they do for many other services.

Because I'm sure everyone uses a long, complex, and unique password on Slack. So nothing to worry about, everyone.

If Slack data isn't important, then it would have been fine to use one's default low-value password, right? So now some attacker has thousands of copies of the word "dragon".

Recommendations related to Security, should start from words "I'm {name}, known expert in IT Security, my works can be found here: {url}".

Otherwise such advices should be ignored.

Could not agree more. In fact, here is Bruce Schneier's quote[1]:

    Anyone can invent a security system that he himself
    cannot break. I've said this so often that Cory Doctorow 
    has named it "Schneier's Law": When someone hands you a  
    security system and says, "I believe this is secure," the  
    first thing you have to ask is, "Who the hell are you?" 
    Show me what you've broken to demonstrate that your
    assertion of the system's security means something.

[1] https://www.schneier.com/blog/archives/2011/04/schneiers_law...

Could not disagree more. Perhaps one should use their own reasoning and the ability to peer-review in conjunction with 'appeal to experience' rather than shut out discussion.

Is your reasoning that of an experienced, competent cryptographer? Because mine isn't. I'm not a cryptographer's peer, either, and I'll bet you aren't either. Step one--nobody said step all, but step one--is establishing your bona fides to determine whether it is worth burning cycles on your idea, because proving or disproving cryptography is very hard and very time-consuming. It is a heuristic that, generally speaking, works pretty well.

No, his reasoning probably isn't. But without putting words in his mouth, there is this quasi-religious "thou shalt not talk about cryptography" attitude among programmers like crypto is literally voodoo magic. The appeal to experience is an incredibly frustrating part of this. It's like people are willfully ignorant and forcing those of us who may not be experts but also want to have an intelligent discussion to pretend to be idiots along with them.

It's not "thou shalt not talk about cryptography."

It's "cite your sources."

If you're pointing out a gaping flaw, you don't need a source. If you're suggesting a standard security measure, you don't need a source.

To insist on something novel, yes you want a source.

I would agree, except "standard security measures" aren't and you lead to travishamockeries like the pepper nonsense. Which is why I default to, "cite your sources."

I might get down-voted for this, but I'm going to clarify my position a little.

I'm not advocating for shutting-down anyone's discussion. What I, and probably others, are advocating for, is only putting in trust in proven crypto.

The parent comment to all of this basically has the form: "Wow, you could totally solve your password hashing (that isn't broken) by using this scheme I came up with. No one else has looked at it, but boy, it looks difficult to crack to me."

There's a complete difference between that comment, and many of the others: "how about if we do X", and I believe the second is completely valid for discussion; As LONG as you include relevant experts in that discussion.

Joe and Bob talking about encryption isn't very useful unless Joe &| Bob are trained in cryptography and have experience in applying that crypto in the real world.

Down-vote all you want, but that's my view-point at least.

The best summary I've read recently on peppering was posted to the PHC list by Thomas Pornin last week. It's worth quoting in its entirety;

  Adding an additional secret key can be added generically, in (at least)
  four ways, to any password hashing function:

  1. Store: salt + HMAC_K(PHS(pass, salt))
  2. Store: salt + PHS(HMAC_K(pass), salt)
  3. Store: salt + AES_K(PHS(pass, salt))
  4. Store: salt + PHS(AES_K(pass), salt)

  I have used here "HMAC" to mean "some appropriate MAC function" and
  "AES" to mean "some symmetric encryption scheme".

  These methods are not completely equivalent:

  -- With method 1, you forfeit any offline work factor extension that
  the PHS may offer (i.e. you can no longer raise the work factor of a
  hash without knowing the password). With methods 2 and 4 such work
  factor extension can be done easily (if the PHS supports it, of
  course). With method 3, you can do it but you need the key.

  -- With methods 2 and 4, you must either encode the output of HMAC
  or AES with Base64 or equivalent; or the PHS must support arbitrary
  binary input (all candidates should support arbitrary binary input
  anyway, it was part of the CfP).

  -- Method 4 requires some form of symmetric encryption that is either
  deterministic, or can be made deterministic (e.g. an extra IV is
  stored). ECB mode, for all its shortcomings, would work.

  -- Method 3 can be rather simple if you configure PHS to output exactly
  128 bits, in which case you can do "raw" single-block encryption.

  -- Methods 1 and 3 require obtaining the "raw" PHS output, not a
  composite string that encodes the output and the salt. In that sense,
  they can be a bit cumbersome to retrofit on, say, an existing bcrypt
  library.


  The important points (in my opinion) to take into account are:

  1. This key strengthening (some people have coined the expression
  "peppering" as a bad pun on "salting") can be done generically; the
  underlying PHS needs not be modified or even made aware of it.

  2. Keys imply key management, always a tricky thing. Key should be
  generated appropriately (that's not hard but it can be botched in
  horrible ways), and stored with care. Sometimes the OS or programming
  framework can help (e.g. DPAPI on Windows). Sometimes it makes things
  more difficult. You need backups (a lost key implies losing all the
  stored passwords), but stolen backups are a classical source of
  password hashes leakage, so if you do not take enough care of the
  security of your backups then the advantage offered by the key can go
  to naught.

  3. For some historical reasons, many people feel the need to change
  keys regularly. This is rather misguided: key rotation makes sense in
  an army or spy network where there are many keys, and partial
  compromissions are the normal and expected situation, so a spy network
  must, by necessity, be in permanent self-cleansing recovery mode; when
  there is a single key and the normal situation is that the key is NOT
  compromised, changing it brings no tangible advantage. Nevertheless,
  people insist on it, and this is difficult. The "method 3" above
  (encryption of the PHS result) is the one that makes key rotation
  easiest since you can process all stored hashes in one go, as a
  night-time administrative procedure.

  4. Key strengthening makes sense only insofar as you can keep the key
  secret even when the attacker can see the hashes. In a classical
  Web-server-verifies-user-passwords context, the hashes are in the
  database; one can argue that database contents can be dumped through a
  SQL injection attack, but a key stored outside the database might evade
  this partial breach. But if the key is in the database, or the breach
  is a stolen whole-server backup, then the key does not bring any
  advantage.

  5. If you _can_ store a key that attackers won't steal, even if they
  get all the hashes, then you can forget all this PHS nonsense and just
  use HMAC_K(pass) (or HMAC_K(user+pass)). The key must thus be
  envisioned as an additional protection, a third layer (first layer is:
  don't let outsiders read your hashes; second layer is: make it so that
  your hashes are expensive to compute, in case the first layer was
  broken through).

P.S. As the inventor of blind hashing (which would have prevented this breach entirely) I have a serious horse in this race. We launch publicly at RSA 2015 in San Francisco. Hope to see you there!

> 3. For some historical reasons, many people feel the need to change keys regularly. This is rather misguided: key rotation makes sense in an army or spy network where there are many keys, and partial compromissions are the normal and expected situation, so a spy network must, by necessity, be in permanent self-cleansing recovery mode; when there is a single key and the normal situation is that the key is NOT compromised, changing it brings no tangible advantage.

This is a really strange advice from Thomas Pornin. People rotate keys because not doing so weakens most symmetric encryption schemes. For example while using AES-GCM with 96-bit nonces one needs to rotate keys after encrypting roughly 2^32 ~ 4 billion messages; otherwise the IV collision probability will be higher than 2^(-32), which is already high enough in most large scale systems (and really bad things happen when the IV is repeated).

Given a salted hash is being encrypted, who needs a nonce? The salt's already taken care of that, right?

Also, if you have 4 billion hashes stored, and you rotate the key, and you still have 4 billion hashes stored... What's changed? You would need a key ring or derivative keys I guess but I think this is actually a case where ECB does the job.

But I guess we've now proven the point that even a pepper is non-trivial.

We have an approach to this which doesn't modify the password hashing at all, so it can't possibly reduce the strength: we store users and password hashes (currently bcrypt * ) in two separate tables, and the key used to look up a given user's password hash is an encrypted version of their user id. The encryption key for doing this transformation is loaded dynamically at app startup and has some extra security precautions around storing it, so it's not baked into either the source code or a config file.

The upshot is that if you get a db dump and are able to brute force some bcrypt hashes, you won't know what usernames they go with. If you get a db dump and our source code, you're still out of luck. If you get ahold of an old server hard drive, you're out of luck. If you root a running server and inspect the process memory, you can obtain this key.

This scheme also allows the mapping key to be rolled, which would immediately invalidate all passwords in the system.

*we also version our password hash records so we can migrate from bcrypt to a new scheme fairly painlessly if it's warranted in the future.

That's a really interesting idea. I was thinking it might have performance implications but I think that will only apply to the password validation. How do you make sure the mapping key doesn't cause collisions when you roll it to reset everyone's passwords?

That's a great point about rolling the mapping key. We would likely be dropping all the rows in the password hash table for such an extreme event anyway, though, since otherwise there's no way to garbage collect the orphaned ones.

> you won't know what usernames they go with

But if you've got a list of all usernames (probably a relatively small number) and access to a running system, isn't it easy to just try each password against each user until you find a match?

The common practice of limiting logins from a single username wouldn't help with that either.

Yes, but we currently have 2.1 million users, so that's still no small burden. And don't forget that's only after you brute forced the passwords.

So you basically increased the cracker's workload by six orders of magnitude, which is equivalent to increasing bcrypt's work factor by 20. Cool!

If you increase bcrypt's workfactor then legitimate requests take longer.

It's not the security of the passwords I'm thinking about, it's whether or not you've really enhanced it much by obfuscating the relationship between password and username. I'm sure you've thought about this more than me, but I'm a bit skeptical, since if you have a lot of users, you probably have had to create systems to make the log in process extremely efficient. If all of your users wanted to log into your system within a 24hr period, could they? Maybe a week? If they could, then an attacker can attempt to log in with each username over the same period of time.

Is there some reason that you don't just hash the whole column or table though? It does everything you described, except also makes even the bcrypted passwords available in the event of a db dump. It just seems like a half-measure where a whole-measure is just as easy to implement.

Hmm, interesting suggestion. We wouldn't want to encrypt the entire row since we want to be able to query on specific columns (version in particular). My only excuse otherwise is that the goal was "disassociate the hashes from the users" not "encrypt the hashes." :) You're right that we could have done more, although whether that makes meeting the original goal a "half measure" depends on your perspective. :)

Well, let's imagine an attack scenario.

As an attacker, I get SQL access to your DB (meaning no access to the encryption key). I then download the user names, and the hashes. I then attack the hashes offline. I recover only the weakest few percent (since you're using bcrypt). But since the weakest few are those most likely to be re-used (both by different users and by a single user across sites), they are going to be both more valuable to me and easier for the next steps:

Then, I take the highest frequency passwords and the user table, and I start validating them online in your system. Now if I do that too quickly, you'll notice and I'll be shut down. And if I do that all from the same IP, I'll be shut down.

But what if I had a botnet that I could distribute the load across. What if I kept my request rate small enough to stay under the radar of even a moderate scale system.

I would expect to start seeing meaningful results within days.

If you had 1000 users, then I could surmise that you don't have much traffic, and hence keep the request rate down to perhaps 100 per day. In 10 days I'd have at least a few u/p combinations that I know for a fact worked.

If you had 1000000 users, I could ramp it up quite a bit higher, to perhaps 1000 or 10000 per day.

And since they all came from separate IP addresses, it could be rather difficult for you to tell an attack was going on unless you were looking specifically for it.

Does that mean you should stop immediately? No. It's not that bad of a scheme. But be aware that it doesn't give you (or your users) the level of protection that it may look like on the surface.

So after a breach, you have our (currently) 2 million hashes, and let's say you recover only the weakest few percent of the passwords, which is 60000 known good passwords. Instead of owning 60000 accounts now, you have 60000 passwords, each of which is going to require on average one million attempts before you guess the correct username. Is this not self-evidently better?

You forget the cases where username=password (or username+CelverP3rmutation)=password).

Well, let's look at it realistically: http://arstechnica.com/security/2015/01/yes-123456-is-the-mo...

The #1 password out of 3.3 million was 123456, which was used 20,000 times.

So extrapolating that for your 2 million hashes, we'd expect the top password to appear roughly 12,000 times.

Running those numbers, we'd expect each guess to have a 1/12000 chance of matching. Or more specifically, a 1988000/2000000 of not matching.

With some quick running of those numbers, we'd expect a 50% chance of finding a match after trying just 115 random usernames.

I'm not saying it isn't an interesting approach, I just don't think it's nearly as effective as if you encrypt the hash directly (which has no attack vector unless you can get the key).

From your blog:

   hashed_password = scrypt(password, salt + pepper)  
   store(hashed_password, salt)

Why aren't you storing the (N, r, p) parameters too? Is it because your library's "scrypt" function automatically encodes all the parameters it uses in the returned string? (This is not hypothetical; it's exactly what many scrypt implementations do.) If so, congratulations: You just stored the pepper bits in your database, because they're part of the "hashed_password" value.

If you care about security then just use an HSM, they aren't that expensive, and eliminate the possibility of a database dump.

> No financial or payment information was accessed or compromised in this attack.

This wouldn't be my first concern. It would be all of the confidential communication that happens within slack.

Agreed. The content of the chat's would be potentially much more important in my mind.

Which leads to the question if slack encrypts the chat data in the database.

That would make implementing search quite hard so I'd say - it's pretty likely they don't encrypt it.

If anyone from Slack is reading this, the encryption should be an option, even if it means disabling or substantially slowing the search feature.

If they encrypted it, Slack would have to hold the key, so that all users in an org can then read existing messages.

No, it could be a private key shared among users.

That's not right. There is no need to store text body in order to index it. Furthermore, you can implement an index of token hashes, rather than an index of tokens.

It would remove a lot of nice search features, however. If you just index tokens without positional information, you have a much harder time performing phrase matching. If you include positional information, you can probably crack the encryption because some tokens are statistically more likely to appear next to each other than others.

If you index shingles (phrase chunks) instead, you lose out on sloppy phrases...you can only match exact phrases. I imagine you can perform a similar statistical attack too.

Hell, just getting the term dictionary would probably allow you to reverse engineer the tokens, since written language follows a very predictable power law.

Hashing also removes the ability to highlight search results, which significantly degrades search functionality for an end user.

Basically, yes, you can do search with encrypted tokens...but it will be a very poor search experience.

This reminds me of the plot of Silicon Valley

If they dont encrypt storage they are highly negligent. Index and search are done in RAM,which is slightly harder to steal than disk data.

Is there a good reason to keep chat data longer than it takes to deliver it to the recipient?

They archive chat messages so that you can search through them later.

That alone would be a great reason not to use them.

It's also a great reason to use them, isn't it? Your searchable chat history basically becomes the knowledge base of your company.

And a great target for discovery in any sort of lawsuit.

As is email.

To me, that is something that you should keep internal, on internal systems with vetted free software.

It's why I love Slack. If I remember a conversation about something two months ago I go to the room, search and find exactly what I needed.

https://slack.zendesk.com/hc/en-us/articles/203457187-Settin...

It's configurable for paid accounts, and can be set as low as one day. However, one of the best features of slack (and products like slack) is message history and search. Otherwise, IRC isn't all that different (WRT messaging).

Maybe it's time for Slack to adopt the Axolotl ratchet, too.

I'd love for them to do that, but there's a couple of problems that they'd have to overcome first.

First: Slackbot. This is a Slack-run bot that's in every channel; team owners can customize it to do various things, like scan messages for keywords and give out canned responses. Even if Slack adopted some variant of encrypted chat, each message would still need to be readable by Slackbot, so Slack would still have the means to collect every message.

Second: channel history. When I join a channel, I can see the messages in that channel from before I joined. This means that Slack (the server) must be able to give me those historical messages. In an encrypted group chat, the messages are encrypted only with the keys of the participants at that time, which means newcomers can't read them.

I'm sure there are other features in conflict with end-to-end encryption, too; these are just off the top of my head.

The first could be solved by having the activation part of the bot run on the clients themselves, and only send those messages in a readable way to the server.

As for the second, the server could ask one of the clients to re-encrypt the channel history with the newcomer's key. It would only fail if nobody was online the moment you joined the channel (and you still could get it later).

The post notes that the breached database is the user table, which would not contain chat history. I agree that making this abundantly clear makes sense.

This makes it sound like other data was compromised for some specific users. Since they didn't go into how they know it was only for only these users, I'm not very confident about this.

> As part of our investigation we detected suspicious activity affecting a very small number of Slack accounts. We have notified the individual users and team owners who we believe were impacted and are sharing details with their security teams. Unless you have been contacted by us directly about a password reset or been advised of suspicious activity in your team’s account, all the information you need is in this blog post.

I would suspect things like "being used from a completely new country" or something similar. Could be those are the accounts with weak passwords that the attacker tried the top 10,000 passwords against.

This is actually an interesting point. A compromised user table could conceivably be used for all sorts of nefarious purposes. If the attackers "having access" to the information in that table includes the ability to modify that table, then it is pretty much open season on Slack. For example, an attacker could replace a target user's password-hash with a hash that the attacker knows the plaintext of. Depending on the implementation of the random salt, the attacker may have to replace the salt as well. Then, the attacker logs in as the user, downloads the desired chat history, logs out, and sets the password hash to the original. Not enough information was really given in the blog post, but by the sounds of it, some teams experienced more targeted attacks.

If you get the user table, you can log in. If you can log in as (some) users. If you can do that, you can see (some) chat history.

edit you can log in if and when you crack some of the hashes.

Incorrect. You can't login with a password hash, you need a password.

No kidding. That's why I put (some) users. Because brute-forcing the hashes will give you some password plain texts.

I guess that I missed a step in the explanation where you attack the hashes.

However I see that they say that they are using some best practices (bcrypt, "salt per-password") so this attack will be largely mitigated.

If you get the user table, you can crack the password hashes offline, at your leisure.

While technically true, this seems like it would be computationally infeasible, or at least impractical, given that they were not just hashing but also salting the passwords.

Of course, I barely know anything about computer security, but at least it should prevent attacks using rainbow tables I think?

No, a simple password like "slack123" should be easy to crack with any usable password storage method.

With a separate salt for each password not even the NSA can crack that (that we know of). With a single salt for all of them, maybe.

Sure they can. Anyone can. It just takes a long time per password to crack (that time is a function of the cost/# of rounds of the hashing function).

Not necessarily easy. If they're using a decently high cost for their use of bcrypt, we're talking hours to days (or more) per user, even when only considering weak passwords like that.

True, I guess it's possible to crack a password for a single user, especially one with a weak password. I was more thinking that it's unlikely they'll be able to crack the passwords of everyone who was in their database, and given that Slack has so many users it's unlikely for any single person that his/her password will be cracked.

Of course, even if they can't steal everyone's passwords, maybe the hackers will try to crack the passwords of higher profile targets.

GPUs are fast enough to crack a very large percentage of passwords in a short time by brute force, if a simple algorithm was used, even with salt.

Depends on the nuances of the system. If you can pass-the-hash, you can get in.

My concern are the usernames, emails and phone numbers that were probably not encrypted

ultimately passwords can be changed; internal chat messages regarding personal and confidential data can not be taken back.

User metadata can be used for social engineering, and people are typically the weakest link.

Exactly!!!

Encrypting user data should be a common practice like hashing passwords.

Why? Encrypting e-mail addresses would break password reset features and phone numbers are generally public anyway (yes you can go X-directory, but the real issue here is why these services require a valid phone number to begin with)

Why would encrypting email addresses break password reset? You can encrypt the database at rest such that the application has a private key that can decode it. That way both the application and the database server need to be breached to obtain anything usable.

Or just the application. Generally, it's much easier to convince apps to give you the data instead.

It's often a bug in the application that exposes the database, so the same bugs might also be used to expose the private key.

It's also worth noting that it wouldn't just be the web servers that require your private key, it would also be any mail servers you use for sending your newsletters and such like (assuming these aren't run on your web servers - which often isn't the case). Then there's your telephone support staff, who would also may need to know your e-mail address so they could do their job effectively. And any other operators that might compile data extracts, eg for 3rd parties where users have given permission for your details to used / sold.

Quickly you're in a situation where your private key is more available across your infrastructure than the e-mail would have been if it wasn't encrypted to begin with.

Now lets look at the cost of such a system. There's an obvious electricity / hardware cost with the CPU time required to encrypt / decrypt this data (after all, CPU time is the general measure for the strength of encryption) and the staffing cost with the time wasted jumping through those extra hoops. The development time, code complexity, etc - it all has a cost to the company.

So what's the benefits in any companies doing this? They don't gain any extra security? This is really more of a privacy policy for their users; and users which are that paranoid about their e-mail address being leaked should either use a disposable e-mail account or shouldn't be using a cloud-based proprietary messenging network to begin with. What's more the chat history might well have your e-mail address in anyway (eg "hi dave, I'm heading into a meeting shortly, but e-mail me at bob@example.com and I'll have a look tonight")

Don't get me wrong, I'm all for hashing / encrypting sensitive data. But pragmatically we need to consider:

1) are e-mail addresses really that sensitive? Or instead should we be encouraging better security for our web-mail et al accounts (eg 2 factor authentication) to prevent our addresses being abused. Given that we give out e-mail addresses to anyone who needs to contact us, I think the latter option (securing our email accounts) is the smarter one

2) instead of encrypting phone numbers and postal addresses, should we instead be challenging the requirement for online services to store them to begin with? If they have my email address, why do they also need my phone number? Postal address I can forgive a little more if there's a product that needs shipping or payments that need to be made.

Third party authentication should be the norm. Leaving authentication to providers that absolutely know their shit, just like we leave payments to third party services.

Of course, that requires a decent protocol, and Mozilla is doing the world a disservice in not marketing Persona better seeing as it's the right solution....

Major privacy issues, single point of failure etc etc. We leave payments to third party services because nobody wants to deal with the compliance nightmare that PCI-DSS is, not for security reasons. Payment is also mostly less sensitive to availability and latency issues than authentication.

So in a world where PCI-DSS isn't a thing, you're fine entering your credit card data directly on the forms available on random websites?

Why's a password so different, seeing as most people reuse those passwords? Why do we essentially allow (and yes, I am excluding those that use password managers in this statement, I'm one of those) access to our webmail and other critical services to random websites on the internet? What makes this right?

> Payment is also mostly less sensitive to availability and latency issues than authentication.

That's patently untrue. Latency issues are nonexistant in both areas, and availability issues are critical in both areas.

Yes, I have no problem entering my credit card data directly on the forms available on random websites.

Credit card payments online are so ludicrously insecure that it baffles me it's even legal. I only use them when dealing with the US (although some of the major retailers like Apple have finally started accepting 21st century payment methods), and I simply assume my credit card info has been leaking all over the place for ages.

The whole basic premise of credit cards is "we know it's totally broken, we'll just refund you the money because it's cheaper than fixing the problem".

> So in a world where PCI-DSS isn't a thing, you're fine entering your credit card data directly on the forms available on random websites?

Yes. It might be a hassle should someone misuse it, but the status-quo effectively means if I didn't make the purchase I'm not responsible for it.

More importantly, this was proven before PCI-DSS was a thing.

You mean like how Authy specialised in two-factor authentication, but still managed to have basic string concatenation bugs that rendered their entire 2FA system bypassable?

... no? no I don't mean like Authy.

Huh? This is the first I've heard about this, and searching for "Authy concatenation bug" isn't turning up anything useful.

i believe he's referring to this: http://www.scmagazineuk.com/authy-patches-after-2fa-bypassed...

Here's the write-up from Homokov. The guy is a pen-testing genius: http://sakurity.com/blog/2015/03/15/authy_bypass.html

But if you just want the money shot: http://sakurity.com/img/smsauthy.png

Yes. Typing '../sms' in the field bypassed the 2nd factor. Just, wow.

Huh. Well now I know. Thanks!

Amazing what you can do with improperly-implemented input sanitation :)

This probably could've been prevented by disallowing non-number inputs, no?

"In fact the root of the problem was default Sinatra dependency 'rack-protection'".

They were doing the input sanitation, but it wasn't the very first thing in the processing pipeline, since "best practice" was to pipe everything through 'rack-protection' first.

Homokov was first to state, this was really a black-swan type bug which 99.9% of the time makes it into production. Apparently, they were doing the "right thing" and still got burned.

The parent meant "This probably could've been prevented by disallowing non-number inputs" in SDK libraries. Yes, if SDK would cast everything to digits it wouldn't be possible. It is also quite obvious security-in-depth for a 2FA API. Now they do it.

*HomAkov

:-( Sorry, typo. And HN won't let me edit now, grrr!

Or even just input validation on the form itself before passing on to the API, which is more of what I was getting at. I don't know about the details of Authy's setup, but I know that AJAX (for example) supports enforcement of specific value types in text fields.

Basically, the form itself could have (and maybe even should have) required numeric-only values, seeing as Authy's codes are either 6 or 7 digits long and contain no alphabetical or special characters.

hey, that causes some immediate stir in my mind as a user of Authy. Could you share any reference to the incident you mentioned?

http://sakurity.com/blog/2015/03/15/authy_bypass.html

You never decrypt a password however. You only compare the hashed version of the claimed one to the stored hashed version, a one-way operation.

What could you do with a one-way encrypted phone number? I'm not able to enter a phone hash to make a call.

Encryption isn't the same as hashing. Encryption is two-way.

The previous comment did make the encryption / hash distinction - though I can totally understand how his post might have been misread that he was recommending the same mechanisms for both sets of data.

OK, so slack stores a username, name and email address for each user. This is visible to everyone else in the same Slack team at minimum. You also need it for e.g. password resets, perhaps billing.

We can assume they aren't total idiots and there's a Internet facing application server that connects to a internal-only database server that has this data. Also, assume SQL injection is not the attack vector.

How would you apply encryption to protect the username, name and email from an attacker that has gained access to the application server? I've gained some shell on the server and have 24 hours minutes to extract data. I can see all the files on the server but maybe as non-root but just the user that runs the application. How can you, as a security sensitive application developer, stop me if I've gotten so far?

I wouldn't. I don't agree with his point either (see my response to him: https://news.ycombinator.com/item?id=9277659).

> Exactly!!! > Encrypting user data should be a common practice like hashing passwords.

I get the feeling that you've never done this before and you don't understand the technical challenge and implications of the added complexity you propose here for an essentially free to low-price all-in-one communication online service.

Slack is not the NSA, encryption is not the answer to every security problem out there.

It's still worth mentioning, even if it's not your "first concern".

>"If you have not been explicitly informed by us in a separate communication that we detected suspicious activity involving your Slack account, we are very confident that there was no unauthorized access to any of your team data (such as messages or files)."

Under their FAQ on the post. It could be inferred that there was some unauthorized access to certain users' communication logs?

Assuming (no evidence, it's just very common) that this was a SQL Injection, here are some ways to protect yourself:

* Use http://en.wikipedia.org/wiki/Database_activity_monitoring. If you don't list users on your site and you get a query that would return more than one user record, it's a hacker

* Add some http://en.wikipedia.org/wiki/Honeytoken s to your user table, and sound the alarm if they leave your db

* Use Row-Level Security

* Database server runs on own box in own network zone

* Send logs via write-only account to machine in different network zone. Monitor logs automatically, and have alerts.

* Pepper your passwords (HMAC them with a key in an HSM on the web server (then bcrypt). Don't store key in db). https://blog.mozilla.org/webdev/2012/06/08/lets-talk-about-p...

* Use a WAF that looks for SQL injections

* [Use real database authentication, per user. Not one username for everyone connecting to db. Yes, this is bad for connection pooling]

I think it depends on the structure of the system... If you separate authentication from profile information, then you can run each on separate systems. They were already using bcrypt, which is a fairly strong salted hashing system. As to restricting access, having all access to the database restricted to API servers that provide the limitations you mention, you get a similar level of control, without the complexity of managing per-user database logins. With per-user database logins, you are then subject to whatever system the dbms uses. If you are using systems with looser less fine grained controls, you can be even more limited.

Why would you assume that? There are plenty of ways to hack into stuff without sql injections.

It's the most common vulnerability. https://www.owasp.org/index.php/Top_10_2013-A1-Injection

It's the most common vulnerability on the web. It's certainly not the most common vulnerability in projects built under popular non-php frameworks. Under that model, it's harder to create a situation where a SQL injection is possible than not.

Edit: Slack's in PHP, I thought it was in RoR for some reason. Oops.

Slack is a web service written in PHP, so I'd say elchief's assumption is reasonable.

Additionally, Slack has had SQLi attacks found against it in the past, which is proof that they aren't defending against it systematically.

The most important thing to stop sql injection is to validate your parameters on the server side.

Yes, but:

1. Not all SQL statements are parameterizable (dynamic identifiers vs literals)

2. Stopping SQL injection doesn't stop Insecure Direct Object References

3. Developers make mistakes

4. Plugins are a risk (example: http://www.zdnet.com/article/over-1-million-wordpress-websit...)

For parameterization to work you need to be perfect, always. My suggestions are for when someone else fucks up.

I am an application security professional, and I created this account in order to make this post after reading many of the comments on this thread.

Many of the comments have great suggestions. However, very few talk about the most important part of creating mitigations and designing key management/crypto. What is the security target?

Before throwing new designs at a problem, the attackers and attack vectors must be defined. If you don't know who you are guarding against and what they will do (and what data they will steal), then how can you possibly say what is a good mitigation??

One might argue that the threat is obvious, but I'll guarantee you that there are dozens of threats here. List them. Prioritize them. Then mitigate them. It is helpful to fully understand the problem/solution space before jumping in with pepper's, salt's, extra databases, and solutions.

It's refreshing to 1) see a breach notification including the actual password hashing algorithm, 2) see they're using a strong one like bcrypt (presumably with a reasonable cost factor).

Regardless, this is an example of why cloud communication (and ticketing and database off-loading [see MongoHQ] and...) systems probably won't ever become commonplace in most of the government space and the finance and health sectors.

> Regardless, this is an example of why cloud communication (and ticketing and database off-loading [see MongoHQ] and...) systems probably won't ever become commonplace in most of the government space and the finance and health sectors.

I agree. We might not like rolling out our own instances, but it prevents hackers from being able to grab ALL THE DATA in one fell swoop. It really amazes me that some EHR systems have gone the cloud route.

It's heartening to me. I've seen small practices with atrocious IT security. No WAY is self-hosted (for the thousands of small practices with maybe a couple of clueless help-desk types) even a billionth as secure as a professionally secured cloud service.

Also, "cloud" for services like this means "your own private instance of the software running in a private VM in our datacenter" not "your own customer_id in a shared database."

If you're small, cloud may be better, but if you're large it often isn't.

Large does not mean you have better security, as I understand. (see Sony)

Yes, but you at least have the money and resources available to have better security. If you choose to squander those resources and not dedicate a large enough budget to your security department, that's your fault.

Small companies typically can't afford competent and professional security analysts, engineers, penetration testers, and auditors.

It's true that large will have more resources to do security right, but also they become a bigger target. If a small company self-hosts, they are less likely to be targeted than if they are a customer of a big cloud service where hackers might incidentally steal their data because it's there with thousands of other accounts.

I guess what I'm saying is that regardless of who you are, there is no easily discernible best practice playbook, just a sea of tradeoffs generally made by people with a woefully inadequate grasp of the risks involved. Heck, even the best security people are at a disadvantage in the asymmetrical battle of infosec.

OTOH, if you're small, you are not as interesting a target as a huge cloud provider that hosts everyone. Which means, while small practice's security should be good, it doesn't actually have to be as good as the big cloud behemoth.

It's why your gmail account is more likely to get hacked than my piddly self hosted imap server. Google's network security is unarguably better than mine, but you are never going to social engineer your way into changing my password, which is actually doable with gmail (happened to my sister in law).

Also, it is far easier to harden one or two hosts than entire farm of different devices.

If you had a vulnerability in an EHR that was run locally at many different hospitals a hacker would still have to target every single hospital that uses it and wade their way through a bunch of different custom configurations. It's not as juicy a target as a cloud-based system where a single vulnerability can get ALL the data of ALL the hospitals EVER in one location. (Like the Anthem hack.) I agree that most locally run systems are more vulnerable than the professional cloud based services. But cloud services are more exposed to attack and are a more profitable target for hackers due to their size.

I think you have to assume that you're going to be hacked if you're a big enough target. You don't know what you don't know about your vulnerabilities. The better question is how you're going to design your data and platform to minimize the damage a major hack can do.

bcrypt is only strong if their cost / work-factor is set correctly

Exactly this. If they used ten rounds, it's dire, and just saying "bcrypt" doesn't say much unless you also specify the number of rounds.

It says a lot more than your passwords are safely stored behind unsalted MD5 :)

10 rounds of bcrypt is "dire"?

Yes? 16 rounds take 1ms on my (old) machine. In Python, no less.

I hate to suggest that your observation is wrong, but 16 rounds should take orders of magnitude more time than 1ms. 16 rounds using Mindrot's Java implementation of BCrypt on my admittedly old 2009-vintage i7 consumes 6.3 seconds to hash a 10-character password.

That's because you're conflating "rounds" with "work factor". "Work factor" is actually 2^rounds, you're using 65536 rounds. Try 4.

Thank you for pointing that out. I suspect many of us in this thread are referring simply to the single parameter to BCrypt.gensalt as the "work factor" or "number of rounds" interchangeably. And you're right, the work factor is what is actually provided to gensalt.

Nevertheless, in all implementations I am aware of, the default for that parameter is 10. And earlier, you wrote:

> If they used ten rounds, it's dire, and just saying "bcrypt" doesn't say much unless you also specify the number of rounds.

tedunangst and I both assumed you were referring to the default 10 work factor of BCrypt and were calling it "rounds" as many of us are doing.

The obvious question that tedunangst is asking (and others in this thread) is whether a work factor of 10 is considered too low.

No, a work factor of 10 is usually fine. I generally use PBKDF2, which uses a parameter for actual rounds, and set that to about 20k, but don't think about rounds, just see how many authentications per second you need to be doing at the most, time your servers and use a parameter that gets you those authentications. 200ms is usually okay for most applications.

10 is obviously the log rounds number. It's not even a power of two! Nor has any implementation of bcrypt even supported such a low number.

How is it "obvious" when the unit is rounds? Ten was an example, round to 16 if you like. The point is still the same, using few rounds is a risk.

Because the set of values that make sense as linear round counts doesn't overlap with the set that makes sense as log base two work factors. Every implementation takes the log number; it's the only number people ever discuss.

And do they call it "rounds"? I've only heard it called work factor.

As a shorthand for work factor? Sure. It may be technically inaccurate, much like talking about centrifugal force, but you'll see "10 rounds" far more frequently than you'll see "1024 rounds". There's another thread on this post that refers to it as rounds as well.

The default cost for most libraries and languages is between 10 and 12, which is considered too low for 2015 but still pretty good. As long as they're at the default or above it, I wouldn't be too concerned about an attack against the whole DB.

Targeted cracking attempts against specific hashes are definitely still an issue though.

If I set bcrypt cost to 11, hashing takes 0.1 seconds. At 12, it takes 1 second roughly. Setting it to anything higher leaves my service open to Denial-of-Service attacks, so I'm very hesitant to increase the cost factor.
To you have a credible source for the "10..12 is too low for 2015" claim?

HHVM 3.6 on a small Ubuntu server

You have either a very slow server or a very bad bcrypt implementation. Running bcrypt in python on my 5 year old server has these results:

>>> timeit.timeit("bcrypt.hashpw('this is a password', bcrypt.gensalt(11))", setup="import bcrypt", number=5) / 5

0.13497538566589357

>>> timeit.timeit("bcrypt.hashpw('this is a password', bcrypt.gensalt(12))", setup="import bcrypt", number=5) / 5

0.28287739753723146

>>> timeit.timeit("bcrypt.hashpw('this is a password', bcrypt.gensalt(13))", setup="import bcrypt", number=5) / 5

0.5341608047485351

>>> timeit.timeit("bcrypt.hashpw('this is a password', bcrypt.gensalt(14))", setup="import bcrypt", number=5) / 5

1.069920015335083

>>> timeit.timeit("bcrypt.hashpw('this is a password', bcrypt.gensalt(15))", setup="import bcrypt", number=5) / 5

2.151028203964233

That's five repetitions of a bcrypt hash with the work factor passed in bcrypt.gensalt(). The resulting units are seconds.

You are right, my times are apparently somewhat dated. HHVM 3.6 actually gives me 1.88 seconds with costs of 15.

Good thing you made me re-measure :) That makes 13 my new bcrypt default.

Considered to low by...?

I think this just goes to show exactly why these systems will become more commonplace. There are only so many security experts to go around. Having all the very best concentrated on a smaller set of services seems like it makes more sense than trying to get a security expert for every service.

I agree, but can't deny self-hosted means your security lapses see less fanfare, which has value to the biggest and most lumberingest risk-adverse organizations

not to mention: you can host a chat server in your company network, somewhat protected from random people on the internet, and your ops people should already be securing that from external intrusions anyway.

I'm mixed on this. It's undeniably true if everyone is in the same building but in my experience this is rarely actually true as people need to work from home, use mobile devices, other offices open, partnerships or acquisitions happen, etc. That tends to lead to people requesting holes in firewalls or using VPNs as a sort of aftermarket spray-on security measure, which inevitably makes things much worse because now you have exposed services[1] which were setup and operated by people whose threat-model was “Sheltered internal service accessed by trusted staff”. It's much better to start with the assumption that your services are exposed to the internet and secure them accordingly.

1. VPNs are almost always terrible for security because people tend to get them for one need (i.e. email, one line of business app, etc.) but when their computer gets compromised the attacker now has access to the superset of everything which every VPN user has ever needed to access and in all likelihood a bunch of other things which were never intended to be exposed but were never adequately firewalled internally.

You have to look at it from a regulatory and compliance standpoint. While I agree from a technical standpoint that the average company's data is probably going to be safer at Slack than in some internal system, the accountability risk is just too high.

You can't prove your cloud provider is using security best practices, while you theoretically can prove (or disprove) the same internally. Few companies do proper auditing, reviews, and pentests, but they have the capability to do so.

> You can't prove your cloud provider is using security best practices

But you don't have to. They do. Look at Amazon. They publish their security audit each year, and now every company that uses them doesn't have to do their own audit. They can point their auditor at Amazon's report and say "see our datacenter passes".

Also, do you do a security review on your power company, or do you assume they've done it?

I have a friend who works security, previously for the government, and now for Visa. In his opinion: if you haven't been hacked, you just aren't an interesting enough target for the right people.

I don't know how common this line of thought is in security. But if it is common, then if you're a small company, aren't you better off not hosting my stuff at these large companies, because it's putting information you collect with someone who is more likely to be "interesting" to the right people?

I think a more correct assessment would be:

If you think you haven't been hacked, you probably have (or you are so small that you may have only been probed by bots).

If you haven't actually been hacked yet, it is only a matter of time. Ideally you start designing security layers now, before you are compromised.

As the famous quote goes: there's two types of companies - those who have been hacked and those who will be. Actually, there is a third kind: those that will back hacked again.

If you haven't been paying attention, the US Govt (including the DoD) is moving out in the commercial cloud space in a big way.

How does one discover that they were hacked? The post states that the breach occurred during February, and this is the end of March... did it just take them a long time to react and write a post about it, or did they likely discover after the fact? If so, how?

That's a really good question and one that probably has a different answer for every breach. In this case it's also probably a question that only Slack could answer for you. In regards to the second half of your question, being that they only recently went public about it, I suspect that they most likely did discover it after the fact.

At a place I worked at we discovered it in a couple ways. One was a routine scan done by our host provider, looking for malicious files meant to do things like create a web-shell. After they found the malicious files they had logs to determine the time frame of the attack(s).

And in another instance the hacker emailed us asking for ransom.

Logging every API request through every layer into ElasticSearch/Logstash or something similar for starters.

logs -- perhaps they hired a security firm to do periodic audits.

I recommend "The Cuckoo's Egg". It's a great book about traking down a hacker and explains one of the ways it was done.

Step 1) Discover a hole in your code, Step 2) go back to logs and see if anyone ever used that hole, Step 3) panic.

True story:

Log into server. Why is server slow? Run `top`. Hmm, `./exploit` is consuming 99% CPU...

./exploit ...? Seriously??? LOL

The wording of the article suggests that they released 2-factor auth -after- the breach happened. This is purely speculation, but one possibility is that they wanted to get their ducks in a row (i.e. have some enhanced security options in place) before announcing the breach. Mitigate the PR damage.

Usually, if the attacker doesn't dump your info or otherwise blatantly advertise themselves, the FBI tells you.

Host your own IRC if you care about the privacy and security of your communication.

There is no reason why you can't take 10min to setup a IRC with SSL on your own.

Yes, Slack is awesome, lots of features, but it's not yours!

Not sure why this is being downvoted. Setting up IRC on a .onion isn't even difficult. Setting it up with SSL is only a touch harder.

Setting up an ircd is not a bad idea. Then we just need to find the ircd and client that does what Slack does. (Which is totally possible, but I'm not sure it exists?)

Slack also has webhooks/bots that interact with other services for notifications... pingdom, github, etc... which can be done with bots, but it gets more complicated. Not to mention search archives, file upload preservation and other display niceties.

I don't personally use it, but this might help your https://hubot.github.com/

Careful, that has basically zero in the way of authorization/authentication. Putting business logic in there on a standard deploy is asking for trouble, especially if malicious accounts have access.

I use ejabberd as chat server and as interface to hubot. Solves both authentication and backlog issues. With a ssh tunnel (or your choice of vpn), it also solves out of network access. Jabber and ssh clients exist for Android (and iOS I presume)

I am interested first in persistent backlogs -- do you know an IRCd that does that? I am sure it couldn't be implemented client-side.

I remember "SILC" which is not IRC, but if I remember correctly it had this. Mainly when you join a room, you should not get tabula rasa. You enter into some context, and if the last message was posted 4 days ago, your context starts 4 days ago, and the datestamps all reflect this.

There are other features of Slack that make it worth using, but this is the one thing I don't know if IRC can support at all, that I see every other serious chat system doing.

See above, I use jabber for chat (supports backlog) with ejabberd and for other automation tasks I use hubot which integrates nicely with XMPP.

SSL does not protect against a variety of attacks, among them:

* breach via a compromised (guessed) password * breach via a security issue in the service and/or the host

Hosting your own service may make sense depending on your knowledge and/or threat model, but it's not a silver bullet that solves all issues. Hosting on a .onion address buys you exactly nothing in terms of security - and in the case of a company/team chat server probably not even in terms of privacy.

I know that sounds secure, but it's no more secure than just an irc server anywhere out here.

But the funny thing about all of this is there are already voice comms and chat comms clients out here in spades. The game industry created them. No need to go to IRC. Just use a more modern comms setup. I have a private mumble server myself.

It's presumably being downvoted because it's just lazy propaganda, not a serious answer.

It'd take more than 10 minutes simply to setup a server which ircd would run.

Then, of course, you'd need the time to setup all of the other services which Slack provides (i.e. durable file storage, integration with your applications and other services, etc.) and pick or develop clients for all of the platforms people use.

Then you have to actually support all of this: that's basic stuff like redundant servers and backups, account management, and the ostensible purpose for doing all of this: staffing your own security team at or better than Slack's level.

None of that can't be done and if you already have a good ops team it might even be worth doing. It's just way more than a 10 minute job even if you cut a bunch of corners.

Agree with you in one sense of being responsible for your own security, but by this logic I should keep all my money under the mattress instead of the bank, no?

Yes we do, keep millions for our customers too; Bitcoin company here. :)

Ah, in your case, only because your customers aren't smart enough to keep theirs under their own mattresses.

In our case customers don't need to trust us, they can generate their own keys offline.

Big fan of customers having a way out and not have to trust the service provider.

I know you're being facetious, but the answer is possibly yes, depending on your threat model.

If you have great physical security at your home and don't trust the banks (e.g. their fee schedules), it might be a safer decision.

For most people, it's not.

Or if you experimented an economic measure that froze your account like this one [1].

They froze all accounts and they even threatened the population with opening the safe deposit boxes.

[1]: http://en.wikipedia.org/wiki/Corralito

Not the same thing. Banks are insured against robbery and theft, so if something like that happens, customers don't lose their money. In addition, there's an insane amount of fraud protection in the banking industry, and billions of dollars of vested interests to make sure criminals are caught and prosecuted.

Can you say the same about cloud services?

Good point. Even in the worst case scenario, banks can fail and get bailed out by the public. But once information leaks, there's no "bail out" remedy possible... you can't really put the "information" genie back in the bottle.

> but by this logic I should keep all my money under the mattress instead of the bank, no?

More like "by this logic I should keep all my money under the mattress instead of in a jar in my cousin's pantry, no?"

Slack isn't a secure vault for data, it's a communications tool. They obviously have some amount of security but that's not their purpose, whereas a bank is intended to store money.

I think running your own IRC is a perfectly acceptable solution if you value your data.

It will take you more than 10 minutes to just choose the ircd to use. Not to mention configuration and maintenance.

I looked through the ircds available in Debian repositories the other day and they didn't look very fresh. So you might also have to package them from source and make sure that stays up to date.

Hosting your own services has some appealing security qualities (like being able to put them in your VPN) but it's far from a panacea and definitely harder than signing up for Slack/Gmail/GitHub.

Saved you the search; ngircd

I was looking at InspIRCd and UnrealIRCd which seem to be popular among smaller IRC networks. Any reason for that? Are they better suited for public networks?

The popularity of InspIRCd mostly comes from its modularity and out-of-the-box modules [1]. If you're looking for an ircd to replace Slack in terms of feature-parity, InspIRCd would probably give you the best start.

I used ircu until a few years ago, but these days it has mostly rot (maintenance, git/release tarballs code not matching, ...), and InspIRCd is pretty great if you need the customization.

[1] https://wiki.inspircd.org/2.0/Modules

Seconding ngircd

Hassle-free configuration

This goes for any sort of SaaSS:

https://www.gnu.org/philosophy/who-does-that-server-really-s...

I don't agree with "10m"; I've hosted my own IRC server for many years, and there is configuration involved, especially if you're focused on maintaining a solid, secure system. But I haven't had to touch it since.

> There is no reason why you can't take 10min to setup a IRC with SSL on your own

For the VAST majority of people this would not take just 10 min. Not only would I first need to research the different IRC servers out there but I'd have to get a server to install it on (which is not the fastest processes where I work). Then I need to get an SSL cert (which is like pulling teeth here) unless I want to use self-signed and listen to everyone bitch about dealing with that (and some aren't tech people and I've have to walk them through that). Then I need to find clients for everyone (Windows/Mac/Linux) also now chat is only accessible from inside the company unless I want to expose it publicly then I need to worry about security.....

OR I could pay $X/mo and have it up and running in seconds... Slack is not the end-all-be-all but I quite like it and use it with friends as well as public slacks. IRC is great but let's not pretend it takes seconds to set up everything you need...

And then you have you setup something like ZNC so people can catch up to history while offline.

Thanks for the ZNC ref.

This is a persistent pain point for me. I don't just need owned chat, I need chat that does multi-presence cleanly and supports push notifications to mobile devices. My team is distributed and works inconsistent hours and we rely on notifications to coordinate effort when needed.

I've run internal IRC and XMPP servers before, and they do one client admirably, but multiple presences for the same user rapidly becomes problematic.

With this logic, don't use any SaaS service then.

> security of your communication

Unless one is a security expert I highly doubt that a home-grown IRC system will be more secure than a professionally-run operation, other than simply being more obscure and not on a hacker's radar.

Surprisingly, they didn't force a password reset on all accounts. Even though the passwords are hashed and salted, targeting a couple users and checking for weak passwords can now be done offline, with no rate-limiting or network calls necessary. In breaches like these, it should still be mandatory to issue service wide password resets. Anything less is unacceptable.

I was curious about this too. I don't use Slack, do they enforce a password complexity and use a bcrypt work factor high enough to justify not requiring a reset?

There isn't a work factor high enough or password complexity high enough that it doesn't justify a reset, in my opinion. People will use bad passwords, and we can't really stop that. Allowing offline attacks against them will never be okay.

I'm definitely not an expert, but my understanding is that a per-user salt, which Slack was using, protects against brute-forcing bad passwords. So assuming a reasonable work factor, it really should be computationally unfeasible to retrieve these passwords. That said, I'll change mine, because why not.

A per-user salt doesn't prevent brute forcing bad passwords. The purpose of a per-user salt is to prevent brute forcing all of the passwords simultaneously. Without it, you could just hash a password once and check against all the passwords. With the salt, you have to hash each guess for each user. It certainly increases the work, but with bad passwords, its likely you'll always crack a few in a large dump.

If you have a strong password, a high work factor should prevent brute-forcing it. But really, if it's computationally infeasible to crack a bad password, its also probably computationally infeasible to log in.

In cases like these, even if the risk of password recovery is minimal, a reset should still be forced because the attacks are offline and only become easier as time goes on. Forcing a rotation discourages people from continuing to work on cracking passwords for the next n years until they finally get an interesting account.

Of course, thanks. Guess I wrote that without actually thinking about it. Now, a pepper would prevent brute forcing weak passwords, assuming it wasn't compromised, but that's not relevant here. And regardless I agree with your logic regarding forcing a reset.

No, using bcrypt and per-user salts, forces an attacker to actually do brute-forcing -- not being able to use rainbow tables, or easily test one password against all users. So if they have 2M users, and you can check ~20 passwords per second (what I can do on my workstation with a bcrypt work-factor of 10, single-threaded), it'd take 2M*11/20 seconds ~ 12 days -- to try "username[0-9]?" (e12e, e12e1, e12e2 ... e12e0) against all accounts. And to try any given password, like "Password2015" against all accounts, it'd take 2M/20 seconds ~ 27 hours per guess.

Note that the actual times here are off by a magnitude or two; first they're single-threaded, secondly I happen to have an AMD r9 290 -- and oclhashcat reports 197 hashes/second -- which cuts the time to ~3 hours per guess).

If I were Slack, I would pretend to get hacked. Slack critics often point to its centralized architecture as a weak point, because rational corporations should not entrust security of their internal communications to a third party. Particuarly when that third party aggregates communications of its many clients, it becomes a target of hacking. Why hack a single corporation when you can hack Slack and get all their clients at once?

This is a valid criticism. Slack can do all it can to mitigate security risk. But at the end of the day, there is always at least one vulnerability, somewhere.

As Slack matures as a company, it needs an answer to this criticism. Because security is so naturally unpredictable, it would be disingenuous for Slack to respond with anything resembling "our security is perfect." Because, of course, as we see time and time again, no security is perfect.

Now that Slack has captured the low-hanging-fruit of the market, it needs to pick the high-hanging-fruit. The most profitable clients for slack will be the largest, conservative, enterprise clients who will join the Slack platform and then never leave. The long term survivability prospects of Slack depend on capturing these large enterprise customers.

Strategically, Slack needs to find a response to the criticism that its security is prohibitively weak, so that it can convince these large enterprises to join its platform.

Perhaps, the best response to security criticism is that "we got hacked, but our internal policies mitigated any cascading effects and customer data remains safe." [0] [1] So would it be in Slack's best interest to stage a hack on itself? Or to report a hack occurred when it really didn't?

It seems feasible that by setting precedent for its reaction to a hack, Slack has a chance to demonstrate the competence of its security team. Now investors can point to this incident as one handled well by the security team. In a world where, unfortunately, corporations will always get hacked, Slack was able to survive with some dignity.

[0] or, as safe as it can possibly be according to computer science.

[1] debatable.

Note: My comments and thoughts are geared specifically to Slack or a very similar entity (tech start up, which sells a service).

I strongly disagree with this idea:

"Now that Slack has captured the low-hanging-fruit of the market, it needs to pick the high-hanging-fruit. The most profitable clients for slack will be the largest, conservative, enterprise clients who will join the Slack platform and then never leave. The long term survivability prospects of Slack depend on capturing these large enterprise customers."

I think this is a large misconception held by people. Often times, and I say this from experience, enterprise customers will demand a large amount of support/coddling because of the gross amount they're paying.

That "$1 million" a year contract sounds less and less good when you realize it requires constant attention of two, now dedicated, engineers, two support reps, sales person, and on occasion an executive... Compare that to 1,000 individual customers/subscribers who require next to no special attention, or dedicated engineers. Large, conservative, enterprise customers distort profit margins, and should be avoided.

Think about this: some enterprise customers are so large they may hire a person (or persons) to _only_ deal with your product. Now you've got a team of cogs (people) at your enterprise client, working, to take up your time and cost you money.

What's worse is that in most corporate cultures, partners/suppliers/providers are not looked at as "family" but as disposable. There's no vested interest in your success or failure. So there is little to no concern over how your smaller company may be abused by a larger one.

This is why humans invented contracts. No two businesses enter into a $1 million deal together without a contract to describe the expectations of the partnership. For this reason, no service provider should be "surprised" by the level of commitment required to fulfill its end of an enterprise deal, because all the expectations should be in the original contract (literally, the service level agreement). There are no surprises.

Also, (pedantic) in your example, $1m a year is beyond sufficient to cover the costs of two engineers and some support staff. Besides, realisically the biggest support issues will be problems with availability, which presumably will be network wide and not limited to one client. I highly doubt slack needs to hire dedicated engineers for each new enterprise client. Instead they can reallocate existing engineers when needed, and grow their total labor capacity as it becomes constrained.

So a database gets hacked, they add MFA and people are arguing about peppering passwords. What about the part on how the hackers got access to the database in the first place?

Passwords are not the only sensitive info that can be stored in a database and most of the time, that info isn't hashed.

I would be more interested in how the hacker got access to their DB and nothing else. Maybe the DB is remotely accessible (unlikely) or there is SQLi vuln. in Slack.

One thing that bugged me about this today was that after I changed my password on desktop, my mobile session wasn't invalidated. Apparently it's an option for mass password resets, but it really should be mandatory.

My team logs with our Google accounts. It's not addressed in the disclosure, but should we be deauthorising Slack. Have our tokens been breached?

Possibly no because the token is app-specific and it's not likely that the slack key was compromised, but I also agree that they should have disclosed that.

> Slack’s hashing function is bcrypt with a randomly generated salt per-password which makes it computationally infeasible that your password could be recreated from the hashed form.

I'm happy to hear they didn't just use MD5 with no salt as this would be the same as storing it in plane text...

bcrypt + random salt sounds to me like the best practice nowadays, is it still holding? or are there some advanced in GPU cluster costs on EC2 that make even bcrypt hackable. I think I heard something that it has a way to "adapt" to advances in computing, is that by simply adding more iterations based on the current CPU speed or something? how does that work?

Well, bcrypt + random salt is great but it still can't protect you from bad passwords. It's vulnerable to a dictionary or bruteforce attack the same way everything else is.

You _can_ increase the difficulty of bcrypt, which essentially increases the iterations (as you mentioned). Most bcrypt libraries default this value, but the programmer can override it. The downside is that this requires you to update all of your hashes, which limits how often this is done.

Note: if your hashing algorithm is too compute intensive, you have to take other measures to prevent your login system from being a DDOS vector... for example, with SCrypt's recommended defaults for passwords, the .Net library takes almost half a second of time on a modern CPU... if you get more than a few dozen requests per second, per system you can be brought to a crawl without other mitigation in place.

What makes it slow? Is it implemented in C#?

I would avoid slow implementations of password hashing algorithms. You want the overall operation to be slow due to the computations you're performing, but you want the implementations of those operations to be fast. Because the attacker's implementations of those operations will be fast.

I didn't compare to the C/C++ implementation, it was harder to get that working on windows... I did compare to a JS implementation[0] running in node, which was about half as fast though... mainly because at the time many node modules requiring a build step didn't run well in windows, and my target environment was windows.

[0]: https://www.npmjs.com/package/js-scrypt

Yes, you can set the work factor. It seems like the default is around 10 at the moment.

http://wildlyinaccurate.com/bcrypt-choosing-a-work-factor/

There are some thoughts on the matter here: http://chargen.matasano.com/chargen/2015/3/26/enough-with-th....

But I would say yes, bcrypt is still best practice. Other commenters are right that bad passwords will still be recoverable, but using one of bcrypt/scrypt/PBKDF2 is due diligence. I would use whichever one is most easily available on your platform.

Why do articles talking about this always talk about specific work factors and choosing a correct work factor, as if it's fixed? I thought good practice was to choose the work factor dynamically so it's calibrated to whatever hardware you're running today.

Uhh.. it doesn't? While it does give some specific values as 'a reasonable starting point' the article suggests tuning these to the specific environment.

"Each of these algorithms requires setting of an appropriate work factor. This should be tuned to make digest computation as difficult as possible while still performing responsively enough for the application. For concurrent user logins, you may need <0.5ms response times on whatever hardware is performing the validation. For something like single user disk or file encryption, a few seconds might be fine. As a result, the specific work factor values appropriate for your application are dependent on the hardware running your application and the use case of the application itself. Thomas Pornin gave some great guidance on determining these values in a post to stackexchange"

The stackexchange post (which is linked in the article) can be found at http://security.stackexchange.com/questions/3959/recommended...

Yeah, what I'm getting at is that those posts don't actually come right out and say your code should be calibrating itself regularly. They talk about selecting a work factor by benchmarking your current hardware, but they leave it there, and one might come to the conclusion that once they've measured their hardware, they put "13" in a config file and call it done. I'm advocating for advice like "Don't think about work factors, think about time. Your configuration should be a time value, and when your app starts up, it should compute its own work factor based on this time value." If your code does this, there's no need for rules of thumb like "11 is a good starting point for a bcrypt work factor."

Ohh, okay. I see what you're getting at now.

Yah, that does seem like a reasonable design. The implementation wrinkles there are that the hardness params have to be encoded in the digest itself, or otherwise stored alongside it. Since implementations commonly do this anyhow, that doesn't seem likely to pose much problem in practice (if any). The other issues would be details around when to calculate the hardness. On app initialization seems obvious, but you'd have to sample over some period of time to get a representative benchmark. I worry that this could exceed administrator tolerance for how long an app can reasonably take to start up, but this doesn't seem like a show stopper either.

Ultimately though, I'm not sure the factors are THAT variable. I mean, you want to reconsider work factors as hardware advances, but I don't think the line is so solid that running a work factor of N versus N+1 will make that much of a practical difference in the span of a few months or even a few years. Still, with the goal of making it as hard as feasible while still being suitably performant in the context of a given system it makes sense.

This doesnt make any sense: the goal of this work factor is to make things slower for the opponent, not my own servers. My estimation of the computational power available to my opponent has nothing to do with how large the web servers I chose to use happen to be.

The recommendation is to make it as slow as you can tolerate, which is based on how fast your web servers are.

  bcrypt + random salt

bcrypt incorporates a random salt by definition, so it's redundant to add " + random salt".

http://en.wikipedia.org/wiki/Bcrypt

Was an admin account compromised in a situation where 2FA could have prevented the unauthorized access? If that's not what happened, then 2FA seems a bit hand wavy if it's not directly related to this security incident.

I actually am coming to the conclusion that 2FA is dangerous. It's recommended any time there's any kind of breach as if it were some kind of panacea for security, when in most cases the cause of a security breach is a database intrusion.

2FA keys are stored unencrypted so if the db is breached they're already cleartext (no boil the oceans bcrypt collision needed) and the extra layer of security theatre from 2FA only encourages bad practices like password reuse & easy passwords.

2FA can prevent future password phishing attempts when your users table is compromised.

It's a bit of a redirect in terms of what happened with Slack though and not worth mentioning after the fact.

Agreed: 2FA protects against password leaks on other sites from being applied to the system in question. In this case it would have been more reassuring to hear them disclose what was the attack vector (e.g. social engineering, email trojans, etc) and explain how they will reduce the chance of a similar compromise in the future.

I wonder how many people send sensitive credentials or other operational details through Slack. It'd definitely be a target (along with mail systems) if you want to attack better-protected customer systems.

I think that's the point of slack, being able to communicate sensitive information. Where would you relay something like an Amazon AWS Access Key?

We always use onetimesecret.com even over Slack

PGP encrypted email.

We've been waiting for over a year for Slack to create a self-hosted version that we can deploy to our intranet specifically because we can't expose ourself to things like this. They've kept insisting that it's around the corner but it doesn't seem to be happening. Hopefully this will spur them to prioritize self-hosted Slack.

Literally was arguing with someone like two days ago that using Slack for sensitive data was a bad idea, guaranteed to blow up in your face sooner or later.

Nothing sweeter than "I told you so".

We haven’t seen anything blow up yet other than Slack itself. While it sucks that the user table was compromised, I think the actual affect on businesses is more benign than they’d like to believe (ie: sensitive chatlogs that could be used for blackmailing).

given the use of bcrypt for password salt/hashing, I'd say any attacks from here may be targetted to specific users, or those with really weak passwords (top 10k password list) run through wouldn't take too long on a distributed cluster per user. How much that opens up, and how that corresponds or overlaps with slacks password requirements will vary.

Last year someone found a Slack security hole and was able to see all the companies using it and the room names, some of them pertaining to products under development.

People are suckers for still using this company's products.

Slack encourages 2-factor authentication:

> Download and install either the Google Authenticator or Duo Mobile apps on your phone or tablet.

Hey Slack, I don't have a smartphone. What am I supposed to do?

Slack is from the startup culture. Do not expect any support for people who are not "up with the times" and crud.

If security is that important to you, buy one. A cheap Android tablet isn't exactly going to break the bank, is it?

There are plenty of security reasons not to buy a smartphone.

Get an OTP app for your laptop. It's an open standard.

These usually fall back to regular text messages.

https://www.authy.com/

https://chrome.google.com/webstore/detail/authenticator/bhgh...

https://code.google.com/p/winauth/

and others, I'm sure

Buy some small and cheap one, no SIM card needed.

While I appreciate all the downvotes and "get with the times" comments, a significant portion of the population does not have a smartphone, assuming everyone has a smartphone or will instantly know what to do when presented with official instructions that only mention smartphones/tablets seems like a bit of a security oversight on Slack's part, no?

Those pointing out PC-enabled authentication apps: thanks. That's USEFUL feedback!

The vast majority of the population that comprises Slack customers will have a smartphone.

And only focusing on the "vast majority" is a GREAT security solution. There is some significant, non-zero number of people who use Slack and do not own a smart device. I am one of them. (reasons for choosing not to own a smart device are many and I will not go into them here)

I'm not trying to rake them over the coals, just point out this is a very real blind-spot and they should promptly update the notice with instructions for people who don't have smart-phones.

Can you expand just enough on the reasons for not owning a smartphone, so we can give constructive advice? I don't want to suggest things like "buy a Yubikey" or "buy a tablet or iPod Touch with no cell radio" if you're objecting to buying new things, for instance.

I think it's hard to give generic advice about this situation.

Smartphone - 58% of all US adults

source - http://www.pewinternet.org/data-trend/mobile/cell-phone-and-...

It's totally fine to use web based otp generator. Don't listen to others, you don't need a smartphone.

Lots of options.

* You can buy a hardware token. https://www.duosecurity.com/product/methods/hardware-tokens

* You can have Duo call/text you every time you want to log in.

* You can use some other device you have that runs a mobile OS. I had Duo set up on my wifi-only iPad while I was using a feature phone for a few months a year ago. (I eventually gave up on that and got a smartphone, though.)

* You can buy a used/cheap smartphone, iPod Touch, or similar, install either the Google Authenticator or Duo app, and not use mobile service at all. You just need a bit of wifi to enroll.

* You can write an OATH client as a J2ME MIDlet. When I was using a feature phone, I spent some time figuring out how to compile J2ME apps in 2015, by piecing together ten-year-old tutorials. It works fine; if you want me to dig up details let me know. (Alternatively, someone may have written one already, but I didn't search very hard.)

* You can, technically, run an OATH client on your computer. But at that point, you take responsibility for your two-factor auth being slightly less than two-factor.

To you last point... it's funny the number of companies that are using virtual 2fa clients on their laptops for VPN connections because they wanted to save money on hardware tokens. kind of negates the second half of 2fa.

It doesn't. The only case when it's worse is when your laptop is stolen and you don't know about that - very unlikely for corporate laptops

Or, monitoring software on your computer, the origin of which now has access to the computer, your password, and the token generator.

No, if your computer is infected (=monitoring software) there's no difference because token is transaction-agnostic. 2fa won't help

If you have a hardware token, not on the computer.. and use that as part of your VPN, if your computer is compromised, your account won't be able to be used to reconnect to VPN while you are afk without that hardware token... if the software/key are on the machine, they have your password, and the generator for 2fa... they don't even need your machine anymore... that is definitely less secure.

I have a Nokia 1020 Windows Phone. There is a Duo app for it, but it's single account. Duo hasn't updated their WP app since 2012.

I also have a Windows Phone. There's an official Microsoft 2FA app that I use instead of Google Authenticator or Duo, including now for multiple Slack accounts. http://www.windowsphone.com/en-us/store/app/authenticator/e7...

Slack was nice enough to point this out too. Works perfectly with the QRCode.

Get an abacus!

Buy a smartphone. You can get a Huawei on eBay for $20. You don't even need a sim or a contract, just connect to wifi and download the Authenticator app.

You can use http://gauth.apps.gbraad.nl/ on another computer or on same machine (wouldn't protect if computer is stolen, otherwise the same)

Can we go back to IRC now, please! Slack is not only distracting, proprietary, but it is also pretty expensive. Let the mere mortals use it, but we should stay away!

I don't think IRC quite offers the feature set that Slack users would be expecting, does it?

I'm not talking about users who care about custom Emojis and animated gifs. That's why I said - let the mere mortals use it; we can do better!

Is this actually the sort of people that get attracted to HN? Thinking of others as "mere mortals" and yourselves as gods?

Sad picture.

I was joking. Sometimes emojis help as we're missing a whole communication channel in here (body language).

Probably not in that condescending way, but it is useful to keep in mind that people are fundamentally different from you. The trouble is when you think they are inferior beings (their tech skills on the other hand are inferior, no reason to not state that plainly).

For what it is worth I have coded Erlang at work and am quite partial to cat pictures, myself.

Well, let's talk about the reality now. By definition, most people are mediocre, not very smart, not very healthy, not very pretty, etc. Most! That's whom Slack targets - most.

That cow left the barn a while ago.

Oh, yeah? You haven't heard of Let's Chat [0], for example?

[0] http://sdelements.github.io/lets-chat/

P.S. To clarify, Let's Chat is not IRC-based, but it's slackish and supports XMPP.

That's awesome. Would love to do a container for it.

Out of interest, where where the per-user salts stored I wonder? Where would people normally store this if not next to the hashed password in the same table?

I'll answer my own question, given the attacker wouldn't know how the salt was used unless they had access to the code, it doesn't matter.

Salts aren't meant to be secret; they are generally stored next to the hashes (and used in a known, standard way). The purpose of the salt is to raise the cost of brute-force attacks, by forcing attackers to brute-force each password separately.

https://en.wikipedia.org/wiki/Rainbow_table

Yeah it's Usually the same table, but salt is used to prevent hackers using already generated hash map of popular passwords to diff with your password hash directly without computing.

I am slightly confused. How can they have use their "hash map of popular passwords" to diff against your table if they do not have the "pepper" used originally for bcrypting the passwords?

Pre-computed mappings of popular passwords (rainbow tables) aren't really used nowadays due to parallelization being more cost effective, but the idea in both cases is that you want collisions with known values: the popular password hashes in the case of the rainbow table, or the computed values in parallel enumeration.

If the password hashes each have a stored unique salt (bcrypt will), you have to compute the hash per salt per password that you test. Instead of computing the hash for "password" (+ stored work factor for bcrypt) to be X, and checking all database entries for X, they instead have to calculate each hash per entry.

For a table of 1000 users, it would take around 1000 times as much work to determine the users with "password" as their password. If you just wanted to target a single user though, the salt doesn't really matter for enumeration (though if you were using rainbow tables, you likely wouldn't have those specific hashes computed).

Lot's of hype (IMO) around Slack, but lot's of money thrown at them so I kept thinking that I'm missing something! Just being skeptical as usual. The other day an invitation arrives to use Slack. Great! Let's see it, this killer feature or killer combination of features. What have these smart people come up with that hasn't been done countless times in the same space to make them so successful?

It's literally nothing. I can't believe that's the product.

Anyway, on top of a completely underwhelming experience comes this news. I can't see why a company would use them, to be honest. But then I haven't built a billion dollar company, so not many people will be asking me for an opinion.

You could do something similar with an IRC server and bots with webhooks... Just the same, they did do this, and offer a web based interface that people are more comfortable with (no need for an irc client), and does more than IRC clients do.

IT's not that anyone else couldn't do this, it's that they've done it relatively well.

Absolutely. It's just so much better than any IRC client. People looked at the iPhone and said "but all of this technology already existed!" That isn't the point. It had never been put together in a user-friendly way.

Slack hasn't invented anything new, but the entire product is a delight to use.

I think a key differentiation with Slack is their design, and that they provide clients for both mobile, web and desktop. I can't think of another product that has done this well.

I would say that their ux is actually really good. I'd go so far as to say they have one of the best on boarding experiences I've ever seen. But that's just my opinion.

It fills a necessary gap on large and remote teams. Email is too cumbersome, verbal communication isn't always practical or possible, nothing does group chats very well.

And it's not hipchat (or bought by Atlassian). Let's not forget that.

Can I ask why that would be a problem? You can actually host HipChat yourself (behind your own firewall), which seems like a great asset against this kind of breach.

https://en.wikipedia.org/wiki/Egg_of_Columbus

Neat story, but I don't see the analogy. OP says he/she doesn't see any innovation, not "this innovation is obvious". At least tell us an egg or two that Slack cleverly stands on end.

Now it occurs to me I may have been making a distinction (about the nature of innovation) where there arguably isn't one. Still, I think it's fair to ask exactly what Slack is doing right. If there is a real innovation, someone should be able to tell us (there've been some good responses); otherwise it's just lucky network effects or a fad.

If you use Slack for a week you can't go back. I would switch back to an IRC client for a few team members who would not switch and was filled with rage every time I had to use the old client. :)

It's the same criticism people had for Dropbox or the first iPhone: there's nothing new. The key is that it's very well packaged with a focus on UX (especially on mobile) and it requires extremely little setup effort.

"As part of our investigation we detected suspicious activity affecting a very small number of Slack accounts. We have notified the individual users and team owners who we believe were impacted and are sharing details with their security teams."

Assuming the password hashes can't be reasonably reversed, what would have caused suspicious activity on some user accounts? Is this a situation where certain users may have been targeted specifically, meaning that only a couple hashes needed to be reversed, making the task feasible?

Dear All - Your passwords should be considered compromised. Hashing is merely a deterrent, it does not prevent cracking.

6500 bcrypt(5) hashes/second with custom FPGA:

http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=703252...

I didn't say it would be easy.

But dictionary and password list-based attacks are expected to be quite effective anyways.

http://www.openwall.com/presentations/Passwords13-Energy-Eff...

That's exactly what I thought the moment I read the post. The fact that Slack used bcrypt is good, and if they used a high number of iterations that's better, but neither will help you if your password happens to be on the crackers' list of 10,000 most common passwords.

So they didn't get the password, ok. But they still have my name, my email, my skype.

This completely sucks.

Yeah, when they call, can you tell them I have a whole flock of ducks that need cleaning? :P

Hope the Slack team will properly implement SSO two-factor authentication policy - right now none of the Slack apps request re-authentication after thirty days once an account is signed into via SSO. Sign in once on OS X, Windows, iOS, Android and no further authentication needed. Looking at Google Apps SSO specifically but am assuming it affects other authentication providers.

Might as well plug my tiny Slack auto-install/update script for the OS X app while I'm here - hope someone else finds it helpful:

https://github.com/opragel/install_latest_slack_osx_app

Can anyone from Slack elaborate as to where this hack originated if known?

Slack has now sent me 6 e-mails about this, to the same address :(

They have different names associated in each one (i.e. some have my last name, some have an alias, etc), but all to the same target e-mail address.

See my other comment on the same point in this thread.

Also see that mine predates yours by an hour :)

Can we please reference to this event as the "Shlacking"?

Interesting that the title says "March 2015 Security Incident" but it turns out it was in February. Also interesting they don't say which days in February.

Why is everyone worried about what hashing algorithm was used and not if their company's private chat logs are about to become public knowledge?

Because obtaining a hash of a password does not mean the attacker can access the accounts and the chat logs.

If the passwords were not hashed or hashed in a weak manner then that means the attacker could figure out the account passwords, then gain access to the accounts, which would be bad.

I completely understand the risks associated with user passwords and hashing algorithms. But you know what? That shit gets leaked all the time. It's not uncommon and unfortunately, users are accustomed to it by now. It's relatively easy to fix.

You know what's a big deal? Having your company's internal private communications suddenly public or sold to the highest bidder. That's the sort of thing that doesn't have a "reset password" simple fix.

If you have a reset_token you still can log in, even when passwords are hashed. Likely reset tokens were stored in the same table

> Slack’s hashing function is bcrypt with a randomly generated salt per-password which makes it computationally infeasible that your password could be recreated from the hashed form.

Is this true even when the attacker is specifically focusing on a single account, or is it only computationally infeasible to recover passwords for accounts in general?

Coincidentally, I was just looking into your question; this should answer your concern.

"Since every user has their own unique random salt, two users who happen to have the same password get different salted hashes. [If] the dictionary attack is foiled, the attacker cannot compute the hashes of every word in a dictionary once and then check every hash in the table for matches anymore. Rather, the attacker is going to have to re-hash the entire dictionary anew for every salt. A determined attacker who has compromised the server will have to mount an entire new dictionary attack against every user's salted hash, rather than being able to quickly scan the list for known hashes." [0]

[0]: http://www.developerfusion.com/article/4679/you-want-salt-wi...

Yeah, salts mean you can't use rainbow tables. But you can still attack a single user. The question is what "computationally infeasible" actually means. How much computing power would it take to crack a single user's password? How about if it's a weak one? A strong one?

If the answer is "it would take $1000 worth of Amazon EC2 computing to crack a single weak password", well, that's certainly feasible to do if you have a specific target in mind.

And given th usernames and emails are out there, it reveals (a) all the companies using slack, (b) potentially very high value targets. I wonder who they had specifically emailed. My guess is those people aren't using slack anymore.

The answer depends on the tuning parameters used for the bcrypt hashing (cost/work factor) and the length of the user password. Obviously a 1-character password will fall even with a high workfactor and a dictionary word will probably fall as well. This paper http://www.emsec.rub.de/media/crypto/veroeffentlichungen/201... tags some numbers on breaking passwords with a low cost factor. Assuming a cost factor if 5 (12 would be more "real" world) and an 8 character password with an alphabet of 62 chars (uppercase, lowecase, 10 digits) the estimated cost to break a password within a month is in the millions with dedicated hardware designed for brcypt breaking (Fig 5). With a work factor of 12, an 8 character password will not break on EC2.

I knew bcrypt was tunable, and I guess I was hoping someone from Slack would actually pop up and say how their bcrypt was tuned (which I guess means what the cost factor is). But you do have some good info, I wasn't aware of what expected cost factors were and how secure a single password would be at those cost factors.

I'm assuming that your "8 character password" is assuming a randomly-generated password. I'm curious what the expected cost would be of breaking a particularly weak one (e.g. a human-generated password, based on dictionary words although perhaps with mnemonic devices or letter/number substitutions). The paper you linked says their hardware computed 6511 passwords per second at a cost factor of 5, and based on the cycle costs listed, I'm thinking it does 52 passwords per second at a cost factor of 12. Assuming a particularly weak password, I don't know how many passwords a brute-forcer would expect to have to try before hitting the correct one.

running the top 10k passwords on each hash will likely get quite a few hits, and not take much compute time... the overlap to accounts that are owner/admin accounts is unknown.. just the same entirely possible. (not counting for slack's password complexity requirements)

Going by the paper linked in Xylakant's comment (https://news.ycombinator.com/item?id=9277780), if the cost factor was 12, using dedicated hardware would let you test 52 passwords per second. So that would take 192 seconds to try 10k passwords against a single hash. If you were to run this for a month straight you'd have tested 13675 accounts. Slack has over half a million daily active users (and I'm not sure if that stat actually means daily active accounts, or daily active people (who may have multiple accounts on different teams)).

The paper goes into estimated costs as well, but I'm not going to dig through it to figure out how much it would actually cost to run that hardware for a month straight. And of course it's talking about dedicated hardware, which Slack's hacker almost certainly doesn't have.

It's feasible if you use a commonly used password or if the attacker knows specific information about you or your password. Otherwise, it is pretty much impossible.

// this is about focusing on a single account. Nobody would ever bother with trying to crack the whole DB.

Not only that, but along with the email you may have higher value targets that you will spend more time checking or trying to brute force.

Even if it's a gmail user, if you get them, and they use the same password on their mail, poof, you're in.

What gets me is that banks of all places have the worst password abilities (for their web logins)... case insensitive, only letters and numbers.

The salt helps to defeat anyone who has pre-computed bcrypt tables for large numbers of passwords. It forces the attacker to have to compute all the hashes themselves, even for a single user.

quick answer - It's probably computationally infeasable to gain access to any specific account, or all accounts in the database. It's probably feasible to gain access to at least one account in the database, if it has a weak password.

Hashing functions like bcrypt are intentionally set up to be computationally expensive, so that when brute-forcing a password, attackers can only try, say, 1 thousand passwords per second, not 10 million, on any given computational unit. The exact numbers of attempts-per-second depends on the attacker's computer, and the exact setup of their password hashing.

When trying to find the password for a particular hash, which is a hash of a strong password, there's really no better way than trying all possible passwords, hashing them, and seeing if they match.

When attacking a whole database at once, sometimes you get lucky and some accounts have weak passwords, and those you crack quickly.

Yes, depending on the work factor chosen. See http://wildlyinaccurate.com/bcrypt-choosing-a-work-factor/#b... for a table of work factors and associated time to check a single password. The work factor is a trade-off involving burning CPU cycles, load on the servers processing logins, and user experience. If you choose a work factor resulting in each password verification taking half a second, it's essentially computationally infeasible to brute force any decently chosen password.

Someone could go through the list of the top million passwords in a week of CPU time and, because each hash has its own randomly generated salt, the effort spent cracking one user's password yields no insight into anyone else's password; the entire effort needs to be repeated per user. If your password has sufficient entropy (eg: long and randomly generated, not appearing in any password lists) then -- barring some revolutionary mathematical breakthrough -- you don't need to worry about someone discovering your password before the heat death of the universe.

It would be good to know exactly how this sort of thing happens so others can try to prevent it in their own systems.

Well, I can't even reset my password. I suspect that's because my registered e-mail address uses the "foo+bar@gmail.com" format (which I use for easier filtering), and something on Slack doesn't like sending those (no e-mail on my inbox, spam folder or filter matches)

Looks like they require Google Authenticator or Duo Mobile app to do two-factor auth. I'm not interested. Why can't they be like Github and just send me a text message? I don't want a dependency on some other company's product to make Slack more secure.

Those are only two examples of apps that support TOTP[0]. TOTP is a standard with many implementing apps, including free and open source that will accomplish the same thing. The two apps listed on the website are just well known examples.

[0] http://en.wikipedia.org/wiki/Time-based_One-time_Password_Al...

I hate to break it to you but SMS is some other company's product. Google Authenticator is not a cloud service; it's simply an implementation of the TOTP RFC and there are others available if you hate Google so much that you don't trust a purely local app written by them.

An open source one at that!

https://github.com/google/google-authenticator/

SMS is far more insecure than Google Authenticator. Also, you can use other apps like authy if you don't want to use Google Authenticator for whatever reason.

Google Authenticator relies on an open algorithm for one time passwords. It has been implemented by many apps, and it's so simple you can do it yourself. So it doesn't require you to rely on other company's product.

You can use any other TOTP-based application instead of Google Authenticator. Plus, unlike SMS, it works offline.

Google Authenticator is a free-standing app with no dependency on you having a Google account whatsoever. It doesn't even have Google account integration as an optional feature. It also uses algorithms documented in public RFCs, and there are about a thousand compatible, alternative, implementations because it's so damn trivial. Stop whinging.

SMS is the worst kind of 2fa, better not use it at all.

> I don't want a dependency on some other company's product to make Slack more secure.

How is dependency on SMS not dependency on some other company's product?

You can use other apps such as Authy too.

Wish Clef would integrate with Google Authenticator, https://getclef.com/

Feels like a much nicer way to provide the "something I have" part of authentication.

Remember kids, it's 2FA before the fact, not after. 2FA is not a magic bullet though, and neither is salting. Salting makes it _expensive_, but not impossible for pass recovery. Always aim for impossible. You want to be able to throw away the key during an incident.

The only impossible system I know of would be one that uses client side certificates, but I would be surprised if anybody those in a successful business.

US Government systems typically use client certificates as a component of access control in the form of the Common Access Card[1]. Not that that's a business, or even a success from the point of view of the users, but it's one of the largest deployments of client side certs out there.

[1]: http://en.m.wikipedia.org/wiki/Common_Access_Card

Unfortunately the only reason they can do that is that they are the government.

For those that store the user tables in the same db as everything else, what is the big deal about protecting passwords in particular? The attacker already has much more. If it is because of password re-use then that is what should be prevented.

Got the email from them about the issue, as a Slack user. Also got 2 other duplicate emails from them (which went into the same Gmail conversation, since same subject), that were empty.

Seen this a few times for other services, not sure why it happens.

They probably send a message to you for every team you're on.

Could be, thanks.

This is why you need to hash reset tokens too

http://sakurity.com/blog/2015/03/27/slack_or_reset_token_has...

In devise at least reset tokens expire, so they'd need to have been set in the last day to be useful, which narrows that attack considerably doesn't it?

Why wait and not use it right away? If you have read access now you can exploit now.

Oh I see - you mean they have read access, then trigger password reset, then use the token straight away? That does mean they'd be firing off emails which would alert users though.

It would. They didn't do it probably because they didn't try this trick. But they could i think.

BTW actually new version of devise hashes tokens.

ppl gotta stop thinking encryption in a db saves you from compromise. Its being very naive or ignorant. It only reduces your exposure to data leak very lightly - in some circumstances, which are generally not even likely (like make a dump and post it publicly)

Its like 'but it says AES on the box so its secure right?' and shouldn't be a thing among developers anymore.

Obviously the database data has to be decrypted for the app to use it, and generally, you hack the app, not the db.

What tangible things were hacked? apart from password was the communication history, files share etc were hacked too? We sometimes share code blocks and zip files etc..

If someone was able to get access to the user table I would believe it is trivial for them to download the chatlogs/ mine other information from database.

May be slack wants us to believe only a small part of the data is hacked , I dont know .

We have been using Slack for many projects over last year and it helps improve productivity

Slack is growing their security team, if you want to help them improve the security of the product: https://slack.com/jobs/dfd75111/security-engineer

"So yeah, we got hacked..." - Landing Page

How many rounds of hashing did they use?

PHP default is 10 if that helps

Wait a second, you got hacked 27 DAYS ago (at least), you know they got data, and you are NOW telling people?

Dong, Dong, Dong, that is the death bell of a startup

It could be a bad idea to let intruders know you're on to them if you haven't determined the extent of the intrusion.

Sadly the universe isn't fair.

Nobody gives a shit about security, but yeah it should be illegal to sit on the information more than 48 hours. If you can't determine the complete and full extend of the damage before that you need to be out of business.

They added team-wide password/security stuff.... but it's paid users only. Lame.

Agreed. Proper security shouldn't be an optional, paid "feature".

I feel like it should be mandatory for all tiers of a product, whether they're free or not.

Only owners can see it, not admins apparently.

https://twitter.com/SlackHQ/status/581544119677374464

Why the fuck am I reading this on hacker news and not from a notification in slack?

> If you have not been explicitly informed by us in a separate communication that we detected suspicious activity involving your Slack account, we are very confident that there was no unauthorized access to any of your team data (such as messages or files)

They sent you an email.

uc3dp

is there anything that slack does you can't do with skype?

I find lot of these new startups are just creative ways of reinventing the wheel and convincing you need it to appear cool & hip....kind of like fashion for high schoolers

skype doesn't have admin/team lead functions that companies typically might want. my skype mobile app crashes all the time. also inline gifs are cool.

could be a bad phone. my skype mobile app used to crash all the time but I switched to a faster Android phone and its smooth.

Skype is really bad for text chat.

- It is closed-source, obfuscated, and untrustworthy overall (https://www.eff.org/secure-messaging-scorecard).

- The supported (Windows and Mac) versions have obnoxious ads, and the Linux version is buggy and 32-bit-only.

- Since it’s P2P-ish (which makes sense for audio and video calls), you can’t send a message to someone who’s offline and expect them to be notified (say, via e-mail, which Slack does). Worse, if you go offline, they won’t get it until you’re both on at the same time!

- Files shared on Skype don’t last. You have to use an external service.

- There is no permanent archive; logs are client-side and easy to lose. (You can get messages that are a few days old from peers, but there’s a limit – I don’t know exactly where.)

- It is closed-source, obfuscated, and untrustworthy overall (https://www.eff.org/secure-messaging-scorecard).

and Streak is open source?

- The supported (Windows and Mac) versions have obnoxious ads, and the Linux version is buggy and 32-bit-only.

I have the Mac version of Skype, where are the ads?

This part is true, it's pretty limited.

- Files shared on Skype don’t last. You have to use an external service.

Skype is more P2P

- There is no permanent archive; logs are client-side and easy to lose. (You can get messages that are a few days old from peers, but there’s a limit – I don’t know exactly where.)

Skype is for encrypted P2P communication.

Why not just use IRC?

Yes, I know. What point are you trying to make here? curiously asked:

> is there anything that slack does you can't do with skype?

So I named some things. Mostly being not P2P. I also prefer IRC over Slack, although you do need to have some system in place for staying connected all the time or receiving logs to make it workable for everyone.

Anyway, re: your points:

> and Streak is open source?

Slack? No, but I don’t have to allow arbitrary code to run free on my computer to use it.

> I have the Mac version of Skype, where are the ads?

Maybe they’re gone. Windows certainly still has them.

> etcetera

Sure, it’s P2P. That’s a fine excuse, but it also doesn’t make Skype good.

Yeah, about 90%.

Slack is many-to-many entities real-time-optional communication, where one particular subset is that all entities are humans and it is entirely real-time. That subset is handled, badly as always, by Skype.

Yes, the /giphy command. :)

I dont really care. Never shared any critical information on slack any ways.

Why do I have to install Google Authenticator some sort of other app for 2factor here? Why can't you send me a text like everyone else does?

EDIT: Slack responded that they do not support SMS yet.

Because SMS is not even remotely a secure communication channel, and the point of two factors is proof of possession of two different classes of things (that you have some device and that you know some secret), and using an insecure channel to send a message weakens the proof of the have half of that, in much the same way as using a weak or published password weakens the know half.

Ideally, you want as strong as practical proofs of each half within the constraints set by UX considerations.

> Because SMS is not even remotely a secure communication channel

Neither is 2FA, it's a key stored completely unencrypted in both your phone and the the host's db.

That's not particularly weak as proof to the person who controls the DB that you have access to the phone, which is what the device factor of a 2FA scheme is intended to prove, since you have to have access to either the DB or the phone to get the key.

Conversely, SMS's weakness is in the communication channel, which can be compromised without compromising the things for which the factor is intended as proof.

Phones get hacked too. If you ever access a site with your phone, it's not two factor auth. Malware can read your 2FA key and read your password as you type it in.

And thus why we have Yubikeys and other hardware tokens.

Why not both? Google's pioneering approach (TOTP, SMS, automated phonecalls, and offline backup codes) is so thorough that I've been disappointed by almost every other implementation.

Because you can't trust SMS-based two-factor authentication.

http://en.wikipedia.org/wiki/Multi-factor_authentication#SMS...

http://webcache.googleusercontent.com/search?q=cache:UiwfUal...

How does Google Authenticator prevent malware on my phone from approving a login?

That's outside the scope of the tool. You're supposed to prevent malware from compromising your phone. At the very least this scheme requires an attacker to implant malware on your phone as opposed to just monitoring/intercepting communications to and from it.

That's my big complaint, too. Just do like Github and send me a text message. I don't want some additional company's app as a dependency.

You can write your own TOTP application - its open source and works offline, you don't need Google Authenticator, there are dozens of applications available.

This sounds like a half-assed hack if they were able to limit the damage. 2/10 try harder next time, skiddos.

What the flippin' heck is Slack? I received an email today telling me of this event. Then I had to recover my team and my password, then deactivate my account, and I have never seen this website until I saw that email. Looking at my team, it seems to be created by the guys at <pyistanbul.org>, or someone using that website as a base for accounts. That one has a people page and there are links to various places. Now, someone has created an account in my name using my main e-mail address and I have not been e-mail-notified for a confirmation of account, or at least not have been notified that an account is created. Apparently either these lot cannot write proper emails that can pass through spam filters or they expressly allow this sort of malicious account creation. Not surprising that they got hacked.

Reply on news.ycombinator.com