Important Security Announcement from PagerDuty
pagerduty.com
pagerduty.com
There have been several breaches in the last months where this was the main cause and it's something almost impossible to defend against – unless you're running your own datacenter hardware, which is very hard to get right.
Few providers properly secure their control panels with 2FA, even though these admin panels are an attractive target and almost always provide full access to the system.
Consumer-style 2FA protects somewhat against brute force, against password reuse by users who just don't care, and against theft of a whole password list. It doesn't actually protect the protected resource very much.
From their CTO (on Disqus in response to my question):
"The salt, pepper, and password were concatenated together to form the string that was in turn passed to the hashing function."
40 + 40 + strlen($password) > 72
Uh-oh.
PHP. This is a known weakness in PHP's bcrypt implementation. From Wikipedia, "Many implementations of bcrypt truncate the password to the first 72 bytes." I would hope that they're using a competent implementation that either supports longer passwords or throws an error if it's asked to hash a longer password.
Besides, bcrypt takes the salt as a separate parameter anyway. So it doesn't really make sense in context that the salt, pepper, and password were concatenated and passed to bcrypt. Perhaps he was talking about the SHA-1 stretching; perhaps they hash the passwords with SHA-1 before passing it to bcrypt.
I don't think we know enough to conclude that they were definitely doing it wrong, but it would be nice to know more details about the algorithm, though.
Actually, it's a known weakness in BCRYPT. PHP did not implement bcrypt, it was ported in via crypt(3). Meaning that ALL versions of bcrypt have this issue.
Some implementations error on > 72 bytes, but NONE of them accept longer passwords.
> I don't think we know enough to conclude that they were definitely doing it wrong, but it would be nice to know more details about the algorithm, though.
Given what has been shared so far, there's enough signs pointing that the chances are pretty high they did something wrong. 40 byte salt? Bcrypt only supports a 128 bit salt. So either they did something silly custom (at which point it's no longer bcrypt), they aren't actually using bcrypt, or they did something silly like concatenate the salt + pepper + password and pass it to the password field.
I presume they wanted to have all of the facts before they notified customers, but it is totally unacceptable that they waited 3 weeks to notify me about an incident with confirmed external intrusion and confirmed theft of customer data (including my own).
> Within hours of the start of the intrusion, we were able to detect and remove the attacker, and shut down the attack.
If this were some highly regulated banking/securities/government system, 21 days MIGHT be ok for public notice, but not for an infrastructure tool with highly technical users.
People get hacked (not good), but responding badly is the true badness.
And now additionally we're in WTF hell because no one knows if linode was compromised meaningfully (again), or if it was just a PagerDuty credential which got compromised.
https://support.pagerduty.com/hc/en-us/articles/202830570-Wh...
Instead, the post's comment makes a reference to "an administrative panel provided by one of our infrastructure providers". It's not clear if this refers to the provider's internal tool, or the administrative panel they expose to customers for administering their accounts. If the former: I'd prefer they name the provider. If the latter, it might be worth clarifying the wording so it's more clear.
Most cases like this the company claims they found the hacker but that isn't the case.
PagerDuty had to learn this lesson the hard way (circa 2012), as they were entirely in US-East during the catastrophic AWS outage in June. They proceeded to go offline along with a lot of their customers, who subsequently were not alerted.
Multiple region redundancy is not trivial, but it would certainly be a little higher up on my to-do list if it was basically the entire value proposition of my service. I was a little suspicious of PagerDuty's operational chops before, but now I simply don't trust them.