Hacker News Reader

Yeah, they really gloss over the fact they have no idea how the TOTP secret key was compromised, which worries me the most.

They changed the 2FA to use a microservice, so whatever the vulnerability was before, if the 2FA is now on an isolated server, that vulnerability shouldn't have access to the new 2FA key.

But given that they don't know what the vulnerability is, there's no way of knowing that.

When it comes to the security of who's hosting my servers, I want a little more reassurance than they shouldn't have access. I need to know that they don't.

I think it's fairly important to note that they're NOT currently using the microservice for the 2FA, and they're NOT using bcrypt right now.

The blog post states they're "working towards" these changes, they're not currently in place. It's fairly unlikely that they're using the same secret key as the one they found on the server, but it's fair to assume that they are still using salted SHA-2 for your passwords and the same 2FA setup right now.

They likely won't roll out the major changes until they roll out the "new and improved" Linode dashboard they're coming up with.

The article didn't state that. The article stated they are rolling out soon. The new dashboard will be an open source project. So you'll know when that gets released. There is no link to the project yet so assume that part isn't started yet. So the microservices should be released in a timely manner. Let's hope with the new focus on transparency if there are any delays they will keep us posted.

Isn't that exactly what I said? o.O I said they will "likely" get rolled out with the new dashboard, not that the article said they would lol. But, they never stated when it would happen anyways, so "delays" aren't really a thing when there's no deadlines.

Yeah, the thing you are missing is why the story doesn't add up :P

Seriously, in one paragraph they state their TOTP keys were taken, in another they state nothing is wrong because they can't figure out how an attacker might have taken those. WTF?

It sounds they have an idea how it could have happened — the Lish vulnerability — but they don't know if that's how it actually did happen or if there's another undiscovered vulnerability lurking.

Ok, now help me understand how that can be true given this quote from the advisory:

The findings of our security partner’s investigation concluded there was no evidence of abuse or misuse of Linode’s infrastructure that would have resulted in the disclosure of customer credentials. Furthermore, the security partner’s assessment of our infrastructure and applications did not yield a vector that would have provided this level of access.

Did they get compromised or not?

"no evidence" or insufficient logging. The wording is spun to their favor throughout the entire advisory. Like "Security Investigation Retrospective" making it sound like they have the whole thing wrapped up. Also that everyone will be fine once their passwords are reset, when it sounds like they still don't really have any solid idea if they fixed the problem or not.

My reading was that:

1. Their security partner (whoever that is) didn't see the Lish vulnerability, either because Linode's security team had already fixed it or because they just missed it.

2. Their records weren't sufficient to show the breach itself.

I'm not just trying to be argumentative. Here, let me quote more specifically:

no evidence of abuse or misuse of Linode’s infrastructure that would have resulted in the disclosure of customer credentials.

I feel like I must be misreading something. Didn't they say earlier that they found secrets from their account credentials database on a customer instance that was used to attack (apparently) PagerDuty? That's not "no evidence of abuse or misuse of Linode’s infrastructure"?

Honestly, it is likely because they are spinning/lying that it is confusing to you.

If you just assume they are attempting misdirection it makes perfect sense for them to insist things are fine.

you forgot "that would have resulted in the disclosure of customer credentials".

I'm speculating, but having been a fly on the wall at this kind of meeting before, here's my theory of how this went down:

Manager: "So somebody got the key to generate one-time password tokens for PagerDuty. How did that happen?"

Engineer: "I have no idea."

Manager: "What about that Lish vulnerability? Could it have been that?"

Engineer: "Could have been."

Manager: "We don't know?"

Engineer: "I mean, they could have gotten it that way, but we don't have logging for security vulnerabilities, because, you know, we would just fix them instead of logging that they occurred..."

Manager: "Okay, I want a second opinion. Time to bring in $SECURITY_PARTNER."

$SECURITY_PARTNER: "What Engineer said. And here's a bill for our time."

Manager: "What about if the customer screwed up and lost a device? Could it be that?"

Engineer: "It could be that."

Support: "PagerDuty said something about wiping their phone."

Engineer: "If they did lose a device, it would be indistinguishable from the evidence we have."

Manager: "So, maybe we weren't hacked after all?"

Engineer: "Maybe not."

Manager: "Any other theories?"

Engineer: "I just want to go on record... AGAIN... that I don't trust our ColdFusion infrastructure at all. The people responsible have all left, and none of us understand it. We need to rewrite in a cooler language, like Python. And hire some actual security people."

Manager: "Sigh. We've gotten enough flak over this that I guess I'll put it in the budget. Okay, good meeting everyone. I'll write this up in a blogpost, because we need to show transparency. That way customers are at least as confused as we are."

In our case someone logged in to the Linode Manager first. Whomever logged in was confident they knew the password of the user and the MFA token.

That would indicate that Linode's password hashes may have been cracked offline.

Hah, pretty close except the only person responsible is still there.

from the disclosure:

> After examining the image from our July investigation, we discovered software capable of generating TOTP codes if provided a TOTP key. We found software implementing the decryption method we use to secure TOTP keys, along with the secret key we use to encrypt them. We also found commands in the bash history that successfully generated a one-time code. Though the credentials found were unrelated to any of the unauthorized Linode Manager logins made in December, the discovery of this information significantly changed the seriousness of our investigation.

so, it was an inside job then?

As I read it, the 2FA login does some processing of the credentials before submitting it to their authorization server. Someone logging into a machine directly instead of going through their portal would have to do this processing themselves. They found a program designed to do just this on the compromised machine, which a normal user would be very unlikely to have, and thus serves as strong evidence of malicious activity.

Right, there's that, and also (if I'm reading this right) they found key material.