Comment by thaumaturgy | Hacker News Reader

Comment by thaumaturgy | Hacker News Reader

Parent Full thread

They reset the passwords on Friday. Even assuming that they've been working for the last four days on discovering the extent of the intrusion, there's a pretty big gap between Friday's announcement:

"This activity appears to have been a coordinated attempt to access the account of one of our customers. This customer is aware of this activity and we have determined its extent and impact. We have found no evidence that any Linode data of any other customer was accessed. In addition, we have found no evidence that payment information of any customer was accessed. ... Out of an abundance of caution, however, we have decided to implement a Linode Manager password reset,"

and their disclosure today. I find that I'm uncertain whether they would've released today's disclosure if it weren't for the IRC chat log getting passed around online.

I don't get why they thought it would be prudent to reset everyone's manager passwords "out of an abundance of caution", but didn't bother to mention the possible leak of payment information. We don't have any information on the strength of the passphrase used (nor can they provide any, of course); we have no idea how feasible it might be to brute-force it.

Even if the full credit card numbers are never broken, the last four digits are still available in the clear. That has been enough to give unauthorized persons access to accounts on other services in the past (see the Matt Honan thing from last year: http://www.wired.com/gadgetlab/2012/08/apple-amazon-mat-hona...). Depending on what other information was stored in the database, customers may be at risk of having accounts compromised elsewhere using the billing information from Linode. Linode really should have included that information in Friday's announcement.

Also, ColdFusion has had 103 different CVEs going back to 1999 (http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=coldfusion). Adobe doesn't really have a sterling record when it comes to this stuff. A nice example is the recent CVE-2013-0632 (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-0632), a remote code execution exploit that was discovered already in use back on January 4th but didn't have a fix available until January 15th. Linode -- and their customers -- are a really big, juicy target for future CF vulnerabilities.

This is what Caker (founder of Linode IIRC?) has to say about the strength of the passphrase:

"our private key is stored only in encrypted format. The passphrase is not guessable, sufficiently long and complex, not based on dictionary words, and not stored anywhere but in our heads." [1]

[1] http://blog.linode.com/2013/04/16/security-incident-update/c...

Doesn't make sense. Who here thinks they typed that in manually for every single CC they got? They didn't, encryption/decryption was automated, meaning anyone could have done it if they had access to the process that normally does.

Also, the post says salted sha256 passwords are useless. Evidently, they do not have a clue how to safely store passwords, because that is not useless at all.

Finally, it's an outright lie that systems were not accessible, because we already know that they hacked seclists.org/nmap. Even their previous post acknowledged this.

It's a private key. You need the public key to encrypt, which is not password-encrypted.

Although I still wonder about how they run recurring charges.

It could be a manual script that he runs which prompts for the passphrase. I know that earlier in Linode's history, caker made it seem as if it worked that way from the way he acted in their IRC channel when the batch transactions were running.

This seems likely. Once a month, run a script, type in the complex passphrase, and all the billing gets done at once.

I mentioned this elsethread, but I've had changes to my account get charged to my card immediately (within minutes), even in the middle of the night. Last example of this was January 2012, so they may've changed their billing practices since then.

I suppose they could also keep the decrypted key in memory using something like gpg-agent.

I read somewhere (can't remember where now) that the hacker supposedly dumped the key from memory. My guess is they type in the passphrase on boot or something, and it lives in an agent thereafter.

I haven't seen that anywhere, and I find it unlikely.

I understand this, and any system that needed this information would not work if it relied on a human to decrypt.

I think the most likely answer is pretty simple and non shocking: the key is loaded via an agent on boot (subject to the correct passphrase being entered) and stays resident in memory (either permanently or for x minutes/hours). Where's the problem?

If they did it that way, then because the attackers had access to the keys and the server code for an unknown period of time (up to maybe a couple of days), they may have been able to either extract it from memory or use the private key with someone else's credentials to decrypt some or all of the payment information.

I'm really trying to avoid speculation at this point -- we simply don't have enough information to make smart guesses -- but I think we can reasonably conclude that nobody at Linode was manually keying in a password every time there was a charge to a customer card, and therefore it's not insanely paranoid to cancel any cards associated with Linode.

Really the most annoying thing here is Linode's obtuseness about the attack and their processes and what's at risk, and if I end up moving to another service (as at the moment I'm inclined to), that will be the reason -- not because they got compromised, but because they valued their own butts over their customers'.

They're a VPS service for chrissake. They could explain in very technical terms exactly how manager passwords were stored and how the credit card information was stored & accessed and what other billing information may have been compromised -- a full disclosure -- and most of their customers would not only understand most of it, but they would appreciate it.

Sorry, I kinda went off on a tangent there, but the more I think about how this has been handled, the less I feel like I can trust them.

A single private key passphrase is all that stands between a bad guy and everyone's credit card information... Well I know I feel better...

But jokes aside, there is a huge gap between the strongest and weakest algorithms used to passphrase protect a private key. I wish they gave us more information about the certificates.

Including: is it ONE private certificate for everyone? Or one per client/customer? How was the key pair generated? When was the key pair generated? Which algorithms does it utilise and at what key length?

> A single private key passphrase is all that stands between a bad guy and everyone's credit card information... Well I know I feel better...

Where do you get that from? They encrypt all of the CC info using the public key. You need the private key AND the private key passphrase -- and one of them isn't stored on a computer.

> Where do you get that from?

The article and the IRC log.

The IRC dude said flat out that the "bad guys" have the private key and the encrypted data. They also have made no claims that turned out to be untrue yet.

The Blog post infers that the bad guys have the private key by talking about passphrases. Why are passphrases on the private key even a discussion topic unless the private key has fallen into "enemy" hands?

Therefore the ONLY thing between them and unencrypted data is the passphrase protecting the private key.

The private key shouldn't be on any machine that it isn't used on.

If we assume that the key is used on that machine, and is password protected, and is used automatically, then the unencrypted key, or its password, must exist in memory, if not on disk. Perhaps they log in when the machine boots and manually enter the password once, so it is cached and can be used for repeat decryption processes.

If the key is in memory, then a blackhat who knows what he's doing can get access to it.

This isn't a problem unique to Linode though. The same risk is present for every system that needs a password/passphrase to decrypt. This is why effective security (not absolute - which is impossible for any system that has to connect with other systems) can only be achieved through a layered approach.

Linode layered pretty well: Hashed and salted passwords, isolated systems, encrypted CC database and passphrase protected decryption key with no written record of what that passphrase is. They didn't do a perfect job but then who does (or even can)?

I'm pretty sure UnoriginalGuy understands that. What he says is correct, the attackers now have the private key file so all that prevents them from decrypting the CC data is the passphrase. They don't know what this is but they can try and brute force it which is why it's relevant to wonder what cipher the key uses, how many bits and how long, random and complicated the passphrase text is.

Hopefully, they used 4096 bits and a random >20 char string as the passphrase. That combined with a standard cipher algorithm (DSA, RSA) should be enough to keep us safe.

Cracking even a 1024 bit RSA key would cost at least millions of dollars. Credit card numbers are not worth that much.

Tens of thousands of credit and debit cards, many of which are likely to be corporate cards with extremely high spending limits ?

Worth no more than $1 a card on the black market. It'd be cheaper to buy cards than to try to crack the key. There's more supply than demand for stolen card numbers.

That's because the black market for credit cards is a lemon market [1] - known good corporate cards are much more valuable.

[1] https://en.wikipedia.org/wiki/The_Market_for_Lemons

Nobody needs to crack the key: merely the keyphrase. Bruteforcing is relatively cheap.

> What he says is correct, the attackers now have the private key file

I will concede that this is highly likely, but they did not confirm that the private key was taken.

It is implied that it was taken.

The IRC log said that they had taken it and since Linode is talking about the passphrase in their blog post, the implication there is that it was taken too.

If the bad guys didn't have the private key then the passphrase is literally irrelevant to the discussion. Since they would have to break the primary cryptographic key rather than just the passphrase.

A passphrase protects a private key file NOT the data. So why would Linode even bring it up?

> A passphrase protects a private key file NOT the data. So why would Linode even bring it up?

I would bring it when discussing how I cryptographically secured sensitive data. Especially, when people might be confused and try to argue that they should have used pbkdf2 and not encrypted the information (because they were thinking about password hashes).

At least if someone tries cracking the passwords they won't have to bother with a dictionary based attack or short strings ;)

Unless of course it's a double-bluff.

Digression: That would be a single bluff.

Bluff: Saying one thing, doing the opposite ("our passwords are really long" (actually, they are short).

Double bluff: Saying one thing, and actually doing it, as everyone will assume we are bluffing. "Our passwords are really long, honestlyl! (and they actually are, but bad guys will assume we are bluffing and using a short password).

Except you know, system memory when the key material is unlocked... unless you really believe someone is sitting there typing in the passphrase for each non-batch CC transaction, because I do not.

What exactly do you propose as an alternative? At some point the card details need to be decrypted. Security always boils down to somewhere between total lock down and being able to actually get things done. The only semi reasonable alternative would be not to store any CC details and outsource that task to a payment processing company. It's not a great alternative though as you'd effectively be giving up the ability to swap payment processors (since they, not you, now have all your customers card details) for a potentially small and unproven bit of extra security.

Lets be honest here. How frequently do you think a company changes processors?

Agreed, however, imho you'd have to be insane to run a company with hundreds of thousands of monthly billed customers and turn control of the card details over to a third party (who, lets be honest, will be running their backend in a very similar manner i.e. encrypted CC details protected with a passphrase). I'd take my chances the same way Linode chose to rather than punt the issue upstream and hope for the best.

I might be biased though as my default approach to most things is to do (and have control over) as much as possible yourself unless there are extremely good, unequivocal reasons not to.

Honestly, the gap can probably be attributed to them not knowing what exactly happened. Figuring out that you were compromised by a zero day vulnerability isn't an easy process to work through.

> we have no idea how feasible it might be to brute-force it.

Sorry to be morbid, but... there's also rubber hose cryptanalysis. Whether the payoff or actors involved here make that a legit concern I don't know.

Reply on news.ycombinator.com