Hacker News Reader

Comment by sturadnidge | Hacker News Reader

I really wish they would officially comment on the 'cover up' aspect. Security breaches happen, and are forgivable. But attempting to broker a 'silencing' deal with the intruders and hoping your customers will never be the wiser is not.

All this update does is restore my faith in their ability to store my information correctly. It does nothing to reassure me that they won't try to cover anything up again.

View on HN

I feel like there is a fine line between a "cover up" and a "grey hat disclosure". Reading the IRC logs from #linode [0], HTP hacker ryan* seems to say that they made a deal of "we don't tell if you don't tell" but then Linode broke the deal by reporting them to law enforcement.

You could see this as a cover up or you could see this as a disclosure from the crackers to Linode. I see it as a cover up, since Linode should tell us anyway that servers were broken into, but I could see how it could have been seen as reasonable disclosure.

In any case, if what ryan* is saying is factual, then Linode reneged on the deal, a shady one in the first place, HTP has credit card hashes and evidence that Linode stored Lish passwords in plain text (a log maybe?), and the only thing separating them from credit card numbers is the passphrase on the private key, which is hopefully a strong one.

[0] http://turtle.dereferenced.org/~nenolod/linode/linode-abridg...

Perhaps nitpicking, but looks like HTP might have gained password hashes and encrypted credit card numbers, as well as a passphrase protected private key file. That's if I'm reading between the lines of both ryan and Linode's statement.

This is why you build security in layers. If one layer gets broken, there are other layers to protect you.

I'm not underestimating the severity of this incident, but I think it's good to see this layered approach being used by Linode. If the passphrase was strong enough, and not stored anywhere, then I doubt HTP would be able to gain access to the private key, even if they got the actual private key file.

Sorry, 6am here, that's what I mean. It is good to see Linode adopting this layered approach, though it is what you're supposed to do. The other route would be storing the private key file unencrypted, which, thank goodness that wasn't the case. Knowing Linode, I'm fairly confident it's a strong passphrase; it would be imprudent to secure credit card details with something like "swordfish".

Well, the official statement re: the passphrase is:

> @Eivind – our private key is stored only in encrypted format. The passphrase is not guessable, sufficiently long and complex, not based on dictionary words, and not stored anywhere but in our heads.

> in our heads

So it's short enough to remember and likely has some sort of pattern. There's a limit to what a person can remember, lower if there are several people that have to remember it.

It is not very difficult to memorize random strings of arbitrary characters. I use a password manager to manage most of my accounts, but the important ones, like banks and email, I keep in my head. I use my password manager to generate a 15 character string of alpha+numeric+symbols. The symbols would kind of make it hard, except that in my head they are just upper-case numbers, mostly (shift-7, not ampersand). And in any case they are just positions on a keyboard (God help me if I need to enter one from my cell phone).

To memorize, just copy it into your favorite text editor, then type it 25 times in a row and delete. If you are paranoid, make sure you use a text editor that does not store temp files. Do not save this password anywhere. Set a timer and do it again an hour later, then again the next day. 10 minutes of your time and you have a password in your head. I can keep 10-20 of these at a time, maybe more since I seem to be able to type older ones from years ago.

I don't consider myself to have a great memory. I can barely remember lyrics to songs I've listened to dozens of times and it takes me hours and hours to memorize lines for plays. But I started doing this for passwords ten years ago and it is very reliable.

The thing is for your personal bank account a 15 character password is acceptable.

But for x many customer credit card details you're really looking for a much longer password that that. I'm talking 64 characters or more of pure random data.

You shouldn't be compromising for the convenience of being able to remember a password when it secures such critical data in my opinion.

Edit: I do agree though that your method is a very good way of remembering password.

At 15 characters and my character set ( [a-zA-Z0-9] and about 30 symbols) I have about 92 bits of entropy. Mean time to find a collision hash of my password is more than several years using 100% of computing power on the planet, much less do AES brute force. If memory is no issue - 256 bit passwords (usually displayed as 64 hex digits) are wonderful and there is no reason to stop short of that for pass keys that are stored electronically.

If I was responsible for this key I might increase from my normal 15 to 20 characters, giving me more than 120 bits of entropy, and I would expect to be safe from offline brute force for decades, and I could remember it.

It's trivial to memorize an entire sonnet. Actors and actresses memorize many times that amount. It's also trivial to write a sonnet. How many bits of entropy do you think a sonnet has?

Are you saying that you are happy to type in an entire sonnet when prompted for a password?

Being realistic, to expect someone to type in such a long password regardless of if they can remember it or not is clearly unreasonable.

For a consumer, perhaps, but for protecting thousands of individuals...

You can't assume this. It could be ridiculously strong, and, with a lot of use, has become remembered.

I'd argue that actually it's better to assume the worst case here, not what it potentially could be.

That and the fact that an offline attack can be run on this key is not promising.

You've assumed that everyone involved has the entire passphrase.

They were storing LISH passwords in the clear... Does it really sounds like they care enough to use some sort of multi-party brokered passphrase accountability system?

Not really, just more than one person as is implied in the grammar.

Edit: Theses "making an assumption" arguments are silly. It is good practice to assume the worst case, to assume the best in this situation is bad.

I'm impressed by your ability to extrapolate "in our heads" to mean whatever you wanted it to mean.

Deleted comment

Even if they have a passphrase it has to be strong enough to withstand brute force for months. If you're not smart enough to keep the private key separate from the crap you're trying to protect, why should I think you're smart enough to ensure your passphrase is good enough?

why should I think you're smart enough to ensure your passphrase is good enough

Key strengthening.

OpenSSL doesn't use key strengthening on password protected RSA keys. GPG does, but I don't know how much it does by default.

Except that LISH passwords were plaintext...

> All this update does is restore my faith in their ability to store my information correctly

How is your faith in that regard restored, when they just said they stored passwords in plain text? They didn't store passwords in plain text by accident, they chose to. They knew the risks and didn't care.

"Linode Manager user passwords are not stored in our database, but their salted and cryptographically hashed representations are."

Sounds like it's not stored in plaintext.

And then "There were occurrences of Lish [the Linode Shell] passwords in clear text in our database."

"There were occurrences of Lish passwords in clear text in our database."

Which is entirely different from "all passwords were stored in plain text". We don't know whether there were 1, 100 or a majority of such passwords. It's a bad oversight in each case, but the latter is quite a bit worse than the former.

Right, we don't know because they failed to disclose this.

Wouldn't you rather have them working on assessing, containing and repairing the damage rather than catering to the impatient internet crowd that is so used to immediate updates on everything that they can't fathom putting together a responsible, useful, correct response might actually take a while? Honestly, these people don't understand what it takes to run a company and handle such an incident. As far as i'm concerned, the indignant vocal minority could shove it while I was working on resolving the problem for the majority of customers that exactly that from us.

Quoting Linode:

"There were occurrences of Lish passwords in clear text in our database. We have corrected this issue and have invalidated all affected Lish passwords effective immediately."

Presumably, they know how many customers are affected, because they invalidated their passwords. They didn't tell us.

I don't understand where you are coming from here.

Why do you think companies can only do one thing at a time ? And where have you worked where software engineers are drafting press releases or conducting security audits ? And why shouldn't keeping customers (who have their own apps) informed be their top priority ?

  Why do you think companies can only do one thing at a time?

Amdahl's law. Recovering from a catastrophic hack isn't parallellizable. Nobody can draft a press release before the security audit has been performed. You can't inform anyone before the security audit has been performed. The software engineers can't go to work before certain parts of the security audits have been performed. Certain parts of the security audit can't be performed before other parts have been performed.

  And where have you worked where software engineers are 
  drafting press releases or conducting security audits?

A bootstrapped startup? Depending on the scale of the company (and I have no clue how many people work at Linode), the work may be sequential simply by lack of manpower.

I believe the correct approach for us is to assume all Lish password where in clear text until a good explanation is given. It's not usual to store some password in clear text and encrypt the others, right?

Could be that the passwords were hashed, but something logged a password, or there was leakage somewhere during the login process.

All it takes is one debug log statement to leak through from development and you can be logging plain-text passwords on every login.

Storing passwords in clear text is definitely not normal.

Also the social engineering implications could be huge. Many people use the same password across various sites so in theory there could be a lot of compromised email accounts. Which could then mean compromised internet banking sites, trading sites etc.

Bear in mind this can be as simple as a handful of people emailing their LISH password to support while asking if it is correct etc.

Or someone mistakenly logging "Bad LISH Password: 'FOO' for user 'BAR'" somewhere that logs to the DB.

Or users accidentally typing their password in the username field.

Nevertheless, this isn't the first time Linode has needed to spend their benefit-of-the-doubt points.

Also (and correct me if my understanding of this is wrong since I've never used it), LISH is just a remote terminal service. You still need to know the VM's root password. And frankly, if you're doing it right, the root password should be a long, unique random string that you store somewhere safe and never use since you should be using keys to login to your box on a day to day basis.

It's quite likely that someone has left root logged in on hvc0 (the LISH console) on their Linode, while logging out of LISH. Probably more than one person.

Also, access to LISH even without a root password provides access to some scrollback output, which could expose sensitive information.

Also, LISH allows multiple connections, all of which see the same console, so the attacker can just connect to it and wait for a root prompt to appear when root logs in next. (Does changing the LISH password prevent this attack if they're already connected? I doubt it.)

Also, most distribution boot processes can be messed with at boot from the console. For example, you can ctrl-c to stop important daemons from loading. In some cases you may be able to get a shell without the root password.

Also, Magic SysRq can be accessed over a serial console by sending a BRK. You do not need to be logged in to do this, and it could be used to kill processes, reboot, etc. I don't know if LISH allows sending BRK.

Yeah but most of these are generic problems with providing a remote serial console (excluding LISH allowing multiple shared connections to one console, which is obviously bad).

They've now reset all the passwords and fixed the bug that meant some of them were being stored without encryption. So what we're saying is there was window of a week or so where only LISH users affected by the clear text bug may have been open to an attack if they happened to use LISH during this time frame and the attacker was targeting them. Not great, I agree, but could have been worse.

Hopefully, they'll change their setup going forward so that each LISH connection to the same VM gets its own console.

Shoot, you are correct. Definitely an attack vector.

The connection is to the same TTY when I log in via the web terminal and SSH at the same time, so if you are logged in via LISH anyone with the LISH password has access to the logged in user's console + scrollback.

I wasn't able to send BRK from Putty though.

I'm pretty sure this is correct. The LISH passwords just give you console access via ssh (as an alternative to the web-based terminal in Linode Admin), which is essentially the equivalent of getting to an ssh prompt. A login/password would still be required to access the machine.

It doesn't appear that you can reboot or enter single user mode from the LISH console login prompt.

I'd imagine the 'cover up' is because law enforcement is involved and this 'ryan' character is being looked for.

As these situations, especially on the internet, come out as "he said, she said", I think it's probably more important to keep focused on what directly affects you.

Sure, but in this case it wasn't Linode who initially came forward - it took a public announcement from the intruders for Linode to notify us.

Devils advocate: preparing a press release from a company after plugging holes and auditing that makes the appropriate admissions and apologies probably takes more time than writing up a successful hit after attacking a page.

Linode didn't just take their time to announce this, just a few hours ago they were apparently telling customers definitively that their credit card information hadn't been accessed.

And that is "still" the case

My guess is that they probably went straight to the authorities, while playing along with the intruders as a way of buying time and collecting as much information on them as possible. I'm not surprised that they wouldn't make an official comment on that, though.

Reply on news.ycombinator.com