Hacker News Reader

Comment by danso | Hacker News Reader

From a purported abridged chatlog with the alleged hacker:

> 05:42 < ryan||> credit cards were encrypted, sadly both the private and public keys were stored on the webserver so that provides 0 additional security

> 06:00 < ryann> They did try to encrypt them, but using public key encryption doesn't work if you have the public and private key in the same directory

http://turtle.dereferenced.org/~nenolod/linode/linode-abridg...

View on HN

Here is what Linode replied to me when I asked them about that chat log in a support ticket:

  Hello,
  
  Thank you for reaching out. We appreciate and understand your concerns. At this time the evidence suggest that this activity was targeting a specific customer. We are unable to release any additional details regarding this incident at this time, as there is an ongoing investigation.
  
  We have no comment regarding ryan*'s comments in #linode. You are of course free to take any steps you deem prudent or necessary to ensure the integrity of your online presence.
  
  I am sorry that we cannot provide more information at this time. As always feel free to contact us at any time with any future concerns.

  Regards,
  Quintin

These guys are looking totally incompetent at this point.

If you believe this Ryan guy, credit cards stored on the same server as the key to decrypt them, Lish passwords stored in plain text, they've known for some time and lied about what actually happened and now they're saying "we won't do anything about it" via email?

"You are of course free to take any steps you deem prudent or necessary to ensure the integrity of your online presence."

Unbelievable.

Edit: not to mention they "made a deal" with the hacker not to tell anyone? What the hell?

> If you believe this Ryan guy

That's a rather key assumption. If you don't believe him, then all you have is a trolling (or at least self-aggrandizing) hacker whose credentials consist solely of logging into an IRC channel, refusing to identify who he was working with, and offering no tangible proof of having compromised any CC info.

On the other hand, it's conceivable that if ryan managed to get into the files a customer was hosting on Linode, and that customer was improperly storing CC info, then their customers' info would have been vulnerable, and ryan's claims would be sort of half-true. Even so, that wouldn't directly affect other Linode customers or put liability in Linode's lap.

Regarding the "made a deal" assertion, I wouldn't take the IRC log hook-line-and-sinker. There's probably a mixture of truth and lies.

for the record, i am the person who started the WHT thread.

there is a mixture of truth and lies on both sides, to be honest.

i am annoyed with it, because i reached out to several linode employees privately to given them an opportunity to explain what was going on -- they either said 'no comment' or said my linode was fine.

based on the irc log, that is clearly not the case. which is why i decided to raise my concerns publically.

luckily for me, my linode was not doing anything mission-critical, just some secondary monitoring and running an ircd for a network i like using, but there are others who are using linode for mission-critical work, and they deserve more transparency than this.

what makes you think you are more special then anyone else? Why would they tell you more details, than the rest of the people?

Sorry if you get offended that they didn't tell you much more. But seriously? you are not special. This whole thread is a lynch mob.

Is he more special than other people, or should we ALL have been informed properly?

To be fair the hacker didn't say the keys were stored on the same server as the credit card numbers, he said they were stored on the web server. It's most likely the database containing the CC numbers resides on a separate set of boxes than the web servers.

The Cigital-recommended way to hash your passwords is to use an HMAC/scrypt combo, with the HMAC key stored on the app server (not the database).

What Linode did may, or may not, be dumb. They are being tight-lipped so we can only guess.

Why would you use an HMAC for password storage? It's not like length extension attacks are relevant in that application.

the database server is local, read the entire log...

Despite what the other replies here are saying, this seems like a perfectly acceptable response to me. This comes off to me not as they're refusing to talk about it, but they _can't_ talk about it, presumably because of an ongoing investigation. I'm not sure what else people here are expecting them to say.

Then why not say that?

They did.

> We are unable to release any additional details regarding this incident at this time, as there is an ongoing investigation.

The pasted text from Linode appears to say exactly that:

> We are unable to release any additional details regarding this incident at this time, as there is an ongoing investigation.

Edit: Oops, didn't refresh before replying. jmilloy got it first! Sorry about that.

That is an absurd response. I don't care if they believe a specific customer was targeted, I want to know what happened and what information may have been compromised.

They probably can't say anything just yet. It's fair to be mad about them not realizing (or worse, covering up) the breadth of this breach, but do keep in mind that if they're working with the FBI, they've probably been asked to keep a lid on their official response for a few hours.

That is not the way to handle this issue. I've found my one problem with Linode is they are arrogant. It comes off pretty strong if you ever ask them questions in chat.

One thing to note is that the irc channel if that is what you mean by chat, is mostly populated by non linode staff. And many of us there tend to be sarcastic as we idle there and chat about all sorts of stuff while we are bored at work or whatever. Unless it's someone with ops, you aren't getting an official reply and even then for something official they usually refer to ticket system.

This is ridiculously unprofessional.

Well the steps I would like to take regarding linode aren't exactly legal. Satisfying yes, but not legal.

But I just checked and my credit card haven't been used anywhere I didn't use it.

"Don't worry, all the doors are locked."

"Where are the keys?"

"In the locks."

Wouldn't doing that be a massive PCI violation? Aren't there extensive audits for this sort of thing?

Extensive PCI audits. Heh.

Like the "scan" that said our linux boxes were running an out-of-date version of IIS.

That compliance web form I absentmindedly clicked through sure had a lot of buttons.

Can't think of the exact word to describe that practice (what you did in response to the long list of questions which I've seen) but on the part of the company requesting you to answer the questions it's more or less a "absence of malice" type of thing that allows them to appear that they are doing the right thing while fully knowing that people are doing what you are doing. It's a "we will look the other way until we need to show that it's not our fault because we have passed the liability to you - look you acknowledge doing all the right things".

It does not mean anything until they decide you are not compliant and you need to prove you are compliant. (I've never had to but I'd appreciate insight from people who have)

Companies are split into PCI Levels based on how much money/customers they handle. Level 1 are big companies like amazon, level 2 are medium sized online retailers generally, and level 3 are smaller retailers.

The 'lower' your level, the easier the PCI audits are. If you are level 1 you have mandatory external audits. If you are level 2 you have a 'self assessment' which is basically a checklist which says "Yes, I promise I'm in compliance".

If you have a confirmed breach, you are upgraded to Level 1 merchant audit requirements. This is generally quite costly as the external audit is extensive and must be paid for.

I've done PCI "audits" for several companies I've worked for; it's a checklist you go down yourself. That's why its called a "pci self assessment".

Actually, if you're processing cards directly, you do in fact need to have an PCI-qualified outside firm† (a QSA) audit you for PCI compliance. But those audits are notoriously superficial; PCI audits are a race-to-the-bottom affair.

† We are not one of those.

Note that you hire those firms yourself, and they work for you. They want you to pass the audit and will work to make that happen.

Every one of these reviews that I've been involved in has been conducted by a couple of guys with laughable abilities.

The quality of PCI security audits is a continual aggravation to everyone I routinely talk to in my industry. I've told more than one client: if you need a QSA audit, get the cheapest one you can. If you need a software security assessment, don't use a QSA firm.

"But those audits are notoriously superficial"

Will add that just having gone through an ICANN registrar audit (which by the way were specified and supposed to be done literally 10 or 12 years ago but never requested by ICANN) with a third party company hired (accounting firm) it's total compliance theater.

Add: "hired by ICANN after a bidding process". Same happened with data escrow which was just implemented a few years ago and is operated by Iron Mountain.

Most PCI audits are not mandatory unless you have had a breech that compromises data. Before that "self-certification" reigns.

The audits are toothless and ultimately the audit only happens once (if ever) and people keeping pub/secret key in the same place unprotected... well.... They're just unlikely to get security at all...

The paste-bin link he provided seems to time out for me.

But if the things he claim in there is even half way true, nobody involved with linode should ever be allowed to be in business every again.

pastebin is a directory listing of linode.com - trying out a few of the files checks out, including very difficult to guess file names such as:

http://www.linode.com/y_key_57284cb2de704e02.html

Am I the only one who is more confused about why there are compiled java classes and AMI BIOS updates in the www directory than about the hacking itself?

> Am I the only one who is more confused about why there are compiled java classes and AMI BIOS updates in the www directory than about the hacking itself?

Sysadmins being lazy.

Somebody needs to get a file from a workstation to a remote machine. There's a firewall in the way somewhere that prevents SSH directly between them or one of them is a Windows box that isn't running an SSH server.

The "correct" solution is complicated and takes 5 minutes to setup. So the sysadmin just copies the file to the web server and downloads it with a browser on the other end. Because port 80 is always open.

Something that took me a long time to notice was that you can actually copy files directly over a RDP session from/to the local machine or other RDP sessions using the clipboard.. Even nested RDPs. Has been an absolute timesaver to know about when doing Windows admin work.

Doesn't work for me when using an RDP client on Linux. Is this an RDP spec thing or a microsoft only feature?

I think it depends on how you mount your clipboard/drives on rdp connect. To get it right with windows you just check the share clipboard/share drive checkboxes and off to the races. With rdesktop you have to throw the -r flag and mount a clip board and then the -r flag and mount a drive. Not sure about other Linux clients but I'm sure there's a similar option in all of them.

Excellent, thank you very much. I use KRDC and a bit of googling shows that that uses rdesktop in the background. I shall investigate.

There is literally zero evidence to suggest they compromised anything more than a webserver.

That's totally shit.

Its also why we invoice and take wire payments rather than storing CC details. There's just so much to go wrong.

Also PKI is shit for this sort of thing. As demonstrated, the moment that public key is gone, then the whole system falls like a house of cards. For the non believers of this fact, why else would there be a certificate revocation list and root CA updates for windows periodically...

Using a processor who stores the card number outside of your infrastructure (ie. Stripe) can also be helpful.

Even better to use someone that isn't your payment processor, so that should you need to change payment processors you don't also have to re-acquire all the billing info from your customers. You can use Stripe today, PayPal tomorrow, and Braintree the next if that's what works best for your business.

Card vaulting as a service: https://spreedly.com/ ($10/mo for up to 5000 cards)

Isn't that just adding one more point of failure? I don't trust myself, I barely trust Stripe or Paypal, and I've not even heard of spreedly.

There's always going to be single points of failure, but which is more likely: you want or have to change payment processors (you've been terminated, your fees have gone up, you want to switch to a lower cost provider) or you want to change flat-rate vaulting services? Plus, Spreedly will give you your data if you leave, whereas there is no way to get stored billing info out of most payment processors.

Until they have a security breach.

Better rely on someone who's sole job is securing that info than doing it yourself.

Storing credit card info just helps make you a bigger target. If your a small company, better let someone else store card info, let them be the target.

Also you're fined by the credit card companies if you lose card information. I believe it's a per card fine, so it get expensive really quickly.

Actually I don't get why any company would choose to store credit card information, when most payment providers will do it for you.

Stripe is amazing... I trust them, someone hacks me, awesome, you got password hashes and stripe customer keys, all worthless.

Not exactly worthless, depending on the hack someone could still charge an awful lot to your customers and make you have a bad day.

But yes, significantly better than other situations.

True, but I'd rather fix the problem that got them in, force reset of passwords, and delete all customer keys and require them to create new ones than be like "uhhhh, our data was hacked and your credit card is safely encrypted... But we had the encryption key on the server too, oops"

But you still might need to consider data residency issues and making your customers aware of where their data is being stored.

Yes

But if that happens, it's not your responsibility (at least not 100%), it's theirs

Following the traditional responsibility/accountability dichotomy: They are responsible for storing the cc number securely but you are accountable when something goes wrong (because you gave them that task)

Much like Linode are responsible for hosting my clients site but I sigh am accountable when something goes wrong.

In what ways are wire payments better than using credit cards? In wire payments aren't you using the actual bank account numbers along with routing numbers which is also very sensitive information ?

Also I do not think, but I am not sure, that fraudulent wire payments/transfers are reversible.

They're not. You're right.

Wire transfers often are not able to be undone once they happen (and are accepted by the other bank). This is the reason why there's so much verification that happens in wire transfers. (I helped develop 2nd factor authentication used for authenticating wire transfers for a financial company)

Credit card charges can be reversed.

http://www.reba.net/news/wtransfer

Reply on news.ycombinator.com