Linode hacked, CCs and passwords leaked | Hacker News Reader

Linode hacked, CCs and passwords leaked

slashdot.org

753 pointsDiabloD3404 comments

Ah this is so shit. I want to support Linode, I've had nothing but a good experience. But I just had to check my credit card to be sure they hadn't lost my details. I've NEVER had to do that before with anyone - they've got to respond fast here because if I don't trust them with my CC then I can't leave five-figure contracts at jeopardy hosted on their servers.

I've been living comfortably on Linode servers for over three years. This is like suddenly being evicted and having to pack my stuff up and find another apartment.

I have to wait for some sort of verification for this but if true then I have to leave Linode. I have client sites hosted here - not for cost reasons, just because I like Linode.

For the sake of $5 a month I can't even take the slightest risk of being criticised for using Linode. And this lack of transparency could be a nail in the coffin here.

I don't want to waste a couple of days on this but that's what's going to be involved if this is true.

I've now heard from a number of people using Linode that have suspicious activities on the cc which they used with Linode.

I just called up my bank to tell them to 'block' it as a precaution (I will now have to give them a visit later today to get a new card). I encourage all other Linode customers to do the same, because it'll be easier to just spend half an hour doing this instead of spending hours upon hours disputing specific transactions.

Linode customer support keeps saying they have "no comment" on this issue (which I suppose does make sense -- I'm assuming they've been ordered by law enforcement persons to not share details), so as we're not being given much information to work with... just treat this as a worst-case scenario (all names, addresses, credit card numbers, etc. have been compromised). Do operate now with the assumption that all of this data has been compromised and may very well be public soon.

"because it'll be easier to just spend half an hour doing this instead of spending hours upon hours disputing specific transactions."

I live on the internet. Put my credit card out on many services. Over the last 5 to 8 years I've had my credit card numbers taken I believe 4 times.

Never had to dispute it once. These Credit Card companies and Banks have a stake in not allowing your account to be drained.

I think it would be a waste of time to go out and cancel our CC until hearing from Linode that yes, CC information was taken.

It depends on how sophisticated the identify theft is. I had a good friend who was taken for about $9000 in credit card fraud in 1998/1999, with Well Fargo. It took him the better part of six months, and endless correspondence with WF to prove all of the purchases were not his. There are lots of stories of people who were financially wiped out, to the point of bankruptcy, because of Credit Card/Identify fraud.

With that said - almost everyone seems to feel comfortable handing out their credit card to random taxi drivers, waiters, sales staff - with no idea whether a copy of their information is being taken down. Heck - if you give them the Credit Card, they even get your CCV as well.

I did it, not worth the worrying. But then again I only bill two things to this CC, one of which is annual.

I've had roughly the same experience: in the last 8 years, I've had suspicious activity on my CC about 5 times. Each time, the bank caught and trapped it before I noticed and issued me a new card quickly. I've only had to fill out paperwork for a disputed charge once, and it was a 2-page, 2 question, sign-and-mail-it-in deal.

My advice is different, though. I notice that I tend to get lucky in places where people can have very aggravating experiences. I'd say that if you've had problems before, then anticipate problems this time around too. If you haven't, then don't bother.

Wow, four times? You should probably be more careful about who you give your number to. Personally, I usually get a new card every 3-5 months. If someone ever sat on my card number, it's useless to them now. Never had any issues either.

I would be utterly shocked if nobody using Linode had suspicious activity on their CC. Linode has lots of customers, and at any given time, some of them probably have suspicious activity going on.

There's baseless speculation and then there's I have some information speculation. I'm operating on heuristics which rely on information that is handily available. Yes, in the end you're right, I'm just speculating. But hey, it's better to err on the side of caution.

No weird activities on mine either but I will give a call to my CC company anyway. I have had to cancel the card I use on linode twice in the past few months because of suspicious activities. I just didn't think it would be coming from Linode

No weird activity here (no declined transactions or anything)

Mine is not a CC or a debit card, it's a 'prepayed card' so liability is limited.

Still, I'm considering calling the lost/stolen line of the card.

So here's what Linode support is actually saying when asked about the breach:

   Thank you for contacting us. We have no evidence at this time that any payment information was compromised.

I asked about the security measures and they answered with:

"We appreciate the response, and we can assure you that we have implemented all appropriate measures to provide the maximum amount of protection to our customers."

Yeah I did the same thing, but new card takes 5-7 days in my case. Very disappointed that I had to find out about this incident here, imagine all the customers who dont happen to look on slashdot or hacker news.

Do this immediately if it is a debit card!

If it's a debit card that can be used as a credit card (and it must be, otherwise it couldn't have been used to pay for Linode), then it enjoys the same protection as regular credit cards when it's used as one.

What about for people who did not use a credit card to pay for linode but instead relied on PayPal. Should they follow the same steps? What about other cautious steps?

I don't think they allow paypal as a payment method

Quite. I like the company and their servers are good - but we need a detailed response, and we need one now.

I'd say support tickets or posting on their forum[1] may help try to get a response. But based one one of the support ticket responses posted in the comments in this HN story already, it sounds like Linode isn't allowed to release that kind of information yet. They may be waiting on the police and/or their lawyers to allow them to talk publicly about it. And if that isn't the gating factor, they are probably trying to determine what they are required and should share about the incident.

[1] http://forum.linode.com/viewtopic.php?f=20&t=9978

They may be waiting on the police and/or their lawyers to allow them to talk publicly about it.

Waiting for their own lawyers would be a particularly weak excuse. This is a priority, and they are responsible from conveying that urgency to their lawyers.

its a recursion! the forum points back to here

Anyone know of any good way to export linode images to other VPS providers? Seems like I'll have to be doing it manually.

If your using lvm you can create a snapshot to do this while online, if not just read from your disk (assuming sda here):

   1. (offline) Boot new and old VM servers from live CD
   2. old server: dd if=/dev/sda bs=8M | pbzip2 -c | netcat <newhost> <random high port>
   3. new server: netcat -l <same port> | pbzip2 -cd | dd of=/dev/sda bs=8M

Compression: You can use something besides pbzip2, maybe pigz of if you only have a single core use bzip2 or gzip. Security: You probably want to add encryption to this pipeline.

Generally speaking, this is one example where having a good deployment system starts to look extremely valuable (along with tested backups and restores for non-deployed data).

Check out http://www.cloudconverter.com - move from Linode to pretty much any other provider out there.

You mean the comprised image?

Please do realize: lack of suspicious charges on your credit card is not evidence that it was not stolen.

Aah!, the self-fulfilling nature of paranoia.

How about you guys cool it and stop organizing a lynching mob devoid of any real data? It's embarrassing. HN is supposed to be populated with lots of very smart, data-driven analytical folks. Yet, every time something like this happens out of the woodwork come people who would ran you and your children down in the event of an emergency rather than turn around, carefully evaluate the situation, and help you. Don't be a moron. Stop it. For all you know there's a serious law enforcement effort under way that prevents Linode from talking.

For the record, I am a Linode customer and just got a new server to migrate a couple of sites into. My plans have not been altered at all by this. I have no data to suggest I should.

I am a customer too, and I wield no torch or pitchfork, but I grow increasingly frustrated at the lack of response from Linode.

I understand your point about the idea that they may be unable to speak to the issue due to law enforcement efforts, but for the moment, acknowledgement would be satisfactory. I would be happy with, "We're aware of the rumors regarding the intrusion at Linode this past week. We are working with law enforcement and cannot comment on details at this time. However, we will provide a full postmortem once we are able to do so."

The problem is when the explanation never comes. It's OK if it's not this second, but tell us it's coming, and then follow through. Complete silence is frustrating.

Here's the response I got from my support ticket:

>Thank you for your inquiry, and I certainly understand your concern. We are still conducting an active investigation and unable to disclose most information at this time. This being said, we do not yet have any evidence that any payment information of any customers have been compromised. We will be releasing further information regarding the incident soon, so please keep watch of our website and blog for said information. If you have any further questions, please feel free to ask.

That actually sounds pretty close to what you're asking for, although I have to say it didn't make me feel much better. It would be nice if they would make a public statement too.

I get it. It's just that these things keep getting exaggerated, twisted and blown out of proportion in the best Fox News, CNN, MSNBC, et al. style. It's sad to see it happen on HN, where things ought to be far more analytical and data (aka: facts) driven.

Remember this is the SECOND time it has happened.

Before it was the widespread hacking that resulted in Bitcoins being stolen. And users were never told exactly what happened and what was done about it.

People have every right to expect the worse with a company with a track record as poor as Linodes.

Mob-mentality. Unfortunately naturally occurs whenever a group gets to be about the size of a mob.

I don't think that quite explains it. Someone who calls himself "ryan" on IRC makes unsubstantiated claims, and in response many dozens of people say loathsome, sneering things about the security practices of this company. The appearance of a mob emerges after a thread exists, it doesn't create the thread.

The IQ of a Mob = (The lowest IQ in the Mob) / (The size of the Mob)

This from a Terry Pratchett book. Can't remember which one, but it's one of the earlier Discworld books.

Reading through the IRC chat log it didn't look like he'd proved he had the CC numbers.

I mean, if I was a linode customer I'd definitely be on the phone to the bank, but this guy presented no evidence I'm aware of that he had the kind of access he claims.

From a purported abridged chatlog with the alleged hacker:

> 05:42 < ryan||> credit cards were encrypted, sadly both the private and public keys were stored on the webserver so that provides 0 additional security

> 06:00 < ryann> They did try to encrypt them, but using public key encryption doesn't work if you have the public and private key in the same directory

http://turtle.dereferenced.org/~nenolod/linode/linode-abridg...

Here is what Linode replied to me when I asked them about that chat log in a support ticket:

  Hello,
  
  Thank you for reaching out. We appreciate and understand your concerns. At this time the evidence suggest that this activity was targeting a specific customer. We are unable to release any additional details regarding this incident at this time, as there is an ongoing investigation.
  
  We have no comment regarding ryan*'s comments in #linode. You are of course free to take any steps you deem prudent or necessary to ensure the integrity of your online presence.
  
  I am sorry that we cannot provide more information at this time. As always feel free to contact us at any time with any future concerns.

  Regards,
  Quintin

These guys are looking totally incompetent at this point.

If you believe this Ryan guy, credit cards stored on the same server as the key to decrypt them, Lish passwords stored in plain text, they've known for some time and lied about what actually happened and now they're saying "we won't do anything about it" via email?

"You are of course free to take any steps you deem prudent or necessary to ensure the integrity of your online presence."

Unbelievable.

Edit: not to mention they "made a deal" with the hacker not to tell anyone? What the hell?

> If you believe this Ryan guy

That's a rather key assumption. If you don't believe him, then all you have is a trolling (or at least self-aggrandizing) hacker whose credentials consist solely of logging into an IRC channel, refusing to identify who he was working with, and offering no tangible proof of having compromised any CC info.

On the other hand, it's conceivable that if ryan managed to get into the files a customer was hosting on Linode, and that customer was improperly storing CC info, then their customers' info would have been vulnerable, and ryan's claims would be sort of half-true. Even so, that wouldn't directly affect other Linode customers or put liability in Linode's lap.

Regarding the "made a deal" assertion, I wouldn't take the IRC log hook-line-and-sinker. There's probably a mixture of truth and lies.

To be fair the hacker didn't say the keys were stored on the same server as the credit card numbers, he said they were stored on the web server. It's most likely the database containing the CC numbers resides on a separate set of boxes than the web servers.

Despite what the other replies here are saying, this seems like a perfectly acceptable response to me. This comes off to me not as they're refusing to talk about it, but they _can't_ talk about it, presumably because of an ongoing investigation. I'm not sure what else people here are expecting them to say.

Then why not say that?

That is an absurd response. I don't care if they believe a specific customer was targeted, I want to know what happened and what information may have been compromised.

They probably can't say anything just yet. It's fair to be mad about them not realizing (or worse, covering up) the breadth of this breach, but do keep in mind that if they're working with the FBI, they've probably been asked to keep a lid on their official response for a few hours.

That is not the way to handle this issue. I've found my one problem with Linode is they are arrogant. It comes off pretty strong if you ever ask them questions in chat.

One thing to note is that the irc channel if that is what you mean by chat, is mostly populated by non linode staff. And many of us there tend to be sarcastic as we idle there and chat about all sorts of stuff while we are bored at work or whatever. Unless it's someone with ops, you aren't getting an official reply and even then for something official they usually refer to ticket system.

This is ridiculously unprofessional.

Well the steps I would like to take regarding linode aren't exactly legal. Satisfying yes, but not legal.

But I just checked and my credit card haven't been used anywhere I didn't use it.

"Don't worry, all the doors are locked."

"Where are the keys?"

"In the locks."

Wouldn't doing that be a massive PCI violation? Aren't there extensive audits for this sort of thing?

Extensive PCI audits. Heh.

Like the "scan" that said our linux boxes were running an out-of-date version of IIS.

That compliance web form I absentmindedly clicked through sure had a lot of buttons.

Companies are split into PCI Levels based on how much money/customers they handle. Level 1 are big companies like amazon, level 2 are medium sized online retailers generally, and level 3 are smaller retailers.

The 'lower' your level, the easier the PCI audits are. If you are level 1 you have mandatory external audits. If you are level 2 you have a 'self assessment' which is basically a checklist which says "Yes, I promise I'm in compliance".

If you have a confirmed breach, you are upgraded to Level 1 merchant audit requirements. This is generally quite costly as the external audit is extensive and must be paid for.

I've done PCI "audits" for several companies I've worked for; it's a checklist you go down yourself. That's why its called a "pci self assessment".

Actually, if you're processing cards directly, you do in fact need to have an PCI-qualified outside firm† (a QSA) audit you for PCI compliance. But those audits are notoriously superficial; PCI audits are a race-to-the-bottom affair.

† We are not one of those.

Most PCI audits are not mandatory unless you have had a breech that compromises data. Before that "self-certification" reigns.

The audits are toothless and ultimately the audit only happens once (if ever) and people keeping pub/secret key in the same place unprotected... well.... They're just unlikely to get security at all...

The paste-bin link he provided seems to time out for me.

But if the things he claim in there is even half way true, nobody involved with linode should ever be allowed to be in business every again.

pastebin is a directory listing of linode.com - trying out a few of the files checks out, including very difficult to guess file names such as:

http://www.linode.com/y_key_57284cb2de704e02.html

Am I the only one who is more confused about why there are compiled java classes and AMI BIOS updates in the www directory than about the hacking itself?

There is literally zero evidence to suggest they compromised anything more than a webserver.

That's totally shit.

Its also why we invoice and take wire payments rather than storing CC details. There's just so much to go wrong.

Also PKI is shit for this sort of thing. As demonstrated, the moment that public key is gone, then the whole system falls like a house of cards. For the non believers of this fact, why else would there be a certificate revocation list and root CA updates for windows periodically...

Using a processor who stores the card number outside of your infrastructure (ie. Stripe) can also be helpful.

Even better to use someone that isn't your payment processor, so that should you need to change payment processors you don't also have to re-acquire all the billing info from your customers. You can use Stripe today, PayPal tomorrow, and Braintree the next if that's what works best for your business.

Card vaulting as a service: https://spreedly.com/ ($10/mo for up to 5000 cards)

Until they have a security breach.

In what ways are wire payments better than using credit cards? In wire payments aren't you using the actual bank account numbers along with routing numbers which is also very sensitive information ?

Also I do not think, but I am not sure, that fraudulent wire payments/transfers are reversible.

They're not. You're right.

Wire transfers often are not able to be undone once they happen (and are accepted by the other bank). This is the reason why there's so much verification that happens in wire transfers. (I helped develop 2nd factor authentication used for authenticating wire transfers for a financial company)

Credit card charges can be reversed.

http://www.reba.net/news/wtransfer

My Visa card that I used with Linode was stolen and used on an Amazon order I didn't authorise last week, my bank successfully blocked the charge. Someone else reported their Visa had also been compromised in the thread 2 days ago, looks like that confirms the suspicions: https://news.ycombinator.com/item?id=5542015

Poor show Linode. (edit: worth noting I use the card with other things too, I have no confirmation it was leaked through Linode other than the compromise happening at the same time these supposed leaks happened).

I have two accounts with them, one for my day job and one through my LLC. Right now neither card is showing unusual charges.

My day job had some Google Apps account compromises last month, and this is making me paranoid that the database that contained hashed/salted passwords for our Intranet hosted on Linode was the culprit. The time frame doesn't seem to line up, but we didn't see any evidence of phishing or compromised desktops being involved.

(Don't want to spread fear - I haven't been able to find any evidence our Linodes were compromised either.)

I'm normally huge a Linode evangelist, but I'm severely disappointed with the lack of transparency on this. I'm debating right now whether to rebuild all our nodes from scratch as I'm not sure they can be trusted.

I had a CC problem, and I use it for a very restricted number of online services. Luckily the fraudulent attempt was blocked by the CC company. Linode was not at the top of my list of suspects (there was another company that seemed to be storing passwords in cleartext), but now I'm wondering. This was a couple of months ago though... I wonder if that's the right time frame?

Switched to https://www.digitalocean.com/ last week. Excellent service and pricing.

Yeah, a week sounds like plenty of time to assess a hosting provider.

Oh, and the years of Linode assessment were adequate?

Until they're hacked, too...

Or until they let other customers see all the data on VMs that you've shut down. Oh wait, that already happened: http://www.wired.com/wiredenterprise/2013/04/digitalocean/

Unlike Linode, they accept PayPal.

was not convinced by the website, looks nice but prices are kind of very cheap ?

also a twitter feed for the customers page ?

Great, now I am feeling paranoid although I don't see any unauthorized charges on my card. Does anyone know if debit cards are legally protected the same way as credit cards with 0% liability.

I'm pretty sure that depends on who issued your card.

Visa has a zero liability program for debit card - http://usa.visa.com/personal/security/visa_security_program/...

These are the FTC's rules [1], I'm not sure if Visa or Mastercard can make them 'better' (give you a larger window). They have an interesting tidbit below their chart -

>If someone makes unauthorized transactions with your debit card number, but your card is not lost, you are not liable for those transactions if you report them within 60 days of your statement being sent to you.

Isn't it free to get a new card? That'd be the easier way than worrying.

[1] http://www.consumer.ftc.gov/articles/0213-lost-or-stolen-cre...

Generally these days they are, but the problem is the money is removed is from your account by the time the charge appears, (at least a period of time), whereas on a credit card you have 30 days to review charges. This could cause overdrafts, etc. depending on timing and amount. Those too can generally be reversed, but the whole thing becomes more of a headache. I never use debit cards for any kind of recurring charges, and I avoid using them online in general.

Debit cards have less protection.

Wouldn't hurt just to ask your bank to re-authorise it anyway? It will change the three digits on the back.

they typically have the same protections, the only difference is that while credit cards only check that the charge can be made during the authorization period, debit cards lock up the bank's funds immediately which may cause your checking account to be temporarily unusable even if all the charges are successfully disputed.

a good practice that bankers constantly tell me is to have a separate credit card for online purchases for the fact alone that it is one step removed from your checking account.

They don't, protections are exactly the same.

I would cancel that card right now. IMHO You should never ever ever ever use a debit card anywhere else other than the ATM. Credit cards give you way more financial protection.

Unless said ATM has been compromised :s

You should talk to your bank as it depends on the network your bank uses. Most of the time it's something like a 48 hour window to challenge charges but it's far less at some banks.

No, it's covered by the FDIC. Banks can offer better protections but at least 48 hours within discovery is mandatory. http://www.fdic.gov/consumers/consumer/news/cnfall09/debit_v...

They are. Protections are exactly the same (in the US at least.) You have protection from the moment that you learn of the problem, not when it happens.

Not that having your account drained doesn't suck, but your worst case scenario there isn't terrible unless you fail to check stuff and be responsible.

Nothing on my UK Mastercard.

FWIW, no charges on my card used with Linode at this point.

If this is true then all the trust that Linode has built up over the years was just thrown out the window. According to the hacker they've known for 2 weeks and made a deal with the hackers. Ultimately, they were as far from transparent as it gets and on top of that they did a horrible job with their security.

Hopefully, they own up and start being transparent.

If this is true then what alternative hosts should I look at, besides AWS?

Two alternatives often mentioned on here are DigitalOcean and RamNode.

I've only used DigitalOcean. My anecdotal experience from running a Chef Server on a 1GB instance has been pretty mixed. The price is good, but network and CPU performance feels very variable to me. A month ago their Amsterdam servers were unable to be resized, and there was nothing about it on their status page. I tweeted and was told they'd be working "some time later today". Doesn't fill me with much confidence in general.

I'd still choose Linode for anything of importance - their long reputation is well earned in my opinion. But, if this breach is true, I hope they handle it well.

>But, if this breach is true, I hope they handle it well.

That is unfortunately the problem though. If this is true, they have already handled it terribly as it has already been 2 weeks since the attack.

Yeah, I just migrated from Linode to Ramnode (just in time!) and I'm quite happy with the performance. Great network connectivity.

Yeap same experience here with DigitalOcean CPU performance. If CPU power is important to you, Linode's CPUs are a LOT better.

Don't be so logical please. This can happend to anyone in the industry.

"credit cards were encrypted, sadly both the private and public keys were stored on the webserver so that provides 0 additional security"

That's just poor security and 100% they're own fault. I accept that there are security issues with every platform, but basic security measures and being transparent is still expected. My biggest issue with them in all of this is not being transparent.

What are they supppose to say?

Looks like someone who likes attention on some random IRC channel who is apparently a hacker may have hacked our system and we don't know who/when/where/why/how or what they may have got. Nor are we sure we were even hacked???

It takes time for people to investigate stuff. It's not just a couple hours. Also some random guys words on IRC (who could very well own INSERT RANDOM HOSTING COMPANY for all we know and looking to scare people off Linode) should be taken with a grain of salt.

"Logical" is not a fancy synonym for "severe" or "unforgiving", and is probably not the word you wanted here. There are sometimes good, logical arguments that you can make for cutting people some slack.

I've been using digitalocean for a hobby project, and am planning on launching a more serious project with them. For the past two months I've used them (so, not much experience, but some) I haven't had any issues, and they are very reasonably priced.

I am using digitalocean, check them out.

Just like I can have application-specific passwords for my Google account, I wish I could have application-specific credit card numbers from my CC issuer.

If I had these, I would immediately cancel my Linode-specific CC# and reissue a new one. I would not have to worry that my other recurring bills will go unpaid, or spend hours dealing with tracking them down and changing them.

Bank of America provides this [1], as does Citibank [2] and likely others.

Paypal at one time provided this service as well, but it doesn't seem to anymore [3]

1: https://www.bankofamerica.com/privacy/accounts-cards/shopsaf...

2: https://www.citibank.com/us/cards/gen-content/messages/van/i...

3: https://www.paypal.com/va/webapps/mpp/security/general-freet...

Crazy, thanks for sharing. I use BofA and am interested, but here is the issue:

> Set your Valid through date for up to 1 year in the future

I'd really like it to be up to the expiration date of my card. It's probably worth doing anyway, I suppose. Thanks for the heads up. Given the zero-liability status, I'm surprised that banks don't promote this feature more visibly.

I've used the BoA and it's fantastic for one-time purchases. Like another commenter mentioned, it's only good for 1 year and you give a global cap. It would be nice if it was more permanent and allowed for a weekly or monthly cap so you could use it with subscription services or something like Amazon.

Nice. I wonder if any UK providers do this

My bank in Sweden does (Swedbank), so it's not a US-only thing. They even have an iPhone app for it.

The portuguese ATM network operator provides this for free (its called mbnet btw). You can even set expiry times and value limit, it's the best thing to use when paying for stuff online. Want to buy a 9€ game? Just create a 10€ card and use it.

Yup, I use this all the time. They even, recently, added support to multi-use cards with bigger expire dates.

I wonder how hard would it be to make a startup like this.

>I wish I could have application-specific credit card numbers from my CC issuer

Do some research on "virtual account numbers". I haven't used them myself, so I can't verify how well they work.

Visa have this (probably others too) that you can generate and either set a time or value limit on. Don't think you can set a monthly limit though unfortunately. They call it e-cards (in sweden at least).

Either this, or a assignable CVV codes. Something like that would be awesome.

That wouldn't work since CVV codes aren't sent with recurring transactions (they can't, since they cannot be stored).

Why can't they be stored?

Is it a legal requirement to prevent merchants without the CCV from using the credit card?

...oh wait

__should__ not be stored

Good point. Regardless, there should be __something__.

Remember when the Linode customer service portal was compromised which exposed everyone's VPS?: http://julianyap.com/2012/03/01/compromised-linode-vps.html

In that case, specific Bitcoin users were targeted.

It's pretty much why I don't trust Linode.

You can't trust a company which puts random AMI BIOS files on the main index directory on the main web site. You can't trust a company that can't even lock down their own Linode customer service portal (which could lead to a breach of each and every customer's VPS).

Perhaps history is fuzzy for people when new announcements come out or low prices are around.

Off topic but still relevant, but doesn't it seem a bit primitive that companies have to store you CC# for recurring payments? The one number that uniquely identifies your account and everyone you want to re-use it has to keep a copy. Couldn't the credit card company issue some unique ID to each vendor for recurrent payments? Ex. the vendor issues your CC# to the CC Company for charge and recurring process. The CC Co. responds by replying back with authorization and an ID unique to that vendor that says "Use this number for charging this customer again, but it will only work coming from you, so if you lose it, it can't be used elsewhere". The vendor then discards your real CC number.

> Off topic but still relevant, but doesn't it seem a bit primitive that companies have to store you CC# for recurring payments?

You really really don't have to. Any payment processor that isn't horribly incompetent does the unique token authorization scheme.

Storing CC #s for recurring payments is solely the domain of incompetents who have no business accepting payments from anyone.

What if you want to change processors? If you weren't storing the CC details, wouldn't you have to have customers enter all their details again? I imagine this could cause a drop in revenues due to people either forgetting, procrastinating, or just not bothering.

It's never cool to be actually- or quasi-locked into a vendor.

Most decent processors have processes in place for the transfer of CC numbers. I was involved with this process at a decent-sized magazine, it involved armed security, an encrypted hard drive in a locked container and millions of dollars of insurance. It's not an easy process, but is possible.

Some will let you export the data, but it's a hassle. They burn it to a DVD and ship it to you and you have to sign tons of stuff freeing them of any liability.

Otherwise you just ask customers to re-authenticate. Often you have a few months headway for a switch like that.

Said processors allow you to export credit card data to another processor without having to store it yourself.

It seems there are a LOT of incompetents then. Are the payment processors you're talking about something the vendor interfaces with in the background and not a third party that is directly utilized by the customer (eg. Stripe, Paypal)?

This is basically what Stripe does. The "CC Company" basically doesn't offer more than a simple yes/no API -- "approved w/ #" or "declined". It'd be great if they could do more, but that's where the opportunity for folks like Stripe lies.

I guess it just surprises me that the CC companies don't do it themselves. Is the additional overhead of such a feature so high such as to warrant the choice between no payment processor (x% fee per transaction) vs. third party middle guy (x% + a bit more)? Not to mention the money the CC companies would save by not having so many fraudulent transactions (assuming they come out of their pocket, I honestly don't know).

Well, you can get AVS checks back too. But in general the entire system is so far behind the times. It really needs to be overhauled with security as the focal point.

It's amazing that it doesn't work that way. You could argue that it's because of the amount of infrastructure already in place, but why couldn't a new system be gradually and optionally rolled out?

Stripe (and probably others) has functionality like this where the seller's server never sees the CC number, and developers can store a unique token to re-charge the customer at a later date.

Depends on what part of the chain you are. Most gateways/processors offer some sort of token that the end user uses.

You then put your trust in the gateway/processor to store the credit card. Which I assume is most likely behind the best possible security stuff money can afford. Since that's their entire business. One screw up and their gone.

The majority of CC names have "Virtual CC numbers" which is precisely this - You generate a new number, which links to your account, but can only be used for the merchant you specify.

That's what Stripe do.

But isn't that just between Stripe and the company requesting payment?

e.g: Acme, Inc. sends Stripe your CC#, Stripe sends them some unique token, and they store that; correct?

So Stripe still has your CC#, and is at risk.

So this is really just risk mitigation; what I think TP is suggesting we need is unique authorizations at the banking level.

Something on the order of virtual credit cards, or temporary tokens, which are ultimately verified by your bank [or in other words: the lender(s) making anti-fraud guarantees, etc.]

(e.g: this token is authorized for 24 hours up to this limit; this token is authorized indefinitely up to $xx/mo.; this token is authorized for 1 year; etc.)

No. Customer sends Stripe their CC number, via AJAX in the browser. Acme, Inc. never has it even transiently. Stripe return a token to the browser, which is sent in a POST to Acme, Inc., then they verify it server side with a private API key.

Edit: yes Stripe has your number, but since their sole business is about securing that information, they probably do a better job of it than your typical online merchant.

I had a VPS on linode.

I think that Linode did a big mistake here. Let's wait for a formal communication.

But this is the moment to support them. Yes, maybe sounds crazy.

When you host on any third party datacenter, you take risks that something like this could happen. So, deal with it. Check your credit card, if your receive something wrong, call to your card and that's all. But we need to support also the good work, and this guys do great work in the hosting business. Just my opinion.

Well said. It's a fact of life that companies get hacked. So it's no surprise that it eventually happened to Linode. If you flee somewhere else, all you're doing is hoping that the other company you run to won't get hacked rather than using any logical thought.

I can think of two good reasons why you should flee Linode. It remains to be seen if either are actually true, and until indications say yes, then panic is unwarranted:

1. If it becomes apparent that Linode is far more vulnerable to hacking than other hosting providers. But one hack alone does not prove this.

2. If Linode grossly mishandles the situation. There have been a couple of allegations to that effect so far, but nothing substantial. I don't see any reason to claim that they've done this yet.

Linode has already grossly mishandled the situation by not coming out with a complete statement about what exactly happened. I only read this news because it was posed here -- no email notification, no update on their homepage, no twitter, no nothing.

The alleged hacker has made serious and specific claims, and Linode has done jack shit; without more information, how should I proceed? I don't want to call my bank and waste time getting a new credit card (not to mention replacing a million and two services) without a confirmation and I can't get a confirmation because Linodes people are having a circle jerk (or whatever the hell they do).

> I only read this news because it was posed here -- no email notification

There was an email notification a few days ago.

Linode already said they can't comment because of an ongoing investigation. What exactly do you want them to say?

> 2. If Linode grossly mishandles the situation. There have been a couple of allegations to that effect so far, but nothing substantial. I don't see any reason to claim that they've done this yet.

Linode's handling of the Bitcoin incident last year was sub-optimal. This too has been sub-optimal, given that credit cards were exposed but all we heard on Friday was to change our passwords, and even that was claimed to just be a super-careful precaution.

Linode needs to start giving us some frank talk ASAP. They've already burned through a very generous helping of benefit-of-the-doubt.

> This too has been sub-optimal, given that credit cards were exposed

If that's even true.

If this is true, they've blatantly lied thus far in their communications with their customers. I don't think that deserves support.

The service is not the problem, hiding what happened and the continued silence on these accusations is.

This is not a mistake/technical issue, it's becoming an ethics/service one.

So what happens now to all the goodwill Linode has amassed through the years? Does it all turn to shite, almost overnight?

This sounds very very bad, and as a customer it's very off-putting.

[Warning: imperfect analogy follows.] It's one thing if Linode is like someone who gets drunk and crashes their vehicle. That's 100% their fault and they've burned any goodwill. In this case, however, Linode is like someone who was carjacked. Perhaps Linode shouldn't have been driving that type of vehicle in an area known to have people attempting to carjack every single vehicle that drives by. Perhaps they should have installed thicker bullet-proof glass. Or even have taken measures not to trust any locks that the manufacturer insists are secure but have zero-day exploits. Regardless, Linode is still the victim of unscrupulous criminals. Maybe they could and should have done more but the bigger question is now that they've been carjacked, what are they doing to ensure that the carjackers haven't installed anything malicious that still remains in the vehicle?

I think a more accurate analogy is to say that Linode is moving your important files from one location to another in their moving van. They park at a 7-11 to run inside and grab a snack, leaving the van unlocked. An intruder comes along, opens the unlocked doors, makes a few copies of your files and leaves. Linode gets back in the van, notices the intrusion, does nothing except tell you that "you have nothing to worry about, but you may as well change your locks" and then when the truth comes to light, they basically stop returning your phone calls.

"No comment."

This analogy is more confusing than illuminating.

If the allegations are true, then Linode was keeping encrypted CC numbers, with the decryption key in nearly the same place.

Trying to make the analogy more sufficient by incorporating this type of fact would only make the carjacking analogy more absurd. At the end of the day, an analogy is not needed.

I think we should really wait to hear from Linode until we completely dismiss them, at least.

Seems like it usually takes an event like this for a company to really crack down on their security practices. I would wager that Linode will be one of the most secure providers in the coming months. Whether or not people will trust them is another story.

That's pretty much how trust works.

I'd give another trust vote to Linode, anyway this could happen to anyone.

It seems to me that the fundamental problem is not that they got hacked (although it seems that storing a decryption key in the same directory as the encrypted data is over the top careless), but their response to the disclosure that they got hacked. I realize that there may be limitations on exactly what they can say, but they should be as open as possible on what may have happened, what they are doing to protect their customers, and what their customers should do to protect themselves. Customers taking action when there wasn't a breach is less of a problem than not taking action when there was, in fact, a breach.

That's true -- it could indeed happen to anyone who keeps encrypted data adjacent to the keys.

Linode really needs to make a statement about what happened with this hack, stating if credit card information was taken. A lack of communication does not help me trust them. I'd rather have them speak up as to what happened and know if I need to have my CC reissued.

FYI, I just learned from Linode support that accounts have both a default LISH password AND a default API key, which means that even if you've never set a LISH password or generated an API key before, you still should still go and reset them. This is not what I would consider expected or desirable behavior.

I've never generated an API key, and Linode showed it as blank. I generated a new one anyway, but I can't imagine how they'd have a default key and somehow not show it in the UI.

I know. I've asked for further clarification, especially since their email on Friday said API keys should be reset "if applicable."

Edit: Groan, here's their clarification. It's starting to look like they don't know what the heck they're talking about:

"Thanks for getting back to us. To be extra cautious it would not hurt to regenerate your Linode API key. You can do that in your user profile. Please let us know if you have any other concerns we can address."

After seeing your original post here, I also asked for clarification, and received a similar reply from support:

  The Lish password is set to a random string by default, however we would still recommend resetting this password even if you had not set one manually previously.

I had expected that if the password was not set, then password auth was disabled. I've told them that's what I want and have asked when it will be implemented.

Hacker News, wherein I find further motivation to immediately get my off-site services off of Linode...

Just rang my bank to cancel my debit card. Hate doing that. Now I have a week or two of failing payments, bills, etc to look forward to.

I will probably be moving away from Linode after this. The poor response to this and lack of full disclosure, plus reading that they're using ColdFusion (wtf?), means I don't feel I'll be able to trust them any longer. It's a shame because their UI and service is generally fantastic.

There's nothing wrong with ColdFusion, especially if you've had it around for a while. It's not as glitzy as Rails, but it works and it's still supported and modern. Besides, this isn't ColdFusion's fault. Leave because Linode violated your trust, but not because of the programming language they wrote their site in.

It's closed-source, made by Adobe and seems to have a bad security record - there are 3 things wrong with it.

Besides, it's not the reason I'm leaving - it just makes me question them. I'm not after glitzy. If anything, I'd have expected Linode to have been written in Perl or something.

(made by Adobe) ⇒ (closed-source) ∧ (has a bad security record)

The problem more lies with Linode not properly dealing with a disclosed ColdFusion vulnerability: http://breenmachine.blogspot.com/2013/03/cool-coldfusion-pos...

It may not have been disclosed yet when it was exploited.

That seems unnecessarily pre-emptive, especially since you are protected as a card user and don't know your card was compromised.

Not with a debit card - you don't get the same protections as a credit card and I'd rather just have a few days of hassle and then know my card is secure than be unsure and have to constantly check my account for odd transactions. Also, whilst I may get money refunded if taken from my account, if I miss bill payments as a result - that would affect my credit report and I don't know if that would be removed when I report the transactions as fraudulent.

Yes, you do.

The Debit Card was used in a Credit transaction, so Visa's general protections still apply. You can dispute any of the transactions if they were done through credit (which online ones are nearly 100% of the time).

Love the whole side discussion about bitcoin from the supposed attacker:

06:07 < ryannn> They say there's no 'central weak point'

06:07 < ryannn> Yeah there is, there's the developers

06:08 < ryannn> There's been bugs in the client that have allowed the blockchain to split previously

06:08 < ryannn> One could just backdoor the bitcoin client binaries, not the source.

06:08 < ryannn> Nobody would figure it out until it's too late

http://turtle.dereferenced.org/~nenolod/linode/linode-abridg...

I'm curious how he's going to backdoor the binaries running on my computer built by the launchpad servers without anyone noticing. Granted, I'm not checking the commit logs every time apt gives me a new version, but all the same.

Well I'll wait for a response from linode, but it certainly looks like they were very dishonest. I think I will close my account.

Linode is always bad about communicating with customers over breaches and outages.

So you're unfortunate enough to be a customer who had their CC leaked. So you spend 5 minutes changing your password (you use unique, non-formulaic passwords, right?) and 15 minutes on the phone to CC company to ask for a new card. Then you use your backup card for 2 weeks (you have a backup card, right?)

A month later, spend 30 minutes on the phone with CC company only if strange transactions appeared.

Not the end of the world. The CC industry is set up well to handle this kind of thing.

To dismiss this breach seems odd to me. The tech community in general has placed a lot of trust and faith in Linode over the years. The shareowners at Linode have surely been great beneficiaries to that. Part of that "unspoken agreement", if you will, is that Linode be competent at what they do and that means keeping your data and information secure.

If even an iota of what I read in the abridged IRC log is true, Linode doesn't seem to care much about security or protecting Linode customer data. I mean, storing "encrypted" card numbers alongside private/public keys? Really.

Sigh, really? Ok, you typed your credit card number into a web browser at some point. If your sole reason for doing so was "I absolutely trust the people on the other end of this socket not to do what 99% of all people handling credit card data do whether they pretend otherwise or not", instead of something like "hmm that reminds me, I haven't scanned last month's statement yet", then the problem lies squarely with you, the uninformed consumer.

I will happily dismiss this breach, not because they didn't make some amateur crypto mistake, or because they weren't using freaking ColdFusion, or because they were storing data in some nice compartmentalized form, I reject because this happens every single day and has done for decades, and there is an entire sub-industry built around its after-effects. If you don't understand this you shouldn't own a credit card.

If you type a credit card number in online not expecting to recuperate any damage caused from your card company, call them up now for clarification or cancel the damn card. That's equivalent to stuffing cash in an envelope and posting it to Nigeria because some prince promises he'll keep it in a safe for you. It's 90% the reason you should be using credit cards in the first place. Think.

Linode should not be rubbished here. They've got one of the largest VPS installs around, so they most likely know their shit. They make an ultra-common CC mistake that has happened daily for almost 20 years now, by companies large and small, got pwned due to a bug in someone else's software, and you think I'm going to play along with the righteous indignation bullshit here? GTFO.

Let he without sin cast the first stone. Despite 20+ years' experience I still cannot cast that first stone. I make bullshit mistakes like this every day, and despite your grandiose delusions you probably do too.

As for whiners complaining about their data suddenly being insecure, well, data security 101: you're making the same bullshit mistake Linode are making, and despite that you're complaining about it. If you care about data security in the "cloud", hosting it on a freaking VPS is not the way to do things.

You missed the step where you have to find all charges going to your old CC and then deal with moving every one of those accounts to your new one when it gets there. Hopefully you don't incur any late fees while you're going through the process!

Kind of sucks to have to spend hours doing that for someone else's oversight. It's not the end of the world, but it paints a clear picture about where a company's priorities are.

Maybe I'm weird, but I know exactly which binding credit agreements I'm in and how they're paid, and definitely none of them get paid using another binding credit agreement. :)

Its worth noting that charges can be credited to cancelled cards under certain circumstances. Happened to me and the bank said it happens regularly. True, its usually a painless process; but its not that simple.

In the real world our time matters, sadly wasting it is not something you can sue over.

The fact of the matter is that OP shouldn't have to deal with any of that. No matter how much an inconvenience, it's still an inconvenience.

Found it interesting that Linode uses Coldfusion. Wonder if Adobe has anything to say about the apparent 0-day.

If the hacker's claims are true (Would appear so, the directory listing checks out) then Linode really need to address this ASAP. Passwords are one thing but to have CC details leaked is even worse. I'm not familiar with CC processing but it seems like bad practice to store the encryption keys on the web server.

It wouldn't take a zero-day flaw in the Coldfusion stack for a CF application to have an undocumented vulnerability; in fact, it's much more likely that the vulnerability is in the application code than in the stack itself.

A patch has recently been issued (09 APR 2013) by Adobe for the various versions of ColdFusion:

"This hotfix resolves a vulnerability that could be exploited to impersonate an authenticated user (CVE-2013-1387).

"This hotfix resolves a vulnerability that could be exploited by an unauthorized user to gain access to the ColdFusion administrator console (CVE-2013-1388)."

http://www.adobe.com/support/security/bulletins/apsb13-10.ht...

Yep that's bad.

The hacker claims it to be a CF 0-day vulnerability:

> 05:05 < ryan_> manager.linode.com was breached with a coldfusion exploit

...

> 05:33 < Ruchira> ryan||: give us the link to cold fusion vulnerability that you are talking about

> 05:34 < ryan||> Ruchira: 0day

> 05:34 < ryan||> linode staff apparently failed to deduce it themselves and relied on chmodding CFIDE to 000

Depending on who you're talking to, an app-level vulnerability in a Linode management console might be called a "0-day". But it's true that a CF stack flaw is not impossible.

This is not a brand new 0-day. This is a bug that Adobe communicated and patched months ago, a bug that affected a lot of folks who didn't follow the standard practice of locking down an administrative directory on the website.

http://www.carehart.org/blog/client/index.cfm/2013/1/2/serio...

The basic overview is this: CF servers have an administrative portal at /cfide/. A bug in the scheduler code (think cron) allowed remote attackers to upload arbitrary code to the server and then execute it. Savvy attackers could upload their own backdoors directly into the administrative folder on the site and then execute that code to gain additional access.

As a Linode customer (admittedly only for a small VM I play around with) I have to say I've been impressed with their service and their prices of course, and I'm waiting for further confirmation about the depth of this hack. I was unaware Linode was using ColdFusion. It should be pointed out that CF is a very mature language, akin to ASP.NET. It is actively maintained by Adobe and used by a huge number of websites globally.

Source: I'm a long time ColdFusion developer.

I wonder if Linode has a requirement to have their CF admin site accessible outside their network (assuming, of course, that the attacker didn't first gain entry into the corporate network, and then attacked the CF installation)?

As someone who still maintains a very old CF application, I am sure to lock down access to the admin site via IP restrictions.

Was it a 0-day? The purported linode chatlog makes it sound like poorly configured CF:

http://turtle.dereferenced.org/~nenolod/linode/linode-abridg...

--------- 05:43 < ryan||> Well linode also had terribly configured coldfusion

05:57 < ryann> <cfif ListLen(cgi.script_name, "/") gt 2 AND ListGetAt(cgi.script_name, 2, "/") eq "linode" AND NOT ListFind("index.cfm,linode_edit.cfm,linode_resize.cfm,label.cfm,cancel.cfm,dc_choose.cfm,su.cfm,pastdue.cfm", ListGetAt(cgi.script_name, 3, "/"))> <cfinclude template="/members/linode/common/dsp_topNav.cfm"> </cfif>

05:57 < ryann> this code

05:57 < ryann> It's so dirty I feel bad reading it

I've never used CF so someone correct me if I'm wrong but I just read that code as rendering the navigation bar in the members area?

Also if you check out my reply below I've C&P'ed the chat logs where he claims it is a zero day:

https://news.ycombinator.com/item?id=5552992

Plus another commenter has linked to a security advisory for exploits in CF that was issued a few days ago.

Linode hasn't been very forthcoming in the past where security "incidents" are concerned:

http://arstechnica.com/business/2012/03/bitcoins-worth-22800...

http://forum.linode.com/viewtopic.php?f=20&t=8509

I had really hoped that they had changed their stance on incident management. If it's true that they suppressed information about a possible wide-scale compromise where customer data could have been affected, then despite everything else about their service that's so great, there's no way anyone should want to continue to be a customer there.

Given Linode's past behavior and the information provided in the IRC chat, I think there's reasonable suspicion that customers' password hashes were stolen and Linode wasn't completely honest in their recent email to customers.

Did you see http://www.linode.com/linode4.css and http://www.linode.com/linode3.css?

Nevermind the security concerns. These guys aren't using version control!

Renaming CSS files with numbers or whatever is one legitimate way to handle cache invalidation (not evidence of not using version control)

It is not the best way to handle cache invalidation. Why not /linode.css?v=<file_timestamp> ?

Is it just me or is each passing minute without an acknowledgment of this issue bad for Linode? There's tens of thousands of customers right now who would kind of like to know if they need to request new credit cards or not, or don't know about this and deserve to know that their VPS provider's credit card database has been compromised.

The chatlog does provide some evidence that it is indeed the hacker, but does little to convince me that he got CC info and Linode is not telling us the whole truth. The evidence he provides is just simple source code snips and the directory listing, which would be expected based on what Linode has told us.

This could very well be the hackers own submission to /. trying to get more attention for his hack by claiming he has CC numbers which I doubt he has.

Doubting the claims of a hacker after it's known that they compromised a system is really bad OpSec. Consider everything burned.

That's a fair point. I'm speaking more as a judge/jury view on the situation though. I don't think Linode users should panic and run for the hills just yet based off this alone.

I know the /. submitter (As he's the one who told me he submitted it to /. and he had the first /. link I saw), he's definitely NOT the hacker.

Except Linode didn't tell us that. Linode said it was a downstream customer's info that was compromised. Nothing about Linode itself.

I'm looking forward to seeing an official response from Linode on this. Hopefully they are fast and honest about it. I've been a happy customer for quite a while, but this is definitely a concern.

They weren't exactly fast and honest the last time a break-in happened. In fact even to this day nobody is quite sure what went down aside from tons of Bitcoins going missing!

They are already way overdue. Their support staff is pretty damn fast, so I'm sure they could have commented on this if they wanted to.

Alas, they have not.

I just blocked my credit card, and Linode will not get anymore of my money. Too bad, i enjoyed my stay very much, but no matter what their response will be, I doubt I will be able to believe them. =/

You think linode is going to be transparent about things?

I guess this is why they wanted everyone to reset their password 2 days ago.

https://news.ycombinator.com/item?id=5541915

Not sure how useful that reset was. All I had to do was type in my old password and then choose a new one. No email verification, no reset token, nothing. So if the password was indeed compromised, couldn't the attacker do the same?

Yes, he could. But he didn't. If he did that, the page would tell you that your password is wrong, and you'd contact support. The attacker probably didn't change any password, because it would be futile.

Also, the attaker could also change the account email. Neither tokens nor email confirmation would help.

The biggest problem is for people that used the same password on Linode and any other important service. (And, of course, all the CC stuff...)

It addresses the "attacker has a list of hashes that'll take him a while to compute matches for" aspect of things.

Not only that, it also makes me wonder about the free RAM upgrade from almost a week ago. Some people are reporting their Linode credit cards being used for fraudulent purchases as far as a week ago, so this might have been a move to gain some pre-emptive goodwill.

I don't know though... will wait until more details are available but will be keeping an eye on CC statements / VPS alternatives.

"Someone hacked us and stole customer details... quick, give everyone more RAM!"

Doesn't sound very plausible to me.

Logically, they couldn't have planned it all in such a short period of time. However if they did, from a business perspective it is very good plan. Without the upgrade, today some customers would have had two reasons to leave Linode, now they only have one.

I tend to think the timing is just a coincidence. They also upgraded CPU and bandwidth over the last month and there were rumors of upgraded RAM for a while too.

I believe that to be a coincidence. They have been performing upgrades for a while now. (Increased hard drive space, increased CPU, and finally increased ram.)

I think it was just an unfortunately timed third phase of their upgrade plan.

That may explains the seclist hack too... http://seclists.org/nmap-dev/2013/q2/3

If that's the same one, then that link implies Linode already fixed the issue on or around the 13th. So, I'm wondering if the silence that has some folks here up in arms is indeed because they've been instructed by law enforcement to keep quiet pending the investigation.

Now if we could only stay the torches and pitchforks for a while before this gets sorted out...

Holy cow!.

Is there any confirmation on this?

[edit] Just looked at twitter, this tweet doesn't look good: https://twitter.com/Jamiesingleton/status/322730588459114500

But it may just be random coincidence.

[edit again]

Links from slashdot article:

IRC chat: http://turtle.dereferenced.org/~nenolod/linode/linode-abridg...

Link in IRC chat (i think it is of linode.com's web directory): https://bin.defuse.ca/hq0Ay8RzpKdR6vQwYxnmhc

That tweet would suggest to me that someone's credit card was stolen via another method and used to purchase Linode services, rather than the other way around. Ie, I don't think that was related to Linode breach.

There's URLs in the leak that could never have been guessed, like http://www.linode.com/y_key_57284cb2de704e02.html (mentioned by Tomdarkness above).

Putting aside any opinions on his morals or maturity-level displayed in the chat, it's a fun read. I love how ryan keeps reconnecting after each attempt to kick him out. He must have a pre-verified list of proxies/compromised-boxes he can connect from so he's just burning through them as he gets banned. I found that amusing.

So, they asked us to change the password for reasons they cannot yet tell. The fact that they cannot lay it all out for our curiosity is driving most of people here off the rails. I don't have any problem with letting them verify & deal with the issue properly before releasing any public statement. Anyone who's ever been with them has never received less than perfect customer service. Can we give them a break, wait and hold our judging wands for a moment? It's not like they are staying mum out of spite towards their customers and the press…

I am an ex-customer of Linode and I'm still worried about this incident - Do they still store your card after you've quit the service? This is terrible :(

I've asked them this question. Here is the answer:

Credit card information continues to be stored in our database in an encrypted format, and the decryption key is not stored electronically. We are working on a process on remove the credit card details of past customers on request and can handle this for you soon if you would like. If you have any further questions or concerns please let us know.

So the answer really doesn't answer the question.

(It appears the answer is no, but given they haven't been forthright in the actual details of the hack, possibly not)

Thank you :)

I'm in the same situation...

As someone who has been a very happy and loyal Linode customer for a long time now, this whole situation paints an image of Linode I otherwise would never have thought. The fact they apparently had both the private and public keys for the credit card hashes in the same location as one another is beyond belief. The very fact that Linode failed to mention they made a deal with the attackers and then reneged on it all without telling anyone makes me sick. I don't want to bash Linode purely because everyone else is, I am legitimately concerned here that my personal details have been compromised.

I thought Linode was different but based on their lack of transparency in this matter, I'm seriously considering just moving all of my sites to DigitalOcean, Rackspace or even AWS instead. This makes me wonder who originally cleared them for PCI compliance in the first place. This is a huge violation of trust and now I've got to keep my eyes focused on my credit card statement for fraudulent transactions, the bank I am with ANZ however has great fraud detection systems and considering I'm in Australia any transaction should be easily reversible, but the fact there is a possibility my card could be fraudulently used saddens me.

Linode needs to come clean about this situation now.

Really off-topic, but still sad: This is a link to slashdot, but it's on HN's frontpage before it's on slashdot's front-page (if it'll ever get there). (And IMHO that's sad, because /. used to be top notch).

I've noticed before that stuff from the HN frontpage appears on /. one to three days after, but I've never seen it for links to slashdot :-)

I rather wish this was a link instead to the original thread rather than to Slashdot. It isn't a big deal, but it certainly would have saved an extra click.

Also off-topic: I've noticed that as well with Slashdot, which is why I lurk HN pretty regularly now. Plus, some of the front page material on /. does more to insight angry discussion, and the community has become increasingly more vitriolic.

At least here, even if someone's brash, they're fairly honest about it (in general). I've even seen a number of disagreements that have been respectful and cordial. It's sad to say, but that's a rare thing these days.

I've noticed the same thing for almost every tech-related subreddit, especially /r/startups and /r/programming. A consistent 2-3 day lag.

Just got a response from linode: somethings not adding up?

----------------------

dportalatin 30 minutes ago Hello,

Thanks for getting into contact with us about this. Linode has found no evidence that payment information of any customer was accessed. We have implemented all appropriate measures to provide the maximum amount of protection to our customers. If you have any other concerns we can address, please let us know.

Regards, Dolores

They're doing canned responses right now guys. Just got this exact same message myself.

you seem to be new to this, most companies lie all the time about the nature of their security compromises.

To those of you who have claimed that your CCs have been abused -- I checked mine (which I used to pay for Linode) and it hasn't been used to do anything funny.

I checked mine and nothing untoward has taken place. All the same, I put it on hold. Everyone should assume their CC is compromised.

(not knowing anything) - wouldn't it be possible to give the cc company a white list of clients that can continue to use it and block all other requests?

yet.

Is there another way of learning the URLs like

http://linode.com/googledebcc14d3c9f777a.html and http://linode.com/y_key_57284cb2de704e02.html

If not, it seems likely that he did have read access to the main website filesystem at least

The best part is at the very bottom of the log. A customer enters the IRC channel to ask for support after ryann (the hacker) finishes explaining the attack.

Customer: "hello, i forgot my password and linode's email reminder service doesn't work. i checked spam box but there's no email from linode." Linode Guy: "ryannn: can you give him the password?"

And right now we got confirmed on IRC that data is out, we do not know how much, 4 last numbers of CCs are out but that doesn't mean that full CCs are out.

E-mails and logins are also out it seems.

Last four digits of CC#s are often used to identify them to users ("Visa ending in 1234"), and are specifically OK to store in cleartext. So that's not necessarily a big deal.

OFTC has disabled Linode IRC channel:

* mode/#linode [+m] by tjfontaine

<tjfontaine> this is what I'm going to say, as a network representative

<tjfontaine> regardless of what has or has not happened with linode, OFTC cannot tolerate release of sensitive information with itself as that mechanism

<tjfontaine> this channel is moderated until staff determines otherwise

Channel now unmoderated. Edit: HTP came in and trolled (pretending to be ex-Linode), re-moderated.

Is this why Linode doubled the RAM? To bribe us and make us stay. I'm pretty pissed off about this and will be exploring other options. I'm not pissed off they got hacked, I'm pissed off they are hiding and not being forth coming about it. A simple, "We fucked up, we are going to take steps 1, 2 3 to fix it and reduce the likely hood of this ever happening again" will make me happy. I understand that any server can be hacked. I'm stunned that they are storing CC details on the servers, there are ways to go about this without storing them if you want recurring billing.

I'm stunned that you, and plenty of other people in this thread, are taking the anonymous IRC person's word as the gospel truth.

A link to an unapproved /. submission. Fascinating choice.

That explains why my LLC's business debit card was turned off due to a fraud alert.

Well, here's a data point: https://news.ycombinator.com/item?id=5554071

Since Linode has proven in the past that they aren't the worst at communication, I can only assume some entity really has them over a barrel, considering the curt and callous responses there.

I just requested a new CC. Can never be too safe. Not sure I'm going to stick with Linode now... sucks because they seemed to be doing so many things right.

I have the same sentiments, new cc and might move off linode, who I have been with for 4 years.

This is disappointing and scary. A friend on another forum posted that some guy on IRC told him the last 4 digits of his CC and his e-mail address. I just called my bank and cancelled my current CC and give me a new number. I really liked Linode too :(

FYI, the last 4 and your e-mail address are both visible in plaintext from your /account page in Linode Manager. Obviously, still disappointing and scary, but it doesn't necessarily mean that whoever has that information also has the full CC number.

I echo everything everybody has said. This sucks because we don't know one way or the other. This may be some asshat FUDing away, or this may be a genuine hack.

It's hard to get angry at anybody but Linode needs better auditing around sensitive data so they can tell people one way or the other.

I resigned to the fact that I'd find it easier to change my card details in 30-odd online shops than it is fight my bank to get my money back. Now I can't make any purchases for 7-10 days.

I don't know the skill level of the Linode folks, but could it be that they left a honey-pot CC DB out for this hacker to discover on purpose? Keys sitting right there?

I wonder if they've deleted CC details of previous clients. Is Linode going to contact all relevant customers? Seems like the right thing to do. Not everyone reads HN.

I was just told by a customer service rep at linode that I "shouldn't trust everything I read on the internet" when I inquired about the possibility of deleting my personal information from their server. This seems like an extremely inappropriate way to handle this situation...

I stopped using linode in mid February and I have not received an email. They also still appear to have my credit card attached to my account (and I had to change my password when I logged in).

Does any one know of other VPS providers who are about as good as Linode in terms of quality and price but do not have a history of security issues ?

Hmmm, Linode claims they emailed their customers about the password reset, but I never got an email (nor in my spam folder)

Saturday morning:

    From: "Linode" <support@linode.com>
    Date: Sat, 13 Apr 2013 00:11:09 -0000
    Precedence: bulk
    Return-Path: 6723614.1706014@e2ma.net
    Message-ID: <knuab.c9dae.xxx@e2ma.net>
    List-Unsubscribe: <http://e2.ma/optout/c9dae/xxx>
    X-Test-Mailing: no

    Dear Linode customer,

    Linode administrators have discovered and blocked suspicious activity on th=
    e Linode network.=C2=A0 This activity appears to have been a coordinated at=
    tempt to access the account of one of our customers.=C2=A0 This customer is=
     aware of this activity and we have determined its extent and impact.=C2=A0=
     We have found no evidence that any Linode data of any other customer was a=
    ccessed.=C2=A0 In addition, we have found no evidence that payment informat=
    ion of any customer was accessed.

    We have been advised that law enforcement officials are aware of the intrus=
    ion into this customer=E2=80=99s systems. We have implemented all appropria=
    te measures to provide the maximum amount of protection to our customers. O=
    ut of an abundance of caution, however, we have decided to implement a Lino=
    de Manager password reset. In so doing, we have immediately expired all cur=
    rent passwords. You will be prompted to create a new password the next time=
     that you log into the Linode Manager. We also recommend changing your LISH=
     passwords and, if applicable, regenerating your API key.

    The following represent best practices in creating new passwords:
    -- Avoid using simple passwords based on dictionary words
    -- Never use the same password on multiple sites or services
    -- Never click on 'reset password' requests in unsolicited emails - instead=
     go directly to the service

    We apologize for the inconvenience. If you have any questions, please do no=
    t hesitate to contact our support team at support@linode.com.

I got one, so there were at least some of those emails sent out...

Welcome to GMAIL?

So glad the CC I had on file expired this month. +1 for procrastination. Here's to hoping Linode gets their shit together.

Again! I seriously start to lose trust in the startups' security of the Silicon Valley...

They're based out of New Jersey, actually.

I'm most concerned about Lish access. I'm not liable for fraudulent charges to my CC, so all I'll suffer is inconvenience. I'm most concerned about access to my VMs, which can be a business terminating event if things go extremely badly.

No suspicious charges on my end, yet, but I did just find out that my bank (USAA) is finally offering cards with Chip-and-Pin, which causes the card number to get re-provisioned. No time like the present, eh?

I recently sent an email to linode support, and got a very murky response. I used my debit card on linode, and it was recently used on transactions I didn't make in random parts of the world that I'm not in, so I had to cancel it. My first guess was that it was linode, and all of the posts here make it more likely.

Essentially: I am a linode customer. My cc details were somehow leaked. Adds a data point here.

Why would you store credit card numbers? Almost every processor has some method to store them with the processor and all you need is a token.

When did this breach occur? The implication is that it was a couple of weeks ago. As it happens my CC expired at the end of March.

I've seen no evidence of passwords or credit card numbers being accessed.

The purported "evidence" is a list of supposed entries in a public_html directory.

https://bin.defuse.ca/hq0Ay8RzpKdR6vQwYxnmhc

Has anyone seen any evidence that this is not simply a hoax?

Well, yes: try some of those files out:

http://linode.com/MyAddress.class http://linode.com/y_key_57284cb2de704e02.html http://linode.com/ispcheck http://linode.com/k5_3

etc.

the card with my linode account has a minimal amount total allowed to charge. I'll go through my cc usage, but at worst they could've gotten a couple of ebooks. as a long time customer though, Im conflicted and motivated to move off linode to another provider.

If this is indeed true I will be very disappointed, I have had an otherwise great relationship with Linode.

I await for more data and hopefully an official response from Linode.

If the worst case is true, what are some good alternatives for Linux VPS hosting?

this is why i always use my "virtual credit card" on web sites that store CCs.

I'm just curious, is this in any how, related to the recent RAM increase ?

Doesn't look like it. If you believe the IRC logs, the attacker broke in using a "0day" exploit of ColdFusion, which is being used on the site. Not because of a flaw in RAM upgrades. Although I believe that Linode beginning to offer free trial accounts is probably responsible for the increase in attacks lately...

I dropped linode two months ago. Hope they have purged my cc information from their system. Well I logged in just now with old credentials. Cc information is still there, not so good...

Sad day to see that Linode is not immune to data breaches. That being said, I don't necessarily support the witch hunt that seems to have taken place on its reputation.

http://blog.linode.com/2013/04/16/security-incident-update/

When was this alleged hack supposed to have happened?

Fortunately my CC information on file is expired.

A funny comment on IRC:

    "One bit of good news from all this hubbaloo: more space will be opened on the new E5 hosts from customers leaving \o/"

Being a former customer I decided to contact Linode support. I wanted to know if we have anything to worry about as we used to have a credit card on file with them. Notice how they say "Decrypted" which to me is not the same as NOT COMPROMISED.

Here is their response:

Thank you for reaching out to us. We do archive customer credit card details. At this point there is no evidence that customer credit cards have been decrypted.

Reply on news.ycombinator.com