Linode Manager Security Incident
status.linode.com
status.linode.com
Only 8 accounts were affected. Do not worry. Minor breach. Not much harm done.
It seems to me the truth is the attacker looked for bitcoin wallets and emptied them. The fact he could identify 8 accounts and access them suggests the attacker could have accessed far more accounts if they wished. I think this is the most worrying thing about the breach.
I don't really understand how bitcoin works but it seems that people with wallets need to set up multiple wallets on multiple providers and limit the amount of bit coins in each wallet to limit any losses from breaches like this.
If I was a linode customer I would be thinking about moving. This message, while fairly open, doesn't give me much confidence there aren't other security issues with the platform.
Basically, this announcement gives me no confidence that they've done due diligence in fixing this problem. They haven't explained what the vulnerability actually was, nor what they have done to avoid it in the future.
Of course, this does speak to the dangers of using hosted services for anything that needs a high level of security. Anyone with appropriate admin privileges on the host system can compromise any user. That increases the attack area considerably; you don't need to attack the system directly, nor the users of the system in question, you just need to find one person who has admin privileges who is vulnerable, steal their credentials, then attack any users at your leisure.
Basically a lot of people were renting storage rooms in an apartment complex run by Linode, you get your own key to enter the door and retrieve and store things -- whatever. Some people left their wallets inside these buildings, with cash therein. Someone else used some unidentified systematic security flaw, but we don't yet know what it was. Maybe there is a ventilation system which is easily navigable once you know how to get in; or maybe all of the rooms have unlocked windows for no good reason; we haven't been told yet. (There are some suggestions that they stole a key from one of the janitors who cleans these rooms up.)
What we have been told is that some burglar stole eight wallets, and that "All activity by the intruder was limited to a total of eight customers, all of which had references to 'bitcoin'." That suggests that the burglar did indeed peek in the windows beforehand somehow, to find out that these 8 rooms had wallets inside. Otherwise, presumably they would say something like, "The intruder broke into many of our customers' accounts but didn't actually do anything in 99% of cases." In that sense I think the scary bit isn't that he accessed the 8 accounts, it is the fact that he identified them in the first place.
Amortizing the loss across many points of failure may be a good idea, but it wouldn't seem to solve the central problem. Suppose I put $20 in two accounts with 5% chance of compromise, rather than $40 in one account with 5% chance of compromise -- either way, I should expect to lose $2. What I've changed is that I am more likely to lose some of my money (9.75%), but I am less likely to lose all of my money (0.25%). This may appeal more to risk-averse people but it is not fundamentally changing the situation.
Perhaps a better approach is to keep a BitCoin wallet encrypted, since that's pretty simple to do in day-to-day life. This is something that you can't do with your wallet -- you cannot turn your wallet into a steel vault with two-foot-thick walls.
This isn't all that surprising. There are basically two reasons why you would have a Bitcoin wallet on a server: if you are mining using the CPU power of that machine, or if you need to send Bitcoins from an online application. For example, one of the people who mentioned having coins stolen was from a mining pool; you need some automated system to pay out the earnings to the people who have been doing mining, and so the wallet for that automated system was on the server, and was stolen. I suppose one further reason might be as a backup, but in that case, I dearly hope that it's an encrypted backup without the encryption keys in the sever.
Given these reasons for having your wallet on the server, it's not surprising that people found them. These require network-facing services, that are easy to trace back to the server in question. The mining pool is a public service; anyone can join, and find the address of the server. Furthermore, when you make payments, you announce them to the full Bitcoin network. Someone sniffing transactions can watch where transactions originate, and target that. If they already had a compromised customer service account on Linode, they probably watched the Bitcoin network for a while, made note of transactions originating from IP addresses in the Linode range, and then targeted those accounts.
One way to protect yourself from this would be to proxy your Bitcoin transactions through a host other than the one that has the wallet, obfuscating where the transactions are actually coming from. You could even go so far as to make all of your transactions via Tor, which would probably make it fairly difficult to find where your Bitcoin wallet actually lives.
> Perhaps a better approach is to keep a BitCoin wallet encrypted, since that's pretty simple to do in day-to-day life. This is something that you can't do with your wallet -- you cannot turn your wallet into a steel vault with two-foot-thick walls.
The problem is, if you need to make payouts from your wallet, then the machine that does that needs to be able to decrypt the wallet. That machine can then be compromised to be able to steal your keys. Encryption doesn't buy you all that much, unless you are just doing a backup and don't need the machine to be able to do online transactions at all.
Perhaps another solution would be to encrypt each key in your wallet separately using a k-out-of-n encryption scheme (where produce n keys, any k of which can decrypt the wallet). You can then distribute those keys to independent hosts, which hopefully should not all be subject to the same vulnerabilities. Then any time you do a transaction, k of those hosts will need to produce their key to decrypt the key in your wallet and perform the transaction. That way you would have to compromise several different, independent hosts in order to steal the wallet.
Of course, this would drastically increase the cost and complexity of the system; and you would need to ensure that whatever system that authorized payments was likewise distributed, which if you had, say, a web-facing service would be difficult.
The easiest thing to do to reduce the risk is to only leave enough value in the wallets that are on the servers for a couple days worth of transactions. Then you transfer Bitcoins from a more secure location once a day to keep the coffers full. This is not much different than a physical store; yes, you are at risk of being robbed, but if you only have one days worth of cash there, with the rest somewhere more secure, you reduce how much risk you have.
Your more general solutions would protect from an attack that rooted a live system rather than just resetting the root password while the machine was offline.
I'm not disagreeing with anything else said here - it's just that it seems that the memo was very conclusive in stating something that couldn't be known unless there was more VPS introspection than they claim...
Implementation of two factor authentication for your customers and requiring it for a root password reset would go a ways to preventing similar attacks.
If someone went to that much effort just to steal some bitcoins, they set their sights too low. Linode must host more valuable stuff.
Such as? bitcoins are valuable and easy to run away with. Stolen credit card numbers are such a hassle to monetize that they can be bought with only $2 or $3 of e-currency.
I have no idea how Linode's support team access accounts, but I would hope it is less public.
I think someone found a way to gain access to any Linode customer account through the customer website and from there shut down the instance, changed root password and rebooted (you can do that from there).
I just wish they posted a little more, it feels vague.
Ah crap. It looks they have been moved to Amazon EC2, ~8 months ago they were hosted on conventional Linode VPSs. Points still stands though.
What I'm asserting is that the servers that store the actual banking and customer data have very high security standards. It's one thing to store front end website code on a VPS, it's a totally other thing to store your database with customer & bank data on Linode.
The bitcoin breach seems analogous to Bank of America storing your account information on Linode and trusting it as the Real Data. Does that make sense?
//reply to tomg, but seem HN stops nested replies beyond a certain level
At the end of day you can have millions of dollar of security, auditing, PCI compliance tests passing, developers that celebrate every Friday that everything is secure, data is hosted on premise etc... But if you leave the login page javascript to a third party hosted on Linode then you might as well be BoA storing your data on a mySQL linode instance. So in a nutshell it kind of undermines the work you guys do.
I also know there are "levels" of PCI compliance. The highest one, which reputable banks should be following, is very strict AFAIK, and includes provisions for controlling who has access to the physical hardware, encryption levels, etc. The fact that a Linode VPS can be 'rooted' via their management software by a sysadmin working for Linode would, from what I can tell, make them unqualified to be used to store banking transaction & customer data, though perhaps I am wrong.
The user who was affected by the incident quoted an email from linode that stated "Our investigation has revealed a customer support interface was used to access your account.", based on that and all the information of that post you get the impression that through the 'interface' the attacker was able to change the vps root password.
Now a reply from linode comes and says "The portal does not have access to credit card information or Linode Manager user passwords". So if the portal doesn't have access to Linode Manager how the attacker gained ability to change the root passwords ?
Thy should give more details on the incident, i do have a certain trust in the ability of linode to have a secure environment & i can understand that things like that will happen at some point to everyone. However its one thing for someone to get access in your system because you had your roots password to 'password' and another if there was a bug that got exploited.(yea this is an extreme example)
They didn't say that, they said it doesn't have access to the passwords. They have an interface to change details, they just can't read them. So they can reset your password to "hunter2" but they can't see if it's "hunter2".
The point is that exploited interface had a backdoor access to the virtual machines (to be able to change passwords or w/e)
Suspicious events prompted an immediate investigation and the compromised credentials used by this intruder were then restricted
Unless you believe they're lying, then Linode has the same thing. Some interface where they can access their users' Linode Manager accounts, but that interface does not show Linode Manager passwords or credit card information.
You can expect for your landlord to keep their copy of the key in some sort of lockbox, but you aren't going to expect them to keep it in a pressure-sensitive safe, guarded by movie-style lasers and a German Shepard.
You expect a little more security out of your bank.
The real question is: why were services that act like bitcoin banks storing their coins on Linode in the first place.
The only (realistic) way to generate bitcoins today is with GPU or other specialized hardware that doesn't exist on webservers.
These servers that were compromised were used to manage generated bitcoins. One was used by a pooled mining service (mined coins were sent to the server then payed out to miners) and the other was a faucet service which would give a little bit of bitcoins to new users. The other 6 servers that were compromised are unknown to me.
Linode falls into the second category, cheap, uninsurred storage space. You want storage space insurred against digital robbery, a) good luck b) expect to pay a lot more than $20 or $30 a month
As a policy, it would neither be good business nor appropriate for Linode to assume all of the risk in a situation like this.
Also, if Linode did put a policy in place to assume some of the risk (some sort of insurance policy) they open themselves up to scams (just get your friend to rob your bitcoins and cash in on Linode's good will insurance policy).
Creates a precedent and greater future liability.
EDIT: first report from bitcoinica was "over 10k btc". Most recent report is 43,554 BTC, which would be worth almost $200k if liquidated on MtGox at the moment.
And some selling is going on: http://bitcoincharts.com/charts/mtgoxUSD#rg1ztgSzm1g10zm2g25...
1337000000000.00 1 (1) 272981 1405085033491The payment card industry wouldn't certify this hardware to process credit cards for a mom-and-pop online business, yet these guys use it like their bank? Come again?
Why would a hosting provider take on the liability of what you host on it? If the underlying filesystem had an error that Linode could avoid and you lost data, would you expect them to replace the value that was lost? Why is it not limited (at most) to the value of the VPS itself? (I can't imagine them compensating for hardware failure either).
OP's expectation was that Linode should have unlimited liability where no guarantee on their part has been made. If you expect Linode to take on the risk then you would want (and probably need) that in writing.
Otherwise, you take on the risk: which would be true wherever you hosted it without any guarantees regardless of whether it's a VPS, dedicated server, or your basement.
It seems to me that this is a classic example of security failures following inevitably from a lack of peer review. Maybe Linode didn't consider its LM software to be peer-reviewable, but I bet the victims of the bitcoin thefts wish that someone else had tested the code (and human systems surrounding it) for vulnerabilities.
Is this not exactly what Bruce Schneier frequently points out? Anything that must withstand attacks to protect the valuables within should be tested by attacking it. A lot. My hunch is that the vulnerability exploited by this attacker would have been found and fixed already if the LM software were more open.
I would be willing to bet that they have had the system tested by external security contractors and scanned with automated scanning tools. This seems to be a people problem and features problem not a vulnerability problem.
I guess we just don't know at this point. You may very well be correct. I guess if you want to use an open source provider, just make sure they are running OpenStack (openstack.org).
Gaming (i.e. casinos), at least some of them, do a reasonable job with some very similar security problems.
osGold, etc. were probably better parallels. But yes, this whole field has had a long series of technical, business model, governance, and government related failures.
osGold was just an internal fraud; digital currency, give me your (gold) and issue tokens -- then one day it disappeared, taking the original value with it.
There were hack attacks by third parties against some other gold currency systems at the time, but I don't remember the details.
It shouldn't do significantly. In the public's mind the well is some distance off, many not even knowing it exists, so there is plenty of time to get security better sorted before the average man on the street has his money invested.
Also, everyone knows that cash and other forms of investment are far from safe anyway. Hopefully people will eventually see online currency as being no less safe than other forms, and if the security is done right they'll see it as more safe (as it could be).
But this seems to me to be a general security issue with decentralised anything, not a bitcoin specific problem. If you remove central control, and take as full control yourself as possible, then you remove responsibility from other people and that is something you need to seriously think about. Providers like Linode will not be taking responsibility for financial loss in these cases and paying for hefty insurance policies to underwrite the possibility of said loss: they'll just add clauses to their contract disavowing themselves of responsibility if such clauses don't already exist.
One way to protect yourself is to spread the money around multiple places, then one hacked provider doesn't put all your resource at risk. Again this isn't bitcoin specific: if you have more then 50K to invest over here (in the UK) then you split it between multiple organisations as only 50K per registered organisation is guaranteed protected by national safety nets provided by government.
Back onto "poising the well" this could of course work the other way around: if the virtual currency is worth the effort of stealing then it might be seen as more valuable in the eyes of the public - much like a sign you have a good product is that knock-off copies start appearing.
That said, there's no way that resetting the root password should be something a customer service rep can do. Particularly when Linode are very explicit about not being a service for newbs - and are in general unwilling to provide help setting up a Linux system.
According to the admin of bitcoinica, he didn't have the wallet encrypted because the ruby gem he was using to process withdrawals didn't support it.
I share a Linode with a friend and at work we deal with 4 so this is a concern.
That doesn't imply they will be posting a public update. As a Linode customer, I would like to know details of how this breach occurred and what actions will be taken to prevent it in the future.