Compromised Linode, thousands of BitCoins stolen
bitcoinmedia.com
bitcoinmedia.com
"hosting critical sensitive operations" in particular. If you are doing "critical sensitive operations" you need a more secure solution and process which will cost you more money.
Under no circumstances can a hosting provider assume the liability for something like this.
The tradeoff you make for the low cost you pay is that you might have an issue like this because someone screws up.
You pay more for a safe to store your money (and for a safe deposit box to store your valuables) because it's important and you understand the risk involved in not doing that. If you have valuable jewelry many times the insurance company will only insure if you keep it in the safe when you are not wearing it and even the amount of days is specified when it can be out of the safe.
It's unreasonable to expect (and linode's contract clearly states as other's have mentioned) a hosting provider to have a liability over what you are paying them. Edit Add: Unless you specifically have an agreement in advance or that is what they promised or charged you for.
Before anyone reacts to this with any harsh criticism please think for a second what liability you would want for any mistakes that you make with your web startup or idea. You could either be charging zero or charging a small $5 to $20 per month charge. You might make a mistake. Are you willing to accept and even be able to insure for thousands or even millions in liability for those mistakes?
If a bank gets robbed, I'm not liable for the cash they steal. But how about if I've got cash in a safe deposit box and someone uses a fake id to get into it, and the bank doesn't recognize the fraud? That's trickier. And if someone robs my house and I've got a bunch of cash under my mattress, that's another story too. I know the analogy doesn't quite hold up because it's kind of like a bank and a customer engineering a safe together (eg both could be at fault for a break in), but there's got to be some responsibility on Linode's part.
If you co-locate your hardware at a data center and your staff competently secures your systems.
Three letter agencies and foreign governments could attack your data if they took it off line, but you monitoring should detect that.
I'd take issue with that summary. If I put things in a self-storage unit, and it gets robbed because some employee left his master key under the door mat, regardless of how bad of an idea it was to store my money there, that's still their issue.
Think about it this way. I could store my money under a table at McDonald's, in a self-storage unit, or in a bank. Clearly the self-storage unit should provide me some more security than McDonald's. So when an attacker gets access through some really trivial method that they really should be protected against, that's their fault; it doesn't matter that there was a better security option, because it's still below what I was paying for.
A good point but keep in mind that courts don't have the level of expertise to judge what in terms of security is trivial and what is not.
The person trying the case and/or the jury may very well be someone who uses "football" as a password.
Also there are multiple cases of the very best companies with supposedly the highest levels of security getting hacked on a regular basis (might be a small percentage but it always makes the news). Consequently any company defending could make an argument that "this stuff happens even with the best and brightest" and it might be believed. (Well anyway that's what I would argue if I was a lawyer..)
So the public could easily be convinced in the case of a technology company something that would never fly as far as a screwup at the self storage - something physical that they can relate to (like leaving a door unlocked which is easy to understand).
For a hosting company this is different. Especially because--it's hard to draw a line but I feel it's there--there's a difference between storing sensitive data and storing (what are practically) valuables/money.
I wonder though, I'm not clear on his set up (or business, even), but basically he was literally storing his bitcoin money on a Linode server? Since they're not a bank, nor a safe-deposit service, or are in the business of storing valuables (as opposed to sensitive data). I can't come up with a realworld analogy (they usually break down anyway), but wouldn't you want to wrap this data in an extra layer of encryption or something? It's not that hard to come up with some scheme so that people with root access to the Linode server can't do anything with it either. Since this is about (almost) real money, that's what I'd do.
Well of course that's never going to happen and the truth is the contracts of the more expensive provider no doubt also limit liability. (And all of this is in the TOS/contract etc). So what we are really talking about is who will do a better job protecting what you have AND more importantly who has more to loose if they screw up. (Small linode has more to loose but they also don't have deep pockets to pay. Amazon has deep pockets but access to magnitudes greater legal help to prevent having to pay.)
I think there's a _lot_ of "grey areas" here, and while I feel sympathy for the guy who's out ~$13k worth of bitcoins, I can't help but think he was "doing the wrong thing" relying on the security of an inexpensive vps to keep them safe…
You have to trust somebody at some point unless you're keeping the server locked in your own closet. It seems really bizarre to me to say that a hosting provider doesn't have a responsibility not to steal your stuff.
Firstly, I'd start asking whether a $19.95/month shared hosting* account is a "reasonable" place to store $13k worth of (effectively) cash. I'd be _very_ careful if I had that sort of folding-money-type-cash on hand, and would under normal circumstances automatically deposit in a bank account to mitigate the risks involved with carrying it around. And I'd usually take steps to not ever have that sort of value of cash build up or be required - the only transaction I've ever done of that sort of value in cash is selling or buying a car from an individual - and that's always been a direct from transaction to the bank type of arrangement.
If I had enough bitcoin value that it'd hurt to lose it, I would not (at least now in hindsight) store that on a machine that other people I don't know/trust have root access to. Maybe I'd keep my wallet on a usb stick in my pocket or in a safe at home? I think though that at somewhere near the $13k value the "right" thing to do is convert it to cash and take advantage of the existing banking system and its time-tested security and insurability.
(* Which is fundamentally what a linode VPS is, at least from anyone with access to the hypervisors point of view.)
There's a saying in many businesses with different variations:
"Price, quality, speed" pick any two.
So this would be:
"Price, security, bandwidth" pick any two.
People pick on price and bandwidth security is taken for granted to be commercially acceptable. The colo can easily figure out price and bandwidth they are clearly defined. Losses from a security breach not as easy - to many variables. Same reason insurance companies love to write life insurance but hate to write disability insurance. Life insurance is absolute (you know when someone is dead and tables exist to compute probabilities on when they will die). Disability is open to interpretation, fraud and other things. It's not black and white.
With respect for #3 it would be reasonable for a bank safe deposit box to contain a $100,000 ring maybe but not to contain a 10,000,000 ring.
That doesn't mean Linode has any legal liability in this case, just that your analogy is off the mark.
Really? (I'm reading that as saying you think it _is_ a reasonable thing to store $13k worth of effectively-cash-value in a $19.95/month vps account?)
Does anyone know what regulations like HIPPA or PCI have to say about the security of data stored on managed-by-3rd-party servers like VPSs?
Second, where are you getting $19.95/month from, anyway? I haven't seen the plan in question mentioned, and even if this particular VPS happened to be Linode's lowest-end, the last time I looked (a while back, granted), slush had multiple large VPSs with Linode.
Third, really, what does the price of the VPS have to do with it? You think as the cost of the VPS goes down, we're entitled to less assurance that an employee isn't going to bolt with our data?
Finally, HIPAA and PCI regulations are ginormously complex, but violations of them almost inevitably cost a hell of a lot more than $13k.
I'll freely admit I've only been thinking about this since reading this article, so I'm both not-fully-informed and I'm thinking about it as a response to some guy losing ~$13k, but to me it's _not_ reasonable.
(And my $19.95 number is perhaps hyperbolically chosen from their least expensive vps offering - but my assumption would be that the management/hypervisor back end would be shared across their entire infrastructure, so I think my argument holds, in that I'd expect the higher priced offerings to "only" have the employee-reliability-assurance of the cheapest vps…)
If you've got a vault that holds a massive diamond, and a little gold ring, do you become concerned because the diamond "only" has the protection of a little gold ring?
PCI doesn't specifically say anything about usage of a VPS. It does however speak about access to data.
If you have encrypted credit card information, you'll be asked to list those that have access to the encrypted information, they encryption key, and the key encrypting key. Then you'll be asked to justify their access.
I'm sure I could come up with with several other major violations, but this alone is severe enough that I can't envision a way that you'd pass a PCI audit.
As an aside, signs like that are the same as ones that appear in auto shops that say "insurance regulations don't allow you in the shop". I owned a company that did manufacturing and said a similar thing because I didn't want anyone in the machine area. I've dealt with insurance companies for many years they don't dictate things like that at least from my experience.
Your example is correct though the restaurant would be liable in the situation you cited because you actually saw an employee steal the coat. If you saw someone else NOT connected to the restaurant AND the coat was expensive you are probably out of luck. It's your property and there is no assumption that a restaurant protects you from acts of god.
The key is was their negligence on the part of the restaurant.
On the other hand if there was a large party at the restaurant and normally a coat check girl that's a different story (devils in the details with this stuff).
It seems to me that bitcoin wallets are a relatively new and not well enough understood risk. There are very few other "files" like them, in that an attacker copying them can deprive you of their value in a way that you cant protect with backups. I feel a big part of current "internet security best practices" are about minimising the risk of getting exploited - but with a pragmatic limit to how much effort you invest mediated by the excuse of "if we _do_ get rooted, we can always reinstall and recover from backups". It'll only cost you time, and perhaps some reputation, and may put assumed-private-to-you information in someone else's hands, but it hasn't deprived you of access to any of your data. That doesn't apply to bitcoin wallets, and example like this are pointing out flaws in assumptions people are making about appropriate ways to manage them.
It'd suck to be "that guy" who provides the object lesson in why we need to think differently about bitcoin wallets to just about any other file type we might put on an internet accessible machine, but we _do_, and I don't know whether we have an answer to the question "Is there a way to secure a bitcoin wallet on a machine someone else has root access to (either your datacenter's staff with physical access, or the people with hypervisor access to the hardware your vm is running on)?"
I _think_ the answer is "if you can't trust those people, you can't risk storing your bitcoins there". There's a reason people keep their money in banks, and not in train station luggage lockers. I'm guessing inexpensive commodity VPS's should be considered closer to storage lockers than bank vaults. I suspect the finance sector and/or fortune500 companies have hosting arrangements with companies offering bank-vault grade protection and reserve bank style insurance - but sure as hell not at $24.95/month.
With Bitcoin, once it's copied, you can consider it gone forever, irrevocably, in totality.
(Of course, my understanding might be completely off-base.)
But what really is a bit coin? I mean in physical existence. Is it just a file(plain text) with some data/metadata?
And stealing it means copying those files, and then deleting the source?
Which in case how is this any different than traditional bank account. My money in the bank is basically DB record. And that can be stolen.
The bank can then just say to every one 'look this transaction from db such and such is no longer valid'.
Can't bit coin do the same? I guess I'm missing something fundamental.
Can somebody explain this?
So they got access to these peoples keys and transferred ownership of the coins.
Most money supplies are regulated, but bitcoin isn't regulated. No one has the ability to say 'reverse that transaction', but it also makes the currency safe from inflation and interference by money printing governments and privacy snoops.
There's an economic myth that inflexible supply of a commodity gives that commodity when treated as money, stability. It does not, as looking at this graph of US inflation/deflation time shows (1944 is when the dollar stopped being gold convertible):
http://en.wikipedia.org/wiki/File:US_Historical_Inflation_An...
A few points:
1. The money supply around a currency not only contains the mined/minted instances of that currency, but also liquid currency-denominated assets, like customer bank balances. So money supply is not necessarily bounded.
2. Inflation/deflation can be considered measures of the change in demand for the currency. In times of deflation, holding money is valuable because it becomes more valuable.
3. From the above graph, you see that during the gold standard, inflation tended to be mostly balanced out by deflation in the long-term, so long-term inflation was low. But in the short term, prices were very unstable as inflation jumped all over the place, and far more unstable even than fiat money in the past three turbulent years that we've seen.
4. From the point of view of an economy, inflation and deflation are not symmetric; because of the value of sitting on money during periods of deflation, savers do not tend to invest their money but move money from investments to cash savings. This undermines economic activity. But in an economy with a rich range of investment opportunities, moderate inflation does not penalise acquisition of money and does encourage investment.
If you want a non-performing store of value and don't mind big fluctuations in value, gold is there and we know how to secure gold rather well. Bitcoins are another non-performing store of value with far more drastic fluctuations in value, and securing it involves the double vulnerability: physical security of storage media, information security of computations involving bitcoins. And it is much easier to accidentally lose bitcoins than gold, pirate tales notwithstanding.
Only funds that you have daily access to need be vulnerable to the latter point, as physical security (air-gapping) is sufficient when you do not need to -send- funds.
> And it is much easier to accidentally lose bitcoins than gold, pirate tales notwithstanding.
Strongly disagree. Can you keep N redundant copies of your gold? Combined with secret splitting, you could require that at least K of N secure locations be accessed.
I'm talking about protocol risk: e.g., the software that implements the protocol on some machine is flawed, so the cryptography can be effectively breached. Or there is some issue with the protocol, like but worse than the issue Kaminsky found with anonymity.
>Can you keep N redundant copies of your gold?
Have you ever found that your backups didn't contain what they were supposed to contain?
Gold gets stolen, but besides such things as costume jewellery, I think it doesn't often get lost.
So if you are a bitcoin business making bitcoin payments, at moment you cannot avoid the risk of having the wallet stolen if someone gains unauthorized access to your operating computer.
Probably not, _maybe_ trusted computing could help[1]. But it might secure enough if you had a special piece of hardware that stored the bitcoin-key and did all the signing operations. I believe things like these exists for Certifications and other signing-keys. I doubt VeriSign stores their root certificates on just-another-box :)
[1] In theory, TC could "anchor" your data to the hardware chip on the motherboard, and breaking it would require physical tampering.
Of course, sane people don't mix HSMs and VMs.
People who host on a VPS typically do so because they want to run complex applications that scale up, and do so without the cost or headaches of adding more physical hardware. Most people who use Linode probably do not do so to host static webpages, which can be done just fine on a shared server for a quarter the price. While Linode might be cheap, they do claim to be secure, and if they didn't it's doubtful people would host their apps there. And most apps do take some personal information; and a large number of them take financial details.
No one would use Linode to host their apps if they thought all accounts were rootable from a master login panel. And no one would use them to host static pages. So who would their customers be if people didn't expect their instances to be secure?
1. Can this interface be accessed from anywhere on the Internet? If so, why? If not, does that mean other systems owned by Linode were compromised as well?
2. Why can customer service representatives access and update servers without the client being notified and with minimal logging?
"Our investigation has revealed a customer support interface was used to access your account. The compromised credentials have been restricted and we are discussing policy changes to prevent this from recurring."
Linode can't just leave us all wondering about our own security while pouring over over someone else's Pastebins.
More accurately "according to somebody at linode"
That's funny. I know from experience in the script kiddie part of the Internet that it was sometimes exceptionally easier to hack entire datacenters (even ones worth millions of dollars) just to get into a few of their customers, especially if those customers secured themselves.
Hosting companies have very sophisticated websites sometimes, meaning that they're almost always vulnerable to something.
I know of an SQL injection in an very large U.S. datacenter's administration panel which has been there for at least six years. Six years and it has not been fixed, and maybe a dozen people have independently discovered it. The deeper you delve, the more you realize that at least a handful of people also have access to important upstreams/backbones.
It's a lot bigger of a mess than anybody realizes. A bit of advice: if you say you're secure, you're either lying or colocating.
Looks like a class spear-attack.
3 compromised emails? Very unlikely. They are all major contributors to Bitcoin, I think they know a little more than using the same password everywhere.
Linode will only send you a confirmation email if you enable the feature, otherwise tough luck. It's also been confirmed by the vice president of Linode to be a fault on their side.
More plausible to have broken web UI security than an entire bitcoin-community-wide targetting.
IF you have the IP whitelisting feature enabled on your account. It is not by default.
The attack was not via the consumer facing admin panel. It was the internal Linode customer service interface.
This also provides an interesting dilemma when it comes to such events. In this case the damage is relatively easily quantifiable, he got X bitcoins stolen so the damage is X times the bitcoin value at that time. Still, it could have easily been user personal data or credit card information, which would have made an evaluation harder to make.
One of the risks of using such a platform I guess and something that anyone who does it should consider.
As I consider launching my own Bitcoin business, I have wondered about where to host the bitcoind that serves as my business bank account. This event certainly makes me reconsider just any VPS.
They end result is something like bitcoin would seem nearly impossible to insure for in any reasonable way.
Isn't this akin a bit to storing a bag with $12K in a storage locker in a public space and then asking to reimbursed after robbers broke into the locker and stole the bag.
I think the author is hoping for way too much. The world doesn't work that way, nor probably can or should it.
Given that a bitcoin client was almost certainly running on this box, basically anyone could have connected to the bitcoin network and established this guy's IP address as a target.
If you run bitcoin software, you are advertising your IP address as a high value target.
I wonder if there are any insurance providers who'd be willing to provider coverage for this sort of event.
But isn't access to the customer service portal under direct and exclusive control of Linode.com?
Get a dedicated server (not cheapest you can find) and secure it with:
(install in this order)
APF - http://www.rfxn.com/projects/advanced-policy-firewall/
BFD - http://www.rfxn.com/projects/brute-force-detection/
rkhunter
Ideally, install rkhunter on fresh system, right after updates, APF, & BFD. Then update the binary check-sums with this command, if you know server is secure:
Update file properties: # rkhunter --propupd --sk
Run a system check to make sure it is known clean: # rkhunter --check --sk
Lastly, sign up for the security alert mailing list for your version of linux on your server.
If you want maximum security, be sure to password protect your boot loader and use an encrypted file system. This will make it very difficult for ISP to work on your server however!
Maybe if the customer service system had had two-factor security, this might have been avoided (i.e., customer service can access your account only if you read them your hardware token's code).
Requiring SSL/SSH client certificates even for intranet accesses might have deterred this attack.
I hope other cloud providers take note of this incident. This is a very interesting incident.
At the very least, I'd hope Linode implements two-factor authentication for their own logins. A customer-provided OTP would be great but you'd need a customer service reset tool for that when people forget, which would put you back where you started...
There will always be ways to human-engineer your way into any single host. Having a hosting provider just increases the attack surface a little.
This was an attack against Linode's customer service systems, which allow their support reps to reset root passwords. There's no reason for that system not to be protected by two factor authentication on top of heavy logging.
http://www.debian-administration.org/articles/152
You can also require a password AND a cert.
I mean, I'm very clearly not a crypto expert, but I do believe that this would be quite a lot like what Bruce Schnier calls 'the evil maid' attack. Instead of having a bootloader, you have a minimal Linux install, then you get a key to that minimal linux install, and that minimal linux install uses that key to decrypt your encrypted disk.
I believe that if that minimal linux install that does the decrypting is compromised before you log in, in theory, the attacker could then insert something in the code that runs after the data is decrypted to, say, send some of that data elsewhere, or, say, open a back door for them to log in and examine the decrypted data.
I mean, certainly, you are making the attacker wait for you to log in (assuming that they've only compromised the admin interface and don't have full root on the dom0) and that's another step you are forcing the attacker to take, and you do at least have a chance then of detecting the compromise and /not/ sending the key, so I'm not saying that it's worthless.
Of course, this is all protecting against a compromise of the admin tools that does not lead to a compromise of the dom0. If the attacker compromises the dom0 without rebooting it or otherwise disturbing the guests? they have access to your ram. They can snapshot both the ram and the disk and take apart the system at their leisure.
I don't think I've ever met a professional sysadmin who could defeat the evil maid attack. All the ones I've met would refuse to think about an attack vector if it implied they could not boot their servers.
Any sysadmin that claims to be able to protect against a physical access attack or its contextual equivalent is either lying or incompetent. In neither case should that sysadmin be considered "professional".
The evil maid attack would be useful against the system admin's workstation. Securing NOCs is beyond the scope of this discussion, but it is not difficult.
If you can memorize it, it is a bad password.
This may be oversimplified, but it's the correct horse battery staple.
Then that scrap can be securely stored somewhere else. (eg: In your "in case of death" box if you want people to have access when you die.)
If that's even the case here. We don't have much indication that the attacker couldn't have taken a snapshot of a running system (or that that will be how it happens next time).
So, sure, use a loopback crypto filesystem. It just doesn't provide much in the way of a security guarantee (but it adds some measurable costs).
I just read the release from Bitcoinica where they explained that the server accessed contained _only_ Bitcoinica's "hot wallet", and that no code, services, customer data, or other wallets were stored on the server.
If this was the case, why couldn't every access to that wallet, which, assuming the above is true, necessarily occurs on other servers, run a decryption on the file first? Even if you keep the passphrase and/or secret key in plaintext on the machines that run the code, the separation should prevent this kind of rogue access as long as the intrusion is isolated as these people claim.
There is really no excuse just to have a plaintext wallet sitting around anywhere anymore (the official bitcoin client now supports symmetrical encryption). Like credit card numbers, when a wallet is accessed it should be decrypted in ethereal storage like RAM and promptly discarded; it should never hit disk as plaintext. At least the same practices used for PCI compliance and credit card data should be used for btc wallets; preferably better since there is no recourse if your btc wallet is compromised.
Although, AFAIK, bitcoin 0.5+ support keys encryption. I am not sure if this available only through GUI or through API as well, but even if it is not available through API, it might be good idea to implement.
From the sounds of it, this was that other server. All it did was operate on the wallet. And if they used other servers, then those would have been the target of the attack.
And, no matter how much damn encryption they have, they rooted the box that operates on the decrypted data, thats game over. The only attacker you would be able to thwart with more encryption would be the one who is able to root a linnode VPS, but unable to extract the key or decrypted wallet from from software running on that box. Sure, there is probably some number of attackers in that space, but security is a game of diminishing returns, and there are different security measures to take that are a much better investment of time than stopping that small slice of people.
The server software was used for automating bitcoin spending, too, so you couldn't just save public parts of the wallet on the server.
The only thing that could be done would be obfuscating the encryption key to the data very hard in the code, but that's hardly a security, just obscurity.
Simply put, people trust Visa and MasterCard to safely manage transactions, for better or for worse. Regulation like PCI helps ensure that this trust is sound.
The Bitcoin community at large could really benefit from a set of published best practices for managing transactions. Anybody possessing an insecure wallet is ultimately a liability to the credibility of the currency.
*for hackers to get away with the entertaining virtual train robberies we've seen in the last year
Even so, bitcoin seems to attract every shark in a one thousand mile radius.
Afterwards, let's talk about your comparison. Is 'can be stolen' really something that the state can protect you against? Let's discuss it over dinner. Depending on the contents of the wallet I'd pay.
On a more serious note: Your mockery, while amusing, is unrelated to the problem at hand. 'Stealing amounts of $currency from private persons' is not a new idea or something that bitcoin is supposed to change?
1) I don't think PCI is relevant here. If you store bitcoins somewhere and they get stolen then this is, in my world, cash. It's your very own digital cash. Not a credit card. That's why I constructed a (probably poorly implemented) example of someone leaving a wallet full of shiny $currency notes out there.
2) 'What state?' WTH? Can I reply with 'What kind of question is that?' The state I'm coming from is called 'Northrhine-Westfalia' [1]. Now I'm living elsewhere and there are no 'states' here. I can offer the district 'Tel-Aviv'? The point is, 'what state' is invoking aggressive feelings towards your US-centered mindset.
(...) something that the state can protect (...)
US-centered mindset
The fuck? Firstly, I'm European. Secondly, I assumed you were talking about the State[1], not a particular state.
My take: Someone was mocking Bitcoins with "But all that regulation is evil and it's the freedom of bitcoin that gives it the power" and I tried to make a point saying that _no regulation is involved here_ (laws? certainly). This is a wallet, it got stolen. Your credit cards are protected, your cash is gone for good.
You invoked PCI and I was (and am) unable to make the connection, maybe again because of a misunderstanding? I'm talking cash. Bitcoins are cash in my world (or - at least their value is equivalent to cash, if you choose to sell them).
From there we went downhill and I overreacted. Yes, for me 'state' is exactly what you posted. Again, sorry for the lapse.
I invoked PCI because of the thread: the original post was from klodolph, who said:
(...) You can't process credit cards on a VPS and be PCI compliant (it's against the rules), but any moron can do what they want with bitcoins.
And to that gravitronic replied:
But all that regulation is evil and it's the freedom of bitcoin that gives it the power(...)
"All that regulation" only makes sense if gravitronic is talking about PCI, which was the only regulation cited by klodolph.
"In a public cloud environment, additional controls must be implemented to compensate for the inherent risks and lack of visibility into the public cloud architecture. A public cloud environment could, for example, host hostile out-of-scope workloads on the same virtualization infrastructure as a cardholder data environment. More stringent preventive, detective, and corrective controls are required to offset the additional risk that a public cloud, or similar environment, could introduce to an entity’s CDE.
These challenges may make it impossible for some cloud-based services to operate in a PCI DSS compliant manner. Consequently, the burden for providing proof of PCI DSS compliance for a cloud-based service falls heavily on the cloud provider, and such proof should be accepted only based on rigorous evidence of adequate controls."
From: https://www.pcisecuritystandards.org/documents/Virtualizatio...
Amazon:
None of that should be particularly difficult for a VPS provider as large as Linode.
This part isn't doable if you don't own the datacenter. Slicehost has a problem with this because they never owned the datacenters they used.
They could've defaced your site in high traffic times. They could've logged in and delete your projects on the VPS. Depending on your setup (they had root) they could've searched for your backups. They could've read your mail, if that machine is your infrastructure service - and continue from there (password reset, amazon, buy expensive stuff. password reset, twitter, damage your reputation).
In this case the bits modified were part of a virtual currency and had a more or less clear value. I'd say there could've been worse results of that security hole though and 'don't put anything on a vps' is not a solution.
Bitcoins are no real recognized currency. So you can trade them for USD -> Don't store it?
What about this great project I'm working on? All my stuff on the VPS, because that's convenient and accessible from everywhere. I spent a double digit number of days on it. I have a daily rate for working as a programmer. Don't store it?
You totally ignored (so hard, that I think you didn't read it fully) my post about issues that are harder to value even. Access to your mail can be devastating. Even if you don't store 'money' on that VPS. Putting a dent into your online reputation by messing with your life on the net is hard to value, but certainly damaging. Again, no 'money' stored.
Bottom line: You ignored my point or didn't read my post at all. You picked a line out of context and refuted it with a pointer to the argument _I explicitly tried to prove wrong_.
Also the value something is worth is what someone else will pay for it. You can't value a project you're working on as your hourly rate * hours worked, it doesn't really work like that.
2) 'You can't value a project you're working on as your hourly rate * hours worked, it doesn't really work like that.' Right. But it's totally okay to value ~worthless~ stuff you have according to market rates, although you didn't sell them yet? Why are we talking about ~12k USD here? That's just a couple of bits and bytes on a disk. Yes, he _could've sold_ that at a specific time for a specific amount equaling ~12k USD. He didn't. Why do you assign this value to a highly fluctuating 'currency' on your disk, but don't like me assigning value to a 'yet to be successful' project on disk?
Point 2, back in the day when currency was backed by gold reserves would you have said the coins and notes people had were also worthless? As they are not 'sold' yet into gold?
Also I don't say a project on your disk has no value, a project on your disk has worth, the amount it's worth is how much someone will pay for it. Because the project on your disk doesn't really abide by any fixed standards you will probably find it's quite difficult to sell it.
Third report is the biggest, Bitcoinica.com which is arguably the second-largest exchange. Their main site is hosted at rackspace, but their 'hot wallet' was hosted at Linode, and contained 10,000 BTC which were stolen.[1]
1: https://www.bitcoinica.com/posts/warning-please-do-not-re-us...
EDIT: Those not following this incident on the bitcoin forums might be amused that the attacker used the stolen bitcoins to form a transaction with a size of 1337 bytes. That's probably not a coincidence, since the size of bitcoin transactions are usually under 1kb.
http://blockchain.info/tx-index/2893660/d9804de366aa4c2a0156...
Where's the bcrypt/scrypt/whatever police in this comments thread?
Obviously, more rounds and unique salts per user would yield better results, regardless of the hashing scheme employed.
Now, how long would it take for people to take down bcrypt? in the scale of time bcrypt is all the way over there -> from SHA1 and cleartext. Bcrypt will protect you even as computers get faster. SHA1 a couple decades from now will be be compromised for 9 or maybe even 10 characters. No way with bcrypt.
They don't mention a work factor for their SHA1 hash storage, they could be using PBKDF2 for all we know.
If you're looking for something to pick on about Linode here, go after something else, this isn't the worst thing to come out today.
Also, was the nature of the attack just that the were able to login to your linode admin panel and from their root the machines and then loot your wallets?
Someone at Linode did it. Ran a script to see how many bitcoin files there were on all the machines (they probably do these types of queries for anti-virus/whatever anyways) and took a customer support password to log in and get the coins. If he did it right he still might be working there, as it is easy to get credentials from friends/coworkers (even though it should be really really hard).
I'll happily eat my words if that turns out to be what happened, but it is definitely not the simplest answer.
It is technically possible for an employee to do it because it seems (from the linked pastebin above) that is how it was compromised, an elevated account for linode manage was compromised.
As for an employee being the one that did it, that is probably the least likely cause.
I wonder how anyone can trust their linode systems after an admin account being compromised.
It would likely ruin their business to re-install everything, but that is the only way to know root kits have not been installed.
We can only speculate at this point.
The simplest answer is probably that one of the staff was subject to a targeted hack and a 3rd party gained external access to the CSR tools.
Possibly for an extended period of time. <-- This is the concerning part.
It's relatively unlikely an internal staff member would do something this dumb (but, not impossible. we've had this happen _here_ where I work, with credit card numbers, but obviously the person responsible was caught almost immediately).
The way I understand it the attackers were able to get access to the admin panel and invoked some kind of 'change root password' emergency stuff. The machines were rebooted it seems, which makes sense: The interface of Linode has probably/hopefully no access to the root password. Maybe this 'Reset my root' feature (now I'm guessing) reboots the machine in single user mode or passes init=/bin/sh to the kernel to reset the password once and reboots again afterwards.
Only THEN the attacker had access. But yes, he had root. The good (if you want to call it that) part of it is that this procedure rings every alarm possible. The real owner doesn't have the password anymore, as he'll soon figure out. It's everything but sneaky.
I DO wonder why root is allowed to log in at all, though..
He won't figure it out until he tries to login though.
Also admins that only log with ssh keys and don't use root won't be able to notice that, will they?
2) Very good point. In that case it might work undetected for quite a while..
Even if you fully disable root, that's not going to stop the init=/bin/sh script.
Even if you fix that (securing grub?) you're still screwed because it's a virtual machine, and they can just mount the partition to another VM, and pull all your data/reset root that way.
So, maybe if you have an encrypted partition, no root access, secure grub, and real hardware (it's probably possible to dump the VMs memory by snapshotting it, then pulling the key out that way), you would be secure against attacks like this.
With a VM? No, it's not nearly secure enough for very important things.
Total dataloss for me. But i fyou _do_ remember your dm_crypt password, I think you're safe against these kind of attacks
I'm not really sure if the author of the article expects to be compensated but if so, he's dreaming. Just read through their terms.
Next time he won't be so ignorant as to put something so sensitive on a server like this.
At least Bitcoinica is eating the loss, it's not client money that was directly stolen.
What sort of defenses can developers put in place to protect against admin panels?
I've used these sorts of techniques in the past:
1) Separate username/password system compared to the regular website 2) IP whitelist of who may even access the admin panel 3) Failed login attempts send an e-mail alert with a log entry
Any other recommendations or suggestions?
Wouldn't have prevented this but would likely have paged this unfortunate soul when his machine rebooted unexpectedly.
Especially upsetting is that I went to great pains to keep everything as secure as possible.
When that's plainly not true. Surely having a wallet stored on a VPS is a really bad idea, what with admins potentially having full access to hard drive contents? Wouldn't a PGP'd local copy be a better solution, or am I missing a trick?
Because sh*t happens...... whether we like it or not. Even if the technical requirements are light and it runs fine on a tiny linode, that might not be the right place from a security or integrity point of view, depending on the value of the app.
(for me, a digital wallet worth that much, I'd want at my home..... where I can control it)
A person or merchant receiving bitcoins can easily convert them out to USDs and still lose less in fees than the same transaction would cost compared to accepting a credit card or debit card payment. For example, BTC -> USD at most exchanges is around half a percent.
True, but to be a practical currency it is critical that the value remains relatively stable. A currency capable of dropping from $1000 USD to $150 USD in a very short time is clearly not stable.
Now, for merchants this might not seem to matter, as they might be able to always convert bitcoins to a real currency immediately and never hold them long enough for depreciation to matter, but if the currency is excessively volatile there will never be a buyer at the exchange besides speculators, which prevents the currency from ever being really useful.
Not just not critical, highly undesirable.
Bitcoin is up 400% over the last year (from $1 to $5/BTC), which has made it an excellent investment for other (smarter) investors not swayed by a bubble.
One corrolary you'll note is that because they are both illiquid and volatile, they even suck as a speculative investment: if you make a bad bet you can't get out of it.
Sell the BTC on MtGox and withdraw the USD via Dwolla directly to your bank account. No need to use Paypal!
MtGox's withdrawal limit can be raised to $10k per day if you provide a notarized government ID copy (IIRC). Dwolla's limit is $5k per transfer with as many txfer per day. So it would take 2 days for completing the withdrawal, plus a few days for your bank to actually post the transaction (thank the legacy financial system for these unexplainable delays).
Of course the very best way to do it is to actually buy a car in bitcoins... see the Bitcoin Market subforum and find a seller. I remember last year someone was happy to announce he was the first person ever to buy a used car with bitcoins.
I understand your feelings on this. But the fact remains that the using the "legacy financial system" I can move my money between investments on my etrade account with a latency of minutes. I can buy that car on a credit card or with a personal check with zero latency. Bitcoins aren't remotely there yet.
There may be some privacy or social justice reasons behind pushing bitcoins. They may be fun (I'm sure they're more fun for a hacker to play with than mutual funds). But they're not a serious option for someone looking to "invest" their money, and claiming they are is doing the people you're trying to sell on the idea a disservice.
Also I doubt you can sell stocks on Etrade and withdraw dollars to your bank in minutes. When I do this with my stockbroker (TD ameritrade) it takes at least 2-3 days because the transfer is made by ACH which takes a while to clear.
This latency is perfectly acceptable for stock market investors, therefore I see no reason why it would not be acceptable for Bitcoin investors...
http://library.linode.com/troubleshooting/using-lish-the-lin...
I've completely ruined networking and disabled root logins on a Linode VPS, but could still access that same VPS as root using Lish.
Hint: In a scam, there's deceit. The bitcoin devs never deceived anyone. The whole system is transparent, so if there's anyone who bought without understanding the risks, they have no one to blame but themselves.
(Note: No, I don't own any bitcoins).
But I still believe the ecosystem as a whole reeks of pyramids and other scams
Oh, sure, that's kind of inevitable, it's a result of the lack of constraints and oversight. But personally, it's a part of why I like reading about it - it still has that feeling of a "wild west", populated by pioneers and thieves. Kinda like the Internet as a whole a few decades ago. Of course, it also means I wouldn't trust it with my money.
I am sick of reading about it.
Sorry, but then... why not just skip the link? There are a few topics I'm kinda sick of too, but I just ignore them.
But then I also think casinos and lotteries are scams so I am kind of outside mainstream opinion on a few things I guess.
Again, my literal mind jumps when I read that ; ) I can completely understand that you consider them immoral and/or predatory, but there's no need to call them a scam particularly - FSM knows there are plenty of other immoral acts.
Replace "bitcoin wallet" with "medical history" or "credit card numbers".
What I don't understand is why does such a feature exist, why doesn't Linode require >16 character length passwords that are sufficiently random (or eschew password auth altogether), and why does slush (apparently from what I can tell) allow password auth for ssh AND allow root to login on ssh.
Well, depending on how they got Marek's password, it might not matter. If someone went to his apartment and saw it written down on a post-it...
Don't stop reading and comment with 'I call bullshit'.