I'm leaving Bitcoin | Hacker News Reader

I'm leaving Bitcoin

bitcointalk.org

85 pointszhoutong47 comments

"Spectacular failure is your destiny."

I thought jellicle's advice to Zhou Tong was insightful 248 days ago. Time has proven its wisdom.

http://news.ycombinator.com/item?id=2973803

-- systems that work with money are attacked hard and often, by intelligent skilled people

-- in fact some of the people who attack your system are likely to be both more skilled and more intelligent than you are

-- systems that work with money that fail, fail spectacularly ("What do you mean someone withdrew $8 million last night?")

-- banking websites, Paypal, etc. are all like icebergs - you don't see 9/10ths of the things they've done to prevent spectacular failure

-- spectacular failure is your destiny if you don't work very hard to prevent it

-- spectacular failure may be your destiny even if you do work very hard to prevent it

You should plan accordingly.

Wow - didn't realize on first reading that the comment was actually directed to Zhou Tong, who replied that he would "definitely pay attention to every single detail." Forensic's response was prophetic: http://news.ycombinator.com/item?id=2976031

EDIT: panarky has now updated his post above to make it clear that the advice was to Zhou Tong.

Thank you for your original advice. It's definitely useful. But it's not relevant today.

I don't wish to link the Bitcoinica fiasco with my leave because I sold Bitcoinica a few months ago. And I lost my control in January. Basically everything didn't go as I planned. (I'm VERY conservative about wallet security.)

I wanted to build an independent exchange system to replace hedging, so that most funds can stay offline. But the new owner didn't like the idea.

I know it failed like predicted, but my involvement in the failure is highly limited. I sold the site for the same reasons: What if someday I'm hacked, or caught? (The valuation (P/E) was less than 1!)

I'm a web developer, not a security expert. I know how to protect the API keys but I'm not good at encrypting a wallet.dat. That's all I thought since day one.

May I ask how much money in total you personally netted from making Bitcoinica? I'm intensely curious, but I'd understand if you'd prefer not to tell.

Unfortunately, I have signed an NDA on this.

These bitcoin hacks have amazed me every single time. The old-school financial industry goes to extreme lengths in order to protect the money it's in charge of. Physical money is stored securely and guarded. IT infrastructure is carefully guarded both physically and electronically.

By comparison, these bitcoin folks seem quite content to store what is essentially money on run-of-the-mill servers in run-of-the-mill data centres. Linode? Rackspace? Are you people fucking serious? It keeps happening, and I keep wondering why any thinking person would trust plain old data centre security staff with their money like this.

I keep wondering why any thinking person would buy bitcoins. Could be related.

They go through great lengths, except apparently verifying if you're authorized to access an account, as in the Citigroup hack. Or having two-factor auth. Or being able to use passwords with decent length. Or providing a better payment system than a single set of codes that you have to hope no company leaks, or you'll have to get a new CC (see the Global Payments hack), when even fucking Twitter knows how to develop an authorization scheme that can be individually revoked.

Seriously, the banking industry is hardly a paragon of security. Many startups give you better ways to protect your cat pictures than the average bank gives you to protect your money.

I have not seen an online banking without two factor auth though.

I assume you are not in America? PNC, M&T, and TD all do not have two factor auth. (That I've seen.)

Yes, Europe. Maybe there are different laws? Some decade ago you used to get plastic card with grid of passwords. Now it's usually SMS verification for every transaction (and sometimes for login).

I have.

It's pretty common for commercial banking... which is good because in the US the laws protecting you from loss due to hacking are much weaker for a corporate account.

Well up here in Canada I have never seen a bank with two factor auth. I would love to switch to one.

Instead we get online banking with password restrictions like: 1. 8 to 12 chars 2. No non-alphanumerics.

In Denmark its illegal for a bank not to use two factor auth.

You're right, but you're talking about application security, not the security of the infrastructure itself. For most endeavours, it's enough to have a securely written app hosted in an average data centre. When you're looking after currency, there's so much more incentive for people to break in, and you have to put some extra resources into securing your infrastructure too. By the sounds of it Bitcoinica has completely neglected to do this and has been operating as if they were just a startup storing my cat pictures on S3.

Very true, the difference is banks will give you back your money after you're/they're hacked.

In fairness to Bitcoinica, they've promised to do this:

    > The thief stole from us not you.
    > All withdrawal requests will be honored.
    > http://www.bitcoinica.com/

There is a flip side to that. You ever hear of an actual customer losing their money in one of these attacks on banks?

It's all about the cost of doing vs. the cost of fixing it and who ends up paying.

We do, you just don't hear about it because "we're in it together" and it's not really anyone's specific money being stolen. And banks profit so much, it's just a small dent in their profits. And these security costs and investigation is paid by society, not the banks.

I agree with what you're saying. I'd argue that Bitcoin sites are at even higher risk than old-school financial sites because (I assume) the FBI actively investigates bank hacks, but not Bitcoin trading site hacks. If I were a bad guy, I know which I'd target.

There's really not any problem with hosting servers at a cloud provider that isn't shady. Encrypt the data on the drive, use a reasonable firewall policy, then all you have to be concerned about is application layer breaches.

Trust me, lots of payment data goes through secure servers at Rackspace.

/Not a rackspace employee, but I know

I get what you're saying, but my point is this: This hack happened because Bitcoinica doesn't own the bare metal servers on which the money is hosted. The break in was possible because somebody abused Rackspace's password reset functionality. The last time Bitcoinica got owned, it was because somebody managed to exploit Linode's customer service portal. To me this trend would strongly suggest the need to outright own the production hardware so that attackers have to attack you directly or physically break in to your data centre.

Thanks for posting, Tong, good luck with next.

Might I suggest a post-mortem on the various security problems you dealt with, for those of us who weren't following closely? I'm sure you learned a lot, even if what not to do.

I don't think many programmers, especially web developers in the consumer startup scene, are faced with such security pressures as Bitcoinica was, so it would probably be very useful for many to read a behind-the-scenes accounting of that.

I would love to read that, too, but it's worth noting that the particular failure in this case appears to have been pretty boring. According to his account he was transferring operational control to a different group, and one member was relying on an insecure email server which was used to reset the root password on a Rackspace VPS.

Unfortunately, I haven't encountered a single strong hacker (in terms of web).

All my 3-year Rails experience was sufficient to prevent the most direct web attacks: Injection, CSRF, XSS, etc.

The rest is simple, keep wallet.dat securely. It's just everything the hacker wants.

So, the experience isn't very applicable to other startups, I think.

This has no remarkable impact on Bitcoin or the community, it's just the founder of Bitcoinica (which was hacked several times) disassociating himself from Bitcoin because his reputation has been decimated.

If I did something seriously wrong, I would definitely admit it.

The recent hack is not my fault. It just destroyed the only reason that I stay in Bitcoinica.

I am certainly not accusing you of any misbehavior, but even you would admit this incident did not serve your reputation well -- deservedly or not. And that is indeed your motivation for leaving.

It certainly doesn't inspire confidence in his ability to single-handedly securely run a financial company targeted by hackers, but I think he has handled himself quite well otherwise.

His forthrightness with the incidents, covering of customer losses, etc. has made me respect him.

+1. Please give him respect. It's not often that a 17 year old hacker builds a site in 4 and a half days that begins processing over $1M USD in volume. Of course it's going to be a target for hackers. Do you insult the local jewelry store owner because his diamonds got stolen?

Perhaps, if that jewelry store owner had his diamonds sitting on a picnic table outside the store, unsupervised.

Reputation damage is never the motivation for leaving. I wanted to leave long ago. The income was too attractive for me to actually leave.

Without money I can finally rationally re-think my destiny.

I strongly disapprove of the recklessness you have displayed several times, but then again you're a kid and it's understandable.

I hope you are successful and that you have banked well in the Bitcoinica deal.

"generating value for society at large" struck a chord with me. This is also something I strongly resent in my line of work. Maybe rationally rethinking one's destiny requires having first solved the money problem?

Congratulations on your insights.

You said that you're leaving to pursue opportunities where you can "...build products that save people time, money and headaches."

Why not make it easier for people to use bitcoins in day to day transactions? It feels to me like more effort in the bit coin community is going into making financial products for finance people than financial products for normal people doing normal things. Solutions such as those that make bitcoin transactions from one cell phone to another cell phone would be a valuable addition to the community.

If I did something seriously wrong, I would definitely admit it.

Trying to run a financial site without the chops to do it was seriously wrong. Of course, it was wrong of users not to do more due diligence on the service, its record, the team (you, I guess), etc - after all, it was (was) their money.

The recent hack is not my fault.

You've got a lot to learn.

Trying to run a financial site without the chops to do it was seriously wrong.

He was also warned when he announced Bitcoinica on HN, yet he chose to ignore those warnings.

eh. He chose to try anyway. That isn't ignoring advice, it's just being unable to apply it.

If everything the warning serves is to ask me shut down the site immediately after the launch, I have to say that I ignored them.

However, I really did everything possible to prevent security attacks. I failed at trusting other partners too much - especially Linode. (You can say that the wallet wasn't securely stored, which could be my fault, but the hacker possessed the ability to log in to every single Linode server!) The recent email leak is because I added a guy who is responsible for security into our mailing list that was used for password resets.

I would say that almost none of the comments in the original post provided the right warning or predicted the right cause of failure.

Again, they are useful (and the described things could happen instead), but they are irrelevant.

Any business can fail because of various reasons, and predicting something will eventually fail will always turn out right, just like predicting a baby will die one day. I appreciate the warnings because I didn't make any mistake they mentioned - no SQL injection, no mass assignments, no unauthorized back-end access, no CSRF/XSS, no firewall break-in (the hacker got the database from the server backups, not from the live database server). I think I've handled everything that I'm able to handle well.

Maybe my comment came off too harsh. I apologize. You are clearly quite skilled and I expect you will do great things in the future. I also admire your bravery in attempting an ambitious project like Bitcoinica.

The warnings I referred to were the ones saying that it is guaranteed that you will be hacked. It was never a question of if it would happen - it was always a question of when. Thats simply the facts of running a service dealing with large amounts of money. For that I think you were ill-equipped, despite that you knew that this would happen. Now, I admit that I think you handled the fallout quite well, but I don't know if you were as prepared for it as you should have been. From my very distant view, it looks like you weren't, but I don't have all the facts, so...

I would say that almost none of the comments in the original post provided the right warning or predicted the right cause of failure.

I'm assuming this is where experience would have saved you (or if not prevented a hack, reduced the damage done). Its difficult to cover all angles at the best of times, so its doubly so when you are inexperienced. I'm not saying that the problems would definitely have been prevented and I think perhaps I'm being unfairly critical. If your service had been dealing with anything other than money, I honestly would have no problem at all with anything you did - like I said, I think you have skill and you did handle it well after problems occurred. Its just that if I entrust my money with someone, I expect that they are well prepared for anything that could go wrong.

I think I've handled everything that I'm able to handle well.

I agree, you probably have, but thats where the problem that I see is (or was, I guess). Its not the things that you're able to handle that I'm worried about - its the things that you're not able to handle.

So with that said, let me close by saying that I'm glad you came out of this alive (ie not bankrupt - I hope!) and I wish you the best of luck in your future endeavors. I imagine you learned a hell of a lot running Bitcoinica.

Thanks a lot for the clarification. Now I understand your point.

I have officially left the Bitcoin economy so my next project will not be anything financial related. The most important thing that I have learned is exactly what you said - "Its not the things that you're able to handle that I'm worried about - its the things that you're not able to handle."

Also, trust is another issue. Almost the whole Bitcoinica system fell under the hacker's hands, except for one part - AML verification documents (customers' passports and other extremely confidential information). This is because this part of system is still solely under my control - not even the new owner could access the information. I didn't trust anyone with that data, and it turned out to be the most secure system after all.

I should probably identify my strengths and weaknesses carefully next time.

I hope this thing with Zhou and Bitcoinica isn't going to screw up my company's all Bitcoin IPO: https://glbse.com/asset/view/DMC

We're seeking 200k BTC for a self-green powered mining farm at our own DC up here in Maine, so we profit from the mining farm itself, having nearly no cooling needs for the DC, selling power generated back to the grid, and also renting excess space to other companies who want out of the way DC space in a quiet part of the country.

zhoutong, you are definitely an inspiration. Sad to see you leaving bitcoin.

I still suspect he stole the 18k bitcoins, the guy is slippery.

Why would you think this?

I was genuinely curious why grandparent would think that "I still suspect he stole the 18k bitcoins, the guy is slippery.". Why is the guy "slippery"? What makes you think that he stole the bitcoins?

Why would someone downvote for asking for clarification..?

Reply on news.ycombinator.com