Hacker News Reader

Comment by panarky | Hacker News Reader

There is no fucking way 700,000 BTC disappears with nobody noticing. This must have been building for a long time.

Was there never any attempt to compare BTC in wallets to customer balances?

Or did MtGox know about this problem, and hope to cover it up over time?

Either case is at least gross negligence if not criminal fraud.

Yes, we all fuck up at some point. But I don't pretend to be a surgeon and perforate a patient's aorta. Pretending to be competent to gain people's trust is fraud.

I sympathize with those who lost money in this clusterfuck.

No sympathy for Karpeles.

View on HN

Most people who've been around bitcointalk and bitcoin-otc know not to use MtGox since circa 2011 when their stunning incompetence was at it's height. Sadly there were plenty of media shill articles when Btc skyrocketed to $1,000 last year who were promoting them as the "Biggest Bitcoin exchange" without pointing people to relevant bitcointalk threads on what a nightmare that site has been over the years.

If you read MagicalTux's personal blog you'd know to never trust anything he's coded too. http://blog.magicaltux.net/2010/06/27/php-can-do-anything-wh...

I clicked through and thought you were being too harsh. I mean, it can be fun to make toy implementations of things as an exercise. Doing an SSH server in PHP would be entertaining if you liked PHP. You'd learn something.

And then I read that his hacked-together-in-3-days ssh server was for use in production. In a hosting service.

Wow. Just wow.

> And then I read that his hacked-together-in-3-days ssh server was for use in production. In a hosting service.

That sounds like a brilliant technical guy, capable of running with a daft idea to completion (unlike me, with my collection of at-best-half-built personal projects), who should have some layers of protection between him and Real World Production...

That reminds me:

"Now, people who Get Things Done but are not Smart will do stupid things, seemingly without thinking about them, and somebody else will have to come clean up their mess later. "

http://www.joelonsoftware.com/articles/fog0000000073.html

I think it's different.

Inexperienced (young) programmers don't know what's been tried, and what's available. I've been dealing with this a lot at work recently, where we ended up doing poor reimplementations of off-the-shelf stuff due to a mix of ignorance, and honestly, a bit of hubris.

It's a good attitude to have in academia/non-production critical work, but the GP is right, production demands a more conservative approach, especially when money/safety is at stake.

The type of ignorance Joel writes about (see parent) is different, it's more like "slavishly following design patterns" and "writing copy-and-pasted, improperly factored code". Both are ignorance, but the first is better called "NIH", whereas Joel's is, "writing bad code".

The part of "running a SSH server that you wrote in PHP in production" that is scary is not the "in PHP" part.

It's the "that you wrote" part.

No matter what language you write it in, you are going to mess something up. The OpenSSH guys have messed up working a lot smarter and more diligently and with more time than you have.

This is so very true, OpenSSH is probably one of the most secure pieces of software around. It has extremely high value to attackers, yet has had extremely few remote security holes in its lifetime.

They've invested years and many talented people in developing such a piece of software.

If you want to write your own ssh server in php, you should probably consider your motivation and how you can re-use their code or operate through it instead if your purpose if anything other than experimentation.

Yeah, thats what I thought too, until I read this little gem, suggesting that OpenSSL was written by monkeys: http://www.peereboom.us/assl/assl/html/openssl.html

Its kinda hard to disagree with that conclusion.

You shouldn't be using OpenSSL unless you are an expert in crypto and software development. It's not easy to use and it shouldn't be.

I can't say those difficulties he had in using the library were put there on purpose to keep people like him out, but it seems to be a good effect here.

OpenSSH is not OpenSSL.

Dang, guess that joke's on me. Wouldn't have realized it, thx for the correction.

Omg, I nearly missed that:

    What did I create a ssh server for? The same thing I created a DNS server for fun and for KalyHost.

I'm not sure "disbelief" is sufficient to describe how I feel right now.

Yeah, I'm still trying to figure all that out myself.

There was some allusion to needing some kind of database backend for the SSH server, but there are multiple solutions for that now (like LDAP).

I'd love to have this guy work for me in a junior role (because he can really crank out the code), but all his work would need to be reviewed, and I wouldn't want him to be making architectural decisions on his own.

well, is it better to have a hacked MVP released in production, or spend forever making it and never actually releasing it and then missing the window?

I can tell you absolutely, without question, that when it comes to security and people's funds, there is nothing courageous about a hacked MVP in production. There's a difference between someone's to-do app one weekend, and this case. If you are handling people's money directly or indirectly, you need to care about that and take it seriously. Or don't ship.

Well I would argue that the "M" in MVP would necessitate never losing anyone's money. If I were going to create a trading platform, I'd probably start with one that only accepted Play Money.

Putting on my Lean Startup hat for a sec, I would even say that the M could allow for losing money. If it's early on and your customers are all in the 2 1/2% of innovators, they will put up with a little of that. Certainly if you make them whole, but probably even without.

That said, "flawless accounting" would be very high up on my feature list. I think the failure here isn't launching without perfection; it's operating at scale without perfection.

I've been wondering whether the "M" in "MVP" is for minimum quality, or minimum feature scope.

I think minimum feature scope doesn't necessitate poor-quality software, just solutions that don't do everything for everybody. It's much better to ship a small feature set with very high quality, IMO, than a big feature set with low quality.

Great thing to wonder about!

In the Lean Startup sense, the M is about minimum effort, and the V is about viability with customers. You're basically playing Battleship trying to discover where those two Venn circles overlap.

Different aspects of quality map to both those circles. There's build quality, which relates to the sustainability of the code base. There, you have to consider both short- and long-term quality. [1]

There's also quality as users perceive it. That varies widely by domain by market, and by how far you are along the adopter curve.

My general answer is the same as yours: minimal features with highly sustainable code. But for experiments, I think you can get away with terrible code as long as you a) throw it away quickly, and b) you are on it so even if you have a bad MTBF, your MTTR is really good.

I also think that you can tactically discard certain kinds of user-side quality. E.g., if I'm making a product for early-adopter financial traders, I'm not going to worry about quality of visual design, and I might inflict hard-to-learn interfaces on them. But I'd be rigorous about accounting and about UI issues that might lead to mistaken trades.

[1] I wrote some about that here: http://agilefocus.com/2009/06/22/the-3-kinds-of-code/

Minimum viability?

Well when mtgox started off btc was play money, it was always play money until perhaps 12 months ago when it became serious money. And I base play and serious on the value, at $10 a coin it was still fun, at $100 a coin I had to seriously consider how much I should keep on my phone or any other single place.

He did. It was called Bitcoin

Cute, but they also traded and held USD.

Oh, that's easy. The best thing is to use the off-the-shelf SSH server, one that has been written by experts and carefully reviewed by a lot of people.

Or do you mean this as a metaphor for MtGox? In that case, I would say that I would rather miss the window than be the famous asshole who -- oopsie! -- lost $500 million of other people's money.

The problem is less with a minimum viable product, and more with a far-from-viable product, in ways that aren't immediately obvious. The security FFVP is especially dangerous.

Depends on what the product itself is and from which perspective you're asking the question. At any rate, those are almost never the only two choices...

what if the hacks you used to build your MVP result in unrecoverable real world damage to your self, your investors, and your customers?

One way to look at ordering features in early products is as risk reduction.

One of the biggest risks is, "nobody gives a fuck", which is why MVPs are so valuable. It lets you test market hypotheses.

But if you're building something handling real money, then a pretty obvious risk is, "The system will lose money beyond our capacity to absorb losses." Their failure to address that risk here is at best negligence.

But given the size of the loss, I don't think we should rule out fraud. The interesting question is, "When did they know they had a problem?" Sometimes shitty accounting systems are just naiveté. But when they persist over a long period of time in a way that just happens to cover up loss, embezzlement, or theft, then it's worth asking: did they keep the shitty accounting because better accounting would have forced them to admit something they were hoping to cover up?

Would you want to fly in a hacked together MVP plane released to production? A similarly made bike in a park? Maybe.

I wouldn't today, but the wright brothers did, because if they hadn't, someone else would've done it instead. If you wanted to be trail blazers like the wright brothers were, you might have to put up with a hacked together MVP. I m just saying that anyone who lost their money did so knowing the risks (or should have known the risks).

>Most people who've been around bitcointalk and bitcoin-otc know not to use MtGox since circa 2011 when their stunning incompetence was at it's height.

And still, even 2 weeks ago, tons of people defended MtGox in HN threads, and said how it's a temporary glitch and they are very good exchange and such.

Even when it was pointed to them that it's a service build by a guy with no actual knowledge of exchanges and no prior experience at finance services whatsoever -- a mere PHP developer (not to knock the language) that had done nothing spectacular before (no Carmack, or Fitzpatrick or your favorite coder hero).

People trusted their money to a guy that literary calls himself "MagicalTux" -- which to me seems like investing to the hobo on the corner, people call Crazy Bob.

Does this mean we shouldn't trust "coldtea" to develop anything?

I'm the last person to defend Karpeles' competency, but his internet alias has nothing to do with it.

>Does this mean we shouldn't trust "coldtea" to develop anything?

Of course you shouldn't.

If you were to here him (well, me) you'd ask for my CV -- if not an interview also.

And if it was like "developed some random toy stuff" you wouldn't hire me to develop a money exchange playing with other people's millions of dollars.

And if you were to assess if you will put $10,000 in a financial online service made by me, my past work in the area, my general competence would be quite important.

Else, don't be surprised if you lost it all. The chances were way higher than if you had put that money in Citibank, you just ignored the signs.

And for me, not giving the impression I'm a 20-something script kiddie with a fancy handle would also be quite important. I mean, it might be prejudice, but "Ives, Rockefeller and Berstein" as a financial service just feels more secure than "$uper7eetMoneyMakah", "LuvFlamingoes" or "MagicalTux".

> I mean, it might be prejudice, but "Ives, Rockefeller and Berstein" as a financial service just feels more secure than "$uper7eetMoneyMakah", "LuvFlamingoes" or "MagicalTux".

How do you feel about "Bear Stearns" or "Lehman Brothers"?

Much more comfortable than MagicalTux. For one they have the clout to get trillion dollar bailouts from the government.

Oi!

I don't know, investors don't seem too put off by this Crazy Bob's name.

http://techcrunch.com/2011/03/28/square-cto/

Yes, maybe because they didn't ignore all the context of my email, and did checked that he has serious credentials, like:

"he had been leading the core library development of Android while at Google".

And that he is just but one of the players at Square, including guys like a well known VC and Twitter's cofounder.

If "CrazyBobs" was an unknown in the industry guy and his CV was like "I have done some fun projects, like a PHP mailer" and he was the major person behind the company, no investor would have touched it with a barge pole.

people who've been around bitcointalk and bitcoin-otc know not to use MtGox

There are a lot of forums on the Internet. It's not confidence-building, at all, to tell people "if you hang out on the right forum you know what's safe." Especially because "the right forum" is not written in stone.

That's what makes it fun..

Oh lawdy [1]:

    Of course normal frameworks are a no-go. Using
    someone else’s framework will make your world
    slightly better, but until you create your own
    full framework, you won’t understand what I mean.

    The next step is to build applications with your
    framework. The kind of applications that will
    change the world...

Should have tagged the post with NIH.

[1] http://blog.magicaltux.net/2009/09/19/striving-for-a-better-...

The worst thing is he still runs multiple other companies:

https://www.autovps.net

https://www.kalyhost.com

https://www.tibanne.com

http://blog.magicaltux.net/2009/12/22/incorporation-in-japan...

If he was not dealing with other people's real money and data this experimentation and shotgun approach would be fine.

What you wrote made me think, "What would anyone in their right mind write on their blog that would make people think something like that?" So I clicked through, and read the first four words, that struck me in the face like a four layer wedding cake: You were right. Then I read the rest of the sentence, and that was like a ton of frosting being poured on the cake from a dump truck. Point well taken, sir!

I've been around. What is bitcoin?, and who cares about whatever the hell 'Mount Gox' is?? Can we all get back to work on something meaningful to the future, family, or nature... someone we don't know about made up a currency, and some other company we also don't know about made up an 'exchange' of this 'currency' and now we still don't know what's going on and some 'bitcoins' are missing... the most it's all worth is a laugh

> I've been around. What is bitcoin?

From https://bitcoin.org/bitcoin.pdf‎:

Abstract. A purely peer-to-peer version of electronic cash would allow online payments to be sent directly from one party to another without going through a financial institution. Digital signatures provide part of the solution, but the main benefits are lost if a trusted third party is still required to prevent double-spending.

Of course, the coins are only yours if you hold the private keys. A promise of some Japanese exchange to send you an amount of Bitcoin in the future is not the same as owning a Bitcoin.

It's pretty interesting, you should read it.

I think you had the only decent reply. Thank you. I know bitcoin, and was really just fishing for others that may share my view on the current state of affairs for the digital crypto currency. Some of us have been around just long enough to be aware that the field of technology doesn't create a buffer from unclean people and business characters. Also, that just because a lot of smart/clever people made something that a lot of people believe in, that at some point it passes the point of no return or is immune to failure. I think bitcoin failing may be good, could wake a lot of people up, many mistakes won't be made a second time, and the freed up focus and brainpower will go towards a better generation of technology to benefit humanity.

Currently, this event looks like about the third or fourth-largest crisis of the currency and has had very moderate on the valuation, although this might change in the coming weeks.

Many people in the field are very ethical and think that an improvement to the current financial system is of great benefit to humanity.

I mean, you are the one that decided to dive into the middle of a bitcoin discussion. I typically find it rather easy to ignore things that do not interest me, maybe you should work on that.

Maybe you should go back around to wherever you've been around, and not be around here, where you have nothing of value to contribute and no interest in the discussion at hand. The way things work, in case you didn't learn that by being around, is that you don't post messages complaining you're not interested in a topic to discussions about that topic. So stop being around here.

A laugh and a few good tens of millions of dollars that people actually do value and will pay that much for. Maybe cryptocurrencies aren't in your favor, but their value and growth isn't something to scoff at.

Bitcoin is a technological breakthrough, you are just being ignorant about it.

> There is no fucking way 700,000 BTC disappears with nobody noticing.

The history of Barings Bank is interesting, primarily because the main trader Nick Leeson was in the position to mask hundreds of millions of pounds of losses. In that case, no one noticed the roughly $1.3B loss because he was in a position where the internal controls didn't apply

At Barings, Leeson was trading futures contracts and he was also responsible for accounting for his own positions. There was no segregation of duties, no mark-to-market.

At MtGox, it should be much simpler to (1) add up all customer balances, (2) add up all BTC in hot and cold wallets, and (3) compare those two numbers.

If the numbers differ by more than timing differences, shut it down, debug it, and prevent a small problem from becoming a big problem.

The Barings backstory is that the billion-dollar loss started with a 20k bad trade, Leeson tried to cover it up, and it snowballed.

It wouldn't be surprising to learn that MtGox followed a similar trajectory.

>At MtGox, it should be much simpler to (1) add up all customer balances, (2) add up all BTC in hot and cold wallets, and (3) compare those two numbers.

Yes, considering Gox only had 2 bank accounts (and now are down to just 1 in Japan) I don't see how hard it is to compare balances and notice 744,000 coins are missing.

They claim that over time the cold wallet was cleaned out. If this is true then the first few customers would've complained hey I never got my coins, and he could've seen shenanigans happening due to his php wallet implementation and halted all trading. Then thousands more customers. I find it hard to believe 744,000 worth of coins disappeared over months and nobody noticed until recently

Which is just another aspect of the travesty - that someone responsible for routinely trading tens of millions of dollars per transaction should feel the need to cover up a $20,000 loss.

Leeson tried to cover it up because he wasn't supposed to be trading at all, he was supposed to a broker and he decided to try to make some extra money.

You also have to account for the insane levels of indirection in the management structure of an institution like Barings Bank, an exponential sneakernet.

The Wikipedia article is worth a read: https://en.wikipedia.org/wiki/Nick_Leeson#Downfall

Heh, I had only been on the internet for about 2 years when that went down and I spent that evening watching the world's gears seize up instead of finishing the book I was suppose to be writing; it was around this time that I started tog et interested in alternative currencies. I was just thinking that here I am doing the same thing 20 years later.

You must be new here... I have seen PLENTY of situations where internal controls are weak, NO people dont cross compare assets and liabilities.

> Either case is at least gross negligence if not criminal fraud.

To echo the original commenter: Yes, jail sounds like a reasonable end to this — but keep it restrained.

> No sympathy for Karpeles.

No, certainly not for hiding his incompetence, but I think the point was more: don't be too graphic in wishing him to discover what sex in jail can be. That’s an… understandable reaction, but inappropriate on a public forum. He’s got family who cares about his well-being, and the next days are going to be tough for them too. Imagine talking to his mother, or children: the facts, they are going to need them to understand, but nothing violent or tasteless (and I think you did a great job).

> But I don't pretend to be a surgeon and perforate a patient's aorta.

He didn't kill anyone; I'm sure one can save a lot of lives with that kind of money, but… I’d rather see you compare him to Kerviel (I'd disagree) or that Barings' trader (closer). The Head of security at Target, or Sony might be more accurate cases.

Especially if you are paying millions to accounts each year according to that document.

Where did those accountants get their degrees/licenses?

if there is no fucking way, then where is the news

Reply on news.ycombinator.com