I am Mt. Gox's first employee – AMA | Hacker News Reader

I am Mt. Gox's first employee – AMA

reddit.com

189 pointsdmichulke73 comments

Notable tidbits (most of which were already public record if you followed the story very closely) include:

+ Mt. Gox commingled depositor and corporate funds. (Corroborated in the Japanese version of their docs released by the bankruptcy trustee.) Many in the community believed that they had promised not to do this (true) and they this was impressively unprincipled (I lean against this understanding).

+ Mt. Gox had standards of engineering professionalism which were not what one would hope would prevail at a financial institution with $500 million in assets. True and previously reported. Specific examples include lack of a staging environment, source control, testing procedure, engineering leadership, auditing... it goes on. Also might suggest in future not putting all code into MySQL.

+ At one point all BTC deposited at Gox were accessible by VNCing into a particular box, which suggests that claims that Gox had cold storage secured by keys kept in diverse physical locations were not consistent with conventional interpretations of engineering reality. (The new information here is "VNC." It was previously known that at one instant in time Gox proved its reserves by moving all it's BTC in a single transaction. This could be done in a way consistent with what the community thinks "cold storage" should mean; the VNC bit militates against that understanding.)

+ Employees of Mt. Gox believed, on the basic of public evidence, that the firm was losing money even at the peak. (I'd be interested in seeing their math for this; that was not the conclusion I reached in a napkin calculation in mid-2013.)

+ Wages at Tibbanne (which provided 100% of employees for Gox) were consistent with prevailing wages in Tokyo startups i.e. scandalously low. 50% of employees earned less than $2k per month; director-level salaries in order of $4k, except for...

+ ... explicitly alleged looting of customer funds to support director lifestyles.

+ Management was unable or unwilling to answer basic queries regarding finances internally.

+ Gox's bus number was one, and that bus would have resulted in $500 million being unrecoverablu lost. Management, when asked about this, lied brazenly. (Last part is new info, first part obvious.)

There's more at the link.

thanks for the highlights, I don't have time to dig through this properly at the moment and I haven't been following the story all that closely.

You have made better life choices than I have.

While you jest, I can appreciate your interest in the topic and ongoing saga.

Thanks for a detailed recap!

> + Wages at Tibbanne (which provided 100% of employees for Gox) were consistent with prevailing wages in Tokyo startups i.e. scandalously low. 50% of employees earned less than $2k per month; director-level salaries in order of $4k, except for...

I've been told that employees at a certain like-deviantart-but-good startup are paid about as much as a convenience store worker. On the other hand, the conbini won't sponsor your visa.

> a certain like-deviantart-but-good startup

Is this a thinly-veiled reference to pixiv?

that does not tell anything - are they employees in a different country ? Convenience store workers in the US earn about 20K USD per year - that is double the salary what a comp sci fresher from a tier 2 college (non-IIT) would earn in India. And that's a reasonable salary - at 10K USD per year, you can eat out everyday at pizza hut ("regular" food is cheaper), drink a couple of times a week and share a 3 bedroom apartment.

Compared to the convenience store next door to the office!

The "deviant-art-like-but-good" bit is probably referring to pixiv, a popular Japanese art site.

You're not going to live anywhere near a comfortable life in Tokyo on 2 million JPY a year, much less 1 million JPY a year. Convenience store workers here make around 950 yen/hr depending on what shift you're working on.

Assuming you find somewhere super cheap and livable (we're talking slightly larger than a single bed, maybe not even including a bathroom) for 50000 yen/mo, that's already 600,000 yen/yr on housing alone. You can see that there's not much left over for utilities, food, transportation, clothes, medical, etc... and we're not even talking taxes and luxuries.

> Convenience store workers in the US earn about 20K USD per year - that is double the salary what a comp sci fresher from a tier 2 college (non-IIT) would earn in India

Pretty sure Japan is more like USA in cost of living than India.

You don't think it was unprincipled to commingle funds?

Not particularly. A deposit is a liability of the corporation; the cash on deposit is an asset. Where do people imagine these things typically exist? In the absence of legal requirements to the contrary, money is money -- it doesn't typically carry along requirements to physically segregate it by owner, intended purpose, etc.

My company owes money to vendors, customers, and contractors, too, in the ordinary course of business. At any given time we have probably a few dozen creditors. How many deposit accounts do you think we have?

I think, in the case of a bank or banking-like-entity, it's really easy to argue that commingling deposits with corporate funds makes it much more likely for things like.. what happened at mt gox to happen.

If you violate professional standards while providing your services, even in the absence of requirements to the contrary, you're putting your clients at risk, probably to an unethical degree depending on what assurances you gave them.

I don't think the comparison with an ordinary company is useful. They weren't mingling money paid to the company with money owed (which is normal), they were mingling client money, which they have no good reason to touch except under explicit instruction from the client, and their own funds used for operating expenses.

Companies which explicitly hold other people's money are usually held to higher standards in holding it (Solicitors, Banks, Estate Agents, etc). For example deposits are usually held in a client account, separate from other corporate funds, and not usable except for the intended purpose. The reasons for this are obvious, as it makes fraud far harder to perpetrate, and Mt.Gox not doing so is a red flag given the business they were in. Even worse than this, this wasn't a corporate account, it was Mark's personal account!

http://www.sra.org.uk/solicitors/handbook/accountsrules/part...

https://www.reddit.com/r/Bitcoin/comments/3fe92x/im_ashley_b...

Even worse than this, this wasn't a corporate account, it was Mark's personal account!

I understand how you could read that from the employee's statement, but this allegation is contrary to fact. Gox did most of its business through a series of business accounts at Mizuho, one per currency. A list was provided by the bankruptcy trustee to creditors multiple times -- c.f. here: http://www.mtgox.com/img/pdf/20141126_document.pdf

Perhaps the employee is talking of their time there at the beginning in 2012, not a later setup, they weren't there at the end in 2014.

Separating client and corporate funds, if done properly, will also protect the money from company debts. E.g. If the company becomes bankrupt, the client funds cannot be used to pay creditors.

> Companies which explicitly hold other people's money are usually held to higher standards in holding it (Solicitors, Banks, Estate Agents, etc). For example deposits are usually held in a client account, separate from other corporate funds, and not usable except for the intended purpose.

Banks use client deposits to fund their lending, though.

I don't see any fundamental problem with mixing client and corporate funds, provided there is good accounting, solid auditing and sufficient oversight in place to ensure client funds don't go missing. Obviously none of those existed in Mt. Gox.

Banks are pretty much the only institution that's allowed to commingle client funds with their own funds, and they're subject to a whole bunch of extra regulations and scrutiny as a result. Indeed, I've seen people argue that being able to do this is basically what defines a bank.

I get what you are saying but when you deal with managing other people's money I think it's a different story. It's a safeguard to prevent you from dipping into funds that are spoken for. Now you an still dip in (and may need to) but by having the accounts separate it's a conscious decision to dip in instead of a just a CC swipe away...

> legal requirements

Well yes, there are legal requirements when money is handled. BitCoin operates in a fuzzy realm, for sure, but it is still poor practice to not follow banking safety practices.

Any business that handles customer money (for example, a lawyer that holds client funds in escrow to pay for services later) is required to maintain separate accounts.

The only exception would be if the customer is lending money to the business (as when you invest in a bank account), not having the business to hold the customer's money.

Any business that handles customer money (for example, a lawyer that holds client funds in escrow to pay for services later) is required to maintain separate accounts.

Respectfully: this is not accurate generally or in Japan. Lawyers are special-cased in the laws of several US states for this purpose.

Software consultants in Japan, to use one example I am intimately familiar with, are not. If you take a deposit of $50k from a client which isn't your money yet, you book an asset (the money, typically deposited in your bank account, where there is no duty for segregation) and a corresponding 前受金 ("advance payment received") liability. As soon as you provide the service which you've received the money for, you're obligated to decrement that liability and increment sales. (This is important for tax purposes if the two events happen in different calendar or fiscal years, one reason why I have to keep books and report to the friendly local tax office once a year how much of OPM I'm presently holding.)

n.b. My understanding of GAAP here would be that Gox would probably hold the money on the books as a deposit (預かり金) rather than an advance payment (前受金) but I'm not positive about that -- my business never had to worry about it.

Ok, that makes sense. I think I misread your original comment as referring to commingling personal funds with the corporate funds - which on rereading your OP is not what you said at all.

  In the absence of legal requirements 
  to the contrary, money is money -- 
  it doesn't typically carry along 
  requirements to physically segregate
  it by owner

Right - for companies that aren't routinely transferring client money. Here in the UK, the way Mt Gox was run would be extremely irregular - clients' money would usually be stored in a client money account.

To use an analogy, consider if I courier you a parcel. When I hand it over to the couriers they have custody of it - but they don't own it. If the courier company goes bankrupt while my parcel is in transit, they don't get to open all the parcels and auction them off.

Likewise, companies like conveyancers and insurance intermediaries act as 'money couriers' - and they're expected to keep the money in separate accounts so, if the company goes bust, it's clear who is the owner of the money and who just has custody of it.

Obviously, not every financial service is a 'money courier' - but generally getting licenses that let you lose client money is more work than getting the licenses where you can't lose client money.

Of course, I'm not an expert on Japanese financial law, so it's possible financial conduct standards in Japan are different.

Among the most notable of the tidbits I've seen: There was one bank account, shared by the CEO (personally) and MtGox itself.

This is not correct, except to the extent that management appears to have routinely used Mt. Gox's corporate funds for purposes other than those reasonably required to run the corporation. Mizuho Bank, Shibuya Branch, No.1457705 was indeed registered in the name of Mt. Gox, as were several other accounts.

https://www.reddit.com/r/Bitcoin/comments/3fe92x/im_ashley_b...

> So you're saying that the mt gox bank account was his own personal account? I wired money to his personal account??

> Yes

Maybe it wasn't correct in 2014, but apparently it was true when Barr worked there.

Or at least he believed it was

Thanks for this excellent easy to follow summary. The only piece I have difficulty to understand is this:

    The new information here is "VNC." It was previously 
    known that at one instant in time Gox proved its 
    reserves by moving all it's BTC in a single 
    transaction. This could be done in a way consistent 
    with what the community thinks "cold storage" should 
    mean; the VNC bit militates against that understanding.

What would be an example of "cold storage" that allows moving all BTC in a single transaction?

And how does the fact that it is accessible via VNC change anything?

What would be an example of "cold storage" that allows moving all BTC in a single transaction?

Have all the private keys stored on a machine which is airgapped. Sign a transaction on that machine; copy the transaction to a networked machine (could be done via e.g. copying a QR code from the monitor with a smartphone or, in extremis, just typing very carefully); release onto the Bitcoin network.

And how does the fact that it is accessible via VNC change anything?

Management has previously said that Gox's cold storage was based on offline copies of the keys being fixed onto paper and held in 3+ locations, sprinkled with some RAID-esque pixie dust. (I promise -- this is the maximally charitable summary.) If they're all available for management via VNC at any given time, that casts some doubt about whether they are actually striped over a bunch of paper wallets. Although the Bitcoin community is primarily worried about a server compromise followed instantly by a wallet draining, putting them on a machine accessible from the internal network is approximately just as dangerous, since one assumes that an attacker who gets the web server owns the entire internal network with probability approaching 1.

Cold storage is storage of BTC private keys that are not connected to any live computer. Therefore anyone who would hack your live computer systems could not steal the cold storage BTC just by a remote hack. They would have to physically break into whatever location contained the keys, and then circumvent any encryption-in-rest that you had wrapping those private keys.

You can also split up control of the keys with multisig, so multiple keys have to come together to move the BTC.

Since most financial institutions only have small fraction of their balance needed to cover their money inflows and outflows, having most of your BTC in cold storage is a best practice.

>>> Specific examples include lack of a staging environment, source control, testing procedure, engineering leadership, auditing... it goes on. Also might suggest in future not putting all code into MySQL.

This. . . is. . .particularly shocking. The FIRST thing I did when me and my two partners decided to start a business was to get all of these in place before we even started thinking of what we were going to build. All three of us thought it was that important.

Dude, I mean, lack of source control? Who does that? Even when I only had two years of development experience it was drilled in my head this was not just optional, it was absolutely necessary - there is no option to the contrary. It was just basic stuff you should know and this guy was handling millions of dollars in transactions every day? It really makes my jaw drop to think about.

Hi, I'm Ashley Barr. (I'll prove it if necessary, but the tracks to this username have already been proved out by the bitcoin community) This is pretty vivid summary. I didn't actually know that most of what I provided was already in the public record. The reason I'm commenting is that I'm hoping to get more eyeballs on my original AMA, regarding Edit 3. Cheers for your insights, I learned a little too.

"The entire codebase was stored in a database, only accessible via a client Mark maintained, with no version control nor lock controls meaning if we both had a file open we could overwrite one another's files and undo work. There was no pre-production environment. Meaning changes were supposedly deployed untested or made straight on production. I was told I could not touch the backend, although they were working on getting access to this via Mark. Although it was taking time because Mark was more busy with his Bitcoin cafe. They said that despite all this, it was still a decent place to work because there was literally no pressure to perform. "

Just wow.

I'm under the impression that MtGox had some actual real life impact, how many important systems are twisted beasts like this.

Depends. How many times does a Magic the Gathering Online Exchange attempt to become a real financial institution?

From what I understand it was never a magic the gathering exchange.

The original author reused an old domain for the bitcoin exchange afaik.

That doesn't make the business look any more professionally managed.

Many, many more than anyone will admit.

I feel the urge to crawl through wikileaks all of a sudden.

Ya, MtGox had people's savings in it. I really feel sorry for them and hope Mark's arrest and trial give them some justice even though they won't be made financially whole again.

http://www.dailydot.com/business/bitcoin-crash-i-lost-everyt...

The thing I don't get is why anyone would use their own 'code storage'. This isn't laziness, it would take much less time to setup git.

Because it's easier to make things than it is to learn things.

This is the guy who wrote his own SSH server in PHP: https://web.archive.org/web/20140226001727/http://blog.magic...

We don't know that server was used in production, but it's consistent with the complete anarchy we hear prevailed at Mt. Gox, yet again.

If you knew that git existed, yes. Quite often people who are incompetent do not know how easily their problems could be fixed.

Out of habit, maybe? Edit: and ignorance

I think it's a phase most programmers go through - I jokingly call it “code graphomania”. You make up some sort of overengineered project, then keep implementing it even though there might exist better alternatives. You'd experiment with various language features, try to cram in some wild design decisions that have no basis in real life patterns, etc. It's hard to explain concisely, but I do observe this in other hackers a lot.

And I think Karpeles was going through that phase at this time. I vividly remember seeing his blogposts on reimplementing an SSH server in PHP, just to show that it's doable. I wouldn't be surprised to see that code and other similar terribad ideas running in production.

'Second system effect', the mythical man month (of course!)

I think a large part of this is the 1 Hour rule. Where when things are really bad whatever you can do in 1 Hour that improve things slightly get done. Then Repeat.

People basically just keep chasing local Optima until the unholy mess becomes self-sustaining as real improvement becomes more difficult and you can always look back and say, well at least we have "backups" even if it's just a copy on another disk in the same machine.

Imagine what an SQL injection could do to that...

Indeed:

https://www.reddit.com/r/Bitcoin/comments/3fe92x/im_ashley_b...

> Someone brute forced Jed's account (as told to me by Mark).

> I was there about 3 days before it happened, and I received 3 emails about an SQL injection vulnerability (which Mark ignored/thought unworthy).

Can you please provide a link/source to this? I can't find the AMA'er saying this and a google of parts of what you said only link back to this HN post.

Edit: Ok so it WAS in that thread just not something that the only "verified" person said. I'm inclined to believe it's true however it hasn't been confirmed and posting it here in quotes like that given the title (indicating the person running the AMA said this) is misleading at best.

http://www.reddit.com/r/Bitcoin/comments/3fe92x/im_ashley_ba...

https://www.reddit.com/r/Bitcoin/comments/3fe92x/im_ashley_b...

> The entire codebase was stored in a database

Even if this didn't happen at Mt. Gox I'm sure it's happened in other places. I've heard of people storing the queries that they were going to send to the database in a database row.

When you've got a hammer, everything looks like a nail...

SCID - Source Code In Database; http://www.c2.com/cgi/wiki?SourceCodeInDatabase

> I've heard of people storing the queries that they were going to send to the database in a database row.

In fairness, that's not all that different from prepared statements...

For what it's worth, there doesn't appear to be any proof that this person is whom they claim to be.

The question was asked but somewhat suspiciously dodged: https://www.reddit.com/r/Bitcoin/comments/3fe92x/im_ashley_b...

/u/MtGox_Adam is the same reddit account as previously used by the Mt.Gox CEO. Of course it's possible the account has been hacked.

Interesting stuff. Sounds like a bunch of kids that had no idea how to handle $500 million dollars. However, the owner did seem to handle things with malice. Why would he share a bank account with all the customers assets - I'm no lawyer but that sounds extremely ilegal and obviously a conscious choice.

You're given such a large opportunity and to piss it all away. Yikes.

There are other good summaries but here is mine, in no particular order (All is C&P, no editing other than adding in my own commentary in "[]", putting commenters questions in quotes, and adding in "Mark" where "he" and other pronouns were used. Also "..." is a clip):

* As far as I know, no one else at Mt.Gox ever had access to the backend of Mt.Gox, nor the cold wallets. At my time there, only Mark had access to the Database. ... Mark said that if he died there would be hints that one of his best friends could follow to find and unlock the cold-wallets. When I asked said friend, he said he had no idea what Mark was talking about.

* We then looked at the expenses (eye-witness expenses only [AKA, we collaborated and make a list of things we had seen were purchased for the company] so it's not accurate, but surely less than what was actually spent), and used the trading data to calculate some averages around Mt.Gox's profits. The expenditures far exceeded every model we had for income. [This is in relation to the AMA'er being asked to be CEO of Mt. Gox and what happened when he tried to do due diligence, Mark wouldn't give him access to the books]

* Around the same time, we learned that Mark only had one bank account, shared with Mt.Gox's customer deposits.

* Mark was receiving a lot of pressure for "proof of solvency". I was behind him when he VNC'd into... somewhere, used the bitcoin otc app to send the money from A-to-B, and then posted about it. [In case you didn't catch that... THERE WAS NO COLD WALLET. ALL of it was hot and on a machine (running in graphical mode of all things...)]

* "How likely do You think it is that Mark Karpeles was running the Willy bot?" -- To be honest, before you asked this question I had never considered it wasn't Mark. ... I just know my ex-employee contacted me about with the impression that Mark owned this account, I hadn't questioned it, and it was presented as "Mark is at it again"

* I think gross incompetence happened, and Mark tried to cover it up. I don't believe he is outright malicious, but certainly ignorantly-malicious.

* [Mark's] salary was the same as mine at that time. ... I started at 240000yen/month (~$3.1K/mo - ~$37K/yr), and was paid 320000yen/month (~$4.1K/mo - ~50K/yr) from about August 2011. I was the highest paid employee (including Mark) when I left the company.

* "Where do you think the millions of depositors fiat have gone to?" -- Nooooooo fucking idea. I hope to read about it the same as you do. I want answers.

* "Why was Ross Ulbricht's Gox account banned very early on?" -- Honestly, no idea. I know that Mark banned any accounts blatantly linked to SR. Some people emailed that they needed to use their undeposited cash to buy things on SR. If that made it to Mark, he banned it. I think it was safe practice.

* "Since this is AMA: How come you chose this particular time to come out with this AMA? Why now and not earlier?" -- I think I mentioned in different comment threads, I had an NDA that prohibited me from talking about Mt.Gox. It's still enforceable, but I think Mark has other things to worry about at the moment... I hope :/

* I stopped believe anything he [Mark] said after a month of working there.

* "Reuters reported that expensive toys were being purchased by MtGox/Mark. Do you have any more examples of luxurious spending?" -- He bout a NAO for $5k in August 2011, a Makerbot a little later, and... Ahem, I had to talk him down from buying a Lamborghini as his first car. Respectively, the Honda civic was a modest purchase. (how to fuck could I explain a Lamborghini to 50% of the employees making under $2000 a month... it would have killed moral, even moreso

* "I find it incredibly irresponsible that no employee or ex-employee blew the whistle on what was going on." -- There was no proof, no one (mark) gave anyone access to anything directly. Everything was inferred. (Spent 900K, made x00K?) where did that come from? No answers... what can you report on that ... :/ It sucked....

* "Do you think Mark is guilty of actually manipulating data in a fraudulent manner, or merely negligence in operating MtGox?" -- Both. but only the latter do I have an experience.

"Around the same time, we learned that Mark only had one bank account, shared with Mt.Gox's customer deposits."

No offence to his guy but he seems woefully under qualified. A CEO that doesn't have accounting report to him... makes less than 50K/year. Does anyone know what valued he thought he brought to the table?

It sounds like you wouldn't have to bring much to the table, if the main competition was Karpeles.

Mark, the founder, asked him to be CEO[1]. An incompetant current CEO, who isn't giving details to someone who he asked to be CEO.

[1]: https://www.reddit.com/r/Bitcoin/comments/3fe92x/im_ashley_b...

https://www.reddit.com/r/Bitcoin/comments/3fe92x/im_ashley_b...

>The entire codebase was stored in a database, only accessible via a client Mark maintained, with no version control nor lock controls meaning if we both had a file open we could overwrite one another's files and undo work.

>>Like, PHP source stored in mysql tables?

>PHP, CSS, JavaScript, HTML. Errything.

>There was no pre-production environment. Meaning changes were supposedly deployed untested or made straight on production.

>I was told I could not touch the backend, although they were working on getting access to this via Mark.

WAT

What was the company culture like? We hear about a lot of bad but obviously there had to be some good things going on to work there.

This isn't the actual AMA, click the link at the top to get to it.

Reply on news.ycombinator.com