Hacker News Reader

Comment by seldo | Hacker News Reader

Forget the difference in the exchange rates. Are you saying you have no system in place to monitor if transactions execute successfully? I'm not trying to be hyperbolic but that's really shocking for a company that is trying to play a role as a financial institution.

View on HN

>that's really shocking for a company that is trying to play a role as a financial institution.

That is probably the scariest part about the current Bitcoin community. Many of the big names are simply "playing a role". We have no idea of their technical expertise, the viability of their software, their USD/Bitcoin reserves, how they plan to address disputes like this, or even whether they are skimming of the top.

And a majority of the (previous) big players have been compromised on more than one occasion. It's a big kindergarden.

I mean, MtGox stands for "Magic The Gathering Online Exchange". They were literally set up to sell trading cards. The whole "being the biggest bank in a new $6bn+ economy" is a pivot. The whole system is crazy. But even by those standards, taking 35 grand and not noticing you didn't give them what they paid for stands out.

But even by those standards, taking 35 grand and not noticing you didn't give them what they paid for stands out.

That's really the weird part. This is a business that does money in, money out. They should be balancing the books and checking inventory every day.

It can take Amazon a while to realize a mistake because it takes time for somebody to go around the warehouse and count all the boxes to discover they're holding excess inventory. But how long does it take for Coinbase to compare the number in column A to the number in column B?

But that's regulation. And HN is against that. Get with the program.

It's not just regulation. It's basic accounting. If you're not doing it, you literally don't know how much money you have. It's always a good idea to know how much money you have, even if the government doesn't require you to know.

You clearly haven't been dealing with businesses. 99% doesn't do daily checks. Many notice thins after a month or so, when closing books for month. And some notice those kind of things aften closing a year. But there still are many businesses which won't ever notice or care missing 100 grand. Been there and done that, and it's much more common than you think. In some of those cases I personally think that the business is partly illegal and bookkeeper is cooking the books. Therefore they don't even notice honest mistakes. I have seen same thing happening over and over again. People don't care about money. And this is business as usual.

Reconciliation is a fundamental operational procedure of financial entities. It's how they find system failures, process failures, fraud, etc. You take feeds from various systems and correlate them. It's not super complex, but it is fundamental.

I actually work for a startup selling reconciliation software as a service to the financial industry, so it's something I'm learning about myself. Lack of it suggests Coinbase are a little bit clueless.

If this is really is business as usual, then it might be acceptable for a widget selling company but completely unacceptable for financial services. In financial services the same actions that you describe are gross negligence that (depending on jurisdiction) can mean personal liability, possibly including criminal charges, for the relevant CxOs, and a lifetime ban from running any other financial companies in future. You can't entrust consumer's money to companies being run like that, it's pure nonsense.

You think that Goldman Sachs, for example, had exactly 263,674,000,000.00 of cash and cash equivalents at the end of 2012? You don't think that's likely rounded to at least the million?

In fact it seems that most Fortune 500 companies, regardless of industry round things to the million. It's pretty easy to lose a few hundred thousand if you are doing that.

Auditing standards at large banks have a materiality threshold. At a Goldman or Morgan Stanley, that threshold approaches half a billion. But that's at the top layer, on top of many other layers of accounting. At the level of individual customer accounts, even small discrepancies would be caught.

Any sort of financial account should be ACID safe. You can't just receive money on one end, and then it magically doesn't enter a client's account.

Bitcoin is great, but the exchanges still suck.

Except things like ATMs are not. They use eventual consistency.

Very few financial systems are ACID. Why do people still think banks run on bloody MySQL?

I love a non-sequitur as much as the next giraffe, but what does banks using MySQL have to do with banks choosing not to go ACID?

Nothing at all other than the ever surfacing assumption by a lot of people that financial traffic is done on ACID like principles (rather than eventual consistency which it actually is). MySQL was a random example as should have been somewhat obvious.

MySQL is a random example of financial traffic not following ACID principles? I still don't understand.

Is this a reference to MySQL, pre-InnoDB, not being ACID compliant? InnoDB's been around for a decade and has been the default database engine for years.

You clearly haven't been dealing with businesses. 99% doesn't do daily checks.

No way. Not on fin-co ops.

That's like a car tire store without a pressure guage in shop.

That makes no sense. It is self-regulation. Balancing the books is in their best interest.

You mean without regulation, companies could effectively steal money from you and you would never be able to stop giving it to them? The free market is also regulating, by the way, and it should regulate with its full force against companies like this that fail to provide adequate customer service and satisfaction, safeguards, etc to their customers... we don't need the government to tell us these people aren't doing their jobs effectively and we should take our business elsewhere.

I'm keen to learn more about self regulating free markets. Please provide an example.

I saw somebody today snarking about Bitcoins being Dunning Krugerrands. Coinbase appears to be making their point for them.

MtGox is something like 5 months behind in withdrawls with no confirmed date of when it will occur. I'm waiting on $600 since beginning of Sept...

If this is not some sort of error, MtGox is a ponzi scheme.

I can't remember where I read this and I don't remember all the details, however: As I understand the problem is mainly with USD withdrawals to USA, and the problem is not with MtGox but rather with the bank they use having some legacy issues and that they can only process X transactions a day in batch. Or something to that effect.

For this reason I switched to Bitstamp and the experience is night/day different. Bitstamp withdrawals/deposits have so far never taken more than 4 working days, sometimes 3, for me.

Aww, you guys are so cute. You think banks have this stuff under control? Auditors will "find" hundreds of millions of dollars of missing assets on the regs.

Many banks are powered by duct tape and loose Excel spreadsheets. That's why things like Basel exist.

That's really not true. Yes, auditors do find missing assets like that, but some of these banks have trillions in assets. Moreover, these errors usually aren't things like cash in versus cash out. Accounting and tax rules are very complex. So an error might entail having incorrectly computed the value of certain tax assets or the like. More to the point, a major bank will have layers and layers of accounting controls intended to catch even very small errors. These processes are themselves audited every year, and stress tested. It's not duct tape and excel spreadsheets.

Worked at a bank. They have better processes than this. Not gonna say they're perfect, but they can handle balancing their books and not dropping transactions. They certainly don't use a non-ACID PoS like Mongo.

Excel was over-used in many places, but was not used for day to day transactions and operations.

Yet again, bitcoin apologists inaccurately claim (lie) that banks are actually worse to excuse failings by "their" services.

Worked at a bank as what? Working at a bank clearly doesn't make someone an expert on their IT infrastructure. The actual financial transaction infrastructure isn't ACID.

As a software developer on corporate credit card applications, dealing with managing transactions, accounts, and restrictions. Our infrastructure was ACID. We certainly wouldn't have used experimental, flaky datastores like Mongo, and shit was not processed using Excel.

No-one is an expert on every part of a bank's infrastructure -- its too big for that. But definitely, we had software requirements, as a matter of IT policy, that kept stuff like MongoDB out of production.

Yeah, so, you don't work on financial infrastructure. That stuff is almost exclusively eventually consistent. And I wasn't arguing that you should use MongoDB or anything else instead. Merely that claims that ACID is the holy grail of financial software is a myth propagated by people that do not know how international financial traffic works.

> Auditors will "find"

Does Coinbase have external auditors?

No. The joke is that people are demanding a product so badly that theyre doing business with kiddies who dont know how to run a financial institution.

This is literally an economy of "shut up and take my money," but no one wants to play ball.

I know this is bad, but financial institutions haven't exactly been reliable, honest, faultless or good, particularly the last few years. My own experiences have been bad, but the world still hasn't pulled out of the hole that the financial industry dug, so I'm pretty well off and lucky if I'm compare myself to the average. I'd suggest a different standard to hold this company to.

I suppose that even with this, bitcoin banks still appear to be far more reputable establishments at the moment than say, either RBS or the Co-op, unless Rob Ford has founded one recently.

Having spent some time in financial institutions, I can only agree with lastlogin's point. It's important to remember that there are many such mistakes in large financial institutions every day.

If I could post on yc and quickly get a response from my bank, I'd be a happy customer.

One time I had almost 1 million extra dollars show up in my Waterhouse Brokerage account. I thought, "huh, that's interesting." It disappeared 4 days later.

Yup, it wasn't as much, but I magically had 10k show up after I made a deposit through a bank teller that stayed there for 2 weeks before finally disappearing.

My father withdrew cash from an ATM a while back, he got less than he asked for but had the asked for amount deducted from his account. He complained. The manager of the bank nodded knowingly when he complained. He wasn't believed. He was told that at the end of the month they would do a stocktake and see. Some months later it was credited back. This is about the best of my bad banking stories. The time our deposit for a house got lost was stressful several years of hard hard saving vanished during a transaction.

Wait! One of the founders was a currency trader at GS?! Doesn't that make them trustworthy?

The wild west of currency, with everything that entails.

0. This is exactly why auditing (internal and external) and settlement processes exist elsewhere. But it's impossible to have a startup deploy every single control at the very beginning. There are costs (time & money) to get things done, those control don't just magically appear. Eventually, with scale, the growing pains of business process debt (like technical debt) will come due. That's a good thing, because building every single control into an experimental venture that might not survive could be a very big waste. Just like a successful venture is a series of experiments, the business side gets built incrementally. As a company gets bigger (whether $ or buzz), it's susceptibility to being targeted increases because the bulleye on its back gets larger.

1. Let's suppose I employ an old gambit: process a large transaction through CoinBase, receive the coins and say "I never received them." I'm sure they have at least enough controls in place to avoid double-credit fraud.

Giving coins for USD is their primary use case. Monitoring transactions should be the only control they have in place if they must cut corners.

As a coinbase customer and someone who manages a team and large complex distributed systems, this is an oversight I would not expect from an industry leader and makes me nervous to continue doing transactions with coinbase.

How can we be made comfortable that this won't happen again? $50 isn't the root cause fix and could have cost the OP $15,000 due to the delay. What if those BTC was going to be used to make a purchase? He is left holding the bag, leaving a sour taste.

Coinbase is ultimately a double-ended marketplace. But to add more complexity to the fire, one's own systems count as yet another source of truth. So what the sellers say, what buyers say and what we say have to reconciled. It's these reconciliation processes that are the hard part...

I've had to deal with micropayment reconciliation for a minor handset mfgr (multiple currencies, payment processors, timezones, products, on and on). Sometimes the credit card processor up and changes the API without notice, and a whole slew of payments have to rerun. This is where fields in the local reconcilation db start to creep in and all sort of weird flags get made. What the heck is flag F? (IOTW, document the heck out of the db schema AND have lots^10 unit tests.)

A customer of bitcoin shouldnt have to care if the service you provide is hard. In fact, thats why they come to you in the first place.

Complaining about the complexities does not instill faith that you have a solid grasp on your product. Tell me you dont work at coinbase, please.

Agreed. The biggest growth area in banking and finance is compliance & legal. A startup doesn't have the financial resources to make a bulletproof compliance and customer service department.

Likewise, unlike banks, there is no legal system in place or organizations that assess companies like this and determine whether their transactions complete and such. The downside of the Bitcoin market, pretty much.

> Likewise, unlike banks, there is no legal system in place

There may not be a well-developed specialized regulatory system in place with authority over some bitcoin firms, but in most of the places that bitcoin firms operate there is a legal system in place, and entering into a sale contract, accepting payment, and then failing to deliver the product is the kind of thing that is well within the scope of competence of most of those legal systems.

That is the same thing that shocked me when I run into a bunch of different entrepreneurs in the last few weeks, creating local exchanges here, bitcoin ATM, thinking about colored coins and interesting transactions enabled by bitcoin.... and when I talk with them about the technical side, they never even know how a transaction is verified, what are bitcoin scripts, how the network works in general.... I know that I know quite little about bitcoin and it confounds me that people honestly believe they can get away with not knowing how things work. Due diligence, man!

> We have no idea of their technical expertise

why, we do - a team who could quickly build a rails app and connect it to third party payment processor.)

There was a similar discussion with a very interesting comment regarding their use of mongodb: https://news.ycombinator.com/item?id=5428382

PG's reply in the thread is very telling: https://news.ycombinator.com/item?id=5428372

I've been keeping my mouth shut here about Coinbase after starting that thread 9 months ago.

I feel bad for someone that can't put a finger on his 30K, but I can't feel that bad if he didn't do a little research here first.

Hopefully on the next bubble people will be a little more enlightened.

As highlighted in the thread and in PG's reply, the default response to criticism of Coinbase is to nuke the conversation.

As PG wrote:

> Strictly speaking I should have killed the post entirely

Now, it's possible that other complaints on HN were nuked by PG or some other moderator, making it difficult for someone to find it.

As highlighted in the thread and in PG's reply, the default response to criticism of Coinbase is to nuke the conversation

As PG wrote:

> Strictly speaking I should have killed the post entirely

That's not why he said that. He explains further down thread:

HN is a news site, not a customer support forum for companies funded by YC, and in fact the site guidelines explicitly ask that it not be used that way

This objection is pretty flimsy. HN is very frequently used as a customer support forum for large tech companies not funded by YC. A policy that it is okay to use HN as a customer support forum for Google, Amazon, and Facebook but not for Coinbase does not look very good.

HN is run by YC. While it may feel like a public forum, it is owned by a private entity. I'm OK with a benevolent dictator taking care of matters here, and trust PG to do the right thing by all of us.

Then you are an awfully silly individual.

On HN, there is an expectation of a certain degree of impartiality and free speech. Of course, YC doesn't have to provide it, but that's irrelevant: it's just something that is expected.

Why would you trust pg in the first place? I wouldn't trust him.

Apparently, referring to a thread which is highly critical of Coinbase, while participating in another critical thread, is proof enough for you that Coinbase criticism is silenced. I hope when I frame it like this, you understand how stupid it is, what you wrote.

What's apparent is the fact that you are incapable of analysis.

If you bothered to read the other thread, you'll see parallels to the current situation:

- transaction delays on the order of 1 week

- Coinbase is blaming technology (last time it was a database migration, this time server upgrades)

- Customer support has been silent (znowi described it as "information vacuum" https://news.ycombinator.com/item?id=5427985)

- Online account system called into question, users unaware of current positions (e.g. https://news.ycombinator.com/item?id=5428090)

If this were the first time, you can cut some slack if Coinbase responded appropriately and took steps to prevent this from happening in the future. Based on what played out recently, it's not clear if Coinbase actually learned anything from the previous issues.

And in the future, if you ever do decide to start a company and find yourself in a similar situation, I hope you would be a bit more communicative and proactive than Coinbase was in this case.

You're changing the subject to something which is not in dispute.

You said that the standard approach on HN is to silence criticism -- a stupefying statement, given that at the time you posted it, this conversation had maybe 100+ comments and had been on the front page for a while. Also, in that same comment, you linked to another voluminous thread criticizing Coinbase. Amazingly, the existence of your comment strongly mitigates against your thesis being true.

I think you must realize this at some level, as you have chosen to employ the age-old tactic of the inescapably wrong: quietly change subjects and hope nobody notices.

There is zero ambiguity in the PG quote

> Strictly speaking I should have killed the post entirely

And this was 9 months ago. We are left to assume that the policy was enforced (after all, if the post were killed, it wouldn't show up in a search).

And even in this thread: https://news.ycombinator.com/item?id=6931168

> I've been keeping my mouth shut here about Coinbase after starting that thread 9 months ago.

I wouldn't be surprised to hear others echo the same sentiment (concern about expressing grievances about Coinbase) especially after PG's responses in the original conversation.

Others in the original thread refuted the argument you are presenting, so I recommend you start https://news.ycombinator.com/item?id=5428757 before rehashing arguments that have already been addressed

Sign #1 that should have given you pause: in order to disprove your assertion a person would need to prove a negative, i.e., prove that no criticism was nullified. If you don't see what the problem is here, you may need a course in critical thinking.

(pointing to the statement that HN is not a support forum) + (commenting that PG should have killed the thread) = (PG is likely to kill all threads that may seem like support requests)

Given that many defenders of coinbase, including PG, are seeing support requests as criticisms, (threads that criticize coinbase = threads that seem like support requests) in their heads.

Therefore, (pointing to the statement that HN is not a support forum) + (commenting that PG should have killed the thread) = (PG is likely to kill all threads that may seem like support requests) = (PG is likely to kill all threads that criticize coinbase)

If you're to believe the press they've received, Coinbase are basically the most reputable and trustworthy company in Bitcoin and the one that's going to bring it into the mainstream.

Scariest comment of the thread, surely.

If you've dealt with any of the bitcoin exchanges you'd realize Coinbase is the most reputable and trustworthy. It's either deal with them, trust Magic the Gathering Online Xchange or the Russian mobsters over at btc-e.com. Seriously this is a sector that is in desperate need of more legitimate players but so for this is the best we have.

Holy cow, that's amazing.

Q: Does MongoDB support transactions?

A: MongoDB does not provide ACID transactions.

( http://docs.mongodb.org/manual/faq/fundamentals/#does-mongod... )

It is crazy to be doing financial stuff in a DB that doesn't provide transactions. It completely blows my mind that anybody would do this.

Not really. Financial transactions are really almost never implemented with database transactions.

A lot of people truly don't understand how financial companies do stuff. Hell, Financial companies are ones who are adopting Mongo and the like themselves. We all assume it's one way, when it's actually done another....

I'd love for you to expound on this point. I'm curious, what are they doing?

What financial companies are adopting Mongo for accounting/customer transaction storage?

I've heard of somewhere like ADP using it but they are using it as an analytics database, not for actual payroll processing.

They're really only doing that for hipster cred. That way they can easily write articles titled "How I made (really simple/trivial application) with (really fledgling technology)" such as "how I made a twitter client with node.js" or "how I put 2 sprites on the screen with webgl" or "how I managed 2tps with java".

We know all financial companies use databases running on tandem or tuxedo. But that's not cool sounding. How are they supposed to get fun jobs doing nothing with catered lunches? HOW, HOW I ASK!

What is crazy is people claiming a bank uses a transactional database for their financial transaction processing or that all parts of that system are "ACID". Both are false. Don't like MongoDB? Fair enough, it isn't great. But the tech banks use is a lot closer to MongoDB's persistence model than it is to your typical RDBMS.

Not hyperbolic at all. This is brokerage 101. It's absolutely asinine that their system fails this hard when doing some "upgrades". A real brokerage (e.g. Fidelity, Schwab et al) would likely be subject to non-trivial fines from the FCC for this type of gross negligence.

Sounds like they need some people who have actual practical experience building financial systems.

I'm not trying to be hyperbolic but that's really shocking for a company that is trying to play a role as a financial institution.

You're not being hyperbolic at all. You're just used to the standard Hacker News "only haters expect businesses to be competent" type of handwaving response. But this thread is about money, that makes it different.

I'm not trying to be hyperbolic but that's really shocking for a company that is trying to play a role as a financial institution.

And which just ate $25MM in Series B.

I work with some of the largest financials institutions and nothing surprises me any longer.

It might not be a computer system problem.

Perhaps CoinBase (or a rogue employee) were doing some proprietary trading and screwed up badly?

Maybe they were trying to quietly recover their losses and weren't expecting a customer to cause a public outcry and force their hand?

...how? This isn't stocks, there just isn't the same financial instruments that they can play with to recover from a drop in price :/

Take it EASY everyone! They made a mistake, took responsibility for the error, resolved the issue, and compensated the user for his inconvenience. Good for them!

For posterity's sake, here are the sequence of events. Coinbase:

1) stopped replying to support emails even though there was a clear issue

2) only addressed the issue when the victim decided to submit his issue to HN (and only after it sat on the front page for nearly an hour

3) blamed a server upgrade and offered $50 of BTC when the actual loss was well north of $12K

4) only made the customer whole after many HN comments pointing out that the offer was paltry compared to the loss (and that they may potentially be pocketing the difference)

I applaud them for resolving the issue in a way that the victim wanted: "deliver the coins at the exact market price that they are at the moment you are crediting the account". However, this and similar issues in the past have raised red flags that haven't be adequately addressed:

A) why did customer support go dark? I suspect the reaction would be different if they were more proactive

B) as mentioned in https://news.ycombinator.com/item?id=6930748 , why doesn't Coinbase have the infrastructure to detect these issues? This is really basic accounting, and if they can't get it right on a small scale why would anyone trust them with more money?

C) what steps were taken to prevent this from happening in the future? I imagine that there will be more server upgrades as Coinbase expands.

D) What is their future policy regarding similar issues? Based on their initial reaction, one should not expect much if any compensation.

E) Will it take an HN post in the future to get their attention?

And looking back at some of the comments in a conversation from 270 days ago (https://news.ycombinator.com/item?id=5427985) it seems that Coinbase hasn't actually learned from the previous issues.

I think joezydeco said it best: https://news.ycombinator.com/item?id=5427985

> You have a company trying to start up in a very tough space where you need to be super-reliable and super-transparent. Coinbase's actions this entire week were neither.

How did you conclude the actual loss was > $12K? He got the bitcoins at today's price + $50. That seems like there was no loss?

>He got the bitcoins at today's price + $50

That part only happened after many people criticized the original response: (https://news.ycombinator.com/item?id=6930109)

> Edit: your bitcoin credit has now been processed. It looks like as we were performing server upgrades last week a handful of jobs didn't run as normal. We should have certainly caught it and responded sooner so that was our fault. My deepest apologies for the delay and trouble on that. We've credited $50 worth of bitcoin to your account for the trouble, as a small way of saying thank you for bearing with us.

This was understood to mean that Coinbase would honor the original price (868.91) and not the current price. The general impression was that Coinbase made the purchase at the current price (roughly 550) and is honoring the other price (868.91), which would mean that OP lost about (868.91-550)*40 = 12756.40 (and Coinbase would be pocketing the difference).

People correctly noted that Coinbase's move was entirely inappropriate, and they were forced to change course (note the "Edit2"):

> Edit2: we'll push through the bitcoin credit at today's price instead of the original buy price (which should be in your favor) since the mistake was on our part.

The general policy with most financial firms is that firm errors should be resolved in a way that benefits the customer (so in this case, if BTC actually increased in price, Coinbase should honor the original trade price)

Only because HN kicked up a fuss about it - someone elsewhere in the thread claims they experienced the same delay but were only credited Bitcoins at the locked-in price, causing them a big loss[1]

[1] https://news.ycombinator.com/item?id=6931976

Only because the poster brought this to light publicly

Reply on news.ycombinator.com