BTC Stolen from Poloniex
bitcointalk.org
bitcointalk.org
If they had the settlement processed asynchronously on a different system, likely with human interaction, they'd be able to say "Hmm, it seems like account #944325 has convinced us to schedule a withdraw of $250,000 more money than he has on deposit. Well that would be a really bad idea, now wouldn't it. Denied. Now I think I'll sip a cocoa while leisurely planning my review of our withdraw scheduling code."
Many complex systems are complex because they solve complex problems, not just because they have evolved over time from different simple conditions. You can't always (if ever) replace a complex system with a simple one. You can't escape complexity by coding around it. This is the attitude that killed Netscape and kills many Enterprise projects.
The people who are interested in cryptocurrencies come from all walks of life and share different views on many topics, just like the citizens of a country. Nothing's ever black or white.
The natural path of these "disruptive" startups is to start small and lean, ignoring a lot of the complexity of whatever market they are disrupting, and then a slow and steady snowballing as they begin to re-accumulate a lot of the baggage of the industry they came in to disrupt. In the end if you're lucky you wind up with something that is still better than the old thing, but not nearly as much better as the original hype around the new thing would suggest.
And I thought I was a pessimist.
However, nobody wants to put the effort into organising politically to cause improvements to the existing system. Because they don't trust anyone, and only care for the increase of their own wallet.
It's also an example of all the best engineering practices missing.
I thought that checking the account balances and wrapping the withdrawal in a transaction was an introductory example of concurrency issues. What were they thinking, handling money while allowing negative balances?!
However, has anyone stopped to ask why so many exchanges are poorly-coded? No, it's not because everyone in Bitcoin adores PHP. I've met some of the most capable coders among cryptocurrency enthusiasts. Go check out Conformal's btcd, or any of Jeff Garzik or Warren Togami's projects, or the amazing talents of the Bitcoin core dev team. (If you decide to denigrate the abilities of any of these Bitcoin/cryptocurrency developers in a response, please be sure to include a link to your own github)
No, it's not because there's no talent. Rather, it's because nobody who is competent in the Bitcoin world is willing to risk their hides or the welfare of their families on something they know the authorities will eventually crack down on. You see, competent people actually learn about the space, and the relevant regulations before jumping in. And they understand that -- absent several million in start-up funding -- there is no way to legally open an exchange at this point.
So by process of elimination, the only ones left to open exchanges are either scammers, or ignorant, incompetent coders who can barely code up a PHP site, or a very few brave, mostly-competent individuals who have the money and lawyers to at least (hopefully!) keep them out of jail when the inevitable crackdown occurs.
I know personally of several extremely competent entrepreneur-developers who have abandoned Bitcoin projects out of regulatory concerns. And I'm sure that's just a small sample.
So that's the reason for the prevalence of poor coders among Bitcoin exchanges. Your solution to this problem will of course be a function of your worldview.
An unstated major premise of the "wonders of regulations" argument is that regulations exist because sometimes the hand of government is needed to handle situations where the invisible hand is a demonstrable failure.
And to me, at least, it's pretty clear why competent, responsible developers are not participating in this space -- we're too frightened by possible government sanctions.
Where's 'rayiner? He'll love this.
Writing bitcoin software is like writing crypto. You need to get it exactly right.
But instead of starting with a spec written down that the crypto community tears to pieces, instead the developers eat their own dogfood. No, change that: they build critical infrastructure out of their own dogfood. All before it's ever been vetted by the really smart people.
I'd say competent people don't write Bitcoin marketplaces that handle real money for the same reason competent people don't write their own home-grown crypto and then make it a single-point-of-failure for their entire business.
Also, you're pretty uninformed about the history of Bitcoin. Satoshi did start with a spec, or at least a white paper (yes, the lack of a proper spec for Bitcoin has been a major problem). Satoshi presented the white paper to the cryptography e-mail list several months before releasing the software. He got some feedback at that time, and then he got significant feedback when upon his initial release, which he integrated into the protocol.
This list included many of the top cryptographers in the cryptocurrency space, including Hal Finney and Adam Back.
You can find all these discussions, and the original release at: http://nakamotoinstitute.org/
Finally, with the exception of the transaction malleability issue, the failures in exchanges have had nothing to do with cryptography, but rather with basic secure software development practices and architectures.
Every exchange seems to be a bespoke system and that is just begging for trouble.
They should publish their architecture, then publish their source code, and then make sure that smart people have tried as hard as they can to find all the weaknesses in their stuff, and then turn it into a business.
Of course, there are business reasons not to do that: someone else can use your verified source, someone else can get a first-mover advantage, you might not get the smart people to pay attention, you might not know when it's ever good enough, and (the more pernicious idea) is the worry that seeing the source will give the attackers ways of attacking your stuff.
They fail for the same reasons that secret crypto systems fail. I fully understand why they are doing it, but it's still doomed to failure.
I think that in the USA at least, government sanctions are far from guaranteed - the courts have confirmed that BTC is a currency. What that means is that there is no longer a question of if you can legally start a Bitcoin exchange in the USA - it's a question of what hoops you have to jump through to do it. (And liability. . . I'm curious if the Flexcoin situation means we'll soon be finding whether a clickthrough EULA that says, "Not responsible for lost or stolen bank accounts" is enforceable.)
Now, perhaps the hurdles are prohibitively high. I'm more than willing to believe that's true. But if so, then you're being rather melodramatic to frame this in terms of "government sanctions". That makes it sound like people aren't doing it because they don't want to knowingly engage in illegal activity. It'd be more realistic to just say that competent people are staying out because the regulatory environment results in an excessive cost of doing business for anyone who wants to make sure they're operating on the up-and-up.
That's exactly what I meant. Apologies if I made it sound like anything else. It costs millions to start up a money transmission business in all 50 states (like an online exchange would need to), and the federal government has been explicit that they consider a Bitcoin exchange a money transmitter.
I don't think that's clear at all.
In fact I think bunderbunder has it right. Those who make the best products will require a large enough expense for formal design, implementation, secure hardware acquisition, physical security (i.e. no simply running your exchange on someone else's cloud without a lot of oversight), the works. This requires tons of time and resource investment.
Because if any part of that chain is improperly coded, designed, implemented, etc. it will eventually be exploited and you'll be no better than the Poloniex type exchanges of the world.
In the meantime there will be those "incompetent" developers you mention with a shipping product already on the market. And theirs will be much cheaper as they don't need to devote "Space Shuttle computer software" levels of development design and implementation effort. So you'll be both late to market and more expensive.
If you're talking about financial security here then you effectively need to be building a Bentley instead of a Pinto. But you'll be competing in that unregulated market with Pintos with a consumer base full of people willing to take the risk of driving in a Pinto instead of a Bentley they can't afford anyways.
And this has nothing to do with the government yet, either sanctions or regulations.
In fact this type of "tragedy of the commons" is exactly why there is government regulation. They help ameliorate the inevitable "race to the bottom" by artificially limiting where the bottom may be.
But government regulation probably won't help too much here since you can always run your exchange out of a country that doesn't care and people can make their transactions with whatever identity they wish.
Who knows, maybe the industry will self-create and self-adopt appropriate regulation as a market differentiator. But that still would open the question of who does the enforcement; if competitors discover their competition isn't actually following the regs then they'd be forced to "streamline" themselves and then the whole thing goes to pot again.
So those millions are money that could go toward building the infrastructure you describe. Bitcoin companies like Coinbase who have finished their Series A have to spend their runway on licenses instead of developers or infrastructure.
It's a huge problem, whether or not people are willing to admit it.
But thank you for arguing in a rational and non-bullying/non-contemptuous manner.
1. http://payment-systems.quora.com/The-Money-Transmitter-Licen...
Yes, this is unfortunate for those devs out there who could solve the problem of making a good Bitcoin exchange if only the cost-of-entry were cheaper, but that happens in tons of other industries too (and not always due to the government), and it already has an answer.
As somebody in the financial markets regulatory space, your comment aligns with my own thoughts - there is a middle-ground: a balanced but essentially prudent view of Bitcoin and altcoins that is seldom seen in these threads full of naive ideals and cynical strawmen.
Another aspect of this middle-ground viewpoint: both of these polarized sides seem to be focused exclusively on fully automated Bitcoin-based systems. I find this highly unlikely.
If Bitcoin-based systems do become popular, there is no reason to imagine they won't involve some human (e.g. back office settlement systems with STP rules, payment tests and 4-eye reviews of breaches) and legal elements (e.g. declare your BTC addresses to your government tax office, KYC/AML compliance for exchanges, etc.) to solve certain problems that have been in the news lately.
WTF, who are these clowns that purport to be running the equivalent of a bank?
It's like everyone running a BTC exchange either is corrupt or slept through the part of Databases 101 where they explained "this is why transactions are important, here is banking as an example". Seems likely both.
In fact, I think they're just M's.
Oh, wait...
Seriously, if I was an economics or law professor today, I don't think I could come up with a better classroom to teach financial regulation than the mess that is the Bitcoin economy.
If my bank sent me this notice, the FIRST thing I'd do is withdraw all my money - there's every chance it may be "an absolute necessity" to steal more of it at any time...
It's also why I don't understand maintaining a balance with a service like this. Unless you're doing very small trades, the 0.2% fee plus the network charge for a withdrawal seems like a small price to pay to be mostly safe from this kind of thing.
I keep seeing this attitude among Bitcoin fans. Of course, there have been many financial crises with regular currencies and banking systems. But that doesn't mean they're the norm. When you look at the number of crisis against the number of different financial systems and years-without-a-crisis it's obvious that they're exceptional. Sadly, the number of Bitcoin shocks seems to be increasing linearly with scale.
I don't think this is inherent so much as a problem of overconfidence and magical thinking. Right now Bitcoin service providers (in the aggregate) are like builders whose structures catch fire distressingly frequently, but who excuse this by pointing to famous fires of the past. The fact that fires have historically been a problem doesn't mean fire codes are useless; quite the opposite, in fact.
- something bad happens to someone doing the conventional thing (vaccination, banking)
- people panic about it on the internet
- there is a lack of trust in other people and institutions
- people pull away from the conventional system
- in the short term it's fine
- then there is an epidemic (measles, fraud/loss)
- this is much worse than the system they rejected, so they resort to denial
- everyone else says "I told you so"
edit: optimistic
Nice, I wish my bank was so chill about having a negative balance. :)
This reminds me of the early Amazon bug, where you could add negative numbers of items to your cart, and it would credit your account, and then wait for you to ship them the book. ;)
Apparently it's only $50,000 ("only"), meaning no individual lost tens of thousands of dollars.
It's unfortunate that this guy being upfront with everyone puts him above average, but it does, so his business will probably continue, such as it is.
and for good reasons.
The exchanges take a cut on every transaction, so Poloniex should have self insured for the first 3% (or what ever their transaction fee is). After all they made that money on the transaction.
They should carry insurance for the rest.
The 12.3% deducted from everyone's account is "wrong" in my view because Poloniex absorbed none of the loss, and kept its cut of the transaction.
The "right" thing in my eyes is for Poloniex to adjust minus their transaction fees.
-Brandon Wirtz (Not a Poloniex customer)
Consumers could look for the mark as indicating that the exchange is a well run outfit rather than a bunch of cowboys.
(I don't have to trust that my bank can, because my accounts are insured by something that rhymes with "duvvermint".)
I don't know enough about the current economics of exchanges.
A Currency exchange would typically not be a bank. They would have a lot more in transactions than they would have cash on hand. $4M in transactions on $500k cash on hand. They would have little ability to lose a customers money because the customer would have to be in the store when it was robbed.
A bank would not have all of its money in a place that could be robbed, so they don't have 100% risk (or 12.3%).
So for a bank the insurance is about .2% of transactions. For BTC it would likely be closer to 3% because the systems are audited.
The difference between Bitcoin companies and cat sharing companies isn't fly-by-night operational practices (although in some cases, sure).
The difference is that when a cat sharing company gets owned up, you don't hear about it. Only a small subset of security compromises involve password hashes dumped to pastebin. A quiet security incident at a cat sharing company doesn't end that company; they patch the bug, (hopefully) reimage their servers, and get on with their lives.
Even payment companies have incidents. But (perhaps counterintuitively) incidents at payment companies aren't company-ending events. Payment companies don't hold bearer-bonds, for magic cards or anything else, on their servers. Attackers might get a few thousand credit cards out of the rolling transaction feed of a payment company. Why bother? Those attackers can just acquire massive dumps of credit card numbers from major retail compromises. Attackers report flaws to payment companies! They're better off trying to score a $500 payday from a bug bounty than trying to monetize those breakins illegally.
Not so at a Bitcoin company. When Bitcoin companies get owned up, whatever "hot" assets they have get taken. Bitcoin companies aren't stepping stones for attackers the way cat sharing companies sometimes are; they're the intended target.
It bothers me when people caution that Bitcoin companies are scary because they're "targets", because I think the people saying that only grok 75% of what's happening. It's not that Bitcoin companies get owned up because attackers spend more effort targeting them. No. Attackers expend effort on everyone's companies, and are usually successful. The difference is what happens after the attacker succeeds.
People considering starting (or funding) Bitcoin companies need to understand this. I feel like there may be a memetic belief that competent security teams can reliably stave off security flaws if they're just careful with two-factor auth and parameterized SQL queries. That meme is false. When you start a new software service, you need to build it on the assumption that you are going to get owned up, if not by a stupid password compromise than by a memory corruption bug deep in V8 or MRI or CPython or nginx that only a few dozen people in the world know about. It is going to happen. If you give custody of cash-equivalents to a startup, you should be doing it with that in mind.
Using bitcoin is a strong nexus between an individual and the long tail and it puts the individual on their radar. The long tail is being nice when they only steal my bitcoin.
Off-topic, but as someone with too many cats I would like this idea to take off pronto. I could use a full night's sleep.
I'll show myself out.
YC '14, here I come.
SOMOLO FTW
This isn't a problem bitcoin companies outgrow; it's a problem that festers as the company gets more successful. Do you go out like Flexcoin did, or like MtGox? Either way: you eventually do get taken out.
It seems that the only way for this line of business to be feasible in the long term is for the hot wallet :: total assets ratio to be as low as possible. Your income to build assets is proportional to transaction volume, but so is the required size of the hot wallet.
Maybe a massive up-front investment to allow for start-up assets to be suitably large in comparison to hot wallet size... but even then you'd need to be careful not to grow too quickly and to ensure that you proportionally build up your reserves for when your hot wallet gets wiped out.
But this means that you have those stored assets that you can't invest elsewhere, so are you even making a profit now? The only way to reduce assets needed is probably some kind of insurance arrangement, but why should the insurance company offer low fees for this with the risk profile we currently see?
Much higher levels of isolation (such as running the transaction engine in some kind of HSM) would help, but I suspect that anyone smart enough and cautious enough to do it properly is too smart and cautious to go anywhere near the prospect of running a bitcoin exchange.
Nothing in Bitcoin is reversible. Everyone working with them has to be hyper-vigilant, always, constantly, whatever you do don't blink, blink and you're dead.
I think if I were working with Bitcoins I'd get an ulcer.
For thousands of years the only way to pay for something was with non-revertible mediums, either cash or bartering for other goods. Reversible transactions are new in the grand scheme of things.
If I ran a bank and didn't bother to lock the doors or keep the money in a safe and someone comes in and steals the money, I don't get to do a chargeback and get the cash back.
That would be one thing if physical security were just as hard (or harder) than cyber security.
But it's in fact the complete and polar opposite; physical security is much easier and much better understood by the actors who need to engage in defense.
Irreversible + manual = opportunities for people to spot the fraud; volume limited by amount of work required to commit each fraud.
Reversible + automatable = opportunity to spot and undo the fraud.
Then ask, how many of these things are true for bitcoin? Then perhaps ask, could I provide these things, and make money while doing so?
There are literally at least hundred thousand developers who have worked for 20+ years on financial systems - the industry employs a lot of them. If you're going to be storing money of other people, picking up a random such guy - even completely mediocre, boring one - would at least bring up the many issues that are taken for granted in 'that world' but nonobvious if you're not from the financial industry.
At least not if they are populted solely by young, hip developers. Those developers can probably build the system, but what they can't do is specify and validate the system. So, those young, hip developers need to at least spend some time talking to some experienced domain experts.
Or, you know, keep repeating every failure in the history of finance and banking that has led up to the industry practices and government regulations they are ignoring. Sure, you can likely find better solutions to some of those problems, but it would be better if you at least "anticipated" the well-known, obvious problems and solved them, rather than repeating them first.
Only with that mindset can one build a system that doesn't screw over every legit customer when it happens.
If you start a BTC exchange, write half of the initial code yourself, have access to the servers and own the company - then you should ask a simple question: could I myself steal funds undetected? If you're an investor, could the CEO/founder steal funds undetected? If the answer is yes, you have work to do.
There are some theft options by privileged people that can't be realistically prevented, but you can make sure that those scenarios would be detected within a day, and thus those privileged people simply wouldn't do it to avoid jail.
Writing bitcoin software should be like writing crypto: you aren't smart enough, so don't try.
Saying that a company has the most secure bitcoin system in the industry, even after proof it was the case, doesn't change the fundamental design tradeoff in bitcoin. Bitcoin offers anonymous possession in exchange for risk. Because the risk is high relative to normal forms of commercial exchange, bitcoin attracts rational actors for whom the rewards outweigh the risks and the distribution curve of interested parties has a tail that skews criminal.
The missing link for bitcoin is coverage under something like the Uniform Commercial Code- a system that specifies general principles governing transactions and the framework for their completion and resolution of conflicting claims. Caveat emptor is not a solid foundation for a banking system..
If someone with ill intent has the ability to wire money from your account, they can do so with ease. Bitcoin really isn't that unique in that sense.
The real story in the bitcoin world is a recurring tale of gross, epic levels of incompetence, largely because a bunch of people who know some PHP have become the ones building financial systems. People who lack even the most rudimentary of knowledge necessary (this particular "exploit" for instance....good God. This is literally usually the very first "learning databases" lesson because it is so obvious of a weakness).
Also the fact that the people making software seem incompetant when it comes to technology.
Some people see these stories and think "ha, I could do better than that!" Other people see these stories and think "can I really be sure that I haven't made even one fatal mistake?"
All software engineers write bugs. All software engineers write security holes. For most of us, the fatal flaw doesn't irrevocably wipe out a bunch of people's life savings.
Some of these guys are indisputably incompetent with regard to the software and services they are building. While it's true that all software can have bugs, these are extraordinarily unsophisticated attacks that any non-hacker can exploit by, say, refreshing his browser in rapid succession.
That is, there is a sliding scale with regard to the level of competence imputed to, say, a certain type of bug. And, this wasn't a simple coding error (which, can be more easily forgiven). This was a fundamental oversight in the overall approach to the software, with regard to a critical operation. And it involves such basic concepts as transactions and race conditions. While eliminating the latter can be difficult to get right, it appears that they didn't even consider the fact that they could occur.
I am one of the guys in your group who thinks "have I considered everything?" In fact, I would be inclined to believe that I haven't. Perhaps that's pessimism or just realism, given what I've seen from determined hackers attacking my business over the years. So, I can certainly give a pass to oversights or errors. But, at a certain point, the nature of some oversights or errors are indicative of the fact that the developer(s) are not competent, at least with regard to the domain.
Among bitcoin holders there seems to be a tendency to assume the technical sophistication of bitcoin mining and blockchains somehow perfects operations involving the buying and selling of bitcoin. The analogy I would draw is to notes which can be exchanged for gold and backed by a central government.
It can be the case that notes are reliably exchangeable for gold and also the case that a bank takes these notes for deposit without writing down the deposit in a ledger. Likewise the bank allows withdrawal of notes, again without writing them down. If An individual makes the decision to deposit notes with that bank based on the certainty the notes may be exchanged for gold, they have used the wrong set of facts as the basis of their decision.
At best the robustness of bitcoin is independent of the robustness of bitcoin businesses. My suspicion is that in practice their is a negative correlation because stolen bitcoin are largely unrecoverable and do not trade at a discount.
Mt. Gox, Flexcoin, and Poloniex. What do they all have in common? It's not what you think. (:
Now if you're trying to bash PHP (or Ubuntu?) for banking software that does not take an exclusive lock on rows it reads then modifies and a daemon that does not check balances on withdrawal, try again.
Many PHP developers, on the other hand...
Just wait until people start announcing their exchanges and business written in Rails or Python, or "anything but PHP" but still turning out to have been written by webapp developers with no idea how to actually deal with currency transactions or basic security.
To make that decision requires that you be insane, or don't know any other tools.
Setting up a website which is secure and not vulnerable to SQL injection or basic BS is quite doable in PHP.
When I was a kid I imagined that banks were incredibly vulnerable to this.
Later I discovered transactions and thought my kid-self was just silly.
I've learned that my kid-self understood things pretty well, sometimes.
But, this is an outlandishly amateurish oversight. Race conditions and atomicity where balance-affecting transactions are concerned would seem to be one of the first considerations that pops to mind. That, along with sanity checks just before final execution of the transaction.
It's clear that there are a lot of people building exchanges with little to no experience in the financial/transactional software field. But, I am beginning to wonder how much experience they have building any software.
Bitcoin isn't going anywhere until the people who trade in it approach competence.
But apparently they didn't even check or put a proper constraint into the database that your balance should be positive (and if you do find a negative balance, shut the system and investigate). Now that's not a "pretty stupid race condition", it's reckless.
I know it's not easy but it is important to get it right.
Not checking for negative balance??? It doesn't get much more freshman CS than this.
$out = \DB::DAO('Money_Bitcoin_Block_Tx_Out')->searchOne(array('Hash' => $bean->Hash, 'N' => $bean->N));
if ($out) {
if ($out->Claimed == 'Y') {
$bean->Available = 'N';
$bean->commit();
continue;
}
}
On the minus side, $50,000 of innocent people's money just got stolen, and nobody is going to jail.
Another design flaw is that withdrawals should be queued at every step of the way. This could not have happened if withdrawals requests were processed sequentially instead of simultaneously.
Oops!
It's invalid logic to assume that because some bitcoin sites have been hacked, all of them will be, but at this point it seems pretty clear the entire community needs to be a little more careful. Until that happens I don't think keeping a balance with a service like this makes sense.
You can certainly put money in right before you execute a trade, and pull it back out right after. But that introduces significant lag, up to an hour for bitcoin, depending on how many confirmations you wait for, and several days for USD.
That said, I don't understand why people keep significant balances in these places. Keep a small amount to trade with and save the rest yourself!
Yeah, that's what I mean.
You can certainly put money in right before you execute a trade, and pull it back out right after. But that introduces significant lag, up to an hour for bitcoin, depending on how many confirmations you wait for, and several days for USD.
Thanks, that's definitely something to consider. I can see it being very inconvenient, though I still agree that the idea of keeping a balance larger than your trade volume in a service like this is hard to fathom.
"Right now, all markets and withdrawals are still frozen, and they will remain that way until the negative balance watcher is written and in place and balance deductions are calculated."
A "negative balance watcher" sounds like a horrible idea. Isn't this a solved problem? Atomic database transactions.
Does not inspire confidence.