NoSQL Meets Bitcoin and Brings Down Two Exchanges | Hacker News Reader

NoSQL Meets Bitcoin and Brings Down Two Exchanges

hackingdistributed.com

103 pointssilenteh68 comments

While I do recognize this is a marketing piece for HyperDex, I'm disappointed HyperDex only chooses to compare itself with MongoDB and Cassandra.

It would be different if HyperDex was fully open source and free, but its not. The "transaction" feature is only available with a license, which in most cases will price you out of any developer who is looking to start a bitcoin exchange. So unfortunately it comes across as "If only you had spent $x/mo, buying our product, then you wouldn't have had this problem!"

Now that isn't bad, and its the definition of marketing, but then it goes on to compare itself against Cassandra (which has a widely different use case in the wild) and Mongo (which is already the whipping boy of the database community). However it doesn't help the use cases are also extremely well served by something like Postgres, which doesn't need a Warp addon, and has a huge developer community.

What I would be more interested in, is how does HyperDex compare with the new "NoSQL" databases (FoundationDB, OrientDB, Spanner-like Databases) or even the traditional commercial SQL like Oracle.

In short, it doesn't help to test and compare unsuccessful and unpopular use cases with these technologies. Most competent people, the same people who might pay for HyperDex, already know Cassandra is bad at transactions. What I would like to see is tell me why someone like Netflix, or Ooyala (tried, tested and popular use cases) would move from Cassandra to HyperDex.

Cassandra's transaction support has been improving pretty rapidly.

http://www.datastax.com/dev/blog/lightweight-transactions-in...

For what I can only guess are strategic reasons, Datastax is moving away from the schema-less design that made Cassandra popular with early adopters. There are workarounds to this deprecation such as using the old Thrift API, but it doesn't really inspire confidence in the product unless the new schema oriented way of doing things appeals to you. PS those aren't transactions.

"Schema-less" is a hack that appeals to people that don't realize that there is always schema. It's just a question of whether you want your data model to be ambiguous and poorly defined, or explicit.

Schema-less also means that you can afford to have deviations in your schemas not hinder your development needs. I agree with you, but I'd venture to say that a schema-less database means what it does, but that a collection of data always has schemas.

No, the schema is always a property of the system as a whole, data alone doesn't have an implicit schema. The schema is the set of type constraints that data has to conform to so that it can be processed by the system. Thus, a DB being "schema-less" simply means that it won't check/enforce those constraints - the schema still is there, of course, simply because a given piece of software that does anything meaningful will not sensibly process any random piece of garbage you feed it.

So no schema means no validation of data for conformance? I thought these "NoSQL", or rather the ones that explicitly state to be schema-less don't do such. People who take the lack of schema as the licence to throw data into a black hole are somehow misleading themselves. The purpose of a database is to store data for later retrieval and modification. Schema-less, as we both understand it, means there is little or no validation by the database. In light of what you corrected me on data, am I correct in saying that useful data at the very least has to conform to some uniformity, from which a DBA would derive a schema, even if not enforced by the database?

Yeah, that's what I meant by "the schema is still there" - "useful data" is data that has meaning according to how the system (software processing the data) interprets it. But my point is that the uniformity can not be derived from the data, but only from the interpretation rules of the software.

I mean, you can, of course, derive some "rules of uniformity" from the data, but that most likely would be completely useless, as you might just find patterns in the data that emerged by chance rather than because they distinguish meaningful data from nonsense, and you might easily miss important constraints that are required to keep nonsense out of the database. Of course, sometimes you can infer from data heuristically what sensible constraints might look like (because field names have some meaning to you, for example), but that really is just a heuristic shortcut for finding out how the system interprets the data.

We aren't all building cat photo sharing apps.

Sparse columns dbs (AKA schema-less in Cassandra lingo) have all sorts of important use cases in the sciences, unless you feel like paging through 100,000+ empty columns to find or discard the information you need or don't need.

The alternative approach is to alter your table's schema every time you find a new sub characteristic of how a given system is expressing itself.

Cassandra didn't actually get rid of the schemaless structure, and you can still use it. All the introduction of CQL did was make some assumptions on the storage format, based on common use cases, and essentially hid the underlying format for you.

If you read up on CQL3, you will see the schema-less trait is still there, and just as performant, but only slightly confusing if you come from the SQL world.

In a way, CQL3 is a "hack" to make a schema-less database look like it enforces a schema. The Primary Key became the Row Key, the secondary key became the cell name, and the value became all the other fields. The column names are still dynamic, and the schema doesn't really exist on disc except in metadata.

> The alternative approach is to alter your table's schema every time you find a new sub characteristic of how a given system is expressing itself.

Which is what you're doing regardless.

Untyped is just a sum type.

This is not true. See this for example: http://www.datastax.com/dev/blog/thrift-to-cql3

You can do CREATE TABLE schemaless (id blob, name blob, value blob, PRIMARY KEY (id, name)) WITH COMPACT STORAGE; and have a 'schemaless' table, essenitally, just as with Thrift.

This is not a workaround either - this is first-class supported, just as you can read/write from/to 'schemaless' thrift columnfamilies using CQL.

Sure, would you mind showing me how to use CQL to insert data into an old-school style Column Family that was created with the Thrift interface? Linus Torvalds would be up in arms if he saw this kind of client facing API breakage.

CQL is compatible with Thrift-created column families (the inverse is not true, though). Just do a DESCRIBE SCHEMA from cqlsh and see how CQL interprets a particular Thrift-created column family, then use regular INSERTs.

> It would be different if HyperDex was fully open source and free, but its not. The "transaction" feature is only available with a license, which in most cases will price you out of any developer who is looking to start a bitcoin exchange. So unfortunately it comes across as "If only you had spent $x/mo, buying our product, then you wouldn't have had this problem!"

Just use PostreSQL. Open source. Free. ACID compliant.

That's because banks employ systems that guard against this kind of elementary error. It's called transactions with ACID guarantees.

Bank systems are often not, in fact, ACID. Think of how many billions of transactions are processed in end-of-the-day batch runs, for example. Overdrafting is not only not impossible, it is a ludicrously profitable feature of the system.

How is this secure? Velocity limits, sophisticated anti-fraud systems, acceptable losses, and, as a bit of a by-the-way, the fact that people who make a habit of defrauding banks generally have a few weeks prior to being woken up by very serious men with guns.

It is interesting because the transaction log inconsistencies can often be resolved technically and because of the regulatory framework. Overdrafts can be "solved" in business sense.

Now imagine something like an bitcoin exchange and bitcoin transaction log. Transaction errors cannot be fixed easily. So overdrafting an account means never getting the money back. There a single account would need to have ACID properties in way.

It seems ability to deal with and tolerate data conflicts and inconsistencies is different for each business. There was some interest in CRTDs (data types that can automatically converge after a conflict occurs, like imagine a max() function or a set union operation) but still means shortly there could be an inconsistency in the system and sometimes that can be exploited and is just not an option.

Bank systems are ACID. They're often not immediate, using batch runs as you say, but the databases the batch runs work on are definitely ACID.

The problem is not necessarily with overdrafts, as you suggest, but - as the article says - with race conditions between multiple readers/writers causing two near-simultaneous withdrawls to only make a single debit to the source account, while crediting both destination accounts.

Note that this could lead to unexpected overdrafts being noticed later, if the exchange also wrote a consistent log of all transactions, and attempted to reconcile the log with the real-time balances periodically (e.g. on a daily basis), but it's not clear if they even did that. (Although I might be missing something)

That's why banks have an available balance and a posted balance. One is the best available estimate of how much money you can spend. The other is the reconciled balance after transactions have been posted.

It would have been possible for the exchanges to have used a NoSQL db with this strategy, but they didn't. They should have been keeping transaction logs and reconciling balances, but they didn't.

While, I'm no fan of MongoDB, it isn't really their fault. These exchanges were designed by people who didn't know any better.

If I'm not mistaken, an eventually consistent database must also guarantee, well, eventual consistency.

Sure! All nodes will be consistent in that your balance is what they all think it is, and that the attacker's balance reflects the attack. But unless those actions are coupled in a transaction, each account may not be consistent with the other.

The problem is that while overdraft is very profitable for the bank; you acquire a legal liability to pay back that debt.

Overdraft cannot be offered in a non fractional exchange and the exchange can't find you if you take more than your balance. Money leaving the exchange must be reconciled with holdings or they are insolvent.

What we may see is blockchain denoted exchange holdings; your exchange wallet is unique and publically verifiable (only the exchange holds the keys). You could verify with balance with the address and all trades are performed on blockchain. Bitcoin would be inappropriate for this with its long confirm times but a newer faster scrypt or primecoin variant might do it. Exciting times!

The problem is that eventual consistency doesn't mean eventual correctness. The basic issue is that math usually only works if you do it in the correct order, e.g. (X+2)5 != (X5)+2. And if you're using a document store to do mathematical transactions then you're not guaranteed that they execute in the correct order, all that you're guaranteed is that all the nodes will eventually be the same in your database.

But debits and credits do not use multiplication nor division, so for those transactions the order doesn't matter.

If eventually consistent databases stored pending operations rather than pending values, this would matter. Unfortunately, they actually tend to be value based. So when you perform an operation on stale data, either the new operation will be lost, or the old one, depending on the implementation.

Banks are made of a mess of different systems. Some are locally ACID, but overall reconciliation and compensation are what keeps it all together.

Regardless, this has little to do with banking. The systems are too widely different to be compared productively. The only conclusion you can get from this (marketing) story is that when you have an eventually consistent system, you will need occasional distributed locking. Cassandra often comes in pair with ZooKeeper.

The other conclusion is that no matter how trivial they sound, distributed transactional systems are far from trivial.

ACID would have solved this problem though, right?

ACID is simpler to reason about. If there is a strong justification for abandoning it, and enough expertise to still make the system work, then fine. But I doubt they had either the justification (I assume their transaction volume could be easily handled with a SQL system) or the expertise.

I'm bothered that MongoDB strives to make people feel comfortable abandoning ACID who clearly don't understand what they are doing.

The first problem to be had when considering distributed systems is the fact that there is no such thing as a general purpose distributed system.

Some problems require doing 50x more reads than writes. Others have more writes than reads. Sometimes losing a few inserts is not a big deal, but a small delay is terrible. Other systems just can't tolerate data loss, and would pay any price to keep it that way. You might have a lot of small pieces of data, searchable in very simple ways, or you might have multi GB documents that require so much indexing than Solr and Elastic seem inadequate.

In the old days, you just bought a bigger box for the RDBMS, and you just were covered by knowing one tech. Now we need to go through a wide variety of tools that will rarely do everything you want, and have to write a bunch of code to compensate for the limitations of the tools. And then your DB of choice decides that they will only support their self hosted product, then they get bought by IBM, and you wonder if you'll be forced to retool your entire backend (Hello Cloudant!)

Until there is some clarity in the market, and we get to see distributed stacks that are general purpose, we'll see issues like this popping up all the time, as there is a lack of people that both have good knowledge of all the available options and the skill to put them together into something that will solve your specific problem.

Please don't be MongoDB, please don't be MongoDB

...

Sigh

> a failure of distributed systems academics to educate developers and to equip them with clear-thinking frameworks.

Here is the thing, distributed systems are not standard academic courses. Just like networking they are often optional. That is really sad and it needs to change.

The CAP theorem is not something fringe and exotic anymore it should be standard taught to everyone.

MongoDB has really messed up. They promoted and marketed a shitty product wrapped in nice marketing. It created 2 things -- low barrier to entry and a trap. Both wrapped in one nice shiny package. "You should have read the docs to enable actually writing data disk when the user asks them to" doesn't jive with me. A really shitty default setup for something calling itself a database.

Oh well, they reaped the benefits of their strategy, both the short term (success) and long term (failure) - it became the butt of jokes and ridicule.

Have you actually used the database since if you did you would know that the write concern was set to FSYNC_SAFE by every single driver (before it was the default) and given the lack of configurability should have been known by every developer.

It never ceases to amaze me how few technical people actually "get it" even for products designed for themselves. Many, many people like things to be simple and easy. MongoDB fits this criteria perfectly and no matter how many times you call it a shitty product isn't going to dampen its popularity.

I, too, don't like the way MongoDB works and is marketed, but I would not bet too strongly on them failing. Only 10 years or so ago MySQL did pretty much the same, but they managed to succeed, despite there being better alternatives at the time.

Even today, after all the development effort put into MySQL and its derivatives, there are still better and more capable databases, but that doesn't prevent MySQL from being one of the top three RDBMS-es (and we can bet it is the most popular FOSS RDBMS).

"Historically, Bitcoin exchanges that suffered significant losses turned into fractional reserve banks, only to fold later."

That is not what fractional reserve banking is and I wish bitcoin articles would stop misusing the term.

The continued misuse of that term seems telling. If they really think this is how normal banks work, it helps explain why they set up these bitcoin systems in such crazy ways.

It's the same story over and over again. The young wild guys arrive, full of energy and ambition to make the world a better place. They throw all the ancient stuff overboard because it seems ugly and useless. Then they painfully rediscover why the ugly stuff is there and not useless at all. And after a lot of pain they finally arrive at a slightly improved version of what they threw overboard long ago, at least if they did not surrender before.

> The young wild guys arrive

That's exactly the problem. The 20-something hipsters who start these companies don't yet have the life experience to realize how ignorant they are. So of course they don't pay any attention to the lessons of history. And so they, very painfully, re-learn them.

If the hipsters had some humility they would have also hired a few grizzled, experienced old fogies.

Fortunately not all startups assume that anyone over the age of 30 isn't worth hiring. For a non-technical example, look at how Google hired Eric Schmidt to provide adult supervision.

TLDR: Exactly what you might have guessed: the exchanges backed their financial operations with a datastore without transactions, tragedy ensues.

This can't be viewed as an indictment of "NoSQL" though; there are plenty of databases featuring ACID transactions that qualify as "NoSQL" (FoundationDB and OrientDB come to mind). That said, none of the popular NoSQL contenders have them, and the new guys with transactions tend to be commercial, so if you need transactions in an free and open-source datastore, just use PostgreSQL.

A bit irritated at how the author complains about other databases astroturfing on Hacker News, then writes a glowing post about HyperDex without mentioning that he's one of the authors.

That said, I agree with the thesis that current-generation distributed systems provide a terrible API for users who actually want to get concurrency right. I'm optimistic about projects like Summingbird[0] and Bloom[1], which run very high-level declarative programs with simple consistency guarantees.

[0] https://github.com/twitter/summingbird [1] http://www.bloom-lang.net/

Its unfortunate to use the ATM example because just such an exploit was used by carders [1] to get more money out of accounts than the accounts actually had.

[1] http://time.com/48344/hackers-target-atms-for-unlimited-with...

ATMs aren't as smart as the article says either: "the real code will check to see if there are sufficient funds".

I noticed that (some?) ATMs around here tend to implement this flawed algorithm, paraphrasing the article:

  0. database.begin()
  1. mybalance = database.read("account-number")
  2. newbalance = mybalance - amount
  3. database.write("account-number", newbalance)
  4. database.commit()
  5. dispense_cash(amount)

At first sight this might seem ACID, but its not. Consider what happens if the ATM is out of cash: step#5 fails, but the money was already withdrawn from your account!

It doesn't automatically revert the transaction either: you have to go to a branch office and ask for the transaction to be reverted. So much for atomic transactions at ATMs.

Whilst that maybe the way your bank has chosen to operate, it is most certainly possible for ATMs to report a step#5 failure to the banks host which will reverse the transaction.

Wow I've never seen this happen. What happened to step -1:

  if (cash_in_machine < amount) abort

He is talking about a failure to dispense the cash.

I've had this happen twice to me whilst overseas.

Interesting. In one particularly ADHD-inspired case, I've also seen

  5. Dispense cash
  6. Wait a minute or so
  7. Retract cash

where 7 did not lead to a credit on my account until several days after I reported the problem to my bank. That's probably by design, however, since otherwise the machine would require automated defenses against surreptitious replacement of dispensed bills with counterfeits.

Your example could be a similar failsafe against cases where the machine fails to properly dispense cash in a manner that nevertheless allows the cash to be retrieved. Otherwise you have to carefully separate errors that can't possibly lead to cash being dispensed (e.g., hopper empty) vs. those that might (e.g., bills jammed somewhere in the dispenser mechanism).

Quite so. If cash is presented to the customer and then retracted there isn't (or at least wasn't) a way for the ATM to know exactly what had been retracted, i.e. you requested amount X that required the ATM to dispense Y number of notes, but the ATM wouldn't know if the number of retracted notes Z was the same as Y. The ATM would have reported this event to the bank's host and would have been present in the daily transaction log.

As you suggest, a customer initiated account credit is necessary to prevent fraud, and the failure to dispense claim can be verified against the host transaction log.

This article was only written to promote something called HyperDex which he is in some way involved.

Mongo does have document-level atomic update operators, such as $inc: http://docs.mongodb.org/manual/reference/operator/update/inc...

Yes this was completely ignored by the article. This along with a 'where' clause can guarantee the operation will only succeed iff there is sufficient funds. I don't know if other NoSQL DBs offer these sort of atomic operators.

Which makes this statement seem particularly unfair: "The problem here stemmed from the broken-by-design interface and semantics offered by MongoDB." He seems to call out Mongo by name a couple times but doesn't say that the exchange was actually using Mongo.

Sounds like the author of this 'article' just wanted to make a thinly veiled plug for the "HyperDex" product.

Blaming all of this on MongoDB is silly. You can do the kind of idiocy displayed by these exchanges even on generic DBs by doing things like not using transactions and/or not waiting for write confirmation.

The root problem is in the (lack of) design and programming of these systems not their data store.

As soon as I saw the term NoSQL and "Brings Down" in the title of the submission, I knew it was going to be MongoDB. Firstly, I don't understand the hatred towards MongoDB. Like most NoSQL datastores, it doesn't offer things like transactions and ACID compliance: this is well-known thing. There are some things that MongoDB does very well, currently I am using it to store clicks for an internal application which then displays those clicks on a page overlay, it handles the task very well. I would never use it for

MongoDB is not to blame for bringing down the exchanges that decided to use it, the stupidity of the developers behind said exchanges are the sole reason. You can open up a can of tinned pineapples with an axe, but that doesn't make it the right tool for the job.

This post was one big marketing piece for HyperDex and while it is touted as open source, but that isn't entirely true. If you want support for transactions, you have to by a licence. So basically what this post was saying is, "If you spent X amount of money on a HyperDex licence, you wouldn't have had these issues" most people that start these exchanges start them from nothing and probably can't afford or wouldn't warrant spending money on a licence.

If you replaced MongoDB with pretty much every other NoSQL database out there, the result would have been the same. Mongo just has a better marketing team behind them in the form of 10Gen getting people to buy into the hype. However, as far as I am aware of, 10Gen aren't telling people to use MongoDB for exchange platforms and anything involving transactions (I could be wrong though). This is a classic case of seeing people talking about how NoSQL is the future and others getting caught up in the hype.

If you want a database that you can rely on, especially when dealing with transactions: use something like Postgresql, not MongoDB and or insert over-hyped NoSQL database here

> it doesn't offer things like transactions and ACID compliance: this is well-known thing

Evidently not. Which is why people are still using it for financial based systems.

> the stupidity of the developers behind said exchanges are the sole reason

Exactly. The fact that they picked MongoDB for this application in the first place tells me that even if they used an ACID-compliant database, they still wouldn't have known to use transactions, and would be vulnerable to the same attack.

This article lacks a fact base on what actually happened and seems to totally rely on conjecture in order to compare Hyperdex against leading NoSQL solutions.

"They did what anyone would do after reading one too many astroturf articles on Hacker News. Sure, their system failed, but in a sense, the overall system failed them."

The technology didn't fail them. On the contrary, the technology did exactly what it was supposed to do. The devs could have (should have) considered these race condition scenarios.

you can put ACID like transactions on top of NoSQL databases, I did it here, but it took a static analyser to help get the logic right:

https://github.com/tomlarkworthy/firesafe/wiki/Send-Item

Marklogic supports ACID transactions. being that it is also a horizontally scalable system, it allows those transactions not just to spawn multiple objects, but also across multiple hosts. http://www.marklogic.com/blog/can-you-pass-the-acid-test/

They also support data tiering into HDFS, and now most recently RDF with SPARQL

They have been doing ACID for a number of years, not sure for how long.

They are not free (unless your data set is small)

Not quite understanding the focus on NoSQL for this use case set. I love MongoDB in the right place and Redis in others. But when you have a bunch of requirements that lend themselves to traditional DBs, choosing MongoDB or HyperDex feels like choosing a tech because it's cool rather than because it is the proper engineering choice.

Question!

Do transactional NoSQL(others than HyperDex) databases exist? or do transactions make horizontal scaling difficult?

Cornell CS professor, superb work.

"What may or may not have happened at Mt. Gox"

Why wouldn't they use a plain old ACID database? Was their load something that Oracle couldn't handle?

What are the advantages of nosql btw ?

Basically developers came for the scalability and stayed because they are so easy to use/manage.

NoSQL is fantastic for backing item-orientated webapps, it removes the transactional and relational overheads. Conversely I would not use NoSQL for anything concerning FIRE or complex inter-related data - those safeguards become important.

Scalability is something of a red herring with NoSQL. Yes it's easier to scale, but my primary criteria for choosing it would be the nature of the data, not the intended scale.

A more "use-case" example than your other responses.. I believe Amazon uses a NoSQL database for their product data. It's tons of reads and relatively few writes. They don't give a crap about transaction quality for product descriptions, comments, etc. However, when you get to the checkout page and they start doing things like taking your money and updating their inventory... that's and ACID compliant database. When you're dealing with money you want transactions, rollbacks, etc.

I don't think that MongoDB or any nosql is right choice for financial application. You can build for yelp but not for banking.

But, it is webscale!

> It points to a social failure: a failure of distributed systems academics to educate developers and to equip them with clear-thinking frameworks.

So, will the OP do an exchange and show us how it's done using his database of and approach of choice. Because writing articles on other's failures is kinda of... Easy.

I'm a little sick of this attitude. Nothing personal with - what I thought was Sinan Eren at first, confusing the names after so many years - the OP here, but given the fact that he is deliberately calling out amateurs and poor designers people who (apparently) deserved it, he might very well create a secure bitcoin exchange in the real world, using his technologies.. There is a huge market for that, today more than ever.

The attitude is warranted when they're losing other people's money.

Totally get that, but his approach seems a little snooty to me and runs throughout the article that's why I had this reaction/comment.

Reply on news.ycombinator.com