MongoDB vs. Clustrix Benchmark

Just a point of clarification, but it is a per-server lock, not a global lock across the whole database.

> I saw a video the other day of some guy at Google giving a talk about the database behind the app engine. At one point someone in the audience asked about scale and SQL. His response was "Well, how well does SQL scale?" Everyone in the room laughed.

Sell your strength, don't attack others weakness. Demonstrate how your product is actually very similar to megastore (strong consistency, use of distributed transactions) and differs only in the query language ("GQL" itself is very similar to SQL to the point where an application developer could care less about which one is using). It looks like this is case the case (except I believe you guys are using 2PC instead of Paxos, in which case you want to clearly state the availability scenarios which Paxos addresses and at what costs).

Of course the problem with systems like megastore is that the overhead they introduce is seldom worth the compromises the application developers have already made. Incidentally, form what I hear, Megastore is not used very frequently outide of the app engine: application developers at Google itself are comfortable with using Spanner/BigTable directly but aren't comfortable with the additional overhead of distributed transactions as used by Megastore. App engine, customers, on the other hand aren't particularly interested in low a in the first place but aren't used to design patterns used on top of systems like BigTable.

By the way, I think Clustrix is a great product. I remember reading the white papers/specs when it came out and I'll make sure to scan it over again. Just why in the (imo, useless) SQL/NoSQL marketing circus rather than focus on the product's strength?

I personally think MongoDB is a poor comparison choice: it has the buzz, but there are some serious flaws in its distribution architecture (namely ill defined consistency and partitioning model) and the outdated disk persistence story (Mendel Rosenblum wrote his dissertation in 1992; building a log structured database is no less risky than cloning circa-1995 MyISAM). Comparison with another strongly consistent system like HBase or HyperTable would be more fit.

Entirely agree with you, eventual consistency and quorums protocols all have their drawbacks. Difficult of developing against that (and the natural reaction of just "handwaving" the consistency model rather than using optimistic locking with a vector clock at the identified synchronization points) is the main one. I have, however, also seen the situation where developers would fail to understand how to correctly deal with availability and build ad-hoc, poorly thought-out HA layers on top of non-HA systems. The typical memcache + MySQL deployment is the prime example of that: there's neither the high availability, nor an understandable consistency/cache coherence model.

> Note that his "right answer" for performance (not availability, so it's a bit of an aside from your comment) is to dynamically repartition, which is what Clustrix does. Note that we can actually do this type of repartitioning on just the "hot" data, using MVCC to avoid any downtime (unlike Mongo, which blocks all writes).

Dynamic repartitioning is also what HBase/BigTable do ("tablet splitting"). The MVCC approach is interesting way around the unavailability window, by the way.

> By the way, the Google Megastore project I believe you're referring to is not actually eventually consistent in the same way that, say, Cassandra is. Megastore is fully ACID within an entity group, meaning that lowering eventually consistency down to the per-node level was not the solution that Google went with. BigTable is also not eventually consistent.

Yes, I did not mean that Megastore is eventually consistent. It is not, it's ACID compliant and uses Paxos. Real question: was this not stated clearly in my comment (I'd like to edit to clarify if it is)?

Megastore, however, is an example of a "NoSQL" system and I meant to use it as proof that not all "NoSQL" systems give up consistency: ACID is orthogonal to the query language.

Quite frankly I find the whole query language debate (and "NoSQL" marketing gimmick) to be quite pointless, the distributed systems and systems programming aspects are a lot more interesting to me.

> This is why Cassandra is now offering counters as an API feature

This is specific to Cassandra, where the development team chose not to implement optimistic locking via version vectors. That said the counters patch essentially implements a vector of counters, very similar conceptually to a vector clock.

That said, there are some scenarios where quorums (in absence of an agreed upon view of the cluster e.g., a zookeeper-based failure detectors) could mean concurrent vector clocks, which means it's up to the application to reconcile it.

If your application can not reconcile a split brain scenario on its own and depends on a total order then yes, it's not a good fit for EC.

As I've said, I also think that serializable consistency is easier for applications to deal with (they can assume a total order) than eventual consistency but provides the strong benefit of read availability (something, quite frankly, most Internet applications require).

Most complex applications have points where consistency can be relaxed (either somewhat or fully) and points where consistency is needed.

In "we chose consistency" and "we chose availability" cases, the applications will have to deal with the consequences of those choices: much as developers may handwave the implications of an eventually consistent system (since it returns right results most of the time, not thinking whether or not their application is sensitive to a non-serializable order) they might also implement ad-hoc, poorly thought out HA solutions on top of strong consistent systems -- often times losing both availability and consistency.

The typical usage scenario of masking MySQL failures and latency by using memcache comes to mind. VoltDB is very correct to remove the need to use memcached for read latency (a caching layer in front of a database which already a cache which sits on top of a file system with a cache is a caching layer too many), but in many cases I've seen memcached used to accept reads (and sometimes writes) while the underlying database is unavailable.

Finally, the whole debate about atomicity is quite meaningless if (like many applications) you're fronting your data access layer with an RPC framework (your RPC call may go through to a database and perform a write even it times out to the application i.e., implying non-atomicity).

The CALM paper is a big move into the right direction: being able to determine synchronization points at which ACID transactions (or weaker forms of synchronizations if permissible e.g., quorums and version vectors) can be used. Incidentally, I also think a declarative language (whether like Bloom/Bud or closer to SQL) is a good way to express the constraints and let a system (which may not be all that different from a query planner) determine whether those points occur.

On the other hand, allowing potential customers to easily try your product can destroy your chances at gaining them.

Here's a quick example:

A few years back, I was writing Smalltalk web systems and evaluating which commercial implementation to use. The two big ones that bubbled to the top of the list then were Gemstone and Cincom.

Cincom offered a relatively familiar set up which would require using sticky sessions for the web server, and another backend database like PostgreSQL.

Gemstone offered pearls on a platter: shared session state across all the backends, and a build in distributed object database. I thought it would be brilliant, and reading their blogs and documentations gave no hint of any pitfalls.

Cimcom was a download away---as were any of the free smalltalks---but Gemstone then required you to contact their sales staff before playing with it. So I made a mistake and chose Gemstone, and the system was built to target their as of yet unseen architecture. When I finally got to use Gemstone, it was a mess!

If I had that experience to start with rather than just their glowing self-produced reports, I'd have never picked Gemstone and instead gone with the familiar "kludgy" system that was "good enough".

If I want to trial Oracle or Clustrix, well, I'm going to have to start a contact with a sales organization and then they need to pre-qualify me, and then they need to qualify me and then I'll spend a few hours on the phone talking to people (which might take a few weeks in wall-clock time.)

If Clustrix offered cloud-based deployment of their database, they could have a small free tier for people to play around with. Add a console applicaiton for accessing the API. Make it really developer friendly.

Imagine if you could just type:

    $ sudo gem install clustrix
    $ clustrix create my-test-app
    Created cloud.clustrix.com/my-test-app
    username: my-test-app
    password: xd634shx

If they had that, I'd be trying it out in a flash. Then what if they had a per-usage pricing system like AWS? Something where you pay per gigabyte. Something that allows you to test stuff out for a few dollars a mount, and then when you're ready, scale it right up to a few thousand or more.

Small nit: You can download Oracle 11g (or 10g) strait off their website for free — you just need to sign up for a free developer account.

Here's a link: http://www.oracle.com/technetwork/database/enterprise-editio...

Indeed. A lot of the reason I like document databases so much is that you can solve problems differently (and many times, much more easily) than you could in a relational database. Compare:

    db.posts.find({tags: {$in: ["foo", "bar"]}})

to:

    SELECT * from posts JOIN taggings ON taggings.post_id = posts.id JOIN tags ON tags.tagging_id = taggings.id WHERE tags.tag IN ('foo', 'bar');

(Single query tag lookup; naively joins the entire taggings and tags tables before limiting with WHERE)

Or a "better" query with two subselects (yikes!)

    SELECT * FROM posts where post_id IN (SELECT taggings.post_id FROM taggings WHERE taggings.tag_id IN (SELECT id FROM tags WHERE tags.name IN ('foo', 'bar')))

And that's the "find where any tag matches" case. Try the "when all tags match" case ($all in MongoDB), and you'll go grey a few years earlier.

Serious question, because I don't know: what's the advantage over a denormalized SQL table?

Last time I heard, twitter still uses MySQL for the statuses (tweets) table. They did have a plan to migrate to Cassandra, but didn't go all the way through with it. So I find it hard to agree with "you just can't scale those". It may be a lot of work, but for some applications you can scale them.

Edit: twitter cassandra link (don't know if this is the latest): http://engineering.twitter.com/2010/07/cassandra-at-twitter-...

You can't scale MySQL/PostgreSQL? Care to expand? (Although Facebook's schema is likely an abomination flying in the face of every normalized form, it does run on MySQL. And it's not like throwing PostgreSQL on a big box doesn't go far.)

I can scale PostgresSQL pretty well, thanks. PostgreSQL is used on some of the biggest databases in the world, but no one's going to tell you which. :)

Your benchmark is pretty much irrelevant because Clustrix sells hardware appliances, not the software standalone.

Edit: That's not to say MongoDB didn't fail your benchmark. But the issues you highlight (single mutex) are known. Test against a real database and your results don't look that impressive for a 10 node setup.

All that said, Clustrix looks great. Just make it available sans-hardware.

Profitable companies have usually figured out that it's better to buy a solution that solves their problem rather than: (a) pay for engineer time to integrate a solution that isn't quite right, (b) take the opportunity cost hit while waiting for (a) to be done. These costs can pretty quickly dwarf the purchase costs of licenses and hardware. Of course if there's a free solution that just slots right in, double bonus, but for the most part the cost of hardware/software is noise next to the cost of people to maintain it.

I think I see Redis more than Mongo.

I get the impression that Mongo is simple to use and simple to program for; Cassandra is quite good, but also more complicated.

Well, don't trust a benchmark that you haven't faked yourself they say.

I was just trying to point out that MongoDB is too easy a target here. The problems it has under high load are fairly well-known, at least to anyone who tried to benchmark it outside of their MacBooks. Just bulk-load a couple million records and watch it tip over if you don't believe me - I'm not making it up and neither is Sergei.

However if Clustrix wants to impress with benchmarks then they should pick an equal opponent. MongoDB is not exactly relevant for companies that consider the calibre (and cost!) of Clustrix.

You can't randomly throw numbers around without listing your benchmark code, the config files, etc.

Secondly: even though it becomes clear eventually, you should mention up front your relationship with Clustrix. Just because you are a founder of Clustrix doesn't necessarily invalidate your findings, but full disclosure is always a good idea.

Along with licensing costs?

imho, the second, except that some consistency is present ib document dbs: like in the order example, all order items will be within the document and wont be lost without losing the whole order.

no-one would ever use 'eventually consistent' databases for financial or trading data, i think. they're for higly scalable consumer web projects