What MongoDB got right | Hacker News Reader

What MongoDB got right

blog.nelhage.com

144 pointsreqres111 comments

> Let's start with the simplest one. Making the developer interface to the database a structured format instead of a textual query language was a clear win.

I think this is the most significant factor, by far. With Mongo it's turtles (or at least Maps/Hashes) all the way down, without a strange pseudo-english layer near the bottom that forces you to translate back and forth. For some devs that's a big deal.

For the last while I've been experimenting with bringing the same feature to PostgreSQL (http://bedquiltdb.github.io), turns out it's very do-able, but I don't have enough time to make it as featureful as it needs to be.

SQL is still one of the most readable languages in my opinion. Its the one language where I find it easier to read queries than write them.

I should clarify, I'm a big fan of SQL and relational DBs, but it's hard to deny that some/many devs find the abstraction boundary a bit weird.

And really, when you think about it, it is a bit weird, (we send sentences of almost-english toward the DB, then get structured data back) and there's nothing wrong with that.

But many devs do value the ability to stay in hash-map land all day without needing to think about how to cross an abstraction boundary into the DB, and MongoDB actually has a pretty cool solution there.

SQL is fine, no problem with it unless you are embedding queries in a OO application and you are forced to hack mapping layer between the two interfaces, which gives you two options, either compromise performance for the sake of development easiness or vice versa.

I’ve played a bit with Jeremy Evans’s Sequel (http://sequel.jeremyevans.net), and I’m not sure that ORMs are necessarily constrained along this dichotomy (easy to use but with poor performance vs great performance but hard to use). Sequel seems to be about making the hard things easy and reducing the amount of time you need to drop to pure SQL to almost zero.

As someone who has built with Oracle’s Pro C(++) and with various ORMs, Sequel is an ORM that makes me reasonably happy and gives me the expressiveness of embedded SQL without compromising my object model.

> SQL is fine, no problem with it unless you are embedding queries in a OO application

which is essentially every single production deployment of a SQL database in the last twenty years. That's the problem.

A lot of people are using ORM's.

SQL's fine on its own. The problem arises when you try to manipulate it. A query to get messages that were sent between now and a week ago, joining in the user table for both the sender and the receiver to get their full names, is sensible enough when written out as SQL. I can quibble about some of the affordances of SQL, but it's good enough for most queries. The problem is if you try to create some sort of data that represents "get the message body", "messages sent since last week", "get the sender's name", "get the receiver's name", and somehow programmatically assemble a query out of the pieces. (Slight quibble, this isn't actually about OO, this is about "anything that isn't SQL". A Haskell library will have the exact same problem.) An AST-based approach doesn't guarantee that this will be easy, but it's still a step above trying to assemble an SQL query from those pieces.

It's totally possible. There's a ton of libraries that do it in various ways. It's just not even remotely easy, like it should be. If you look into the inside of those libraries you'll find a C'thulian monstronsity of special cases interacting with special cases until the whole thing just explodes into a brain-consuming mess, because SQL was very, very clearly not written for this use case.

In another 10 or 20 years I look forward to data-based analysis that tries to determine how much of the "NoSQL" movement was because the relational data model doesn't work for all use cases, and how much of it was the entirely accidental (in the Brooks sense) problems with A: SQL, the language itself, not its capabilities and B: schema migrations with no essential reason to be as painful as they are. (And something something column stores, but I'm not sure where they fit into this story exactly.) And to be clear on tone, I really am interested. I'm pretty sure the answer won't be either extreme but I'm pretty uncertain about where in the middle we'll fall.

> and somehow programmatically assemble a query out of the pieces.

You can do this today, in various statically typed languages[1] (perhaps dynamic languages as well, though the composition will likely be more ad hoc).

Agreed re: under the hood complexity, but that's more in supporting multiple database engines (and related corner cases) than the transformation of Query > AST > SQL.

[1] https://news.ycombinator.com/item?id=10525040

Having written some of this code myself, I have to disagree; transforming SQL fragments is legitimately frustratingly challenging. I have to admit I don't know how to show a small example that captures the problem, though, because this is one of those cases where the small examples always look easy. It isn't until you're trying to support all of them at once that it is a problem. Combining two fragments that each specify tables, joins, where clause filters, and potentially subqueries with each of those recursively is nontrivial when you get down to it.

It also really doesn't help that "SQL is declarative" is basically a lie, and it very frequently totally matters which "synonymous" query you actually throw at the database, thus eliminating a lot of the obvious clean answers in any practically-useful library.

Definitely not an easy task but the degree of difficulty depends on the language.

Both Haskell and Scala, for example, have sufficiently powerful type systems to allow for building up a typed query expression of arbitrary complexity, which can then be deconstructed via pattern matching to assemble the sql statement. Easy? Not at all, but very much doable, and incredibly elegant...until you need to support various database engines and their limitations/extended features; then the implementation hacks begin :\

Personally I think the work of Stephan Zeiger on the Slick library in Scala is ground breaking. Also, Wadler et al's recent-ish paper on a composable query DSL in F# is worth checking out[1]

[1] http://homepages.inf.ed.ac.uk/wadler/papers/yow/dsl-long.pdf

You seem to persist in believing the problem is on the input side. It's not; it's on the output side. My entire point is that the resulting SQL generation code is what ends up quite hairy, because the way we have found to separate concerns in 2015 and SQL are very, very different.

Let me put it this way... it is precisely because a fluent, Haskell-native SQL querying interface little resembles SQL in either syntax or usage that there is the problem. It is precisely that there have to be these library at all that is the problem. If it were easy, these library wouldn't even exist, or would be little more than drivers, but they're not just drivers... they do a lot of real work.

If SQL didn't stink by the standards of modern separation of concerns, we wouldn't need "ground breaking" work!

Or, to put it another another way:

    $ git clone git@github.com:slick/slick.git
    $ cd slick
    $ cloc .
    ---------------------------------------
    Language   files  blank  comment  code
    ---------------------------------------
    Scala      261    3589   3844     23129

Cut down to just the Scala. 23kilolines of Scala is a lot of Scala! This is not an "easy" task. "Easy" would be something that just wrapped up the existing syntax in a slightly more native form and would clock in somewhere in the several hundred range.

No, I'm saying input is easy, type system does virtually all the work. Output is where the effort is spent (i.e. pattern match on query expression to assemble the statement).

Why these libraries exist is because of string-ly typed programming; in the case of sql: 1) it doesn't compose; 2) is not safe (sql injection attacks); 3) difficult to refactor; 4) untyped, therefore whole class of bugs arise.

And yes, these libraries do a ton of work, well beyond just generating sql statements, which, in the case of Slick pushes the LOC count way higher (non-blocking IO, supports basically every database engine, native function support, jdbc modeled in scala, etc., etc., it's a huge engineering effort, somehow by one person).

Anyway, I'd like to see a much smaller composable query dsl with fewer features and opt-in database support. Compile that to javascript and run in the browser against local database would be very interesting. I think this can be done, but would probably be pretty restrictive in terms of features supported.

I find that SQL is pretty nice in a dynamic language with string templating available... I wrote a simple wrapper for MS-SQL in node using ES6's string templating to convert to parameterized queries[1]. The irony I wrote it while migrating a lot of data out of MS-SQL.. but the interface is probably the nicest SQL client I've worked with (despite being the author).

[1] https://www.npmjs.com/package/mssql-ng

> Making the developer interface to the database a structured format instead of a textual query language was a clear win.

No, it was not. It is an abomiation, just like SharePoint CAML Query or any other "express-your-query-as-an-AST" approach. Sure, it's a paradise for Buzzword-Compliant Fluent Interface Enterprise Query Builder-type libraries.

> Querying SQL data involves constructing strings...

No. Case in point: LINQ. How it is implemented is irrelevant to the statement above.

> ...and counting arguments very carefully

No. Named parameters to the rescue.

LINQ is just an "express your query as an AST" approach. Sure LINQ presents its self as a DSL on top of that AST ( which is exactly what SQL is at the end of the day ), but there are libraries for MongoDB that allow you to do the same thing. There is even a LINQ driver for MongoDB[1]. What you are missing is that by having the lowest common denominator be the AST instead of a DSL you have removed a huge burden from both the server and client side of the operation. From that AST you can build any kind of client side interface without worry about how to parse or compile the result into a specific textual language, even plain old SQL[2] can be used if you like your sever spending its clock cycles parsing text.

1. http://mongodb.github.io/mongo-csharp-driver/2.1/reference/d... 2. https://www.progress.com/connectors/mongodb

I like mongo's interface (most of the time)... but when you get to more interesting queries, it gets far more weird than even SQL... I wrote an early node API almost mirroring the JSON query interface (was easy enough to do filtering on sensitive fields), and it worked really well.

That said, it was really clunky dealing with ordering as JSON doesn't guarantee serialization order (sort fields), and that was cumbersome to work around iirc...

Today RethinkDB would be my first choice for a similar solution... better ops interfaces with replication + failover. Instead of having to do a replica of a sharded system in mongo for distribution with redundancy. Also, while the query interface is a little more complicated to get started with, when your queries get more interesting, it's not nearly as messy as mongo gets.

> or any other "express-your-query-as-an-AST" approach

Well SQL is just an AST abstraction, which was made to be human readable. And that's exactly the issue, SQL is excellent for humans to express querys but really inconvenient to interface with other programs unless you express every single query ad-hoc (which seems the way most people choose since ORM tend to perform poorly)

Maybe I'm weird and just like SQL but the query-as-data aspect of Mongo is actually my least favorite aspect of using it. Ad-hoc queries become torture, as do complicated queries. I might just be lucky to have used SQL Server and C# for so long which eases a lot of the pains you described.

Same. I wish there was a DSL layer on top.

If you use Scala and mongodb, I highly recommend this DSL:

https://github.com/foursquare/rogue

Typesafe queries using SQL-like syntax. Particularly when used with the Spindle library (https://github.com/foursquare/spindle), this is best ORM I've ever used.

That does look very nice. I've never actually used Scala seriously, but coincidentally have been learning how to use it all this week. I'm even more interested, now.

It's pretty easy to build wrappers for things like that, though. Build a constructor that has instances methods of just doing `myQueryInstance.addEqualsQuery(key, value)`. You can then just pass it around and add to the query instance as it moves around, deconstructing it at your database access point.

Not a fan of mongo's query format/language, but I have to agree. It's always bugged me that so much work has gone into relational databases, and yet the only way anybody interfaces with them is by using an extremely quirky programming language from the 70's (or whenever).

>>> extremely quirky programming language from the 70's (or whenever)

Because it doesn't have to be shinny and all, it just works. There are no widely used alternatives, so we can say that people are happy-enough using it. See the situation with JS. CoffeeScript and all, it's quite popular. Is there an analogy (in terms of functionality and popularity) in SQL? Nope. No one bothered to do it well enough, or not enough people found that it's worth enough to learn it.

We don't need to reinvent something every few years just because new JS frameworks come out and others are forgotten every few months.

Well you do have MS Access query builder, which I used for a while. Once you got used to it, it was maybe a bit faster than using real SQL. But then it locked you into the query builder way of thinking, and didn't provide very much beneft over textual SQL.

If you've been programming and using SQL for a few years, it's not a problem. It might take a bit of time to get the feeling of SQL, but it's ok later. And that's fine. rails-15min-blog things are cool and all, but some things have a learning curve and that's ok.

The vast vast majority of SQL in the wild is pretty straightforward and sucks only when the schema sucks (can't blame language for that). Most of the time people hate SQL, they do so because they are working with crap schema (whether someone else created it or they themselves didn't think it through enough). Sane schema + putting a bit of effort write readable SQL can go a long long way.

Given how quickly software engineering as a discipline moves, it's not unreasonable that most users at any given point in time (for any given library) are in those first few years.

Optimizing for the "newbie" case is not a failure.

Absolutely. But optimizing for the long term (and it doesn't seem like SQL is going to disappear any time soon) is not a failure either :)

If a newbie asked me whether to learn MySQL/PostgreSQL, or MongoDB, I'd absolutely say *SQL. Yeah, it's not as straightforward as MongoDB. It's not intuitive for many. But this knowledge will still be useful a year, or 5 years later. Finding a job will be easier too. So, yeah, it could be better, but it's still a good, safe choice.

Am I alone in finding SQL/normalized relational databases to be easier to learn?

In the relational world, you can get a lot of insight into what will and won't work with simple math. There certainly are surprises in every database, but most things are not surprises.

NoSQL databases tend not to leverage such a clear and complete model. You have to study each implementation to learn where the surprises are, and there are more of them.

That's not to say they don't both have their role, I just wonder if people who have a little (and in my care very little) CS theory still remembered find RDBMSes easier than those who have learned development without theory.

> Am I alone in finding SQL/normalized relational databases to be easier to learn?

Not at all. SQL is trivial. Thinking through some joins is tricky, but it's usually because I actually underestimated how logically tricky the thing I was trying to do actually was. Optimizing queries can be very difficult, though.

>If you've been programming and using SQL for a few years, it's not a problem.

Years of SQL injection mitigation don't agree with you. Are you sure you just don't prefer it because you are familiar with it?

It's almost as if constructing SQL queries by concatenating strings from user input is a bad idea.

I don't see how using vanilla mongo in the same fashion would solve the blight you have placed on SQL.

In Mongo, `db.findOne({"username":"$_GET["name"], "password":$_GET["password"]})` won't open your database to the public.

While SQL is a powerful language, the article is right. Mongo query language is useful enough to easily give application developers a place to query and store data without giving them enough rope to hang themselves.

For this dev, it's a big deal. I love writing a query in MongoDB way more than I love writing a query in MySQL.

Granted, I think a big part of it has to do with joins being the annoying part of SQL...

That seems to add to the idea that NoSQL fans just don't know how to use SQL properly.

Are you implying you're not supposed to use joins in SQL?

I think he's implying that finding joins difficult is a sign of immaturity of the developer rather than a problem with the language.

Joins might be tricky if you are unused to them, but they are the secret sauce of relational databases. They are what make it possible to have dozens of views on a single normalized data structure.

This just adds to the idea that sql kind of sucks in terms of usability. For the record I always use sql, but let's be honest with ourselves.

What's annoying about joins? I think practically speaking, whether you're going relational or not, joins are a reality of a data model that supports multiple views and transformations. Assuming that's the case, the only question is whether you're doing joins ad hoc in your application code or you've learned the handful of variations that work natively in your RDBMS.

I actually built a MongoDB query interface for SQL via Python+SQLAlchemy. It allows you to query your relational database just like you would with MongoDB, and in addition use some things (e.g. Joins in queries or "deep querying") that are not possible using MongoDB.

Here's the repo:

https://github.com/adewes/blitzdb

The implementation is stable but I'm still working on finishing Python3 support and documentation.

"So while MongoDB today may not be a great database, I think there's a good chance that the MongoDB of 5 or 10 years from now truly will be."

Either MongoDB will be, or other databases that have learned the lessons, both good and bad, of MongoDB.

RethinkDB appears to have captured the "MongoDB done right" mindshare, and PostgreSQL has gained JSON and is gaining better replication in order to cover the same niches.

I agree- I was a huge fan of MongoDB when it came out because of the unique data structures it enabled easily. However when it came time to select a new database for my new project, I found that the JSON support that PSQL had added gave me all the flexibility I needed while still in a somewhat relational form, and additionally it is dead simple to spin up postgres RDS instance in AWS, and it's a pain to use Mongo there.

> ...instance in AWS, and it's a pain to use Mongo there

Why is that?

Quick google search doesn't hint at problems, but rather at pretty slick marketing pages: (which doesn't mean much, I know) https://aws.amazon.com/blogs/aws/mongodb-on-the-aws-cloud-ne...

It's more then Postgres is so easy to use there with RDS which does all the setup for you.

Running MongoDB effectively on AWS is very expensive. It's the opposite of what AWS is optimized for. It requires a dedicated amount of RAM that scales with your database, and it requires permanent storage that's about 100x the size of the actual data.

It's very much unlike a nice, bursty, CPU-bound web server.

Also, you can't use the big selling point of AWS, which is that you can scale it "elastically". Okay, you could, but it would be a terrible experience with lots of unavailability.

How can MongoDB be a pain in AWS ? It is the easiest database in the world to setup. Download and run ./mongod. I've setup plenty of them in AWS and had zero issues.

And if we are talking about managed databases then it's equally dead simple to spin up a Compose/MongoLab instance in AWS.

Last I checked, the recommended production config of mongo, in the simplest case, required at least 3 separate servers configured in a master, slave, and arbitrator cluster. Compared with setting up AWS multi zone replicated RDS SQL servers (push a button), it is very much a pain.

Yes compose.io helps.

Edit: https://docs.mongodb.org/manual/tutorial/deploy-config-serve...

> RethinkDB appears to have captured the "MongoDB done right" mindshare

Mindshare is irrelevant. MongoDB is killing it in the enterprise right now. They have integration with Oracle, Teradata, Hadoop and countless partnerships with other vendors. You can guarantee MongoDB will still be around in 20 years the way it is positioning itself. Can't say the same about RethinkDB (as great as it is).

> PostgreSQL has gained JSON and is gaining better replication in order to cover the same niches

The PostgreSQL replication story is pretty pathetic given how old/mature it is. And I've seen nothing to suggest that anything is really improving in this area. There are a range of addons none of which are supported or built in. Basic replication is confusing, the documentation non existent in parts and good luck getting any support.

You compare it to MongoDB (or really any of the newer NoSQL databases) and it's like night and day. It takes minutes to setup a replica set and there is plenty of documentation and official support for any issues.

Today's mindshare is tomorrow's market share. It's not assured, but there's a strong correlation. Conversely, lack of mindshare doesn't really hurt sales, but it does hurt growth.

PostgreSQL 9.4, 9.5 and 9.6 all introduce foundational changes to make eventually enable replication, but none of it is really exposed to the end user. They are working on it, but they are being very conservative.

> Today's mindshare is tomorrow's market share.

This is just nonsense. There are plenty of hyped startups/products who went nowhere. Unless you understand how to market and execute you're going nowhere. MongoDB has demonstrated they are seriously good at it and given how well 3.0 has been received (write lock gone, extremely fast performance, Call Me Maybe test fixed) they have a lot of momentum.

> They are working on it, but they are being very conservative

Conservative being the operative word. You would think sometime in the last 20 years they would've tackled it.

I think you're missing the massive technical issues that are impeding postgresql from doing efficient sharding. SQL was not designed with distributed systems in mind, it has numerous features that can't be done efficiently in a distributed system, and even doing them inefficiently is insanely complicated. The language that Postgresql supports is Huge!

The reason nosql databases exist is in recognition of this fact. They jettison features like transactions that can't be done efficiently in a distributed system. If postgresql can implement a fraction of their current functionality in a distributed way, it will be an amazing engineering effort.

>Today's mindshare is tomorrow's market share.

Even if so, you're overestimating RethinkDB's mindshare outside of the HN echo chamber (and I say that even though I'd prefer RethinkDB myself).

If mindshare equals to developers knowing "this product exists", thinking "this product is good, I should use it", or deploying it, then Mongo wins by a factor of 4 or 5.

There's also the mindshare that Mongo has in companies building integrations for it, third party products etc.

And of course millions of CS/IT students today know about Mongo and will use it in this project or another, whereas very few know about RethinkDB. Those will be "tomorrow's market share".

> You compare it to MongoDB (or really any of the newer NoSQL databases) and it's like night and day. It takes minutes to setup a replica set and there is plenty of documentation and official support for any issues.

MongoDB makes the operational side of replication easy, but handwaves a safe, functioning implementation: https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-read...

It really is killing it in the Enterprise, and I'm trying to do my part to remove it. I'm at a client that wants to use MongoDB. It's on the approved product list. They have little to no experience with it.

Every chance I get, I advocate ArangoDB. It also is Mongo Done Right. You get joins, graphs and a thoughtful future plan from ArangoDB team. To help bridge the gap I've written an ArangoDB Hadoop connector [1]. Unlike the MongoDB one, you can read and write.

I've also added better Clojure support to it: from a driver to a Ragtime migrator.

Sadly as it stands Mongo has a better Ops story than ArangoDB. Until that improves, I don't think that ArangoDB will make it into many Fortune 1000's outside of some small prototype style applications. Maybe micro-services in the enterprise will change this, but I don't think a large insurance company wants to support multiple database standards in general, and definitely not within a family.

1 - https://github.com/deusdat/guacaphant

You should give RethinkDB a look... it has a great ops interface, and now that it has automatic failover, is probably my first pick... I wanted to like Arango, but they tend to lag behind in node support.

I happen to like MongoDB, warts and all.. that said, I would choose other options over it, depending on the need.

I did. At the time (haven't looked lately) RethinkDB didn't have GEO support while ArangoDB. Turns out that I don't really need it right now (different project). I stay because it's a great community. The Devs watch StackOverflow for questions. They are respectful and helpful in the user group.

I know that RethinkDB has a good rep in those areas too. It's just that ArangoDB is a good general fit for what I need even now. I guess you could say I came for the GEO, I stayed for the warm hearted underdogs that are the ArangoDB community.

That's funny, my first production use of MongoDB was because their geo support was better than ElasticSearch, which at the time, I couldn't get working correctly... I'm toying around with RethinkDB today for a hobby projuct, liking it so far.

Distributed databases are hard. If you're suggesting UnknownDB as a solution you haven't learnt the lessons of Mongo. I really do hope CAP isn't going to bite you or the otherDB cheerleaders in the arse.

PostgreSQL is nowhere near Clustering / HA / sharding features. Afaik it's only a master / slave architecture by default.

Sure, but the underlying primitives are better. Specifically it supports quite flexible binary replication including the new logical replication features. With this you can build things like Manatee[1] which enable effectively seamless HA. The platform I am currently working on Flynn[2] uses a variant of this state machine to implement an effectively maintenance free Postgres cluster.

In future though there will be support for bi-directional replication in Postgres, i.e true multi-master support.

[1] https://github.com/joyent/manatee [2] https://flynn.io/

Agreed, and now that RethinkDB supports automagic failover, it's pretty much a no brainer... and while I like Mongo's query interface slightly more for most queries, RethinkDB avoids some of the weirdness when you have more interesting queries. And server-side collation/joins is a really nice feature in a document-centric database.

PostgreSQL really needs a MUCH better replication/sharding/failover story... While I would use PostgreSQL in a situation where all I need/want is a single server, where multiple servers are needed for HA/failover, I'd probably just defer to MS-SQL, only because pg is so convoluted in that regard.

As to MySQL/Maria... I haven't touched it in years, and every time I have some weird behavior drives me nuts. I find it funny that people can love mysql, and bash on JS.

I'd also like to acknowledge ElasticSearch and Cassandra... ES is wonderful to work with for what it does best, search, and C* is a champ when you need a really good distributed table/kv store, though I think that RethinkDB is a better option today, if you don't need more than 10-20 nodes (which is a LOT).

Counting arguments very carefully? Nearly every SQL library does this for you.

    cur.execute("INSERT INTO a (b,c) VALUES (%(a)s, %(b)s);",
        { 'a' : a, 'b' : b })

Also, SQL is typed, so even if you did fail to count arguments there is a good chance you'd just detect it the first time you ran it.

The article acts as if treating the DB like native structures is somehow innovative and new - it's not. https://en.wikipedia.org/wiki/Object_database

We mostly abandoned object databases because they sucked. SQL was a huge improvement over them. SQL is a great way to organize and preserve the integrity of a lot of business data.

It's also a fantastic way to avoid repeated trips to the DB:

    SELECT * FROM employees AS e 
        WHERE e.department_id = (SELECT id FROM departments WHERE name = "engineering");

In Mongo, I'm pretty sure you need to first lookup engineering, then lookup the employees in engineering. That could be O(# employees in engineering) queries rather than 1.

> In Mongo, I'm pretty sure you need to first lookup engineering, then lookup the employees in engineering. That could be O(# employees in engineering) queries rather than 1.

Or, you could denormalize, and give yourself all sorts of future headaches maintaining data integrity.

> In Mongo, I'm pretty sure you need to first lookup engineering, then lookup the employees in engineering. That could be O(# employees in engineering) queries rather than 1.

The problem with that summary boils down to bad architecture. The point of document storage is storage with purpose; the intent being to make querying EASIER. This could easily be structured to be a single query. You can structure a document countless ways to represent that query, all of them would likely be different based on the purpose of the app.

Whereas with SQL there is more or less a single canonical way to do it and it's mostly independent of the app. I.e. the data design is minimally coupled to the specific use cases.

Right now I'm building a data store and I don't know the app(s) that's are going to be built on it.

It would be really great if computing could stop forgetting it's history. Object databases failed for a reason.

That doesn't make sense to me. A single query is faster than a join. Designing things for your application's purpose seems fairly tangible to me. It's application dependent. I use both for different B2B businesses I work with. One works very well with Mongo and would simply be slower with SQL. The Mongo app has one point of access for writes, while everything else is reads. It's near impossible to become inconsistent. The other application would be an absolute shit-show if it used Mongo. I'd never sleep at night with a fear of it failing. Nothing is black and white, and choosing the wrong technology isn't a failure of the technology.

Obviously a single query is faster than a join, but it's very hard to guarantee that you'll always be making that single query.

Data inconsistency is not about concurrency. See this example on Wikipedia illustrating why 3NF is necessary: https://en.wikipedia.org/wiki/Third_normal_form#.22Nothing_b...

In that example, "Al Fredrickson" can potentially have 2 birthdays even in an entirely single threaded app.

I'm really curious to hear a use case where Mongo was actually significantly faster than Postgres. Could you give a toy example that illustrates the flavor of the problem?

One of our applications I jokingly call "mad libs". It's effectively a document generator. A single point of entry for constructing the document skeleton (admin). The data structure for it is recursive, while user data is meta driven. Any necessary data on any given page is a single query. The only directly related multiple-query is the skeleton construction on the admin side. That part is fairly complicated (and swarmed with tests). There are potential inconsistency issues which could only come about if the admin was putting in a fair amount of effort into destroying their own app (spam POST/PUT wouldn't even cause it). Since the possibility exists even in the near-impossible, there are eventual consistency tasks running. But again, it would be pretty impressive to actually get it inconsistent. Given that the user side is meta driven anyway, there aren't any lasting effects on their side. Had we gone with SQL, the query for the (recursive) skeleton would have been 3 tables, one of which would be recursive. My use case is where reading drastically trumps writing. This tends not to happen in the unicorn industry, so I fully agree that it's silly in many _publicly discussed_ applications. But many B2B applications can fit the use case.

You didn't respond to the question about speed. Probably because Mongo has been considerably slower than Postgres in virtually every apples-to-apples comparison I've seen.

I don't understand the recent backlash against NoSQL here.

First off, almost all of the complaints would have been valid years ago. Secondly, there is so much more choice out there today if mongodb wasn't the right answer for your project, and so many NoSQL stores have had time to mature and get polished APIs and docs.

We use various data stores for different purpose across microservices, mostly ES, couchbase, and datomic, and "use the right tool for the job" and "do one thing and do it well" feels like the right approach to take. For most applications, a SQL DB feels like a really big hammer that is put to a lot of things that don't look like nails.

>>> use the right tool for the job" and "do one thing and do it well" feels like the right approach to take

Absolutely. However, database is a sort of an extreme example. A lot in the modern software (especially Web) relies on the database, and often migrating to completely different one (because requirements change and it might not be the right tool anymore) is a huge task. So you want to use something flexible enough.

Also, you want to hire people, people leave the jobs, people change teams, etc. If you use some exotic, less common DB, it adds a lot of overhead. And if you apply the "right tool" to an extreme and have a few completely different DBs flying around, your maintenance cost increases a lot.

See, SQL might not be a perfect, most elegant choice, but most often it is just good enough. A lot of people have used it, a lot of people have scaled it. If you run into an issue, often enough,other people did too and blogged about it, etc. Hiring / getting help will be much easier than $insertNoSQLDBName.

And, let's be realistic, relatively few companies have hundreds of gigabytes or terabytes of data that typical relational DBs can't handle.

My rule of thumb is that if you're in doubt, use SQL/relational store (I realize that they are different things but often used as synonyms and mean MySQL/PostgreSQL/etc).

"Do one thing and do it well" is problematic for things that store a lot of data. Especially for things that are supposed to be an authoritative source.

Getting storage right is very hard. Either it's too low-level, and it's hard for applications to coordinate complex operations without corrupting data; or you end up putting a lot of features in and end up with a SQL dbms; or everything does its own storage and you have a mess.

Basically, NoSQL became the big trending thing, everyone was pushing it hard simply because other people were as well. Because of this, many people used a NoSQL database when either a.) They were using the wrong technology for the problem they needed to solve or b.) They have very little/no experience with databases and working with data in general and they really screwed themselves up. Then they took to the forums and went on NoSQL crusades.

Nothing is wrong with NoSQL, used correctly and for the right purpose, it is AMAZING.

"For most applications" is very very misleading.

Most applications believe it or not are business modelling problems, which are overwhelmingly relational. SQL was invented to solve these, so no surprise it is actually the best tool for the job by far.

The main problem MongoDB solves is "I don't want to learn SQL". The backlash is against this use case.

(This article certainly seems to be appealing to this use case, c.f. "counting arguments really carefully".)

Alternatively, "I want to do everything in JavaScript", and not learn any other languages.

A lot of recent "innovation" is mislabeled laziness.

Just a note that in PG'OCaml (an OCaml interface to PostgreSQL), you can write:

    "insert into foo (col1,col2,col3) values ($a, $b, $c)"

and it creates the safe prepared statement with ? placeholders. At compile time. Type-checked against the database to make sure your program types match your column types.

http://pgocaml.forge.ocamlcore.org/

I would be very careful with such SQL statements. I am guessing it relies on some intrinsic fields' order? That could change anytime. Order of fields shouldn't have any impact on you app, but I think in your case it does.

The "..." wasn't literal. I have amended the post to make this clear.

I agree that the three areas outlined in the article are things that MongoDB got right: a structured query language (instead of a textual query language), replica sets, and the oplog.

But the lack of transactions over multiple documents (in the same shard at least) and the lack of joins over multiple collections are a big showstopper for the kind of applications I develop.

I note that solutions like YouTube's Vitess provide something similar to MongoDB's replica sets.

I also note that PostgreSQL's logical decoding provide the same functionality than MongoDB's oplog tailing.

Always wonder what kind of simple apps most people must write, if they not need joins?

I will be happy if i got such simple tasks :)

There is always the possibility to make a map/reduce over multiple datasets. It is not exactly a replacement for join, but you can cover aggregation over multiple datasets with a output-collection. Well... in my opinion this is much more complicated and error-prone than a join-operation on a relational database.

How exactly do you think eBay, GMail, Facebook etc work ? They aren't relying on relational database joins.

If you want to write a truly scalable application you structure everything such that you do joins in your application layer.

http://highscalability.com/ebay-architecture

And in the case of MongoDB you avoid joins since it is a document database. You embed data instead.

Not everybody works at eBay, GMail or Facebook scale.

Most applications fit very well in a single server. For example, Stack Overflow runs on a single instance of SQL Server, replicated to a slave in another data center. In such a case, the convenience of joins and transactions is priceless.

And even at scale, it makes sense to rely on joins and transactions. The perfect example is AdWords that runs of F1 and Spanner:

"Our users needed complex queries and joins, which meant they had to carefully shard their data, and resharding data without breaking applications was challenging."

http://static.googleusercontent.com/media/research.google.co...

Exactly. But many of us do have apps that are beyond the capabilities of a single instance.

And in this situation the rule of thumb is to do joins in your application layer so you can store different types data (e.g. graph/document/relational/unstructured) in different systems and easily cache where needed.

The fact that so many new databases have been created in the last decade suggests that there are a lot of people who do fall into this camp.

There are also organizational requirements that push people away from joins (different teams responsible for warehousing different data in a large organization) and high performance applications which pre-summarize data (adtech).

A lot of us have apps that are beyond the performance requirements of Stack Overflow?

Yes. It's not an especially exotic level of performance. Especially if you're talking about applications that handle traffic from other machines, as opposed to humans. As websites go, Stack Overflow is of reasonable size, but as systems go it's small.

They have about 7.5m visits per day to Stack Overflow. That's about 86 requests per second, which perhaps at peak is several multiples larger than the average. I wouldn't consider that a particularly gargantuan website. That sounds like traffic that you could service with a reasonable fleet of web servers and database fleet, given caching. Quite a lot of actions on the site can be applied with eventual consistency, I'd imagine.

http://stackexchange.com/sites?view=list#traffic

I don't mean to say that the system or the problem that it's solving is trivial - I am sure it is difficult to get right. As websites go, it might be large, but as systems go it's not particularly high traffic among systems that receive traffic from machines rather than humans. Imagine that you operate a data center, and you want to sample CPU, memory, etc. from your machines every minute. If you collect 50 samples per machine per minute, and you have 258 machines, you'll be handling 86 samples per second. Storing 86 samples per second into a time series database is probably considerably easier than SO's website rendering, but it goes to show that high-traffic or high-frequency systems are common in companies beyond small to medium size. It is easy for cross-cutting concerns like this to have massive request volumes, far greater than the human-generated traffic to any website.

7.5m visits per day. What does that mean, page loads? Stack Overflow isn't a static website. One page load is a lot of requests to the service. Stack Overflow is a very dynamic site, and a lot of requests are made after you actually load the page. I'm not sure you're accurately characterizing the kind of load that Stack Overflow is subjected to.

To anyone else reading: no, your use case probably isn't so special that the solution Stack Overflow arrived at just doesn't work for you.

> a structured query language (instead of a textual query language)

Oh crowning irony of ironies, SQL literally means "Structured Query Language". :)

LOL. Yes, I realized this after pressing the submit button :-)

> You can argue, and I would largely agree, that this is actually part of MongoDB's brilliant marketing strategy, of sacrificing engineering quality in order to get to market faster and build a hype machine, with the idea that the engineering will follow later.

Author nearly lost me here with this logic. Placing Marketing ahead of quality in something that is supposed to store a very valuable asset (data) is near insanity.

I get the mindset of "break fast", "release often", etc. in terms of customer facing features, but in something that is supposed to be a core part of your foundation, stability is if utmost importance. Otherwise nothing else works - and you lose customers, business, opportunities - because you can't look them up later.

Its not "brilliant marketing", its just marketing.

This is all true, but the success of MySQL shows pretty clearly that just because something is insane doesn't mean it's not good business.

I think the success of MySQL is due to there being fewer options for a period of time (the "dot.com boom"), and thus it became a popular choice to avoid commercial RDBMS costs.

I'm no MySQL fan when things like PostgreSQL are an option, but its probably more sane than some other currently popular choices.

I have to agree with the author, especially since the points he raises are the ones that helped me greatly on my first "serious" personal project[1].

Coming from postgresql land I would have never thought you can have such great replication with automatic failover. I've had literally 100% uptime for the past year.

And that's on commodity servers (one of them being in a room in my apartment, the other two in a proper datacenter) going through the usual upgrades, downtime, reboots, going from mongo 2 to mongo 3 and such.

Speaking of which, the migration from mongo2 to mongo3 was another pleasant surprise: they've made it backwards compatible. So I could do the upgrade on the servers, one by one, checking everything was ok and after that I could focus on updating the drivers and rewriting the deprecated queries, no need to have everything ready at once.

The accessible oplog was another gem that fit my project really well. Gone was the need to poll the database, I could just "watch" the oplog. That, coupled with long polling on the browser side meant I'd have very little chatter between the db/server/web client when idle. Websockets would have been nice, but adoption wasn't high enough that I'd be comfortable going forward with it.

And all this considering MongoDB was my first NoSQL experience.

I agree it doesn't fit every project, but when it does, it's a really nice experience.

[1] https://graticule.link/

I agree that MongoDB has a great replication story.

But I don't understand that part:

> The accessible oplog was another gem that fit my project really well. Gone was the need to poll the database, I could just "watch" the oplog.

Coming from PostgreSQL, you could do the same using LISTEN/NOTIFY?

> Coming from PostgreSQL, you could do the same using LISTEN/NOTIFY?

I have to admit I was not aware of this feature.

However, from the docs[1]:

> Commonly, the channel name is the same as the name of some table in the database, and the notify event essentially means, "I changed this table, take a look at it to see what's new".

From what I understand, you just know that something has changed, the actual change is not included in the event, so you need at least another query to see what changed.

Did I understand correctly?

In MongoDB you get the operation (insert, update, delete), the document and another few details right in the event.

[1] http://www.postgresql.org/docs/9.4/static/sql-notify.html

When you notify, you can also include a payload (ex json) which can be whatever you want.

I looked into this before and the Postgres docs say somewhere the payload size is limited and intended to be small metadata... in some cases you will not be able to fit the contents of the update into it

Looks like the default limit is 8000 bytes.

Good question :-)

You can do as ddorian43 explained, by including a payload in your notification (using JSON or another format).

Or you can do as you suggested and query the database to fetch more information about the change.

Apart LISTEN/NOTIFY, which is a very powerful feature, PostgreSQL also provides "logical decoding" which solves problems similar to the ones solved by oplog tailing.

https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-read...

I don't know that it's story is so great. EASY, sure, but what good is easily replicating bad data?

Yes, I'm aware of aphyr's post, which reveals serious limitations in MongoDB. I said the replication story is great, not the consistency story ;-)

RethinkDB took all the good parts of MongoDB and added proper engineering.

https://www.rethinkdb.com/

But still no transactions over multiple documents (at least in the same shard)?

ACID transactions in a highly available distributed system are hard and often fail in subtle ways when done wrong at the edges. Any implementation will take years to mature in the lab and in actual production usage. This isn’t a knock on the Rethink guys; their product looks pretty awesome and is moving quickly.

For a solution today, MarkLogic is a transactional distributed document database. Cross-document and cross-partition transactions have been a key tenet of the architecture from the beginning (like, 2002 beginning). Take a look at https://developer.marklogic.com/blog/how-marklogic-supports-... for details.

Full disclosure: I’m a Product Manager at MarkLogic.

I love the point about the Oplog.

There are a few equivalents for common SQL DBs (see LinkedIn's Databus for Oracle and MySQL), but in general, getting access to the write log is really hard. Even though it's sitting there!

It would be wonderful if there were some kind of established API or library that would let you parse the MySQL write log without doing hideous, fragile operations that change from version to version. Sure, change the format, but at least version and document it!

When we chose MongoDB for a project, a dominant criterion was out of the box geo queries. It helped that the storage and query approach had good impedance match with NodeJS. From a query perspective, we wouldn't have benefited much from SQL anyway, since much of the reading is free text or social graph or location based search which we moved to Solr.

It becomes much simpler to setup replication in PostgreSQL than before.

reference: https://www.digitalocean.com/community/tutorials/how-to-set-...

Reply on news.ycombinator.com