12 Months with MongoDB
blog.wordnik.com
blog.wordnik.com
What is the advantage of giving up the ability to use SQL and the associated relational algebra that has long been established in that query language? I'm not asking to start a flame war, I'm sincerely interested. Can someone give me a use case for when NoSQL would have a clear advantage over SQL?
from a development standpoint only, I like the fact that I don't have to write database migrations (as defined in Rails). It means I can iterate more quickly during the development.
As well for data aggregation kind of jobs (such as http://www.toutpourmonipad.com/ where I munge different-formatted data streams), it's really convenient to be able to mix datas that are partly equal, partly different, when it's relevant to you.
Edit: forgot to mention that MongoDB comes with a built-in geographical index (MySQL doesn't have it, I'm not sure for PostGres - I believe it's via some extension).
Edit2: forgot to mention I really appreciate the upsert abilities for what I do (http://www.mongodb.org/display/DOCS/Updating#Updating-Upsert...)
Edit3: anyone with some curiosity for MongoDB will appreciate this book: http://www.amazon.com/MongoDB-Definitive-Guide-Kristina-Chod... - well-written and concise
My point is that I only do the "production" migration when needed and once per release that requires it, while I can tweak the schema at ease while developing.
If you have a large-enough volume of data, you will meet the situation where just adding a single column takes ages, too.
Works like a charm. No migration headaches.
In general though I agree with you. I can whip out SQL queries in seconds that would take me minutes to write against Mongo, even though they're all technically possible. I love SQL. But it's not the right tool for every situation.
So I prefer to not use words like "usually" as it truly depends on your application and use case.
So, either the NoSQL solution you are using is incredibly dumb (and your schema is pretty much "id->blob") or it is internally going to have to do just as many joins against separately stored data objects in order to rebuild a concurrently-modifiable tree.
In the former case your NoSQL solution is a really fancy object serialization framework (and probably one that is not optimal for your app) and in the second case it is implementing a database and has a library on top to help you store and index trees.
To be clear, and to go back to my argument: I feel the former provides no real value and the latter could be implemented as a library over a normal SQL solution without having also had to reinvent the storage layer, the transactional semantics, etc..
I agree that Postgres can do this - I've done it before. But I think you're a bit wrong to dismiss document databases so quickly.
Firstly, the subtree update problem isn't a huge problem. MongoDB allows dot notation to update items within a document. Yes, it is may well have to do just as much work as a SQL database in the update case, but I don't care. I'd prefer it is implemented in the database than something I have to do myself.
Secondly, the schema-free nature of a document database is a killer-feature for me. I have truly schema-free tree data (different levels of the tree have different, unknowable-in-advance data stored against them). Yes, I can implement this in a SQL database schema, but it's going to be an ugly schema (eg, I'll have to use rows to store things that should be columns). It will also be slow because of the hierarchical walking needed in the queries. (Although Postgres helps some here with hierarchical query support).
To your "second": that is not a property of your usage of trees, and starts a new, unrelated discussion. I have nothing against document-oriented databases, and use them often. I feel you are blurring the line between syntax and implementation with your "slow" comment (again: if you are able to concurrently update those schema-less data items you are going to be taking the same hit you would be getting with any other backend for the separate storage and indexing), but will certainly not argue that there are classes of problems where document-oriented databases are really useful. However, trees in particular are not one of their killer features.
And yes, XML/XPath support in SQL databases allows them to act as schema-free document stores. However, they aren't optimized for that, so indexing inside the document is limited. OTOH, SQL DB vendors might be able to add that quicker than NoSQL vendors can improve tool support and querying. OTOH you have to deal with XML instead of JSON. OTOH...
It's a trade off.
I rewrote a very similar application with MongoDB this year and the code was way cleaner, and that's not caused by my increase in experience: the document-orientation is really helping here.
This might fall under scalability, but I've worked on a few projects where we just continually added new tables because applying an alter on the existing table in production would take an unknown amount of time. Another thing we sometimes did would be to have 2 active tables and migrate the data over a few weeks while applying updates to both tables. Both options kinda suck. Its generally not an issue if you only have a little data though, so hence "might fall under scalability."
Simply put: it gives you a much more natural way to persist the objects you work with in the OO language of your choice. It's a joy to use. Until I used it, I didn't realize how unnatural it was to map OO programming to an RDBMS. We work with objects in our languages. We don't work with rows of data.
And I totally get that, I really do, but to me, that's more of an issue of personal preference, and less of an issue of a clear advantage. There's nothing wrong with personal preferences. For instance, I like schemas that aren't easily changed, and a clear separation of logic and data, and I prefer to think of data as rows, not objects. But that's my personal preference. I'm more interested in how the functional approach to a use case would make NoSQL the clearly superior technology in that scenario.
Of course, there's no silver bullet - some times you DO want to group together logically different data in a same query. But I've found that in general, I'm not fighting the DB as much when building stuff using MongoDB.
This is not true at all. At least in PostgreSQL you can have arrays and hashes:
"Tip: Arrays are not sets; searching for specific array elements can be a sign of database misdesign. Consider using a separate table with a row for each item that would be an array element. This will be easier to search, and is likely to scale better for a large number of elements."
Searching an array is a pretty common task, Mongo does really well in its ability to search into objects in a document.
I had to laugh a little at one of the sample queries:
SELECT f1[1][-2][3] AS e1, f1[1][-1][5] AS e2 FROM (SELECT '[1:1][-2:-1][3:5]={{{1,2,3},{4,5,6}}}'::int[] AS f1) AS ss;
Seriously?
About the performance/scalability warning; I don't deal with very large arrays, a couple hundred items max, and when using a GIN index over the array field, search queries are screamingly fast.
http://www.mongodb.org/display/DOCS/Dot+Notation+%28Reaching...
From the "Array Element by Position" example:
db.blogposts.find( { "comments.0.by" : "Abe" } )
PostgreSQL doesn't have JSON support but it does have XML/Xpath support.If you stored XML documents in PostgreSQL the similar query to the MongoDB one would be something like this:
select * from blogposts where (xpath('/comments[0]/@by', doc))::text[] = array['Abe']
Yes it's a little more verbose but not terrible so... at least in my opinion.I still think of data as rows in MongoDB, but the lack of a fixed schema makes life seriously easy. That means my schema is defined in the application (and only the application) and is subject to versioning. Want an extra column? Just add it to your application and you're done. No need to promote a slave and cycle through while waiting hours upon hours for each table to build.
Silver bullets…
Seriously, you owe it to yourself to do their superbly written tutorial -http://www.mongodb.org/display/DOCS/Tutorial.
That simplicity, however, is a front-end simplicity that -- if it isn't perfectly defined and appropriate for all future use -- can cause you tremendous pain in the future. A MongoDB document store tends to be very single purposed, and is horrendous to use outside of the narrow bands of that original intention (a simple aggregation is an exercise of extraordinary inefficiency). Which is fine for a straightforward, simple app like the one linked here, but isn't applicable for most projects. An RDBMS design encourages that you think about and abstract your data to the constituents, which often yields tremendous future flexibility, but with more front-end costs (and ongoing significant costs if you don't make the right decisions).
Once again, though, stories like this really...grind my gears (get off my lawn!). We have no idea how correctly or incorrectly they used their RDBMS, what their pain points are, etc, however they drank some of the magic elixir and all ills were cured. Anyone who questions the assumptions will be told the rote "Apples and Oranges!" quote (which is always humorous when the whole context is talking about moving from Apples to Oranges and how grand it is).
There are tradeoffs with each approach. I've found that that migrating Mongo documents tends to require more software for easy migrations, but also tends to be more flexible.
For example, you can lazily cache data from services and since the schema is flexible, a single collection can seamlessly hold different types of records.
Replication is also more natural with Mongo.
Of course, reporting and getting aggregate data is easier with Postgres.
I recently evaluated a lot of different database systems too for a new project, and settled on MongoDB for a key part of it. While it has many positives (the above, plus performance and scalability) one of the negatives that worries me a little bit is the "running with scissors" feeling of not having database-enforced table schemas. Ultimately, we'll find out firsthand whether that leads to disaster or not, and whether it's a net win. My bet now is net win.
As you've said, scalability is one valid reason for using a NoSQL solution. Sometimes you also have special needs for multi-datacenter replication or whatever. Sometimes a NoSQL DB can fit your problem better than a SQL DB. However, what does it look like from a development standpoint?
If you have an offline-capable app it might be easier to implement it with CouchDB.
Also, CouchDB's MapReduce views could allow you to run queries that wouldn't be possible with SQL. Note that MongoDB also has MapReduce support, but it rebuilds the whole index instead of only the parts that changed, so on a large DB this might take too long and thus be impractical. Anyway, I guess you're talking about NoSQL solutions in general.
In some cases you actually want to store very flexible schemaless data in your DB. SQL would be a huge pain, here.
It might also be easier to switch from an XML DB to e.g. MongoDB.
Migrations can sometimes also be easier with a schemaless DB. This could be important when experimenting or developing iteratively.
Anyway, most of these are all rather special use-cases and that's probably the important development-side strength of NoSQL: Having solutions for special problems.
Some people here say that working with NoSQL is more natural than with SQL. If you're comparing hand-written SQL code with hand-written code for the low-level MongoDB API then that's indeed true. However, in practice you should better use an ORM because that makes you more effective. When comparing, for example, the Django ORM with MongoEngine you'll see that the APIs look similar. It's just that SQL gives you infinitely more powerful queries and an ORM makes writing even complex queries very easy. A NoSQL DB sometimes forces you to jump through hoops to implement something that would be a two-liner in SQL+ORM. So, from this point of view NoSQL has nothing to offer for a large number of problems. Of course, some projects only use very simple queries which map nicely to NoSQL. In these cases it doesn't make any real difference whether you use SQL or NoSQL, but that still is not a real plus for NoSQL.
One other reason that once spoke for NoSQL: Google App Engine and Amazon SimpleDB were among the first solutions which offered scalable, replicated, managed hosting with an acceptably cheap pay-as-you-go model. Today we have Amazon RDS, MS SQL Azure, and soon App Engine will have SQL hosting, too, and there are several other cheap cloud hosts with SQL support. So, this reason isn't valid, anymore. It might still be part of the hype, though.
Yes, NoSQL is indeed over-hyped. Too many newbies use it for the wrong reasons (esp. on App Engine where they use it because they can save $3/month). Only if you need to scale or if your project has special needs you might find rescue in the NoSQL world.
Finally, since many of us are working on a web startup: As long as it's not clear if you'll become successful and thus have to scale you might be tempted to use SQL and develop faster in the early phase of your project. However, when looking at a complete web project the development/code overhead caused by NoSQL (vs SQL) is often not very large (well, depending on your particular project), so you might want to go with NoSQL right from the start and not have scalability woes later on. Anyway, you need to make an informed decision for your particular problem. Don't just pick NoSQL because it's hyped and also don't just pick SQL because you're wary of hype. :)
BTW, if you want to go with NoSQL and still have more complex queries like with SQL you should join our open-source project django-dbindexer. The goal is to automate denormalization and index generation such that you can use e.g. simple JOINs and aggregates with Django's ORM instead of emulating them with hand-written code. This should be useful to a lot of web startups and make it easier to use NoSQL instead of SQL right from the start and not worry about scalability. See here:
In this case the flexible schema of Mongo DB is a great fit for my data. My records have varying numbers of fields which I don't know in advance, usually include simple hierachies of data and I often need to query all this data together. Many queries that would be difficult in SQL become trivial in Mongo DB.
I use SQL for data relating to user accounts, transactions etc because the SQL model fits better for critical, well defined reliable data IMHO.
MySQL's deficiencies aren't inherent to SQL databases. Other databases have faster query parsers and better query planners. It seems that with all the time and money invested into NoSQL solutions, Postgres could be improved to the level of Oracle or DB2.
From what I understand, Postgres doesn't (necessarily) have those kinds of locking issues, but MongoDB does let you fetch documents (especially hierarchies) in a much more simple manner, rather than fetching them via potentially complicated join queries.
i ran into the same question
You can update any field in a document. Obviously you'll need the ID of the document.
SELECT * FROM things WHERE name="mongo"
================================================
> db.things.find({name:"mongo"}).forEach(printjson); { "_id" : ObjectId("4c2209f9f3924d31102bd84a"), "name" : "mongo" }
I am having a hard time finding the benefit of that except you can do it programmatically and not step out of your language of choice and into SQL.
I am just looking into this...so maybe the lightbulb will get brighter as I go through the docs for MongoDB.
- I'm curious why querying before a write makes such a big difference. I would have guessed that updating a document that's not in RAM would first load it into RAM, then perform the update. Does the write get applied to disk without loading the page into RAM first? If you do an update to a document that is not in RAM, is it in RAM following the update?
- Can you elaborate on the corruption that occurred to both the master & the slave during a DAS failure? We have seen something similar in our deployment (high write volume leading to corruption in both master & slave. required repair to recover. ran on a partially functioning slave during the repair), but were unable to identify the root cause.
Regarding the corruption, I got an "invalido BSON object" or something on repair, which tells me some object was only partially flushed to disk when the DAS went down. The slave actually worked fine for simple lookups by ID, but there was some issue with the index and I was unable to run filters against it. Luckily the huge collections are only accessed via unique identifier, so this wasn't a huge issue.
Have you spoken with any of the MongoDB developers about why it's currently the way it is, vs. a more efficient update path?
For us, our DB abstraction layer made this behavior so simple to add that we didn't make much fuss about it.
1) Because it's a strawman argument.
2) Because the reply addresses a very small subset of apps, which 10gen specifically says is not recommended for use with MongoDB.
My point is simply that you gain productivity and Mongo tends to get a lot of hate on HN. My time is limited. For 90% of use cases, I am going Mongo.
Could you provide some examples? Scanning the list I see a couple that have a very periphery relation to financials, but the actual applications have very little financial applicability (and the implementations are trivial).
Though the person you responded to didn't actually say that `"anything that deals with money" has no use for MongoDB"', so you're setting up a strawman regardless.
But there's almost always some ACID requirement when you're dealing with accounting, e.g, crediting payment or billing fees.
I would use something that could handle transactional ACID processing in combination with MongoDB depending on the application.
I find that ironic.
Please have a little faith in other people, that if for any reason they're not using nosql it's a good one!