Goodbye, MongoDB
zopyx.de
zopyx.de
I've launched very high traffic websites using MongoDB, where it was the least of my worries. I've also launched very high traffic websites using MySQL, where it was my primary source of pain.
Shall I now run around screaming loudly about how badly MySQL sucks? On the surface you might assume so, but with reasonable effort to consider all the facts, it becomes clear that there's more to this than just "this database is better than that one because I hate it."
This post is more of an angry payback rant from someone who got banned from IRC for being abusive to newcomers and greenies. On top of that, it is factually incorrect in some spots, and misleading in others. Not impressed.
That said, NoSQL databases are evolving extremely quickly, and have reached a point of maturity that they can excel in the right scenario. Just like relational databases, they can provide big benefits, given that the user takes the time to learn how to use them, as well as making sure they have a use case that merits the strengths of their chosen tech.
I really enjoy working with MongoDB and Redis, and have been a community cheerleader of sorts for PostgreSQL for years. It is possible to have multiple tools in your toolbox, and use them when appropriate. No tool is perfect, and no tool is perfect for every job.
Complaining bitterly that your shiny new piano makes a lousy boat, on the other hand, simply adds to the noise.
For me MongoDB and PostgreSQL aren't really playing in the same field. Yes they are both databases. But I wouldn't use PostgreSQL for document storage (the recent JSON addition is pretty weak). Likewise I wouldn't use MongoDB when I need my data to be more structured. I would have no issue using both at the same time.
I guess everyone is looking for that mythical single stack of technologies they can use for every use case.
My simple mental model for MongoDB is: indices (should!) fit in memory and documents are stored contiguously on disk. That is it in a nutshell. A query involves in-memory lookups and maybe only one disk seek. Writes in place are usually possible.
I have been happiest with MongoDB in two different scenarios:
The first is in developing small web applications where there is no scaling issue, and the fact is that MongoDB is so easy to develop against and simply provides a great developer experience. As needed, I use a cron job to do a mongodump a few times a day; or, if I really need high availability (which, frankly, often I don't: if a system is unavailable one or twice a year for an hour it is no big deal) then replica sets are OK.
The second scenario where I have really liked using MongoDB was doing analytics on a modestly large stream of social media data. A single Mongo master on a large EC2 instance was adequate to handle writes and slaves on other large EC2 instances each fed a different analytics application. This setup of apps reading from a slave on the same server worked really well for me. This was a low hassle experience.
I do have one customer with really large MongoDB setups on multiple data centers, and I am working around right now on some hassles, but we haven't found anything else as cost effective for the customer's applications.
All that said, when I can use it, just using a single (no horizontal scaling) PostgreSQL server is for me the most hassle free developer experience, but I have always used PostgreSQL for small or medium sized applications - nothing that needed to scale.
The first is a small setup. MongoDB is called that because it's supposed to be for "humongous" data sets. Also, in what case is 3GB of data preallocated for journalling, which is one of the points in the article, good for a small installation?
The second is "modestly" sized stream of social media data. Having myself worked with a much larger stream of social media data in Mongo, I can attest that the second you leave the land of a single Mongo server, you have a much bigger problem, sharding. Sharding is terrible in Mongo, writes are shard locked, not collection, your shard keys are immutable (imagine having an indexed field you can only set ONCE), and fraught with data loss. Did you have a drop in network connectivity between your mongos and you config server? You just silently lost data. Safe being on doesn't matter, if the config server doesn't get the write it isn't able to report where the data is for a read, even if it is confirmed it was written to disk.
To your point about mongo ranters not understanding what it's good for, MongoDB tells everyone it's good for "big" and "fast" data. However, it fails at both of these, because it doesn't easily scale up from one node, and it doesn't write quickly when you actually want to make sure that your data is there. What's the point with writing data quickly to something that looses it quietly? Might as well pipe it to /dev/null/
I know nothing about MongoDB and have never tried it. But the message seems pretty clear.
It's really just hipsters and music. We are all really just hipsters.
I think you're about two years behind. Node is starting its hate-period. Rails got it a year or two ago.
There continues to remain a "golden hammer" syndrome where white horses and unicorns run free, but it doesn't exist.
Instead, the vision of "NoSQL" was to tell developers that they did not have to use relational data for everything, but could, instead, use the right tool at the right time. Why is this such a hard concept?
If you are a developer and you don't understand the tool you are wielding (it's pretty clear the author of this blog didn't), then you will incorrectly use the tool and experience pain.
That is the fault of the developer, not the tool.
Are you saying that one of the way he is using MongoDB is incorrect? If yes, what point did he say reflects that?
His rants seems quite specific.
I think a lot of people switched to it because it was cool, and maybe assumed that it could solve any application data storage problem and are now finding out that it may not have been a great choice for them.
I don't think its appropriate to take away from this that MongoDB and/or other document stores are bad. Instead, I think its important to understand how they work and decide how well it applies to your use case. It's not going to work well for all applications.
I'm guessing within the next year we're going to see a similar backlash against Hadoop as people who rush to it begin to discover that map/reduce isn't necessarily the best distributed processing model for their needs. This despite Hadoop continuing to be the effective (non-silver) bullet it's always been.
The main thing that really distinguishes Hadoop is that it's built to only do one kind of distributed processing. In exchange for asking you to don that straitjacket it offers ease of use. However, if your problem isn't naturally a map/reduce problem, or if your main performance bottleneck isn't disk I/O, then the alternatives become a lot more attractive.
> Having no option to perform an operation comparable to UPDATE table SET foo=bar WHERE....
What? db.collection.update does exactly this. See: http://www.mongodb.org/display/DOCS/Updating#Updating-update...
MongoDB fit a nice niche for a read heavy mid-scalability db solution. Every DB has it's niche. Trying to use it outside of what it's good for is going to get you burned. If people just did their research before blindly committing to a platform, we'd see a lot less posts like this.
Agree with your general point, though - this seems like they have a product/tech mismatch. Though I'd argue that this isn't a niche - read heavy, mid-scalability is a lot of the web.
> Now instead of fixing a bad implementation or fixing the underlaying architectural issues, MongoDB is moving to Hadoop.
I don't think that's accurate. They have a new "aggregation framework" coming that is meant to replace mapreduce. It could be a wrapper around hadoop, but I couldn't find anything documented about that. I completely agree that a blocking mapreduce is annoying, however, does any framework have a non-blocking mapreduce? I haven't tried many mapreduce implementations out, so this is a genuine question.
Yes, moving your data from Mongo to PgSql, MySql, MSSql, Oracle, etc. is going to be difficult.
However: why start with a production level Oracle install anticipating "web scale" when you're not going to have more than ~1k users at launch.
People look to Google and Twitter and Facebook for advice on how to scale. They then apply this advice far before they need to.
I don't think you should plan to be large scale from the get-go. I think you should have a plan for what you're going to do if you get big.
My $.02
This is why I think that Linus's tirade against O_DIRECT is misguided: https://lkml.org/lkml/2007/1/10/233
Here's the thing: the kernel is a library. It took me a long time to fully understand this deep idea. The kernel is just a library that has a different and more expensive calling convention (syscalls) and runs at a higher privilege level.
It's also much less flexible than user-space libraries. Its interface is an unholy mix of syscalls, ioctl(), /proc, vdso, etc. There is a high bar to adding new interfaces. Removing or changing existing interfaces is basically not allowed.
The resources that the kernel uses are much harder to account for or predict. How can you ensure that a process always gets at least X MB of page cache, and that some enormous "cp" that some sysadmin is running won't evict all your MongoDB pages that are caching your database? Sure you could mlock() your pages, but now you're basically side-stepping all of this smart kernel cache management that was supposed to be helping you so much in the first place.
User-space management of buffers and caches is more flexible, easier to account to its owner, and more predictable. It can't handle page faults with Linux's current interfaces, but the L4 guys have figured out how to let pagers run in user-space and handle page faults. I hope that someday this work becomes mainstream. Our 20-year-old OS design is showing its age.
But, there's something to be said for the simplicity - for most folks you don't need to manage the memory yourself. When you need it though, there's really no easy substitute.
Yes, and most people don't need to implement printf() themselves, which is why there is libc. Just because you're doing it in-process, in user space, doesn't mean you're rolling your own!
Can someone explain to me why this is actually a big issue? Except for really tiny apps, I imagine that having dedicated VMs for your MongoDB actually would be perfectly fine? Probably even preferred?
We run both large (multi-shard clusters) and small memory (500MB - 2GB) use instances of MongoDB and have no problems.
It would be good to have developers acknowledge that, perhaps, they may not have all the information instead of declaring that something can't be done.
Last I checked linux had no interfaces to partition the pagecache in a meaningful way, short of rather extreme gymnastics involving kernel-patches or tmpfs abuse.
If that has changed then I'd certainly also be curious to hear about it.
From what I've seen of MongoDB I'm not impressed at all. In some carefully controlled cases, performance would be acceptable, but change anything at all (even the order that data is inserted) and it just sucks.
For one particular application, the performance difference between MySQL and Mongo was like the difference between the Space Shuttle and a Chevy Sonic.
create table whatever_attribute ( whatever_id integer primary key, name varchar(255), value text )
if you never need a special index you can just stick new attributes onto "whatever" and never recompile
You need to use indexing unless your app is a toy.
The simplest way to get really good performance for multi-row queries (even if you fall out of cache) is to physically order your data in query-order. (that is, if you are going to ask for the most recent 100 blog comments, order the comments by (blog-post-id, reverse comment-date).
MySQL -Innodb makes this really easy (your data is physically ordered by primary key). In MongoDB it's not possible.
Here is a more elaborate explanation...
MySQL-Innodb stores record data in primary-key order (it puts the data right into the b-tree). This means that if you want to access 100+ records in primary-key order, it's pretty darn efficient. Even if it's out of cache, it could be just one or two disk seeks (depending on how many records fit in a block)
MongoDB stores record data in a heap-table in a semi-random order based on insertion and freespace, with each document receiving a "doc id". You can make an index on whatever you want, but when Mongo fetches multiple records, it cross-references every index entry with the doc_id. If your data is out of cache, this means a seek for _every single document_. AFAIK, as of 2012, there is no way around this, because there is no way to get mongodb to store the document data directly in the b-tree. This is a big part of why Mongo is super-slow if you fall out of cache.
HOWEVER, there are some other systems that also have this problem, including some ORMs that sit ontop of MySQL. Ruby-on-Rails forces you to use an auto_increment primary key for every record -- which means even if you use MySQL, you are forcing your data to be in a semi-random insertion order. Django does this as well. If you want to efficiently fetch a bunch of records (like 100 comments on a blog post), then you want them to be in primary key order.
In the SQL world, Oracle, MS-SQL, and Postgres also normally use a form of heap-table for records... This allows records to have a physical "ROWID" which can be used to directly look them up (it's a physical block address with an O(1) lookup). However, it also means they are in semi-random order. The ROWID was an important join and foreign-key optimization back in the days of nightly SQL jobs on machines with very little RAM. Today, it's not a good optimization. B-tree indirect blocks are always in RAM, so direct ROWID lookups have little benefit over b-trees, and the huge drawback of no natural primary key ordering. One can workaround this problem with fully-covered-indicies, table-in-index, and key-clustering -- all of which have their own frustrating tradeoffs.
Does primary-key ordering have downsides? References to records are bigger (They have to contain the full primary key), there is no guaranteed stable way to reference a record (for foreign key constraints), and if you change fields in the primary key, the entire record must be moved. In the "old days" there was another big drawback, b-tree O(log-n) lookups are much slower than ROWID O(1) lookups.. Today this is not an issue because all b-tree indirect nodes fit in RAM always.
Bottom line, if you want the easiest way to order data properly, use MySQL-Innodb, and choose your primary key wisely. If you are using another storage system, study up on how you can control physical ordering becuase every system is different.
Big advantage in that option (Postgres, MSSQL, Oracle).
Isn't that true for any database? What point are you trying to make? That a large MySQL deployment can be flawlessly be maintained by people that can "hardly spell their name"?
MongoDB memory management is a legitimate concern... but not because it's hard to control memory usage of a single mongod.
"More granular locking" is a temporary, non-scalable solution?
I've run out of energy, actually, but really?
I have been criticizing them for their default settings (no response to write requests, safety turned off by default) for a while.
But actually it is not as much for settings themselves, but for lack of a clear red flashing warning on their front page. As their default settings could (still can?) lead to silent data corruption. The worst possible thing to happen to a database. That is shady and shitty practice if you ask me.
They are trying to market their technology to other developers, I expect them to at least try to be honest and open about the characteristics of their product. If they treat other developers like they are customers for male enhancement pills in 4am commercials, then I think they should also expect some backlash from said developers when they choose Mongo and then hit all the hidden assumptions and un-delivered promises.
Sure it is in the fine print. But if it is important, why not put it in the big print. It is a lot less painful for everyone in the long term.
I've seen many mailing list responses to questions about Riak saying "maybe Riak isn't the best fit for this, try looking at X..." whereas 10gen wants everybody to use MongoDB for everything.
It just seems to be a more honest operation, and people don't get burned as much.
Riak's relative obscurity actually serves well in this regard. Odds are that if you discover Riak, you do so for a reason - you've been searching, you have a specific itch that needs to be scratched, etc. Given that you are already pretty clued when you get to it, you are extremely unlikely to pick it for the wrong reason, and consequently the success rate tends to be astonishingly high. Mind you, I'll add that the folks at Basho are some of the nicest people I know, and that helps a lot :-)
Bottom line - I really don't expect Riak to go away anytime soon...
(Note how the video raises some of the same concerns as the blog post)
As I said, no idea how good or bad mongo is, but I'm guessing, if you are as sloppy in your code as you are in your English, I'll be happy to give mongo the benefit of the doubt...
Kind of a bummer.
That said, a lot of people have started to use Riak for a general purpose tool. There are some development growing pains associated with this (and you have to think very carefully about your keys and data structure) but it's only getting easier with things like secondary indexes and Riak search. If there was a non-expensive way to enumerate all the data in a bucket, I think that'd be the one last item on the checklist before I jumped on it.
Comparing varnish to mongodb is akin to comparing a precision Rolex from Swiss to a plastic mickey mouse watch from a gumball machine.
However, I do feel there is something "wrong" about the approach MongoDB is taking. They need to allocate new files in huge buffers, which completely take up all I/O while being filled with zeroes. There is no logical hierarchy in the files, and it just feels a bit weird.
Perhaps they should've taken the approach PostgreSQL did, which is to simply use files and read from them instead of using mmap. The whole reason they went for a global lock instead of more granular lock is because the whole mmap'ed area is one big blob, and it was the most "obvious" approach.
Out of curiosity, is there a simple way to explain why someone would mmap instead of just reading files directly (I've never done any programming with mmap, so I'm a bit ignorant of its use cases)?
mmap() is a good way of doing things but for different reasons than some people assume. Like all tools, you have to learn how to use it well.
The old-school DBA in me immediately thinks that the "70's tech" of the RDBMS is still widely used because the time to develop sufficient memory management is hard work and takes time.
That said, a competently engineered RDBMS can do everything NoSQL databases can, particularly limited databases like MongoDB. The caveat is that you have to learn how to use those databases; they are very feature rich and powerful but that flexibility makes them more complicated. PostgreSQL is a very good choice from the open source world and is just as fast as NoSQL du jour in the hands of someone that knows it.
I currently design extreme-scale real-time analytical database engines, so I have no vested interest in any particular solution (we are not really competing with the current market). If I was going to build a large-scale web app today and needed a backing database, I would go with PostgreSQL -- it is very capable and well-engineered.
FTFY.
The difference between using a good competently engineered distributed database and PostgreSQL is that when your prime concerns are horizontal scaling and operations costs, the distributed database can be an order of magnitude simpler and faster than PostgreSQL given the same amount of effort.
Any additional information that you might be able to give me that might convince me?
Any more details about the project, or is it under wraps?
Most devs know how to get MySQL running without too much thought, but many don't even know what Postgres is.
If it's only a simple blogging platform, it might well be that MySQL will do the job just fine though.
It sounds pretty powerful and I've been told it is rock solid. That being said, don't buy into the anti-mongo hype so easily either. It's been overstated and for a large portion of the middle-ground on scalability and performance mongo is great.
The perceived "ease" at which MySQL is available, thus a lack of actual understanding about how to work with an RDBMS, is why MySQL is so widespread yet such a maligned platform.
We found out the hard way with our app, with which we wanted to do a lot of associating and joining of information. You either get locked into the document, or locked into writing interpretive code, and neither case is fun to work around.
These two things don't go hand in hand. JSON could be used to elegantly represent complex queries. A problem with the query system isn't necessarily a problem with JSON.
I think it could be used to represent complex queries, I don't think elegantly would be it. I'm thinking it would be a small step up from an XML representation.
All other aspects about it I love, though. It really makes development and deployment so much faster. Replica sets aren't perfect, but setting up MySQL replication w/ automatic failover on three or more machines is a recipe for disaster unless you have a DBA to sit there and baby sit it full-time.
[1] To see an example of the difference this makes see http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
It seems like the problem is that you're not using MongoDB in a sharded setup to begin with. For good or bad, MongoDB targets the scale where you need sharded and replicated setups. In other words, a large enough operation to require multiple servers for data storage. If you need the opposite of that, which is multitenancy, MongoDB is not going to be a good fit.
On the other hand, MongoDB has always been sold as a rapid prototyping and easy to iterate datastore, which is attractive for people working on small projects. Then they have an "oh shit" moment when they run into operational issues.