MongoDB Gotchas and How To Avoid Them

1. The keys in Mongo documents get repeated over and over again for every record in your collection (which makes sense when you remember that collections don't have a db-enforced schema). When you have millions of documents this really adds up. Consider adding an abstraction mapping layer of short 1-2 character keys to real keys in your business logic.

2. Mongo lies about being ready after an initial install. If you're trying to automate bringing mongo boxes up and down, you're going to run into the case where the mongo service says that it's ready, but in reality it's still preparing its preallocated journal. During this time, which can take up to 5-10 minutes based on your system specs, all of your connections will just hang and timeout. Either build the preallocated journal yourself and drop it in place before installing mongo, or touch the file locations if you don't mind the slight initial performance hit on that machine. (Note: not all installs will create a preallocated journal. Mongo tries to do a mini performance test on install to determine at runtime whether preallocating is better for your hardware or not. There's no way to force it one way or the other.)

MongoDB does not support joins; If you need to retrieve data from more than one collection you must do more than one query ... you can generally redesign your schema ... you can de-normalize your data easily.

This is a much larger issue than it seems - nested collections aren't first class objects in MongoDB - the $ operator for querying into arrays only goes one level deep, amongst its other issues, meaning that often-times you must break things out into separate collections. This doesn't work either, though, as there are no cross-collection transactions, so if you need to break things into separate collections, you can't guarantee a write to each collection will go through properly. (Though, I suppose if you're using the latest version, you could lock your whole database)

1) Make sure to permanently increase the hard and soft limits for Linux open files and user processes for the MongoDB/Mongo user. If not, MongoDB will segfault under load and when that happens, the automatic recovery process works incredibly slowly. It's a bit tricky to get this right, depending on your level of sysadmin knowledge. 10gen doesn't emphasize or explain the issue very well in their docs: "Set file descriptor limit and user process limit to 4k+ (see etc/limits and ulimit)" That probably makes sense to just about 0.1% of the people setting up MongoDB: http://www.mongodb.org/display/DOCS/Production+Notes#Product...

2) Make sure to disable NUMA. This 10gen documentation note is a great example of clear documentation: "Linux, NUMA and MongoDB tend not to work well together ... Problems will manifest in strange ways, such as massive slow downs for periods of time or high system cpu time." Massive slowdowns and mysteriously pegged cpu usage on production database systems are definitely 'strange'. I would probably choose stronger and more precise language, but 10gen clearly knows what they're doing: http://www.mongodb.org/display/DOCS/NUMA

tl;dr If you have problems with MongoDB, you aren't using it right. Read the documentation more carefully, and then when that doesn't work, hire an expert.

Nevertheless, a rundown of the gotchas and how to avoid them based on experience beyond simply running apt-get install mongodb is one of the most useful pieces on MongoDB I've seen of late.

The only new-news for me was that SSL support isn't compiled in by default. That's pretty irritating. I wonder if that applies just to 10gen's packages or also to distribution provided mongodb packages.

I thing this will be a good slogan - 'We are PHP of storage engines.'

Even if it's not your favorite technology, sometimes you end up in a position where the rest of the company is using something, and you need to work within those constraints. It's important to understand the technologies you're building on, their configuration options, and to understand the best practices way of working with them.

This, by the way, is not restricted to mongo.

Some people understand the tools they work with. Some people know just barely enough to throw things together and don't tolerate it when something doesn't work out of the box. Worst of all, this second group tends to be very vocal on the interwebs.

I'd almost like to see 10gen not publish the 32-bit package at all. Source is still there. If you want 32-bit, cool, compile it. But forcing the user to compile the 32-bit version assures at least a minimum bound of technical proficiency (an "I understand what I'm doing, why it's not the default and what the limitations are").

So what happens if I have 2 sequential failures? Suppose I have a replica set of size 5 and the master fails? The remaining 4 would elect a new master from amongst themselves, right? But then what if this next master also fails? The remaining 3 nodes are still a quorum (3 > 5/2) and thus (theoretically) should be able to elect a master. But am I to understand that they won't be able to do so?

Any recommendations for such a tool?

Or people are careless about what systems they put into production?

Also, awesome article!

JSON is picky in this regard, and I don't want to convert the whole string to B64 etc encode/decode it going in and out, as I would like to retain regex search capability for the 99% of email titles and names which are not Chinese within mongo from my php application which lives on the front.

But the thing I don't understand is, if people use replicasets, how comes they're not using encryption? It would be easy to sniff data off the instances. But yet, when I search on stackoverflow/serverfault, there are close to no people using SSL with Mongo.

I have been using MongoDB for a long time, unfortunately mostly this has been small applications, so you don't really get to test how MongoDB scales.

On that same note, I would love to see a list of gotchas for Riak (assuming some exist). I keep hearing recommendations for Riak, it would be nice to know how it fares in a large production environment.

(1) There's no need to add a "created" field on your documents. You can extract it from the _id field by just taking the first 4 bytes.

(2) If you are storing hashes (md5 for example), you might want to consider storing them as BinData instead of strings. Mongo uses UTF-8 so every character will be at least 8 bits whereas you can get away with 4 bits per character.

And

"Process Limits in Linux" If you experience segfaults under load with MongoDB, you may find it’s beacuse of low or default open files / process limits