Notes on MongoDB, GridFS, sharding and deploying in the cloud | Hacker News Reader

Notes on MongoDB, GridFS, sharding and deploying in the cloud

viktorpetersson.com

54 pointsmvip11 comments

Also, if you're curious about bandwidth limitations in this particular cloud, take a look at these benchmarks: http://viktorpetersson.com/2012/01/23/benchmarking-virtual-n... http://viktorpetersson.com/2012/01/24/benchmarking-and-tunin... (Using CloudSigma as the cloud-vendor and FreeBSD 9 as the host)

I might be missing something due lack of experience with MongoDB, but what did disk I/O have to do with the secondaries falling behind, and why did effectively eliminating one of the secondaries "fix" that? The description makes me think there's a pathological case lurking in MongoDB that needs to be looked at more closely.

The reason why the secondaries were falling behind was because the I/O was maxed out on the primary. It was barely able to keep up with the regular load, and hence reducing the load (by only having one node replicating instead of two) reduce the I/O load significantly.

That said, reducing the number of secondaries was just a short-term solution. The long-term solution is to convert to a sharded environment and offload the one maxed out node.

That makes sense, but unless bandwidth is the limiting factor, it sounds like there is a potential optimization where the I/O is only performed once for both secondaries.

I'm sure there is plenty of room for optimizing many things in Mongo. I don't know if secondaries can replicate from each other (if one is behind the other), but that would make sense. Mongo is a great product, but there is room for improvement of course.

I'm still confused. Why does replication hit the disk on the primary? If the secondaries aren't already way behind, shouldn't the data be going straight from RAM out the network interface?

In our case with GridFS, the objects wouldn't fit in RAM. Only perhaps the last two objects would. The node had 4GB of RAM and many of the objects sent Mongo were around 0.5-4GB.

Ahhh, and presumably replication doesn't start until the entire object is loaded. Now it makes sense.

There are ways that could be fixed, but at least now it's just a lack of optimization instead of sounding like something's completely turned on its head. :)

Wouldn't the proper behavior be to provide some sort of back pressure so that it cannot decide for you to no longer backup your data to the replica set?

I would expect that if you say you want data replicated, that if it can't handle that under load, it would back off on the load that it is capable of instead of going the pathological route of simply not backing up. If under load the primary goes down with a serious disk failure, you now just lost a ton of data that you expected were being backed up all along.

You can apparently invoke a level of synchronous behavior in MongoDB with the getLastError command: http://www.mongodb.org/display/DOCS/getLastError+Command

Conceptually, its abilities look a lot like Cassandra's "configurable consistency" model.

Consistency and durability are two different things.

Reply on news.ycombinator.com