Redis, MongoDB and the power of incremency | Hacker News Reader

Redis, MongoDB and the power of incremency

blog.mongohq.com

11 pointsliz_mongohq6 comments

There were many potentially other solutions to this issue. The clients could have buffered writes locally, and done batch updates to MongoDB instead of incrementing each time.

Using a local SQLite, I can do "update counts set val=val+1 where id=?;" ~250K times per second on a Macbook Air. It would've been simpler to just do a local aggregate and then a batch update. That avoids Redis durability issues, or a secondary merge process.

FWIW, a local Postgres can do only about 3200 increments per second.

I ahve to agree - this really seems like making a square peg fit in a round hole.

As an aside, even on a 48-core box with 128GB memory, I was unable to get more than ~1000 inserts/sec with MongoDB, even without write confirm - I'm not sure what the bottleneck is with it.

so much for "atomic modifiers for contention free performance" (straight from http://www.mongodb.org/)

also, if it's single threaded: why does adding more cores increase the update rate?

isn't the entire point of mongodb that it's fast? if it can only do 1500 increments a second it's pretty useless...

A theory: One mongod instance per-core on the same box perhaps, using a mongos instance to coordinate? This seems like a complicated way to run a "simple" database engine however, just to get the write rate up.

Edit: Confirmed in the blog comments.

mongos has extremely high overhead - 700% CPU under moderate load on a webserver. The more shards the higher that load, and eventually you run out of local CPU due to mongos overhead. Then you're looking at remote mongos, making it even more complicated.

There's more to lock than just the write, some of this stuff will be lots better in future Mongo releases, but at the moment doing a safe write means a lock is held longer when there's network latency of any kind. Local updates are shockingly faster than even local network updates.

Honestly, the entire point of Mongo isn't that it's fast, it's that it's fast enough and can scale a reasonably flexible schema through the life of an app.

Reply on news.ycombinator.com