MongoDB 2.2.0 Released | Hacker News Reader

MongoDB 2.2.0 Released

mongodb.org

129 pointsfrancesca60 comments

My experience with MongoDB hasn't been the most pleasant in a write-heavy environment. Until they fix the write lock properly, MongoDB is pretty much useless for many high throughput applications in my opinion...

The new DB level locking introduced in this release is a joke. There's not much difference between that and the old global write lock unless you split your database in dozens of smaller ones. What a pain. I wish they'd just stop pretending and addressed the issue properly once and for all.

I really want to like and use MongoDB because the way data is represented and how it can be queried is awesome.

Actually, there is a lot of difference but not just because of the global lock to database lock transition. There is also a lot of work done on yield-before-lock: http://blog.serverdensity.com/goodbye-global-lock-mongodb-2-... And don't forget that replication also happens in a different database...

It's inaccurate that there's no difference between the global lock and database level locking. Whilst it's true that locking is now down at the database level and you get benefits from splitting into multiple databases, the real benefit is from the new PageFaultException architecture. Even using a single database you will see significant performance improvements.

I benchmarked this at http://blog.serverdensity.com/goodbye-global-lock-mongodb-2-... and there is a video explaining how this works at http://www.10gen.com/presentations/concurrency-internals-mon...

That's good to know. It still feels like an half-assed solution to a very serious problem, however.

I definitely won't be looking at MongoDB again until this is fixed for good, and until I know replication is more reliable. Had terrible problems that that too unfortunately.

It might "feel" like a bad solution not granular enough but if you actually test the implementation then locking is no longer a real concern. There are still improvements to be made and this is the first step towards that. Changing from a global to database lock required huge amounts of work which sets the foundations for the next releases with collection and document level locking.

In my experience replication has been extremely stable and reliable for several point releases so I'm not sure what problems you had.

It's not half-assed so much as a first step. 10gen have previously stated that the plan is to gradually increase the granularity or write-locks over successive releases.

Until they end up with MVCC...

aggregation and then MVCC.. the circle is complete.

One of the reasons DB-level locking is actually quite important is that the replication oplog is on the 'local' database and your data is on other databases -- so your application is no longer fighting with replication for the write lock.

David knows what he is talking about, they use MongoDB in really high scale. Also, getting database level locking was like 90% of the work according to the CEO. Doing collection and finally document level locking should be somewhat trivial now. 10Gen just want's to take it slow, and make sure database level locking is working 100% correct before moving onto more granular locking.

Yep they sure are taking it nice and slow... at this pace, collection-level locking is probably only 2 more years away. Other nosql databases are nipping at their heels.

David may use MongoDB at really high scale, but since when is that a reason to call someone an expert.

Moving fast and breaking things in the database world is not a good thing. That said... I highly doubt collection level locking will take too long.

I don't see anyone nipping at their heels. They have some competition in the key/value space, but there are a few different KV paradigms.

MongoDB is a complete replacement for most RDBMS apps and they are way ahead of the pack.

DB Level Locking (https://jira.mongodb.org/browse/SERVER-4328) is a dependency of Collection Level Locking (https://jira.mongodb.org/browse/SERVER-1240). For a critical, tricky core piece of functionality like this it makes sense to tackle one major concurrency issue at a time, do a release, stabilise the code and verify there are no issues introduced by the new locking code and only then release the next step in locking enhancements. Doesn't seem like they're "pretending" about anything as far as I can tell.

They were certainly pretending like it wasn't an issue when I contacted their support some time ago.

Support? Pretending something isn't an issue? Must be entirely unique to this technology company alone.

I have used Mongodb a bit, but only for applications that mostly need to read a lot. What issues might I run into if I need to do lot of writing as well?

None, almost certainly -- if you have enough RAM to fit your working data set, then MongoDB's writes are just changing memory. If it does need to read data from disk before changing it, the new yielding architecture that others have mentioned will help prevent lock contention.

The problems with early rev SQL servers were similar (in inconvenience) but, with feedback from users and hard work from developers, they soon moved beyond MVP.

I'm willing to suck up some inconvenience with MongoDB, it's one of the most inspiring technologies of the last decade. Web application persistance feels more like a fully connected limb now.

I can understand the gripes but do urge people to give 10gen some slack and appreciate that what they are building takes time - it's clear that the team are working hard on features driven by community feedback.

It's more than inconvenience when your whole service is down because some database queries take up to 30 seconds to complete (and often time out), and when replication on some servers is days behind (and some are up to date). We also had to completely flush our data a few times because of some corruption we couldn't recover from.

The truth is, MongoDB is awesome if your dataset fits in memory, but if you're in a write-intensive environment with 400-500GB of data, it's just not there yet.

Not true. We're processing 12TB of data each month with thousands of writes per second and sub-millisecond response times. You simply have to understand how to use MongoDB correctly which isn't difficult with the out of the box settings and by reading some of the documentation. Figuring out your working set is the most difficult part but that doesn't take long to calculate based on understanding the queries you're doing and creating the appropriate indexes.

Since you seem to be fairly negative about MongoDB but light on details, perhaps you should write up your experiences so others can learn from what you did wrong.

Awesome release,

notes summary:

• Aggregation Framework to fix some map-reduce woes.

• TTL Collections

• DB Level Locking (A step in the right direction)

• Better yielding on page faults

• Tag aware sharding (HELL YES)

• Better Read Prefs

• Indexes now handled by mongodump/mongorestore

• mongooplog replay is awesome for getting point in time backups

• Shell now has full unicode, multiline command history, $EDITOR support (all from change to linenoise.c

could you go into why tag aware sharding is a "HELL YES" for you? I was kindof already of the opinion that mongo sharding is much too complicated (http://i.imgur.com/c3Dpq.jpg), this seems to compound that. Is it something you've needed/wanted so bad that its worth it?

I've been playing around with the Aggregation Framework lately (using the release candidate). The performance seems to be pretty reasonable, especially when compared to similar tasks with the old MR framework. A quick and dirty benchmark number in case anyone is interested:

* Obligatory unscientific, probably not meaningful, etc. disclaimer.

Mongo Version: 2.2.0-rc1

Hardware: MBP, Snow Leapord, 2.2 GHz Intel Core i7, 8 GB mem

Data: Single collection with 500k records (machine generated time-series event data)

Query Pipeline:

  [
      {
          $match: { ts: { $gte: 1293858000000, $lt: 1296536400000 } }
      },
      {
          $group: {
              _id: 'aggregations',
              sum: { $sum: '$foo' },
              num: { $sum: 1 },
              avg: { $avg: '$bar' }
          }
      }
  ]

Results: The time range matched against above matches 42,466 documents within the collection. The average response time over 50 runs is 419ms. Not exactly "Big Data OLAP" stuff just yet, but plenty fast enough for most use cases involving reasonably small sets of data. Great job to the MongoDB team!

Out of curiosity, did you have an index on the ts field that you are matching with?

By the way, if you (or anyone else for that matter) come up with useful benchmarks I'd love to get a copy of them at mathias@10gen.com. I have a few of my own, but I'd like to get some real-world workloads from the community to test potential optimizations against.

Yes, I had a single index:

  db.events.ensureIndex({ ts: 1 });

I'll try to clean up my benchmark code a little, throw it in a gist, and then I'll send it your way.

Update: already forwarded this to Mathias, but posting here in case anyone else wanted to see the full benchmark test: https://gist.github.com/3518344

Release notes: http://docs.mongodb.org/manual/release-notes/2.2/

Thanks. I clicked the "Changelog" link on OPs page and that was worthless link to jira https://jira.mongodb.org/browse/SERVER/fixforversion/11496

For startup projects I love Mongo because we can get a product up and running very quickly. However I always feared in the back of my head we would have to move off of it if our service got too big. Maybe it is all the complaints from a small portion of heavy users. Regardless, big updates like this are going a long way to help make me feel content on continuing to use it as we grow.

With MongoDB, you are able to shard your system. This means you can grow your databases horizontally. This is not something you can easily (or cheaply) do in the world of RDBMS. You will see much better scalability with MongoDB than with something like MySQL.

You can shard RDBMS relatively easily -- basically you wind up pushing a part of your database structure into your clients, so your clients can decide which shard to use.

The cost, though, is that you wind up having a difficult time doing some things that MongoDB can't do. (For example: Renormalizing your database... Does that even mean anything for MongoDB?)

There's something to be said, of course, for simplifying your design. But it's probably a good idea also to make sure your design reflects your requirements.

Indeed you can distribute data to multiple independent RDBMS, but balancing when new nodes are added is probably a manual process (or a lot of custom code) that is likely to require downtime. To avoid downtime, your application would need to write to both chunks while it is balancing/migrating (and then delete the old data/chunks once it is migrated to a the RDBMS). Essentially, you would need to write what is already in MongoDB.

You would also have to write a parallel query engine.

I too am a fan of simple designs, but I think rolling your own sharding on top of a RDBMS would likely be a massive chunk of time.

There are really expensive commercial products working on horizontally scaling RDBMS... but personally, I prefer open source and document oriented databases :-)

> However I always feared in the back of my head we would have to move off of it if our service got too big

I like to call that "nice problem to have" territory. There is nothing to fear from being too successful, or from having the problems of success. Success problems can be solved by the application of people, time and money. And like all optimization you measure first to make sure you really have the problems you think you do, Amdahl's law etc.

Far more likely you'll have the problems of not being successful, such as no attention or money from customers and investors. Or not being in the business you think you are in. There isn't really that much point putting in infrastructure just in case you get successful, if that same infrastructure takes time to develop, and slows development.

I'd be delighted having a service "too big" for MongoDB! After all it would mean being more successful than the companies listed at http://www.mongodb.org/display/DOCS/Production+Deployments

In all this talk of locking were missing the real benefit to a heavy write environment, better yielding. this is going to be huge for those one off writes that in the past would have held the lock. Also I am looking forwards to playing with the new aggregation framework.

There is one thing holding me back still from mongo or I'd be using it right now instead of Postgres: native decimal support [1].

There are a lot of proposed work arounds, and some surely work fine if you're only dealing with 2 decimal currency.

The solutions don't scale for arbitrary precision based on the field.

The new aggregation framework is a fantastic step forward and from my testing is relatively peppy, even at a 150k document collection.

Side note: Anyone know a nosql solution that doesn't treat decimals as floats?

[1] https://jira.mongodb.org/browse/SERVER-1393

A native decimal datatype would be awesome indeed.

The Aggregation Framework is a great feature, already in production...

Been very excited about this feature since I first heard about it!

This does look cool. Aggregation previously was a joke with its collection size limitation.

I haven't used Mongo in a year. Are Map Reduce jobs still single threaded?

Yes.

Does anyone know if Collection level locking is in the pipeline?

It is, it's next in fact, but it was decided to make sure DB level locking worked first (and gets the bugs shaken out of it) before moving on to the next level of locking. Dwight gives a decent description of the thinking in this presentation:

http://www.10gen.com/presentations/concurrency-internals-mon...

Great news. We use MongoDB for call data collection, distributed control, and as a firmware/application fileserver in for telecoms testing.

The true secret to Mongo's awesomeness for me has always been the dev experience. JSON documents, query by example, etc.: Awesome API.

A friend of mine wrote this a few months back:

http://blog.engineering.kiip.me/post/20988881092/a-year-with...

Almost all of it is still valid. "To be fair, the global write lock is now JUST a DB level write lock. Living in the future guys."

Blog post with the details is up http://blog.mongodb.org/post/30451575525/mongodb-2-2-release...

Still no support for Solaris on SPARC! :(

Isn't that system dead?

I think you'd be quite surprised if you ever found out just how much of the world, including your very own existence, depends heavily on platforms and systems that you consider to be "dead".

I totally agree there are a lot of big apps running on Sparc... but I think that it would be much cheaper to buy new x86 hardware for a all products/systems going forward (i.e., those written for MongoDB). Sun (errr... Oracle) also sells x86 Solaris servers (a platform MongoDB supports).

I would like to see ARM support :-)

I'm not trolling you when I ask this, I'm simply curious.

Of the open source projects you follow, how many still do builds for Solaris? How often do you have to build them yourself?

Lots of big companies, mostly infrastructure like telecom or tv, use Solaris still. Unless the project aims to support those uses, it usually isn't worth their time.

Legitimate question.

Not many do. There are quite a few things I build myself if I really really want to use it. A lot of the time it isn't worth it.

I would totally be using MongoDB for some projects (it fits the bill -perfectly-), but my resources for those are usually limited to Solaris on SPARC.

I do have an x86 desktop with Linux as my main PC at work, so I don't miss out on all the fun completely.

However, I do mostly sysadmin stuff, so a lot of the things I use come in the form of scripts, so it's often cross-platform.

Though I do come across some install scripts that just blatantly assume that everything is running Bash on Linux. Those are fun.

I have very little expectation that the MongoDB team will ever push a non x86 version. They do a lot of optimisations deep inside that rely on architecture specific things. But one can hope :)

Awesome! Looking forward to seeing this propagate through the 10gen official repos like yum.

...and they're out!

Have had a great experience with MongoDB

I use it for my ecommerce platform. Great engine!

Which e-commerce platform? I'm just curious.

I can't wait upgrade.

Interesting

Reply on news.ycombinator.com