MongoDB as a better default data store | Hacker News Reader

MongoDB as a better default data store

blog.gregweber.info

36 pointstrustfundbaby41 comments

> Most SQL databases trade off too much in terms of performance and features in order to have 2 things that are actually low on the need list of most web applications:

> * relational tables

> * transactions

Shudder... I think I'll hold on to my relational tables and transactions as the default core of my systems, and optimize when I see that they're not fast enough, thank you very much.

Having taken a ruby app through mysql => mongodb => mysql there were several reasons that I wouldn't recommend mongodb as a goto solution for persistence.

1) Your tests have to be document aware. Models that are embedded within a document are very difficult to test in isolation from the rest of the document. You need to build the whole document before you can test any persistence related functionality.

2) Refactoring relationships is difficult. When you start building an app you don't know how the model is going to turn out. With a relational model that's fine, you can add/remove relationships easily. With documents your models are organised into hierarchies that they have to be aware of. The code that handles this hierarchy is incredibly brittle. If you move a model from one document to another, or just it's own document, you're going to have lots of code to fix. Maybe it's a case of needing some new refactoring patterns, but rest assured, they'll be far more complicated than refactoring patterns for relational stores.

3) Mongoid and the uncanny valley. It looks like activerecord, but I spent an inordinate amount of time working around bugs and methods that you expected to be there that just weren't. Admittedly this was before the stable version was released(the recommended version at the time) so things may look significantly better now.

I think you're talking about a few different points here that aren't necessarily related.

Persistence (which is available in MongoDB) has nothing to do with testing of a model (a single document inside a collection). Some actually see that as an advantage of using a schemaloose (yes I just made that up) storage like MongoDb. I don't need to have a whole document defined in order to test the functionality of an embeded object in that document. Yes, you're relying on the application alone to enforce the model, but that's a trade off made up front regardless.

As for modeling and the cost of refactoring, I think it depends on more than just the storage facility you're using. The code that handles your modeling in MongoDB is only as brittle or robust as you choose, same as a SQL solution. Yes, it helps if you think more up front about how you are going to query the data that you store using MongoDB as this will help to minimizing refactoring later on. However, I think the pain associated with moving part of a table to another / new table in SQL is no less painful than moving an embedded object or other data structure around in MongoDB. In both cases you're moving data and affecting changes in your application.

Thanks for sharing your experience.

I think 3) is better now that version 2.0 of Mongoid has been released (and is stable). The developer I am working with complained about earlier versions also. I ported hundreds of lines of ActiveRecord model code, and it mostly just worked.

I am not really experiencing pain around 1) either. I am using factory girl and find it easy to build any required data. It could actually be beneficial that you must focus on creating a group of properly linked objects. I have run into trouble before with messed up/difficult to create object graphs in ActiveRecord tests.

2) Is a good point I am now mentioning in the post. I will let others argue with you about its severity :)

I probably need to do some code examples to illustrate the pain points I was experiencing for 1). When I migrated back to ActiveRecord I kept the same model initially and found that I was able to significantly simplify the tests. My main difficulty was with methods that called save on embedded models. I suppose I could have stubbed those calls out but I generally prefer to test the integration with the db. Yeah, I know it's not a pure unit test but I find this is generally the more pragmatic approach.

I should say I think mongodb/mongoid can be very useful. My opinion so far though is that it's most effective as a performance optimisation strategy for a stable application.

Yeah. The idea that I couldn't roll back a transaction if there was a problem, or that part of my data might get through while others didn't is a terrifying one. My customers _care_ about their data being accurate.

And similarly - most of the data I work with is naturally relational. Having the database make checks to ensure that invalid things aren't happening is a lifesaver against coders making a stupid mistake (and everyone makes one from time to time).

While I'd never suggest Mongo if you need transactional support on par with OLTP, Mongo does have single document transactional support through the use of getLastError and its various options.

http://www.mongodb.org/display/DOCS/getLastError+Command

Having the database make checks to ensure that invalid things aren't happening is a lifesaver against coders making a stupid mistake

------------------------------

mongo now has a 'safe mode' option that protects against this, just FYI

I see a lot of negative comments that lead me to believe that few here have seen the problems with RDBMS's that lead some of us to consider NoSQL alternatives. An application that frequently times out (page loads over 30 seconds) to me is a good candidate for a NoSQL backend, especially considering we are mainly storing documents. This app was not developed by us and granted the design is terrible with many N+1 queries (which we are eliminating one by one), but I'm very eager to consider alternatives to MySQL for this and future apps. Any comment that dismisses this need off-hand strikes me as somewhat unaware of the problem. What I would like to see more discussion of is what are the viable alternatives to RDBMS + ORM (specifically MySQL + ActiveRecord). I am specifically considering Mongo + Mongoid. Now if anyone can give me some solid reasons why this is not a good idea, I would be glad to hear it.

"An application that frequently times out (page loads over 30 seconds) to me is a good candidate for a NoSQL backend" -Why? What tells you that the application is spending most of its time waiting on the database? If it's spending a lot of time waiting on the relational database, I would suggest it's a good candidate for: 1) query optimization (as you start to surmise with your N+1 observation) 2) possibly better disk IO 3) possibly a different db architecture

We have New Relic installed. MySQL is not the only bottleneck, but it's a big one. Ruby code is another. New Relic can't show us how much Ruby execution time is ActiveRecord and how much is our application code, but we recently upgraded from Rails 2.3 to 3 and noticed some decrease in speed. We have been implementing memcached and optimizing queries with pure SQL but still have a long way to go to get decent performance. Most of the slow pages are back end admin stuff, so it's not absolutely critical, but nevertheless it's left me with a bad impression of the traditional stack.

But that sounds like a problem with the application, not the stack. What makes you think that document oriented databases are any more immune to bad application design than relational databases?

I believe it's a combination of application and stack that's the problem. I've been comparing Rails 3 apps to similar apps in other frameworks. Locomotive is a Rails CMS and it was taking ~1 second per page load. Calypso is a CMS in Node.js and page loads were nearly instant. Both used Mongodb. Locomotive sped up when 4 unicorn workers were applied, but it still didn't match Node.js.

I did not say that "nosql" databases don't have their place, I said that I'm not interested in using them as the 'default'. If you've built a site that just isn't running well with SQL, well by all means, use what works for you, but to me it seems logical to start with something solid and well known and then optimize later, should it prove necessary.

  > An application that frequently times out (page loads over  
  > 30 seconds)

An application like this is a good candidate for review what the hell is going on, before making any decisions about changing the stack.

We have done months of review and optimization. Still many page loads must be backgrounded (mainly reports and admin interfaces). We are dealing with a large amount of data.

People like you give nosql a bad name. Yes, nosql can help, but you're looking at it for the wrong reasons.

Go back to school, or hire capable developers.

There seems to be a common misconception that relational databases have no in-memory indexes and query caches and therefore are intrinsically slow. I don't understand why this is.

Because the people who write these articles are morons?

That is not very charitable of you, is it? I think it's fair to say that in many areas tools to interface with relational databases are lacking; consider the horror-shows that are Arel or Hibernate. Also, good schema design is not, in itself, a trivial task, especially considering how common ignorance of such principles are. In my experience, a not insignificant number of practitioners learn a "normalization" peculiar to a specific product--or even a specific product version--and apply this forward into the future to disastrous effect.

"Over the next couple of years a lot of developers are going to make good money moving companies off of Node.js and MongoDB." - http://twitter.com/#!/b6n/status/73535660467818496

Here is a job link if you'd like to work with this guy: https://boundary.com/jobs

And an article followup touching on the the basis of his tweet: http://metaduck.com/post/3564002672/asynchronous-iteration-p...

Big words, little substance. This is why I'm not on twitter.

The words show up in a standard font for me...

I rather like redis, once you get the hang of doing things in that way, and if you plan your system so that you know what ways you're going to access your data in advance, it's really not a problem. Besides, say you want to get a list of "events" by date, you can regenerate the index, or do the sorting within the app.

I'm new to Redis and my approach to this problem has been to create a Sorted Set with the "score" being the date represented as an integer (in JS it's getTime()).

Due to global write lock you unlikely can use MongoDB without memcached, if you have a lot of writes. We sometimes have queries waiting for a lock for a second.

Indeed, once your app achieves a certain threshold then Memcached comes into play. Curious if you've looked into sharding as a way to help minimize the lock time needed?

Per collection locking is coming soon as well https://jira.mongodb.org/browse/SERVER-1240

Yes, we are going to use sharding.

But if per document lock could be available, then mongodb performance would be much more impressive. Though as far as i understood it's not possible with memory mapped files for some reason.

We can put data into MongoDB faster than we can get it out of MySQL during the migration.

Yes, with the small caveat that Mongo doesn't respond to any queries[1] while you do that. So better don't try such a bulk operation on a production Mongo.

[1] http://2.bp.blogspot.com/_VHQJkYQ5-dY/TUO3RAn8SNI/AAAAAAAABq...

As the original author of that quote, no. I was running MongoDB in "safe" mode where it doesn't return until the query has been ack'd. Furthermore, I was running with w=2 meaning that at least one other note also had to ack the data.

Well, I've observed the same behaviour on a single-node mongo during bulk imports, without special configuration changes.

Edit: Not sure why it's getting downvoted. You can reproduce it fairly easily by performing bulk writes - esp. deletes, but also mongoimport or just a write-loop. The global lock is a known problem.

No-SQL solutions have proven their worth, but off course are not in any case a replacement for any relational database

Of course...and relational databases might not be the right tool for the right job. Making a blanket statement that there is a "default" for data in a web site is purely ignorance of the problem (i'm not saying you're saying that....)

Of course the original author seems to think that most relational databases lack performance and / or features. There is no one size fits all situation. This is why companies like facebook, netflix, etc. use combinations of relational databases and nosql databases (although i think netflix is almost completely over to simpleDb now...)

I can't help thinking people who say MongoDB should be a "default" for all web apps are people who've never had to set up, maintain, monitor, debug, or fix a real MongoDB instance in production. It's a pain in the ass compared to MySQL or Postgres. Since doing it right involves redundancy, 64-bit environments and lots of storage space, it can be very expensive to host yourself, and Mongo-as-a-service options like MongoHQ are also pricey for the level of storage/performance you get. That's not to say Mongo sucks — I love Mongo, but I've been burned enough trying to use it on various projects to make sure I (or my company) have a solid reason for needing it before subjecting myself to more pain.

In my experience there are always ways to tweak something in your RDBMS setup to improve performance, like tuning queries or your indexing strategy, or skipping using an ORM for super-critical stuff, or looking at I/O latency or blocking issues between your app and DB machines.

The CTO I am working with is responsible for maintaining both MySQL and MongoDB data stores himself in addition to writing a lot of code. After starting with MySQL he has been gradually shifting everything to MongoDB and would like to just get rid of MySQL.

I personally don't have deep operations experience, and I am sure your individual experience is valid.

I think that in part, MongoDB is maturing- it now has single server durability and replica sets. Single server durability means it now can be a default, whereas before you had to commit to 2 servers as you allude to.

People seem to report it is more critical to have enough memory, and that it wants more in comparison that a SQL database, so perhaps that limits its ability to be a default database.

The issue of course isn't whether you can adjust an RDMS based infrastructure to suit your needs, but how much effort that will be in comparison to an alternative.

I don't understand why he talks about an ORM (object-relational mapper).

I added a more detailed explanation, I hope it makes sense:

In a database with a schema, if you saved a string to an integer field, the database could either complain or coerce the string to a integer. Either way, whenever you read a row, you know what type the value will be whenever you read data a row from the database. Another issue with schemaless is simply mis-typing the key (column) name. If you have a field called referrer, but you perform a query using referer, MongoDB won’t know you have done anything wrong. In a SQL database you will get an error on insert and on querying. An ORM can instead be the one to tell you about the error.

An object-relational mapper maps between objects and relations -- hence the name. Thus, without relations (-> Mongo) there can be no ORM.

Perhaps I am using the term to loosely, and I do see people use an acronym with a D for document now. There are still plenty of relations in Mongo, particularly when you expose them through Mongoid, rather than being enforced in a schema: http://mongoid.org/docs/relations.html

The MongoDB docs do say that I am supposed to use the term "ODM". I will update my post. thanks.

I was thinking the same thing. Why do you have to have an ORM to make Mongo great?

Reply on news.ycombinator.com