jasonmccay

Interesting feedback ... I agree on the log messages that look terrifying yet, when googled, are like "oh, that's normal, don't worry about that." I also agree that the 2.x versions of MongoDB have been a great step forward for stability and performance.

On the logging, not sure what you mean by the old logs get clobbered. Is this, perhaps, some house-cleaning job that you have on your server? When we stop/start processes, the logs are appended and keep going.

On the issues with primary/secondary and state changes, do you host your databases on Amazon? We have often found that when we have this issue, it was pinpointed back to some temporary intra-zone networking glitch with AWS ... either with their name resolution or a blip in one zone being unable to see another zone in the window of time that MongoDB has set for the response.

These often (if not all the time) go unreported by AWS until you get them to dig a bit and report back. We effectively utilize priority to keep the member we want as primary and if a state change does occur, it moves back once the full set is healthy again.

Can you offer an example of the creators claiming that it does something that it really cannot do?

Also, while MongoDB does have its faults and some sharp edges, it is far from poor. For the most part, 2.x MongoDB is solid, performs well and generally without issue.

You are making sweeping (but vague) negative characterizations of a technology while, at the same time, accusing the company that created the technology of marketing it strongly.

When DHH created Rails, was it his job to tell you all the reasons why you should not use it over Java/.NET/PHP or, instead, to tell you all the reasons that he created it and the problems that he was trying to solve by developing apps with it?

To say that there are better alternatives implies that you understand everyone's use case. It is faulty logic. One of the reasons for the revolt that was "NoSQL" was an attempt to get people to stop shoe-horning data into MySQL ... so the concept of developers using database engines poorly isn't new. Same with MongoDB and developers that use it based on their experiences with MySQL or other relational (or non-relational) models.

I have seen people use MongoDB to great (even amazing) success and have seen others use it very poorly. In both cases, it was the developer (and not the technology) that was responsible for the way they used the technology.

I agree and I do think that often times people will select a technology and think it should work like some other technology, become frustrated when it doesn't and then, in turn, blast the technology on only the merits they understand.

There are certainly reasons for using Riak, HBase, Cassandra, etc. and there are reasons for using MongoDB. It is when people seem to act confused when their hammer isn't acting like a screwdriver that we get these blog posts.

So, if someone asked you why they should use "Riak and countless others" and not MongoDB, what would you really say? Also, you seemed to imply that Riak was a go-to solution (my wording) while implying that MongoDB was more of a fringe "buzzword" technology ... when, counting features, I think most would acknowledge MongoDB as being more mainstream.

There are a large number of well-established and quality installations of MongoDB. It works really well at both small and large scale and with a bit of tweaking (like any technology), can perform nicely.

Really? Do you work on your own car? Is it because you think you aren't smart enough to do it or because you think that spending your time on other efforts is a more valuable configuration?

AWS is the electric company. Heroku tends to be the circuitry, outlets and light switches for your house. You may take it for granted because it looks so simple, but it's the simplicity that makes it great.

As PaaS providers begin to mature and turn attention to more worthy challenges (like geo-agnostic deployments, easier scaling, for example), then their value is only going to increase.

MongoHQ has a licensing agreement with 10gen for the name. Plus, as already mentioned, MongoHQ works very closely with 10gen to not only promote the use of MongoDB, but to improve MongoDB as a database product.

I think the point he was making was that the company in question never tried to scale horizontally with MongoDB because, in their words, "we believe horizontally scaling shouldn’t be necessary for the relatively small amount of ops per second we were sending to MongoDB."

Yet, they went and scaled horizontally with Riak and experienced pain.

Their opinion that it did not make sense to have to horizontally scale the "relatively small" number of ops they were sending to MongoDB is certainly their own, but then they horizontally scaled with Riak anyway and boasted about their 25MM ops per day scaling ... which, averaged out, is only about 280 ops per second.

In short, it was far from an apples to apples comparison.

I find it a better course of action to work on properly understanding your data problem and then evaluate the options available instead of just sticking with the stand-by solution ("Well, we've always done x!").

Just like with anything in technology ... there is going to be pain as you escalate the level of complexity and what you are trying to accomplish.

I think that you would find MongoDB to work nicely for you ... especially with the need for rich queries (which assumes, on my part, that you have more complex data needs, documents, etc.).

With all due respect to the Kiip engineering team, this wasn't a strong case for using Riak over MongoDB ... but rather the general pain that a engineering team feels when horizontally scaling in the cloud.

The reality is that developers are maturing in their understanding of different technologies and they are learning how to apply them in correct use cases.

There continues to remain a "golden hammer" syndrome where white horses and unicorns run free, but it doesn't exist.

Instead, the vision of "NoSQL" was to tell developers that they did not have to use relational data for everything, but could, instead, use the right tool at the right time. Why is this such a hard concept?

If you are a developer and you don't understand the tool you are wielding (it's pretty clear the author of this blog didn't), then you will incorrectly use the tool and experience pain.

That is the fault of the developer, not the tool.

There are fairly straight-forward ways to control memory usage in MongoDB, especially if this is a concern for you. All you need is a bit of OS know-how and it works just fine.

We run both large (multi-shard clusters) and small memory (500MB - 2GB) use instances of MongoDB and have no problems.

It would be good to have developers acknowledge that, perhaps, they may not have all the information instead of declaring that something can't be done.

Again ... this is more an indictment on the poor IO performance of Amazon EBS vs. MongoDB as a solution. MongoDB can scale both vertically and horizontally, but as with anything you scale on Amazon infrastructure, you are going to have to really think through strategies for dealing with the unpredictable performance of EBS. There are blog posts galore addressing this fact.

I often think MongoDB has suffered more as a young technology because of the proliferation of the AWS Cloud and the expectations of EBS performance.

I have to respectfully disagree. Your comment is a bit sweeping and I really don't think that MongoDB's sharding solution is a bad one ... simply the strategies are different.

There is a large set of nice features that makes Mongo, for most people, nice to use long before you even need to address sharding. The percentage of people that will need to shard is much lower than the percentage of people that can get considerable benefits of how you can query data in MongoDB, for example, vs. Riak.

You are such a Riak-lover. :P

I understand your point, but the performance issues still stem off of poor IO performance on Amazon EBS. As we continue to use it, we continue to find it to be the source of most people's woes.

If you have solid (even reasonable) IO, then moving things in and out of working memory is not painful. We have some customers on non-EBS spindles that have very large working sets (as compared to memory) ... faulting 400-500 times per second, and hardly notice performance slow downs.

I think your suggestions are legit, but faulting performance has just as much to do with IO congestion. That applies to insert/update performance as well.

Nice promo video ... LessConf is enjoyable because it really does focus on making sure that all attendees have a memorable time. Besides learning a ton from speakers, you make a lot of great friends and feel really taken care of.

It is the only conference that I have attended where I was sad it was coming to a close.

Give it a go ... it is hosted at a great location and worth much more than the small amount of money it costs to attend. Steve and Allan do a great job.

The MongoHQ API is making great strides moving in this direction and as was mentioned in another post, the team is dog-fooding it so much that they are building the next version of the MongoHQ web app on top of it.

The added benefit that we see is that we are experts in managing, optimizing and scaling all of the backside data structure for you, while offering solid analytics, performance metrics and visual tools that allow you significant control over your data as well as critical insight into what is happening.

Really excited about this type of development going forward and good to see other offerings in the works!

(full disclosure: I am one of the founders of MongoHQ.)

Seriously bummed. This was a reliable and (very) useful service for all of our environments on EC2. We had it tied in with Nagios and it worked quite well to alert us of issues.

Since this is official, we will probably start relying on PagerDuty now.

Take the risk and apply. They are looking for brilliant people and are willing to mold and improve your idea or give you an entirely new idea if they like your team. While there needs to be a basic concept/product there, they really do count the team much higher than the idea.

Go for it and good luck!

Assuming the changes that were introduced in MongoDB 1.8 for single-server durability, writes are being put to disk with journaling. So, the data is written to disk.

I am not knowledgeable of your situation, but I am curious about your set up, the intentions you had when you were storing things and other factors in MongoDB not storing your data. Also, I am curious how long ago this was...what version of MongoDB you were using and if you were checking getLastError after the write (assuming you wanted that behavior).

I am not denying your experience, just curious about the whole picture.

Respectfully, this isn't true. Yes, fire and forget is the default behavior, but there is a confirmation step that you can check. It may be implemented a bit differently from driver to driver, but it is generally called a "safe insert". Numerous people use this to ensure their writes across single databases as well as multi-node master-slave and replica sets.

Would you mind elaborating on why MongoDB cares about speed over reliability. Sure, you can go that route (and sometimes that is what developers want to do), but MongoDB, as a database, reliably stores and persists your data.

Just curious what your experiences were that made you think MongoDB isn't a reliable data store.

I have a hard time understanding responses like this and, in the end, they make me think that people a) really don't understand some of these new data technologies and b) are just as much on the anti-bandwagon as those that sit on the latter.

For people using MongoDB daily, their data is in the correct format, it is persisted to disk, it is stored reliably and it is fast. On top of that, it gives them flexibility as their web app changes and new features and data structures are added, it is easily viewed and manipulated in JSON-format, and if and when the day comes that they need serious scaling, it helps there as well. Also, in the world of EC2, it is straight-forward to set up replica sets to offer redundancy.

All these conversations sound eerily familiar to the Java guys bashing Ruby and Ruby on Rails about five years ago. If you want to stick with what you have, then no worries. I just think people should be excited that over the last couple of years, the "golden hammer" approach to storing data has finally been overtaken and developers have a choice about what technology best solves their data problem.

So, in the inverse, if it were not a YC-associated company, you would trust that company more or less? Regardless of your opinion of Flickr, I did not fear the future of my photos when Yahoo! acquired them. Last I checked, they were still there.

That aside, I do see your point, but to say that you will not recommend a really nice service to your friends and family on the sole basis that they are a YC company seems odd.

Admittedly, I think that something was lost in translation during the conversation. We are working with the writer to update that phrasing.

Hey there...I am one of the founders of MongoHQ. We get this question often. I think that the issue is less about getting up and going with a technology (generally, that is the easy part), but the reality that it is something that you still have to manage as you get increasingly busy. So:

1) How are you going to handle monitoring?

2) Do you feel comfortable making the right moves when you are faced with growth?

3) Do you feel comfortable optimizing your database as things change?

4) Do you have time to stay up on all the updates to MongoDB and how they can affect you, what improvements are right for you, etc.? The space is very busy.

As part of both of our shared and dedicated offerings, we offer this to our customers and can assist them as they grow and their needs change. Hosting the data is the easy part...mastering the domain is what takes effort. We do our best to help our users in this way.

One final note...we do offer consulting, installation and monitoring of in-house MongoDB setups. That way, if you did want to run MongoDB on your own platform, we could assist you. I'd love to talk to you more about it if you would like. My email is jason@mongohq.com.

A real estate agent informed me recently that east palo alto is bad news. Gangs, etc. Is this really true?

As a follow up, my initial response came out a bit more snarky than I intended. Sorry about that.

Hey there, Ezra! :)

Agreed...the open aspect of it and how it morphs through the community will be very interesting. Looking forward to it.