Hacker News Reader

Disclaimer, I work for RethinkDB a competitor of MongoDB.

I completely agree with you that NoSQL is not a pure-white solution but I disagree that it's only useful for specific use-cases. NoSQL has frequently been touted as this magic concept that makes databases scale well. It doesn't as many of the poorly scaling NoSQL databases on the market today show you. The most you can say is that it maybe lets you ignore a few particularly hard to scale operations. However there's also no reason that a NoSQL database can't scale well it's just that you have to actually design it in such a way that it does and Mongo failed at this.

Joins are a very useful feature of structured databases but there's no reason you can't have them in a NoSQL database. I speak from experience here, RethinkDB has joins (distributed joins no less). They're just as useful in a NoSQL world as they are in a SQL world.

I think the author's analogy "NoSQL : SQL :: Dynamic Typing : Static Typing" (to use SAT syntax) is a good one. I know some damn good programmers who prefer dynamically typed languages like LISP and I know some damn good ones who prefer statically typed languages like Haskell and I think programmers can be similarly productive with either a NoSQL or a SQL databases backing them up. Currently NoSQL has the very real downside that none of the NoSQL databases on the market have really nailed it (ours included but we're working on it) whereas there are some very well established and well tested SQL databases on the market so it's a perfectly rational choice to say you prefer the better tested SQL route. However I contend there's nothing inherently flawed about the NoSQL concept.

Isn't Google BigQuery a SQL on top of a schemaless database ?

Well, SQL and NoSQL get thrown around a lot but generally people mean 'standard relational' vs. 'newer, potentially schemaless' DB.

By a 'SQL/NoSQL' view, Cassandra has CQL which isn't ANSI standard but close enough to be a SQL.

It’s a distributed near–real‐time querying system for data at rest that supports subset of SQL.

Google has scalable SQL database called Spanner[1]. They also have a proper NoSQL database called internally Megastore[2] that provides distributed transactions based on entity‐groups on top of BigTable. Also known as High Replication Datastore and available via AppEngine[3].

[1] https://en.wikipedia.org/wiki/Spanner_%28database%29

[2] https://www.cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf‎

[3] And hopefully via Google Compute pending announcement in few days at I/O. That is if Google actually wants their Cloud to succeed.

All I can say is this: if the saying "Always plan to throw away your MVP" is true, then I can't see any other storage solution other than MongoDB (or a similar schema-less document storage DB) for MVPs. The speed of development and flexibility are simply worth it. Yes, it is hard to refactor a live product and move it from MongoDB to MySQL / Postgre but was done before and you only do that if you get traction, so its a good problem to have.

I would start with MongoDB, get a grip on what on earth the product is doing, finalize the schema on the fly based on A/B tests, customer feedback and analytics, only once the schema is finalized move it to a SQL database if needed.

If you are not using a good ORM + DB migration system, then MongoDB will make perfect sense when you are quickly iterating through ideas and trying to find a product / market match. You really have no clue how your data schema is going to look like in the end, why confine it at the start? the vast majority of startups sadly won't get even near the phase where they really reach scale related performance issues, so choosing MongoDB for prototyping your business absolutely makes sense to me.

This is really the killer feature of MongoDB its API embeds very nicely in host languages and is very intuitive to use. I work for a competitor called RethinkDB and we've spent a lot of effort trying to make something that can compete in terms of ease of use. The leap we're really trying to make is for people to not have to throw their database out with their MVP.

You make a VERY big "if" in your first sentence, one that (admittedly anecdotally) I've very rarely seen hold true in tech companies. Much more often, the MVP becomes the product, and all those shortcuts and poor design decisions come back to kill your productivity when it becomes necessary to refactor foundational tech/designs that have metastasized throughout the codebase.

I am curious if this is other folks' experience as well, or do you actually throw out the MVP and start all over at some point? If so, at what point do you make the break?

MVP is used to get funding; which extends your runway so you have time to make your product correctly.

In real life, you are totally true. The initial test that succeeds becomes/continues to be the real product. I agree with the parent poster that Mongo is awesome for testing, if not awesome for scale. I think the point here is that tech co-founders/leads need to make it clear that this is a debt that will need to be paid if things take off.

If NoSql's speed and flex gets you to MVP, and MVP gets you traction, then all the hell you may go through to rearchitect for SQL is probably worth it.

Premature scalability is as dumb as premature optimization and premature generality.

Couldn't agree more.

I've worked for 15 years with relational databases and for the recent 6 months with mongodb on my startup. For me, the agility of not having to define my data in advance, and "work" so much to explain the DBMS how my schema should behave is no less than a game changer in my ability to build the MVP quickly. I just can't imagine having to invest all the labor in having to model everything I did with a normalized database and ORM.

Really? Mongo is for me not a database. And I worked like you 15 years with RDBMS and a year with mongo. Mongo has no security. Especially the Mongo Cluster Solution makes it so easy to steal and manipulate datas without notice. Everyone who has access to the server can enter the database. Implemented "security" could be turned off with a restart.

And nowadays it is so easy in a datamodel in a RDBMS to make changes. It is high flexible too.

But another worst thing in Mongo is the performance at complex queries. It is a nice bin for thinks nobody needs or to put documents in but that's it with Mongo.

When I develop against MongoDB in python, ruby, or clojure, I consider it less a database and more a persistence layer for my dicts/hashes/hash-maps, with a handy-dandy query functionality built in.

Hell, in clojure I usually develop against a hash-map of hash-maps stored in-memory until the project gets far enough to re-implement the data layer (which I've hopefully abstracted well enough to not be a big deal).

Think of Mongo less like a schemaless postgres, and more like a persistent redis. Ease-of-use is its true advantage, and I think anyone who exceeds its capacity should be happy to have the problem.

I tend to treat Mongo similarly. At my job, we hate Mongo and I've stood up and cursed it more than once.

Plenty of personal projects end up running on Mongo, because it has a rich query layer for quick hacking. I also never expect these to exceed the amount of RAM on my machine either.

I don't understand how this argument keeps coming up. If you use rails, schema changes and migrations are DEAD EASY and I am not even a professional developer. What time exactly are you saving ??

Schema changes are not dead easy when you have hundreds of millions of rows and/or hundreds of GBs of data.

Please don't conflate MongoDB and NoSQL. NoSQL is about the right tool for the job. For instance many of our Couchbase customers are very technical and risk averse. They choose Couchbase because other solutions (often MySQL or Oracle) have become the wrong tool for the job, as requirements change.

Mongo is a little different bc they actively position themselves as a MySQL replacement for your average Rails app.

"NoSQL is about the right tool for the job."

I never liked that rationale:

* The job almost certainly will change. Data lives a long time, usually much longer than the original application.

* It's likely to conflate marketing claims with actual fitness for a purpose. A special-purpose system with special-purpose marketing may sound great if the special case lines up with what you're doing. But that doesn't mean that it's actually a better fit for that special purpose. And, thanks to confirmation bias, people almost always think their situation does sound like the special purpose for which the system was built.

* A database system is a system, not a tool. A tool is used in a particular instance for a short period of time, the results are obvious, and it's easy to switch tools if one is not working out.

Of course, there are always valid reasons to choose the special-purpose system, and I'm not saying you shouldn't have used Couchbase. I just don't like the "right tool for the job" argument.

I've heard DBA complain that many programmers see data as not all that important: that the data simply exists to run the program. This obviously not the case in any serious real-life environments, but I think it's the sort of attitude that many NoSQL supporters have.

One thing that become much easier with the JSON document model is sync, which is a valuable abstraction over the network layer. http://blog.couchbase.com/why-mobile-sync

Another thing that you win with the document model is lots of primary key lookups, making it easy to model your data access around your hot path. http://blog.couchbase.com/performance-oriented-architecture

   I just don't like the "right tool for the job" argument.

Then say the "right technology for the task" then.

There is no ONE database that works perfectly in every situation.

When installing a database system, it is rarely a "task". It's usually something that will live a long time and there will be many long-term consequences.

Back in the day, many startups died with a gigantic Oracle database that cost them months of runway, because they had to "think long-term".

Build for what you need now, and just make sure your "central repository of truth" isn't lossy. You can always replicate for specific use cases later.

I'm not saying that you shouldn't make short-term compromises. I'm saying that "right tool for the job" is sloppy reasoning.

Even in the short term, it might be much faster to develop against a good RDBMS than a nosql system. I think many developers are so focused on "abstracting away the data store" that they forget what a good RDBMS can do for them.

I think the big problem is that most "NoSQL" engines are naive in their approach, thinking that the people at Oracle, Sybase, IBM, etc. are stupid.

Very few people need scalability beyond what a regular RDBMS can do. So yes, RDBMS have limits, but you'll most likely never reach them.

No , there is something specific to MongoDB. Some NoSQL dbs are fine for some purposes ( multi-master sync for couchdb ,graphs for Neo4j ... )

MongoDB problem is that it promises too much and fail to deliver on a lot of levels. You cant expect to do all the operations one can do on a RDBMS on a NoSQL database and still scale horizontally,while maintaining data integrity. A good NoSQL db should have little features , but features that work very well. I like CouchDB for instance i use it in my apps to store customer infos that need to be synced accross multiple device, but i would NOT build an ecommerce app with it only...

NoSQL dbs solves specific problems. It does not solve the problems relational databases are for.

So selling MongoDB as a replacement for PostGreSQL or Mysql is a blatant lie.It will not work.

You are pretty violent against MongoDB. Any reason why?

I suspect the reason that Mongo attracts so much hate is that it's more heavily marketed/evangelised than other NoSQL DBs - so more in the public eye and people expect more from it. Since the reality is that Mongo is not yet a technically strong product (regardless of its merits in terms of ease of use and getting simple things done quickly), it's bound to attract more negative attention.

Bingo. My brief experience with Mongo a few years ago was in a situation where it got selected in order to use something "new and cool". My job was to produce some reports from the data. Guess what reports require? Joins. The whole mess was slow and sucked, and the client belatedly realized that MongoDB was a shitty choice for what they needed, which could have been handled by Postgres without breaking a sweat.

Having spent the last two years trying to make MongoDB work at large scale and only succeeding because I have a ton of resources (200+ db hosts), I can say it's more than just the spotlight.

It is a fundamentally broken product.

Fair enough - I have no direct personal experience with Mongo myself. Coming from a more traditional DB background, what I've read indicates it has a lot of technological flaws, but I didn't want to be too harsh in my judgement without personal experience.

For my own curiosity, what issues caused you the most trouble? The lack of transactions combined with the extremely coarse grained write locking were what put me off the most.

Read performance compared to a lot of other offerings is a significant issue.

However, one of the biggest issue is the entire sharding design that is incredibly delicate and wouldn't pass even the most basic high availability requirements. I could go into a lot of detail about why it's bad but it would take too long.

After that, it's the hardcoded limitations that prevent true multi-datacentre sharding, the previously mentioned CPU issues with mongos, broken replica selection, inability to control primary/replica setting manually, etc.

The list is virtually endless.

I would have to agree. Stackoverflow is in the alexa top 100, with over 400 million page views per month, and it runs off a single MS SQL Server instance (with a hot replica), with a Redis cache in front of it.

All the high-scale users of nosql systems end up having to learn a lot of arcane technical wizardry to make it scale (as demonstrated by the slidedeck linked here), and I'm not convinced it's actually easier to scale something like mongo than a traditional RDBMS.

I used to spend a lot of time in the RDBMS space, and I'd hear the DBAs bitch about how NoSQL was just a bullshit excuse for not having any discipline around your data model, and I'd defend it, saying, "Well, you have to understand, certain use cases, etc. etc. etc."

And then I see an presentation like this and think, "Yes, of course, you should have put that stuff in a relational database, what were you thinking".

Of course, most of the open source databases do sharding and replication and the like equally terribly, so it's not a perfect solution....

I tend to take the view of RDBMS works until it doesn't. Most cases that things like click data or event data that ends up taking billions of rows, in which case you aren't going to want it in a RDBMS anyways (you'd like to, but hah).

I'd like LevelDB on MySQL personally.

"till it reached some kind of moderate scale where I realized it was a terrible choice going with a NoSQL db (Sometimes, I'd have to duplicate data because there were no Joins, etc)"

That statement doesn't make sense to me. One of the first things you do with a sql database when you reach moderate scale is denormalize data so you don't have to do joins, or shard which prevents a lot of joins. Joins are super convenient, but they don't scale.

You denormalize SOME joins.

On nearly all of the cases, they are very usefull, but once in a while one of them gets too slow, so you make a bit more of work enforcing the consistence by hand, accepts a bit more of risk, and denormalize it.

Throwing all of them away because a few may create problems later is a bit of an over reaction.

How many joins do you think this page uses? A denormalized version that doesn't scale probably uses 25-50. I'll bet the scalable implementation uses close to zero.

I would argue that its actually a multi-tiered question where the choice isn't really "NoSQL or RDBMS" but "NoSQL or ORM talking to RDBMS or going straight to RDBMS"

Here, the naive developer uses MongoDB as an all-in-one that rolls in the part of ORM that too often gets sole consideration, 'How do I persist my objects?', and effectively ignoring everything else. Try as I might to eloquently phrase my frustration with this all, it usually ends up in a rant against those relying on automagic who don't feel the need to understand every decision being made for you.

If you use java stack like JSP or RingoJS, I would just use H2.

http://www.h2database.com/html/main.html

It bets MySQL and Postgresql in performance.

I have to completely agree... in many cases a classic SQL RDBMS is probably best. In some (such as single records having very dynamically shaped data, where most instances can be brought forward based on a key alone, and searches are on a few keys), NoSQL is better. In some, having data replicated from one to the other works well, one for primary records, one for search/display. I've even contemplated scenarios where ElasticSearch could be a source record. It really just depends on what you want/need from your data and searching.

What does "pure-white solution" mean?

In this case, I believe it means all benefit and no disadvantage.