The genius and folly of MongoDB
nyeggen.com
nyeggen.com
They fixed that problem but it was too late. In my eyes they proved they are not to be trusted with data.
Had they called themselves MangoCache or MongoProbabilisticStorage, fine, can silently drop writes, I don't care it is not database. But telling people they are a "database" and then tweaking their default to look good in stupid little benchmarks, and telling people they are webscale, sealed the deal for me. Never looking at that product again.
These are systems designed for the real world, where people don't read the manual until they have to.
When people assume MongoDB was similarly designed with their best interests in mind, that's when things go wrong.
Any time I deploy something as critical as a database, I carefully read about what it does and how it works. Not doing so is like signing a contract without reading it.
I don't understand this reasoning. We are talking about defaults. Defaults are used by people who did not tweak the settings yet. If I am just starting building a thing, I will have bugs and squeaks and I want to make sure I am not fooled by some unreliable data store. I am not likely to need 100GiB/s throughput, but I am very likely to have to hunt bugs, like "I did click on this <like> button but it did not add to the total likes". And I would really really hate it if after half a day of bug hunting I would realize that my data store just didn't store the thing...
No, I just assume that a database has a similar set of features as other databases have had for decades. Mongo does not; it is clearly the exception - and for possibly nefarious reasons, as well.
You buy a car. It comes with brakes disabled because for whatever reasons that also lets it get to a higher top speed. You are expected to read you car owner manual and on page 54 you find that you have to hold "enable brakes" button under the console for 10 seconds to turn on your brakes. Would it vex you that people might be slightly critical of that car. Clearly they are silly for not reading their car manual until page 54.
That "feature" is not something that should be discovered by reading docs or when you get a crash and then load a backup from another week and still get a crash and then you start hitting your head on your desk.
Anything calling itself a "database" should not have shipped with those default settings _ever_. If they did they might have gotten away with it in my book by having a big flashing red warning on the front or download page. I don't remember one.
Don't get me wrong, I'm not saying your assumption is unreasonable. But in the end, it's on you as a conscientious developer to read the documentation. I'm not even suggesting cover to cover - in this case though they are very up front about write concerns. There is no real excuse to find this out any other way, it's just negligence.
It's a nice hypothetical, and that might be your style. Most real-world car purchase scenarios I'm familiar with would make that style impractical.
So, let me get this straight: You laid down tens of thousands of dollars on a vehicle that you only post-purchase read the manual of, and you're raising this as some sort of standard people should follow?
Honestly, asking the right questions (and test-driving) upfront should be what lands the purchase, and not discovering the folly of purchasing a car with such ass-backwards issues you only discover after the fact when you bother to dig out the manual.
You drove it off the lot after you bought it, right? Or did you read the manual in the lot right after signing the papers locking you into the purchase?
Honestly, how can people on HN actually be this against reading? Especially things that are really important? Sure, don't read the contest rules for your McDonald's monopoly. But if the data for your livelihood depends on something, there's no excuse for not reading the documentation.
It's impractical because people live a finite amount of time and this is a terrible use of it
You don't need to read every word of everything, but some things are worth it. Do you sign contracts without reading them too since it's a "terrible use of" your time?
Yes - I am saying in this particular car example, the benefit derived reading the entire manual before purchasing/driving a car is not worth the cost unless your time is worth very little. As others have pointed out, no one is flipping through the manual to check whether the brake pedal actually applies the brakes
But it is a good engineering decision to thoroughly read the docs before jumping into a new datastore like Mongo, I agree. Learning there are things like gigantic global locks and unsafe writes are normally enough to make you say, "hey, I probably shouldn't use this to store production data I actually care about"
A database that has default configured that ends up corrupting users' data silently is like buying a car with the brakes disabled.
Well except that in the car case may brakes disabled won't make the car go faster, but in case of MongoDB I remember fans strutting write benchmarks around comparing it to Postgres, Couch and other database and telling how it is webscale. The reason that design decision was made is shady. That was my initial point.
I did experience one once though: I discovered that a vehicle had traction control when the system activated during a skid. The computer and I disagreed about the best way to respond, and the surprise did make the situation more dangerous than it could have been.
In both vehicles and databases, the situations in which the product might do something unexpected and dangerous should be clearly documented in their own section of the manual. Databases should say "here are the things that could lead to data corruption or loss". Vehicles should say "here are the situations where the vehicle might disregard or override the driver's control inputs".
I suspect some people will believe the results wouldn't have been what I expected without the traction control. I can't prove they would have been, but I did grow up and learn to drive in Alaska. Based on my experience, I think I would have done better than the computer did.
Here's somebody describing the basic idea involved while discussing the joys of mildly irresponsible driving on cloverleafs: http://www.scottgood.com/jsg/blog.nsf/d6plinks/SGOD-66RJ2Y
Heck, even when I have a rental car for a single day, I will read the manual. Maybe not every page, but I will skim it for gotchas (and if I have time, the whole thing).
Maybe it's just the engineer in me, but it's what I do.
Come on guys, as computer engineers/programmers/developers/whatever, surely professional pride at least would mean we at least read the README and/or the manual, before putting something into production?
it's more like launching a shuttle mission
I use MongoDB now in production and I am happy about it. Not huge dataset by any means so MongoDB fits the bill perfectly. It has a few idiosyncrasies (doesn't release disk space after deleting records - what?!?) and you definitely want to read the manual on settings. But it is incredibly easy to use (documents instead of relational data) and allows me to focus on app development instead of my storage backend.
HBase needs hflush to make sure that the WAL edits are resident at at least 3 (default) HDFS data node machines.
Not sure how exactly grand parent lost data. Each edit is first written to the WAL then committed to the in memory store. The in memory store is flushed to disk into a new file at a certain size. If a server crashes and had unflushed data in the memory store that part of the data is replayed from the WAL on another server.
See also here: http://hadoop-hbase.blogspot.com/2012/05/hbase-hdfs-and-dura...
I don't have much sympathy for people who can't RTFM but storing data is kind of a thing for databases.
if you're not trying for that standard at all, it's false advertising.
Today, I would be inclined to use PostgreSQL with JSON support, and some triggers to update an aggregate search table, or look more seriously towards RethinkDB.
With any NoSQL system you give up something.. you just need to be aware of what you are giving up, why and for what gains.
It's kind of funny, albeit interesting that it took this long for much of the industry to start knocking down the house of cards that is built around the database. However many of the databases that are coming out these days have a lot of the same cultural issue of "hide the issues" and "talk about the cool parts".
I'm expecting any minute now for the anti-schema-less pro-schema movement to rise up...
IMHO, it's all about what you're doing at the time, and making the right decision... albeit it helps if the decisions for a database isn't glossing over the issues as insignificant. :-/
btw - "MongoProbabilisticStorage" is a great name for a product!
Most people don't like mongo because 10gen gives the impression that mongo is better than it actually is, many people feel that mongo is not reliable enough for at-scale applications. They're right; it's not. But that's ok, because:
Mongo's really great for rapid prototyping. You don't need to worry about updating the schema at the db level, it can store any type of document in any collection without complaining, it's really easy to install and configure, the query language is simple and only takes a couple of minutes to learn, it's pretty fast in most use cases, it's pretty safe in most use cases, and it's easy to create a replica set once your prototype gets usage and starts scaling.
Mongo does everything well up until you reach the level where you need heavy-hitting, at-scale, mission-critical performance and reliability. Most projects out there (99 in 100?) will never reach the level of scale that requires better tools than mongo. And since the rest of it is so easy to use, that makes mongo a great starting point for most projects. You can always switch databases later, but mongo gives you the flexibility to concentrate on more important things in the early stages of a project.
It's not a database that by default just throws your data at the wall and well, doesn't even care to look if it stuck.
The last time you hacked together a blogging engine in Node.js one weekend, were you worried about future scalability, or just playing with new technologies because it's fun?
And while it's pretty easy to do schema migrations, it's not easier than _not_ doing them. And what is it you really want to worry about? Making sure your DB is production ready, or tinkering with Express.js and Backbone?
So because of that, many people use mongo as their de facto database. It's just what I use when I need a persistence layer for anything I build, because I already have a mongo db running for like 30 different defunct projects on my dev server.
And then, by happy accident, one of your side projects turns into a real product, and then mongo handles you really well for the first year or so, just up til the point of having to hire a real devops engineer; at which point you swap out your ORM layer and switch to postgres.
Just so you know, that's not actually a thing.
Regardless of whether you are dealing with a strict schema or flexible schema, you still have to make changes to how you structure your data as you are prototyping or otherwise iterating on it. MongoDB provides no tangible benefit in this case. If you want to rename a field, then you still need to run an update.
How are ad hoc, manual, historically opaque tweaks to data in any way better than an easily generated and version controlled series of scripts representing a replayable history of changes to the data?
If anything, manual untracked tweaks make "rapid prototyping" more difficult since lots of partially or completely undocumented changes to the structure of the data are harder to revert, replay, reason about, or share with others. It's also more work to do it manually since you need to run the commands in multiple environments, rather than just entering the same command or, more frequently, a shortcut command into a generated file.
I just don't buy this argument, writing and executing migrations is braindead simple and usually takes what, 20 seconds start to finish? Writing the line of code you need for mongo must be about 5 seconds.
edit: I did actually give mongodb a good crack(used on a side-project for 6 months last year) but I found that I actually spent a huge proportion of my time working around things that were missing compared to ActiveRecord. It was a huge net loss for me in terms of productivity.
Even then you can sometimes get away with staying on sqlite for your admin side CRUD and redis for the heavy / concurrent writing from the public facing side (obviously situational).
Many people in the Java world use something very simple like hsqldb, then shift to a new database when out of development.
For a hack-weekend sort of project, fine, but if you are in any way attempting to make a product, it strikes me as the sort of thing that would be really difficult to change later down the line, and so worth investing the very little extra effort it takes to include your schema in the database, and use something like Postgres/MySQL/etc.
I think the up-front benefits of using mongo (especially as a sole developer/devops/sysadmin person) outweigh the difficulty of the changes you'll need to make later on, which will only happen as you hit scale and have more resources to nurture the devops side of the tech stack.
I can't think of any codebases I've seen where intentionally choosing the storage backend you know you don't want to use (if the project is successful) would be a reasonable thing to do. Understate it if you must, but having to change your backend from mongo to postgres is not a desirable situation. Besides, if you're going to use postgres to scale, use it's features and write well optimized queries for it. The difference between a bad massively complex query and a well optimized one can be several orders of magnitude, and that optimization can indeed be difficult. It goes without saying that you wouldn't leave that to an ORM.
That said, I would use PostGres for my MVPs, using it as a key-value store initially until its more clear what the schema should be. That is, if I still bothered using code for prototypes; of late I've been more fond of napkins and Adobe Fireworks.
Edit: "tried and failed" in the political sense. You don't change horses in midstream etc.
PostgreSQL has HSTORE which is a useful but proprietary feature. Cassandra has the ability to have Lists/Maps as data types. Again useful but proprietary.
If you are that concerned about database independence then do what everyone else does. Use an ORM, minimise coupling in your domain model and do as much as possible in the application layer.
I also used it to write a service that had to go from nothing to working in a couple of days. I then spent the next two days swapping it back out again. It was surprisingly painless to go from an object store to a relational model
Most developers who use Windows will just go with something which doesn't require a virtual machine. For example, MongoDB, CouchDB, OrientDB, Cassandra, and ArrangoDB work fine everywhere.
Cassandra et. al. are completely different, in that you don't use them because they are more fun to use. You use them despite their awkward, low-level interfaces because you're going to dump billions of data cells into your database from day one with no end in sight and want all the easy scaling/availability features provided.
> You'll thank us later when you learn a little bit of SQL and start analyzing your data, running circles around the no-sql guys.
That's a little condescending... do you know a single mongo user who doesn't have experience with SQL? Plus, I love the fact that I can literally run javascript against my database. Good for production? Certainly not. But that doesn't mean it's not fun or useful.
Not every project requires such rigor. If that's how you enjoy development, that's great! Very few of my projects put the db layer to the test, and so I'm happy with the balance that mongo gives me. I use it in about 4/5 of my experiments and side projects.
Which is why it doesn't make any sense to claim that using MongoDB somehow eliminates needing to migrate your data as it evolves.
Sometimes you have to do update of records with a certain version number.
My opinions, for the record: MongoDB is a tool with some use cases. I'm more of an SQL+Memcache guy, if possible, but not religiously if a good argument is presented (that don't sound like "let's use .*, I want another keyword on my cv").
And since this is supposed to aid in rapid prototyping, how does it do so? It seems to me that it does just the opposite by introducing a significant and totally unnecessary burden.
(An update routine can be run at any point with low use like Xmas, etc. This is potentially neat, depending on use statistics.)
I'm not saying this is a common thing, but the lack of joins makes the data a bit more flexible -- this can't be too much, if nothing else because then the Javascript will begin to break.
(I do think there are much more use cases for nosql than as a Memcached with more features. Where an old job used MongoDB wasn't one.)
Has any phrase ever struck more fear into the heart of a programmer? Rapid prototypes have a nasty tendency of accidentally becoming products...
> You can always switch databases later
shudder
I wonder if I'm going to be the one who finally switches Google Search over to Go, by way of a quick throwaway prototype...
Reminds me of this: http://www.youtube.com/watch?v=URJeuxI7kHo
Want NoSQL? Use RIAK!
See, you're just perpetuating The NoSQL Problem. :) Riak is well-suited to some tasks, but it is no more a magical fits-every-problem thing than MongoDB is.
- Mostly easy to work with. Mapreduces can be a big pain to troubleshoot because you can't console.log() in your JS. Didn't try it in erlang.
- Being masterless, it has a very good replication story for servers _in the same data center_. It really bit us that there was no good riak solution for syncing data across multiple data centers. There is an enterprise solution for that, but it's quite expensive, which makes riak less appealing if you don't have much budget on your project.
- Errors in general are next to useless. Get comfortable waiting for answers in IRC when you get opaque error messages after running queries. You can definitely work past this, but it wasted a lot of my time.
- Not sure if pro or con, but as the cluster reached load capacity, from a combination of data size and read requests, map functions would begin to slowly fail. After a while, we could tell which completely useless error message (preflist_exhausted, my old friend) could be fixed by a cluster restart, and which would simply begin to happen with greater frequency as more data was added. This was exacerbated by my company refusing to pay for anything more than a three node cluster. You might say I should have fought harder for more, but I had to fight to make them not host all three nodes on a single server. There are places that will hire you that simply do not intend to do anything sane, but I digress. The takeaway: riak is not a super cheap way to scale.
- Bulk inserts? What are bulk inserts?
- Key filtering is just a shim over listing all keys in a bucket. Further, listing all keys in a bucket, or all buckets in a cluster, can be very expensive, and basically you'd never do it unless you had a very small bucket. The bag of tricks you can apply to speed up slow queries is basically "Do you have secondary indexes? Ok, good."
Those points do read a little negative, but actually I would use riak again. To me it works best as a temporary event store living in one data center. If you've got a bunch of items shuffling around your backend in real time, being processed to and fro, you could definitely do worse than sticking in it riak and adding more nodes as needed.
RDBMS performance was fine most of the time as we're not doing big data really. Our problem was developing and maintaining a schema that holds lots of metadata many levels deep. Our app allows for unlimited user defined forms and fields, some of which may hold grids inside which hold some more fields... Our app also handles lots of logs and large file dumps, which slowly made data, cache and fulltext search management mission impossible. Even though we had considerable previous experience with Mongo, it took us a long time to switch because we were utterly scared. It's nice to sell a product that is Oracle-based, as that sent out a message about our "high-level of industry standardization and corporate commitment" bullshit that (we thought) is quite positive for a startup competing against the likes of IBM, HP, etc.
To our surprise, our customers (some Fortune 500 and the like) were VERY receptive to switch to a NoSQL, opensource database. Surprise specially given it would be supported by us instead of their dreadfully expensive and mostly useless DBA departments. It even came to a point where it has changed their perception of our product and our company as next generation, and surprisingly set us apart from our competition even further.
In short, as many people here know, not all MongoDB users are cool kids in startups that need to fend off HN front page peak traffic day in day out. Having a schemaless, easy to manage database is a step forward for sooo many use cases, from little intranet apps to log storage to some crazy homebrew queue-like thing. 10-gen superb, although criticized, "marketing effort" also helps a lot when you need to convince a customer's upper-management this is something they should trust and even invest on. I can't express my gratitude and appreciation for 10-gen's simultaneous interest in community building, flirting with corporate wigs and getting the word out to developers for every other language. Mongo is definitely a flawed product, but why should I care about the clownshoeness of its mmapped files when it has given us so much for so long?
The vast majority of apps just don't deal with that problem. If MongoDB was really only used by people that its a good fit for (like yourself), it'd really be a niche product. They're marketing it as a general purpose product, which is why they've earned scorn from so many.
We are evaluating Mongo to be the persistence for single page web applications, which is how we'd like to start making the majority of our intranet/private enterprise jobs. Are you using it in this context and has it been helpful? Your example of nested fields being easier made me warm and fuzzy- we had a project last year that had growing, fluid, user-defined data structures. We made it work (well) with Postgres but there were several kludges that really bothered me. One of them was handling delete dependencies gracefully on user-defined, nested structures. Did you encounter this issue pre-Mongo as well and if so, did it help with it?
Can you explain why can't you do schemaless with an RDBMS?
From what I understand MongoDB is schemaless by storing all fields as one single JSON document. So what stops you from doing the same in an RDBMS - have a catch-all field "JSON" and store all your data there?
PostgreSQL has also recently added a key-value store type [2] with semantics reminiscent of Redis. The impression I get is that they're gunning for the NoSQL kids in general, and this pleases me; while I grant it is sometimes possible and necessary to obtain new insight in a field by ignoring all that's gone before, I very much doubt this is one of those times, and I am therefore delighted to see a properly engineered database engine gain more or less the entirety of the features which draw interest to the NoSQL crowd in the first place.
[1] http://www.postgresql.org/docs/9.3/static/functions-json.htm... [2] http://www.postgresql.org/docs/9.3/static/hstore.html
This leaves only performance. I think I'm still confused around this area - why do people say that non-relational technologies like MongoDB are faster than relational databases?
Every single MongoDB step has had the old timers groaning.
Even with something solid like Tokutek's storage engine in it, its going to be a hard sell.
I'm confused by your comment. The beginning acknowledges the fact that MongoDB has a weak storage engine, but your conclusion is that, even with a strong storage engine like ours, there is still a problem. What other problems do you see? Are they something we could work on?
The problem is that Mongo advertised itself as a database and wasn't one. Once you do that reputation of the product is dead forever.
TokuMX is a real database as far as I can see, MVCC, great indexing story etc.
By association TokuMX is probably not regarded as highly as it should be. Which is a shame but it's a people problem, not a technical one. People can very easily lose trust in a technology at which point it's effectively dead, it might take a long time to die due to lock-in but it's dead.
For instance I have recently started playing with RethinkDB over TokuMX almost purely because of Mongo association.
Now technically that might not sound like good reasoning but when you think about the kind of person that writes a database that doesn't fsync your writes by default and relies on the page-cache over doing direct I/O when building a database.. doesn't really inspire confidence in the network stack, the query planner.. or well anything.
If anything it makes you insistent on not having ANYTHING to do with that sort of codebase.
Just replacing the storage engine might actually be good enough, but restoring my trust in the rest of the codebase is almost a forgone conclusion at this point.
They can try and rewrite the web and remove all the silly benchmarks, but they were the loudest "web scale" cowboys back in the beginning and we remember them for it.
I disagree with them on what is the minimum viable product for a database. I come a storage and service provider background where failures are treated very harshly (usually death of companies for singular mistakes) so I take releasing a product that stores customer data very seriously.
To be honest this is the biggest attraction for me to RethinkDB. They waited a sufficiently long amount of time with a commercially backed team of very competent engineers that obviously have the required background to sit down and DESIGN a database. The query language generates a non-turing complete language with a clean AST the has all the right deterministic characteristics to implement a powerful planner/optimizer. Their on disk format has been abit in flux but the core design is excellent and you can see that it has been optimized for very fast range queries. Even the API protocol and serialization were designed with care, not to mention the excellent ReQL language and attention to detail when integrating drivers into the host language.
Which is the other thing I tend to dislike about Mongo, it reeks of lack of design. The journalling effort for instance as you pointed out is very adhoc, this goes for GridFS and alot of the other features they have integrated into the codebase. These are smells that I can't ignore when looking at a product that I need to trust with my data.
The counter argument is to not trust it with your data. But I am yet to find a reason where that makes sense where another datastore wouldn't be a better choice.
None of how MongoDB works is a secret. And just like everything else it has sweet spots and problem areas. And like many others, development continues and it gets better.
The database does not get the job done - it is a tool to help get the job done.
Maybe not now, but this hasn't always been the case. The fact that they had (have?) a global write lock was completely buried on the doc site for ages. Benchmarks were waved in front of developer's faces to distract them from the "drivers don't actually write data, they just blast it out in every direction and hope it lands somewhere good" BS.
I don't use Mongo anymore, and I think a lot of it has not to do with the database itself, but with the way 10gen used their marketing machine in a dishonest way. They incurred a lot of trust-debt, and now have a serious amount of work to do to pay it back.
"We value feature-set and expressiveness much more than scalability at our data size, but we want to feel like we're big data too so say some of that stuff"
And that's their brilliance, they listened to what people said they wanted and then gave them what they really wanted.
The fact remains that Mongo just doesn't scale, and 10gen was never honest about that.
I'd say that this is less of a problem for the Dynamo paper databases (Riak, Voldemort, Cassandra), because really to use them at all you have to have some idea of what's going on.
I'm a bit biased here, though, while I've found Voldemort, Redis and Cassandra useful and can see Riak and a couple of others being useful, I could never really figure out a good reason that anyone would use MongoDB besides naiveté. That said, I've never really tried, as I don't have a problem that fits it (part of my issue is that I don't know what a problem that fits it would look like).
Never once while I was there did they publish a benchmark: There was a [publicly] stated company policy to not publish or comment on benchmarks.
If you have evidence otherwise (i.e. benchmarks published by the folks working on MongoDB) fine, but I take this as a deliberately inflammatory (and false) statement.
EDIT: The global write lock was removed ~last August; there is now a database level lock. Future releases will likely make that more fine grained. Additionally, the drivers no longer do "unsafe" writes, but check w/ server.. as of the same release.
But that's not anywhere close to good enough.
FTFY
FTFY
Mongo has repeatedly chosen defaults for their database which make naive benchmarks look better, at the expense of production safety. You seem to be willing to attribute that to Mongo's incompetence. Proverbs are on your side, but it sure ties in nicely with the "leave the benchmarks to the community" policy.
In this case, this article does nothing but elucidate the truth, quite plainly and clearly, in fact. That's important, in and of itself.
How you use it is up to you, but we must not vilify truth.
The new database level lock in 2.2 is also annoying (and arbitrary) but it is better than the global lock.
To your point, I think deployments on modern architectures generally want the ability to scale out and tolerate network partitions, which makes the ability to drop and re-elect masters, reconcile a node that rejoins after a partition, manage shards, avoid hosing remaining nodes, etc. critical. Inability to do so really hurts on a platform like AWS (or in any multi-datacenter deployment, really).
Also, MongoDB does let you elect a primary via setting a priority. Really, it should be a requirement because sometimes mongo nodes will switch due to a dropped packet (or this is all I can assume at least) and the arbiter just randomly picks a node when there aren't priorities.
Most modern architectures make the choice between having nodes serve as master for a subset of the data, and the increased cross-link bandwidth needs and reduced flexibility of a master-less system.
I've used Mongodb for the last 18 months, never lost any data, and it made it obvious that "enough durability" is sometimes... enough. When I switched to MongoDB from MySQL, performance rose 10+ times and I switched from "don't hit the db too much" to "give it more work, 'cause it's idle".
I know it's an ideology problem: otherwise, I can't see why people would complain about a tool they don't use.
* you have to learn to do indexing right later (if you have to scale)
* failure and miss starting to occur (as you scale)
* more code to write to manage legacy schema and optional fields
The last is painful and ugly. Whereas if you start out with a good schema that last point is in a good hand. When you use SQL you always have the restraint that "xyz" attributes are repeating and you can just make a new relation, whereas with mongo you'd stuff 20 fields into a single collection. The refactoring is harder.
I will begin to migrate back to SQL for new projects.
Also ecosystem is richer in SQL. I have not seen a good ORM for Mongo. MongoEngine is fine but implementation + db have a lot of issues make that ORM a bit unusable from time to time. SQLAlchemy is good.
PS: For quick PoC and Hackathon projects sure prototyping with mongo is fine.
Uh, Mongoid is good. I have used it on past projects. It's nice. (mainly use some form of SQL now)
If you use MongoDB in production, you should definitely take he time to learn about the durability options on the database side AND in your driver. By using them appropriately, you can have as little or as much as you like. Data sets larger than 100GB are no problem either -- right now I'm running an instance with a 1.6TB database.
As always, use the right tool for the right job. If you need joins/etc. and don't need unstructured data, Mongo probably isn't a great choice (even with the aggregation framework).
If you hit a db level lock limit, you're probably running a sub-optimal or unindexed query.
I'm not sure what MongoDB returns (or how its clients react) when there are no available connections because of a lock whose duration exceeds the configured timeout. I'm pretty confident, though, that this sort of thing is covered by basic driver config.
I'm really not trying to be argumentative here, I'm just trying to understand what mongodb is for.
Doesn't really matter for the point I'm making. It's a solution for a given set of constraints. Not the solution, or the very best tippy-top solution in all the kingdom, just a solution.
Point being I can't think of a use case where this is true, but if you read the article, the author does include what he says is the only reasonable use case for using MongoDB.
They've fixed it like I said but that whole "we're just using it to validate an idea" thing is a total con. "Nothing so permanent like a temporary [solution]."
For postgres you'd be mapping to a relational schema, and for redis you'd be storing the json yourself as a blob, without any server-side manipulation capabilities (or using redis maps/sets/etc, which are awesome, but aren't as general as json).
I haven't been doing very much web dev the last few years though so it's possible that my first impressions are wrong. I'm just repeating what I've been told, basically.
There has been a considerable amount of work put into postgres over the past few years for getting it to handle your data regardless of what it looks like. The developers seem to have a very good grasp on the fact that not all data is alike, and giving tools that will work well, and together with, all your data leads to a lot fewer headaches in the long run.
Mongo does everything well up until you reach the level where you need heavy-hitting, at-scale, mission-critical performance and reliability. Most projects out there (99 in 100?) will never reach the level of scale that requires better tools than mongo. And since the rest of it is so easy to use, it makes mongo a great starting point. You can always switch databases later, but mongo gives you the flexibility to concentrate on more important things in the early stages of a project.
What's your magic non-db level, supposedly-easier-than-updating-a-schema approach to renaming a field common to all existing documents in a collection, eg, rename an "author" field to "writer"?
PostgreSQL:
ALTER TABLE posts RENAME COLUMN author TO writer;
db.posts.update({}, {$rename:{"author":"writer"}}, false, true);
r.table('posts').replace(function(item) { return item.without('author').merge({writer: item('name')}); })I also use it as a metadata "scratch space" for highly available applications (things where failures are not acceptable and must run for days at a time). Again, with replication and automatic fail overs, I've been able to maintain 100% uptime outside of maintenance windows. Obviously that can't last, but so far it's been >2 years with no major problems.
EDIT: I should point out that although the size of the metadata objects can be highly variable, since I usually had a small number of them relative to the time series, fragmentation was still not an issue.
You want a fixed-size, rolling backlog of time series data such as logs.
MongoDB by contrast will simply mmap that block of file, overwrite the contents, and fsync. Yes, this has obvious downsides.
However, i'm not sure why this should be the case. You mention the complexity of updating a row in MVCC; sure, but all the database has to do before reporting success to the user is to write its intent to make this change to the transaction log (WAL in PostgreSQL, redo log in Oracle). The actual changes to the data files can be written back later on. The transaction log is a single stream being continuously written to disk, so that should be very fast.
MongoDB, on the other hand, is making scattered writes across its mmapped data files, which should be much slower. Except that of course it's probably doing this on a journalled filesystem, which is using exactly the same mechanism as the RDBMSs to provide fast, safe updates.
I'd be really interested to see how a simple update to a single field translates into actual writes to disk for PostgreSQL and MongoDB. If only i knew how to use strace!
Funny punchline at the end there too.
Unfortunately though, I believe Mongo is still beating it at performance, which is the one thing keeping me away.
I think in MongoDB's case, the getting-to-market part pushed a little too hard on the make-it-simply part. Simple is good but a thing should be as simple as possible, no less.
MongoDB IS by every measure a very simple and easy to use database.
I've done this before when I was doing work for a client using an existing simple web host with no built-in options for databases. It works well, and the nice part is that there's a simple, obvious way to do any query. The bad part is that anything other than a primary key lookup is slow unless you add a lot of complexity.
Nobody writes about the filesystem like they do the database, and yet they do the same job - store and retrieve data.
If you've only ever used Mongo, a filesystem, and/or mysql, then you've never really used a database. Postgres (and mssql, oracle, etc) are so much richer; they are so much more than storage systems. I'm not saying you no one should ever use Mongo or MySql, I'm just saying that they are generally far inferior choices for problems bigger than mere storage.
It's high time we started thinking about storage systems and the higher-level functionality of databases separately. We can make different, more informed, and generally better tradeoffs than we're currently making by viewing this broad category of software through such a foggy lens. For instance: Take a look at how Datomic utilize pluggable storage to provide a sensible information model, with raw index access and powerful, pluggable querying.
Different types of databases are useful for different things.
> Nobody writes about the filesystem like they do the database
You must have missed the last decade of people going on about ZFS.
I guess I did, what's the big deal about ZFS?
Recommendation: Stop following all the hype and what all the other blind sheep are doing.
Regarding filesystems... yes they do. Tons of information out there about filesystems you just have to look for it. Read up about ZFS, ReFS and that should get you started.
I think more often its easy to poke fun at _how_ its used.
When any tool or tech is used globally, before knowing its limitations, problems are likely. Attempting to use MongoDB in all storage or persistence scenarios is no more sensible than using MySQL in all cases.
Yes, there is marketing around this product that must be looked at critically - after taking into account that many newly developed technologies won't solve all the problems older tech have worked for decades to solve.
As well as things like that which are actually ridiculous, there is also the substantial gap in features as compared to real databases. Things like recursive queries, user-defined types, partial indices, etc, are commonplace in the more sophisticated databases. You probably won't need them for a simple web application (or even a complex one!), but they can be very useful when trying to do more complex things, or manage a complex system efficiently.
I'm not a big MySQL fan at all, but it's still leaps and bounds ahead of mongo technologically.
http://ronaldbradford.com/blog/understanding-innodb-mvcc-200...
http://dev.mysql.com/doc/refman/5.7/en/innodb-transaction-mo...
> InnoDB does locking on the row level and runs queries as nonlocking consistent reads by default, in the style of Oracle. The lock information in InnoDB is stored so space-efficiently that lock escalation is not needed: Typically, several users are permitted to lock every row in InnoDB tables, or any random subset of the rows, without causing InnoDB memory exhaustion.
anecdotally in the more than 10 years I've used MySQL I've never had any issues. whereas with postgresql I've had a few major downtime incidents. it can be very stubborn and arcane. but at least I didn't lose any data.
Substantially less sensible in many cases. MySQL has its issues (it has a lot of issues), but people have been able to get it to work surprisingly well in roles that it wasn't designed for (albeit sometimes by just building a database on top of it, as with Twitter's thing).
In most node.js apps, the best answer is probably a SQL database. Sorry.
If you're working with timelines or other cases where Redis's data models can help you, consider it, though beware that if your data is large, things will get more expensive fast since you're keeping everything in RAM.
HBase, Cassandra, and Riak are all reasonable in similar cases and have their own tradeoffs.
And yes, Couch fits a similar niche as Mongo. You might even be able to use something simpler like BerkeleyDB (quite mature) if you think you want a document store.
RethinkDB may be a nice choice too. It's fairly young but looks like it's going good places.
But your choice should be mostly dependent on what kind of data you're storing and what kind of guarantees and access models you need.
It should not be based on someone on HN telling you "Riak is the best NoSQL database for Node.js" because their idea of what most Node apps need may not be what yours needs.
Another highly underrated solution is using MySQL/PostGres as a key-value store. Just create one table for each entity type, with the primary key as the key and a JSON or protobuf blob as the value. You're using completely battle-tested solutions, you've got bindings in basically every language, you're doing basically the same work (at the same speed) as your NoSQL solutions, but you have a lot more flexibility to add additional indices and can rely more on pre-existing functionality than a MongoDB or CouchDB solution.
That works for some things. However, it's no more a foolproof magical solution than MySQL or MongoDB or Cassandra or Oracle or... It just has different tradeoffs (non-primary key queries will tend to be a problem, you'll have to make your own replication, sharding will be a problem, etc etc).
The nice thing about doing the dead simple solutions first is that they give you time to focus on the things all startups have to do (getting users, building product) and then fall down at the the things that very few startups have the luxury of needing to deal with (scaling, fault tolerance, reporting, alternative views of data).
Throughout the lifetime of my first startup, I was obsessed with the question of "What are we going to do when we need to scale?" It failed because it had a daily userbase measured in the dozens. Then I went to Google to learn how to scale things. And it turned out the biggest lesson I learned at Google was not how to scale things (though I did learn that too), but that you shouldn't scale things, not until you need to. Because the process of designing for scale slows you down significantly, and makes it much harder to develop a system that's usable and performs well under small workloads. Google products take forever to launch, because they have to scale to millions of users from day 1. As a result, their product decisions are very often questionable in early versions. Most startups don't have the luxury of Google's brand name and billions in cash to tide them over that learning process, and need to hit the ground running.
Focus on the problems you have, not the problems you hope to have in the future.
https://blogs.oracle.com/MySQL/entry/nosql_memcached_api_for...
Wait, what? Even if vertical scaling was a good idea, scaling is far from the only reason you should have more than one server for anything serious.
If you do get to the point where you need some redundancy (and don't yet need to scale horizontally), you can proxy all writes to a second server running the same codebase, have it update its in-memory data structures in the background, and hot-swap it over if the master dies.
That's probably the biggest surprise I learned from working in a fast-growing, well-functioning engineering organization. The half-life of code in a market that's actively growing and changing is roughly 1 year, i.e. 50% of the code you write now will have been removed within a year from now. And attempts to optimize for problems you're going to have in a year, rather than the ones you have now, actively make things worse because you inevitably have a different product direction in a year, and baking in last year's speculative assumptions just means there's more code you have to work around.
You also understand that most of the advice easily accessible on the Internet comes from people trying to sell you something, and so they have a vested interest in you adding many layers into your software stack that you don't need?
If you work in an actual engineering organization that has a clue what they're doing, mmap() is your best friend, and the more layers you can cut out of the stack, the better off you are.
https://news.ycombinator.com/x?fnid=cjVXpi8HxVR5TTze3bqSCa
Unknown or expired link.
My guess: because by relying on in-memory data-structures you can't do what any half assed php forum do, ad hoc queries.
Anything you can do with SQL you can do with in-memory data structures. If you're interested, I'll be happy to take any SQL query and convert it to some Python list comprehensions on arrays of dicts.
BTW, do you miss Java's more advanced structures (say MultiSet) when programming in Python/Go?
I think I'd miss these a bit more in Go because the built-in datatypes are privileges in some of the language statements, but I haven't written enough Go code to really feel their absence.
Sure, but unless you also do some indexing manually, you can't really query your whole dataset when it start to become too big.
Well, this is the thing; 'NoSQL' is really a pretty unhelpful term. It tends to just mean "not relational", and covers a vast number of things.
So, for instance, you might be okay with having to have your data set fit in RAM (with MongoDB you'll suffer if it doesn't, anyway), and not care too much about availability. In that case, Redis might be good. Or maybe you care deeply about availability; in that case, one of the Dynamo paper databases might be good, if you're willing to put in the work dealing with the consistency issues. Or...
I could go on for a bit. 'NoSQL' is verging on a meaningless term.
CREATE TABLE mongodb (
key VARCHAR(256) PRIMARY KEY,
value JSON
);Also, see https://github.com/umitanuki/mongres
Postgresql speaking the mongodb protocol.
This is currently a prototype. The following operations are supported.
db.collection.find()
db.collection.insert()
https://postgres.heroku.com/blog/past/2013/6/5/javascript_in...
I don't know how stable/performant it is (I've never needed to use it), however...
On the other hand, Redis, Cassandra, Riak, and many more are also excellent NoSQL databases. But none of them, including CouchDB, are excellent at everything. What are you planning on making? You can write a lot of different things in node.js. If you're writing, say, a blogging engine you probably should look into flat files, or maybe Postgres, and forget the NoSQL kool-aid. :)
I really don't see how MongoDB beats Postgres for running a basic blog. And while it doesn't prove anything, I note that Ghost (which has been getting a lot of press as a new, shiny, node.js based blogging platform) is backed by SQLite of all things. Why is it obvious that they should have used a document database instead? What advantages do you think that would have given them? Because of the top of my head I can't think of one.
1. If you are doing multi lingual site, you can store your multiple language content in a single document instead of futzing around with {lang, content} tables 2. If you want to do custom form/content, it is trivial to do it in a document database instead of relying on key,attribute tables. 3. Just store your theme in a single document, which can include various html templates, css, etc. To export or import a theme is also easy - just stuff the whole document into the db. 4. If you want to add plugins to enhance the capability of your blog/cms, they can have their own nested document inside their target document. Everything is contained.
Whether 10gen are vapid spin-meisters or not, even whether they have developed a usable product, seems orthogonal to the question as to whether a schemaless persistent storage layer might be a better fit for some projects than a relational database.
[1] - http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis
I'm considering moving away from MongoDB before I have to implement what seems to be an incredibly complicated architecture to get it to scale on the level tens/hundreds of millions of documents.
If you're currently on MongoDB but need more performance, concurrency, or compression, please try TokuMX: http://www.tokutek.com/products/tokumx-for-mongodb It's a drop-in replacement server that uses a better storage engine but speaks the same protocol and query language.
Keystore where the "engine" is ZFS works mighty well and is reliable. There is little need for simple solutions like MongoDB if the filesystem rocks.