Think before you Mongo
blog.runnable.com
blog.runnable.com
unfortunately, this industry is dominated by cocky PFYs (of any age) convinced they don't need to study history of their field and, as a result, don't even recognize they're repeating 50 years old mistakes. sure, SQL has fallen so short of the promise of the relational model it's not even funny, but don't conflate the model with the query language, folks!
if you like hierarchical databases (data trees), consider that relational database gives you a forest: you can treat any datum as your tree root and bloom from there. with hierarchical, you're tied to a single pre-designated root.
both relational and hierarchical databases let you go from department to employee, only one lets you go from employee to their department without enumeration. what purpose does precluding the latter serve?
I argue you ignored the silver bullet criticism and then immediately doubled down on SQL as the silver bullet.
The initial joy and speed of of development is really nice. However, my experience has been you end up paying significant technical debt when the specs evolve faster than you think.
i don't think so, i simply refuse to have my arguments framed that way since i'm very well aware of the shortcomings present in what has so far passed for a "RDBMS". i consider velocitypsycho's argument ("Thinking RDBMS is a silver bullet is just as bad as thinking a document database is a silver bullet.") a non-sequitur: the former is a proper superset of the latter, that sentence simply "does not compute".
doubled down on SQL as the silver bullet
where does the post you replied to mention SQL? are you confusing SQL for the relational model?
Exactly. Just like static typing is a proper superset of dynamic typing. More generally, structured X is a proper superset of unstructured X.
> what purpose does precluding the latter serve?
Whereas, going to a non-relational solution would be something I'd do when I needed to trade away simplicity and consistency in exchange for scaling/performance for specific cases where relational can't handle the write load. Losing the schema and ability to do joins and arbitrary queries and assume transactionality is a big loss that requires lots of work.
A dirty secret of NoSQL is that sure, part of their target audience is legitimate needs, but another part of their target audience is just new developers who don't have much experience with either SQL or NoSQL databases, and are easily mislead into buying the idea that NoSQL will be easier Because There's No Schema or Joins To Think About
Schema on write is a hassle up front because you've got to start imposing a strict data model up front, possibly before you've even got a good idea what your information domain looks like. And every misstep will be immediately punished with a painful migration.
Schema on read is, IMO, a hassle in the long run because now any code that's consuming the data needs to be prepared to have the information come in any of the ways that it has ever been stored in the history of the database. If folks were being disciplined, then hopefully that's a small number. If they weren't, you may end up with either some sort of combinatorial explosion, or a situation where you've seriously got to null-check every little thing. Every misstep will forever be punished with a million tiny little if-statements nagging at you like endless paper cuts.
I suppose it's easy to guess where my crass sentiments lie.
You don't get to just stop worrying about schemas, joins, and transactions in your database. If your database won't do those things for you, now YOU have to.
* No schema just means (as you nicely describe) your app deals with schema changes, not the database. Have fun.
* No joins just mean your app deals with the complex choreography of keeping denormalized tables up to date. You think that's easy? Have fun deciding what updates to make synchronously or asynchronously, what to compute where and when, making sure you don't screw up and either hose performance or end up with stale data in a dependent table... enough of this and you'll be dying to come back to a proper relational database where you can just CREATE MATERIALIZED VIEW and call it a day
* No transactions just means.... oh, who am I kidding, you're not going to bother to write your app carefully to deal with those consistency semantics (and if you do, you'll probably do it wrong), you're just going to ignore them, call it a day, and hope your product doesn't get popular enough that the race conditions start pissing people off.
* No SQL means instead you're probably going to use some protocol that's much newer, much less popular, and locks your app and your mental knowledge into a single specific database product. Have fun rewriting your whole database layer and relearning the whole API and data model when you realize FooDB might be a better fit than BarDB. And it's way easier to go from SQL to non SQL if you end up having to than the other way around
Yes, SQL does have its issues, but I am yet to work on any project where the relational data model doesn't fit.
Plus all the stuff that we can do at the SQL engine level, specially data validation, is just great for the type of stuff we develop.
There is also the fact that no programming language lets you deal with relational data sanely in code, so you have the well known impedance mismatch heaeache. All popular languages I've seen offer hierarchical (maps of maps etc.) and more primitive structures only.
The CAP theorem doesn't force you to give up consistency within a single node. NoSQL databases often do, though.
> All popular languages I've seen offer hierarchical (maps of maps etc.) and more primitive structures only.
So, instead of fixing programming languages, let's cripple databases?
I understand the trade offs in order to scale, but you can give up on joins WHEN it's time to scale. Mongo doesn't give one the choice at first place.
Then there's Prolog, Kanren, Mercury… which are admittedly not especially popular.
The so-called “impedance mismatch” comes mostly from people who don't understand all that SQL can do and inevitably end up replicating it in some other language.
The "Object-Relational impedance mismatch" is not a result of the supposed fact that "no programming language lets you deal with relational data sanely in code", its a result of the fact that industrially popular object-oriented languages, of the time when the term was coined, did not align well with the data model supported by then-existing relational databases, and vice versa.
Just like all those big data deployments that can fit on an USB key and be processed by plain UNIX tools.
(Think about it this way: SQL is Java. NoSQL is your typical extremely forgiving dynamic language. We need a database equivalent of ML and Haskell.)
NoSQL is simply the result of not wanting to think about the logical structure of data. Plain intellectual laziness.
You always see how they bash having to learn SQL on those presentations.
NoSQL was an attempt to scale by sacrificing some of the capabilities of the relational model. Key value stores scale great, at the cost of having almost no query capabilities to speak of.
Now, some developers may adopt NoSQL due to the ease of getting a new project started. But I don't think that was the main motivation of the developers of the major NoSQL databases.
(Although, NoSQL is so broad I'm sure there are counter examples.)
You can give up global consistency without sacrificing local (single-node) consistency. And normalization isn't an all-or-nothing proposition: you can select the kind of schema that best fits your needs. Unlike the case with NoSQL, which just says “lalala... I can't hear you” whenever you bring up consistency.
> Key value stores scale great, at the cost of having almost no query capabilities to speak of.
Far more worrisome is the loss of data integrity guarantees. It's okay to let me selectively disable these guarantees when I don't need them (say, by using a less structured schema), but a “database management system” that doesn't let me enforce the intended structure of my data, under any configuration, is simply not worthy of the name “database management system”.
To be fair, though, it is genuinely difficult to gauge which technologies are worth adopting. Clearly some are, otherwise we'd all still be writing hand-crafted assembler to optimize drum memory I/O, or perhaps heaps of Perl CGI scripts.
It's also unrealistic to expect the average programmer - hell, even the pretty good programmer - to be aware of 50+ years of relevant research. For starters, many of them can't even access the relevant papers without paying some gatekeeper a whole lot for the privilege. If they could, they'd need a significant investment of time to understand them, especially without ready access to domain experts who can tell them which papers are most worth reading and in which order. (All of these are benefits of graduate student life that are often taken for granted.)
I suggest you check out The End of an Architectural Era (It’s Time for a Complete Rewrite)[1].
The "failure" of non-relational databases was for specific application domains.
Not saying Mongo doesn't have any legitimate uses (although I honestly suspect it doesn't, not even the ones listed at the end of TFA, at least not assuming the developer is already well versed in using a good relational database and related tooling. In fact I think I'd be willing to challenge and bet money against a mongo proponent to see who could create "an MVP on a super-tight schedule" faster.)
My question is this, if you tried to use PostGres and designed your tables super super wide and embedded everything in large object data types and then it got to complicated to manage and figure out, would you consider that a failure of PostGres, or your failure as a developer/architect?
I will have to chock this up to a good laugh myself. Lastly, I've used relational databases for most of my career, and I'm very good at writing SQL. But after learning MongoDB and how to use it properly, there isn't a lot that I would want to go back to an RDBMS for. And with the future roadmap of MongoDB, I don't see that changing.
Instead it should be marketed as "good database for dynamic data that requires complex filtering, also we have very good drivers for many languages" because that's the only thing it's good at.
Still, I can't see a situation where I would choose a non-working datastore over a working datastore.
It's like they wanted to be all three but couldn't quite commit to one, and the end result is the proverbial "tankicopter" that's neither as strong as a tank, nor as maneuverable as a helicopter.
I last used MongoDB seriously in 2012-2015. We had myriad operations problems including inconsistent indexing across shards (where some shards had an index created and others didn't, it was baffling), issues with the balancer not moving chunks properly, and more. Also it's just different than other DBs with its lack of transactional consistency (I think they've made progress on building this), but that's part of why it's fast.
However, the bigger problem is that document databases -- in general -- enable a kind of software development where the model sort of emerges over time, rather than being carefully designed from the beginning. Yes, it's flexible, but you pay an absolutely enormous cost down the line dealing with inconsistent documents. It's not like code where if you do something stupid, you can fix it over time with refactoring and "remodeling" -- data has mass. You can get into a situation where, with a large data set, it can take a week or more just to run the migration script required to scan an entire collection and rewrite a few billion documents into a new, better format.
There is no such thing as a "schemaless" database. That's like saying, oh sure, we just have a bunch of 1s and 0s in memory -- our data is "structureless". The question is whether the database enforces the schema, or not. And I think that in a lot of cases, it's a lot worse to have an "uncodified schema" than a rigid, but at least well-defined, one, that's consistent across the data at all points.
Sidenote: It's also occurred to me over the past few years that it's almost impossible to impose a consistent schema on a large enough dataset. If you truly are dealing with "big data" (TB/PB scale) maybe go straight to the document store of columnar because doing a migration is outright impossible, but don't be so quick to write it off for GB-scale datasets.
We had more problems with sharding over the years than you could imagine...the distributed locking mechanism didn't work a lot of the time, the balancer didn't work, weird consistency issues between the config servers, configuration that didn't get replicated across all shards, stupid shard key selection (admittedly our fault but there really should be better guidance on this topic), etc.
+1. I agree that NoSql promotes "careless db design" to an extent.
But yes (as an example) situations do arise where you either need to alter the existing SQL table with bazillions of rows of data in it or refactor your design in an ugly-ass way by adding a 2-column table tied to the 1st table by some FK.
So we have to rely to some extent on the experience of others and our own intuitions / less than perfect inferences.
A challenge there is determining whether it makes sense to massage the data into a common schema for further analysis or to use an unstructured initial approach from the beginning. Sometimes you get to the former from the latter.
This sounds like Proof of Concept, not Minimal Viable Product.
Storing other people's data.
If you need to consume other services, especially if its more than one, it's hard to beat Mongo (or other NoSQL databases). You get a lot of search power (not always the easiest to tap, but you get it), and your app won't break when they change their formats. If you need a LOT of it, even better.
I would never use it as my main datastore, though. At least not for any projects I can think of offhand.
I could always do validation before inserting data, but that opens me up to error on my side which I'd like to avoid=)
(Though if it exists please tell me, I'd be very interested!)
One very good, which I extensively used in my previous position was DevExpress (https://www.devexpress.com/Products/NET/Controls/WPF/Editors...).
Guess again. You can use JSON functions¹ in constraints:
create table foo (
bar jsonb,
constraint bar_count_is_positive check (bar->>'count' is not null and (bar->>'count')::integer >= 0)
);
You could probably write a tool pretty easily to convert a JSON schema to Postgres constraints.
1. https://www.postgresql.org/docs/current/static/functions-jso...
In my particular case clients of different quality are connecting to a local datastore. I'd like to make sure that even if they mess up validation the data in the store still matches the schemas it claims to. Of course, I could have them connect to a local process instead and have that process handle validation before the data goes in the store, but it's always nice to avoid intermediaries.
http://docs.couchdb.org/en/1.6.1/couchapp/ddocs.html#validat...
I am using MongoDB in my production environment for things like consuming incoming rates from different vendors, storing and serving pay stubs and client invoices to our web customers, controlling MSMQ message queues, archiving client emails for historical audits, etc. The key here is that they are document oriented entities, not normalized relational data.
What it comes down to is a willingness to be agnostic in selection of your database platform. Or more to the point, to let your use-cases drive the platform instead of the reverse. If you are going to develop a use-case that requires frequent partial updates, JOINs between multiple data structures, and traditional entity normalization, then a traditional RDBMS is appropriate. If your case allows for a denormalized mode of storage where all of the relevant information is contained within one document structure, and you can benefit from a fluid design, then MongoDB could be a good fit. If you need ephemeral key-value pair structures such as in session state caching, then something like Redis may be more in line with the requirements. They all have sweet spots that they fill well...
We all tend to have our preferred DB "hammer" to drive developmental "nails". What I propose is that we need to have an entire database toolbox from which to choose the right tool for each job.
Others have commented here about the need to thoughtfully plan before you write one line of code and/or choose your DB platform. I have to agree completely. You can map a typical relational use-case to MongoDB very easily to start with, but you do need to be able to enforce some level of control. MongoDB does this now with document validation.
https://docs.mongodb.com/manual/core/document-validation/
You also now have document level atomicity and transaction-like behavior as of MongoDB 3.2.
https://docs.mongodb.com/manual/core/write-operations-atomic...
MongoDB, like the rest of us, is constantly iterating and improving. If you have not looked at it lately and are basing your opinions on earlier versions, I would encourage you to take another look...
Just my humble opinion...
The stack trace is amazing to see.
long store short, it's not the async code per say, but after that.
Also I know document dbs don't aggregate well, but wouldn't it be appropriate to have your app backend in NoSQL and have etl or other duplication to an rdbms or possibly cassandra?
Why, is there something inherently better in MySQL that makes their second choice also suboptimal? Genuine question, I have very limited knowledge on DBs.