Smug self-righteousness
I shall recall these dark days to the next generation as “NoSQL Madness”, or more colloquially, “my schema is my app layer”.
The author of it had ended up as CTO and kept seeking out uses for his work and ended up finding all kinds of odd places for it to live, including as the auth database in a large scale document analytics system and running part of the payroll.
We were constantly running into all kinds of scalability issues where this 12 year old component was often the source of our pain, but he'd never even entertain a conversation to eliminate it and consolidate or replace it with something better.
A competitor bought us for stupid money. They fired most folks (not my small department) had their CTO decide between our fancy new product, and the product he created. There was no question, our product was amazing, his Frankenstein was two pieces of equipment cabled together in three items the footprint ... and still did less, and was quite a ways away from being "ready".
If there was a silver lining, the Frankenstein product doomed that company and we got bought by the far more competent competitor they had.
Lots of hair pulling and I can say that it was really hard to predict the outcome for any individual positive or negative in every case until long afterward.
PS.: We do use Postgres though ;) as it has 1st class json support and at the same time we have the luxury of using its relational capabilities where needed (think of a hybrid model).
Unfortunately, people reacted to being (justifiably) frustrated with schemas by throwing strict schemas out entirely, instead of making better schema management/migration tools.
I interviewed at southwest airlines years ago and i don't remember how it came up but we were talking about bottlenecks or something and i brought up the fact that having to go to a DBA to get a column added to a table, no matter how trivial, is a great source of delay. The whole room just nodded and looked at the floor, it was obviously painful for them.
NoSQL took the DBA out of the loop, now the developers were in full control of what was persisted and what wasn't. If they needed a new field they just made it so. On the flip side, DBAs got really freaked out and cried to whoever would listen.
In my experience you either have a DBA report to Developers or Developers report to a DBA. Never give them equal footing (even implied) because they'll just fight.
In a large company like SW, the developer requesting some change for their app may have no idea how else the db is being used. What if their requested changes took down the db and prevented reservations from working?
My examples are extreme, but I have seen similar things in my years as both a developer and a DBA at times.
That's why it's hard for the two camps to work side by side.
NoSql gave power to the devs at the expense of the experience and wisdom of the database folks. I bet many many applications and systems were completely screwed datawise more than once because of devs and NoSql.
The best I've seen it work is to have a DBA on the team building the application.
Tons of conflation of Mongo's problems with those of nosql in this thread.
Compared with RBMS tooling looks like a high school project.
So Dynamo is an opportunity for Amazon to generate lock-in through their own proprietary software.
But big and likely even medium size businesses are less likely (compared to tiny companies that barely go above the free threshold) to use a new technology without any big well known users, or publicly documented use-cases etc.
One big way companies can provide some confidence to potential customers about their technology is by dogfooding: they use the thing they're trying to "sell" (regardless of whether it's a licence, a service, whatever).
How do you explain the 'NewSQL' approach, which seems to be so clearly borne of what we've learned from NoSQL?
It should be obvious that NoSQL has value, regardless of the issues with one of the earlier NoSQL DBs.
There is certainly merit beyond fashion to the dynamo architecture, and there are workloads where (for example) HBase is simply the correct type of tool despite the lack of polish of its management interface
People who got tired of dealing with schemas are now realizing that having zero schema is way more of a headache and way more work than the up front work of creating the schema.
In my experience Alembic works more better.
EDIT: fixing autocorrect
Edit: scheme/schema autocorrect typos corrected. Thanks!
At some point, you are always going to have to get the data into some sort of consistent model, so that you can operate on it in a predictable and sane way. So there's no question of there being a schema, even if it's only implicit. The question is, do you apply the schema once, when you write to the data store, so that the data at rest is consistently structured? Or do you allow it to be inconsistent in the storage layer, and instead apply the schema and re-validate the data every time you read from it?
There are valid reasons why one might choose either approach.
Which is not to say that valid reasons always play in to the decision to choose one approach or the other.
I don't think that kind of thing is particularly common, but there are definite use cases. In our particular case we use it for financial data where we want the data we are given even if it is flawed. I think the OP is 100% correct. You have to write that validation somewhere or else you are in big trouble. Usually it is easier and more convenient to do it at the DB layer, but sometimes you choose to do it somewhere else.
Anyway this is 2019 so PostgreSQL JSONB fields have got you covered. You can even efficiently query the JSON objects within them.
The main reason I wanted to reply to the question was that sometimes I see people who just can't get past not enforcing a schema at the DB layer for your whole data model. It really is crucial to understand that doing so means that bad data doesn't end up in your DB. This isn't always what you want. Like I said, not super common, but not unheard of either.
The underlying technology is pretty unimportant as long as you can do what you need to do. I've historically never really been a No SQL DB fan (there are very few downsides to relational data!!!) However, we've been using CouchDB for the odd thing and IMHO it has its place. Interestingly, I think it was my boss who originally selected it and he's gone very cold in that direction, where I've warmed to it while using it. I think the main thing is to understand exactly what benefit it is giving you (in our case easily replicated data with immutable change sets) and not give in to the hype of "OMG! You don't need a schema!", which is just not true. I've never asked him, but it is possible that my boss thought it would make life easier not to have to deal with schemas and DB migrations, and when it actually made things harder he got upset. I came into it knowing these things, but not really understanding the other benefits, which is why I warmed up to it.
If we were to start again, I think we would almost certainly go the PostgreSQL route, but I can see places where we would have some problems. It's probably a wash, really -- which is why we've not seriously tried to move away from CouchDB.
Vitess is a really comfortable middle ground of fairly familiar database semantics within a partition.
The main question I would ask is: Is your data schemaless? Often it is - especially when storing what we'd normally call a "document". Heavily polymorphic data is often better stored schemaless. And sometimes you don't necessarily have the schema in advance (common when storing "other people's JSON").
You can store schemaless data in Postgres via the JSONB type, so this isn't necessarily a "Mongo vs Postgres" issue, but more of a general data modeling issue.
As a point of reference, the folks that struggle with schemaless tend to be the ones using Javascript, Ruby, or other type-ambiguous languages. Schemaless is less of a problem in Java and other languages where class structures enforce your schema.
I'd happily agree that's an inappropriate model in close to 99% of cases, and that even if it was the right model one could (and most likely should) still use a decent database for this anyway.
That's fine if you can predict what questions your business intelligence or data science team will be asked ahead of time, but, realistically, you can't actually do that. Using a schema-on-read data warehouse instead is a more costly option, but also leaves you more able to respond to changing business demands.
There are use cases where mongo makes a lot of sense. It's very popular in the node.js / RAD world for sure. I certainly have never been a huge fan by any means. Only relatively recently did they solve distributed writes.
Other languages started prototyping features... that eventually just end up being implemented in Java.
Lambdas were a very good example - if you look at the closest competitor, C#, it got the first take on them back in 2005. Then a major refinement in 2008, adding type inference. By 2010, lambdas were idiomatic in C#. Java, in contrast, released the first version in 2014. And even then, they're still less powerful.
As for alternative JVM languages, while they are cool and have brought many fresh ideas into the platform, they remain a very tiny portion of the Java developers' market.
Java takes a very long time, because backwards compatibility and cooperation among giant companies takes years.
C# has basically Microsoft deciding how the roadmap looks like and rebooting the platform multiple times.
C# 8 won't even be fully supported on the .NET Framework.
Yes, Java takes it slow because its target market - enterprises - want it, and they have good reason to. There's nothing wrong with that. All I'm saying is that there are many other niches where developers find lagging too far behind other available options, and there's a noticeable decline in its use because of that.
And so the analogy with PostgreSQL in this case isn't working well.
They even separate Kotlin/Native graphical debugging from InteliJ, so that developers get to shell out for Clion license as well.
The only variant where InteliJ and Clion come together is on Android Studio, thanks to us NDK users being vocal how 3rd class it felt versus Eclipse CDT.
On platforms where developers can make use of standard Java there is less pressure to adotpt alternative JVM languages.
Just like F# will always be a shadow of C#, never getting to play with all toys, with C# slowly picking up all features that matter.
Code starts getting full of #ifdefs
Siverlight, .NET Core, WinRT, UWP just to give three reboots.
No support for dynamic APIs, appdomains, IL generation on the fly, reflection APIs done in a different way, ...
Actually C# is not fully backwards compatible, variables declared on foreach statements changed their semantics in C# 5.
https://blogs.msdn.microsoft.com/ericlippert/2009/11/12/clos...
MS-DOS is only supported in 32 bit variants, a species in extinsion.
This is not true - there have been several breaking changes in C#-the-language since 1.0. For example:
I think I was using floppies until Win 98?
I write C#, Java, and Kotlin in roughly equal measure. Each has its pluses. But the claim that Java's lambdas are worse because it doesn't let you--and this was a conscious design choice!--do something so potentially catastrophic and difficult to debug is an odd one.
And yet it didn't - which made implementation that much more complex, since capturing mutable locals requires lifting them to extend lifetime.
The official rationale was that they expected lambdas to be mostly used for parallel sequence processing, and wanted to avoid race conditions. Of course, in practice, lambdas are very useful in many other places, where there's no concurrency issue at all - async continuation callbacks, for example, or pseudo-custom language constructs implemented as functions with a lambda for a body.
The ideal solution is to make this a part of the function type - so that APIs that do intend to invoke lambdas concurrently can mark them as such, and then the language would enforce sharing, while other APIs that do not use lambdas in a concurrent context, can use their full power.
Same thing goes for lambdas that cannot escape vs those that can - if you reflect this in the type system, then you can also support safe nonlocal breaks and returns in the former, for example. One of the early Java lambda proposals, the one by Neal Gafter, did just that, and it was awesome.
Because existing memory models have rules for object fields and rules for publishing objects.
These rules don’t apply to local variables.
They could do - but that’s what I’m asking.
That bit is simple - the variable is kept alive as long as there is still a lambda that references it.
Don’t think about stacks - that’s an implementation detail and the compiler is free to use a combination of the stack and the heap to implement local variables.
The issue I’m talking about is if one thread writes a local and another reads it, what does that look like?
Also looking forward to them and disappointed they weren't there since the beginning.
For example adding async to the language isn't necessarily going to change your programs to be async because every widely used library has already adopted threads and doesn't support async yet and often never will.
The real benefit of MongoDB at this point is the ability to easily scale beyond a single machine with shards and high availability using replica sets.
Postgres will get you pretty far, but beyond a certain point the scaling story breaks down and you have to hack some sort of user space sharding solution. At that point all the schema update and backups become a nightmare.
Schemas: https://docs.mongodb.com/manual/core/schema-validation/
Transactions: https://docs.mongodb.com/manual/core/transactions/
On the "relational" front, it has denormalized-only schema validation and limits transactions to replica sets (so no transactions with sharding), while also recommending single-document transactions over multi-document transaction via denormalization. (FWIW, transactions aren't available in any open source release).
On the "database" front, it has a history of misleading users and remorselessly dropping data.
> Postgres will get you pretty far
Postgres is an actual relational database, open source, battle-proven with a good design and a great team behind it. It never claimed to be good at, let alone capable of, doing things it could not actually do (well, or at all).
> but beyond a certain point the scaling story breaks down and you have to hack some sort of user space sharding solution.
Scaling IS a hard thing, and presents itself quite differently to different use-cases. Nevertheless, horizontally scaling Postgres — for when one truly hits the petabyte-scale — is a problem that has been solved correctly many times before (out of core). For a similar-to-MongoDB method, check out Citus, with the assurance that it only adds to an actual database; as opposed to the MongoDB way of doing things: make up and promise magic scaling solutions that "Just Work", then try to build a database on top of it.
* My code depends on this value always existing so make this not null * My code is doing math on this value so make sure it is always a number * This record belongs to other record, make sure other record can not be deleted while this one still exists
Modern computers change next to nothing with the need to validate data. The worlds fastest computer wont tell you how to add a number that doesn't exist.
" RDBMS provides far more opportunity for validation"
This can't be true. The application layer, which ultimately contains all 'knowledge' of all aspects of the business, including data from all other resources, can obviously 'provide more opportunity' for validation than any DB possibly can.
Moreover, 'validation' generally implies aspects which are inherently application specific ergo, doing this purely in the data layer almost implies an intersection of concerns.
Validation in almost every case must be done on the app layer, so anything we get from the DB is an added benefit.
Also, data generally has to be validated when it enters into the business logic, long before it gets into the DB, moreover, there are usually data elements that are not persisted, and must be validated anyhow, again illustrating the requirement for validation above the DB.
I've rarely seen a codebase outlive it's database, but I constantly see databases survive through multiple codebases.
There certainly are invariants that are non-trivial or cumbersome to enforce strictly with a schema, but you can really only enforce them w/ a database that provides serializable transactions.
In many cases, schematization of data in the database is good for other reasons though (for instance, guaranteeing type-normalized data in the presence of multiple deployed versions of an app via accident or otherwise, ensuring your queries and updates are typesafe, etc.)
> 'validation' generally implies aspects which are inherently application specific
Not at all. Taking this at face value implies that some app can write data to the database that is valid according to that app, and then another app can read data that is invalid from its perspective, and have to deal with it. That doesn't make sense - data is data, it's either valid, or it's not. That's why the schema is about the data, not about the app.
> Validation in almost every case must be done on the app layer
For UX reasons, mostly, yes. But it's usually much more basic than what e.g. triggers would do in the DB itself.
I'm not saying that there's nothing to validate outside of the DB, either. But for the data that is in the DB, the DB itself can usually do a better job.
Disclaimer: MongoDB employee. All opinions are my own.
Even if your team executes perfectly and never runs into this, the biggest problem IMO is that you can't really enforce most of your guarantees w/ any degree of confidence w/o a typed schema. Even if you work within a typed language that perfectly validates all the invariants of your application before storing anything, the second you need to perform work that does not strictly funnel data through your application (i.e. an update query), you are effectively gambling on whether or not those invariants will hold. This kind of "read-modify-write" flow of data doesn't really perform well (or even hold validity) for a lot of common use cases, so in reality you need your database to ensure these things for you.
Also the two deployed apps problem is just a special case of two people interacting w/ a database who aren't working under the same assumptions as to what invariants should hold. That can happen in single code bases, even with a lot of care taken.
The database schema (along with stored procedures, views etc.) is an API and database engines are designed to have multiple concurrent writers. Multiple applications and users needing access to the same data is largely why databases exist in the first place.
This is not true.
The objective of the overall app/system (i.e. front/back/middle/DB/storage/services etc.) is to carry out some kind of business logic. A DB schema cannot fully validate stored data against the logic.
Otherwise we wouldn't write backend code, we'd just write a bunch of schemas and be done with it.
Let's use a crude example: a password. (Of course, we would never in reality store a password as a string in the clear, but just as an example ...). When a user sets a new password, we have to validate that it meets specific requirements in terms of format, and then some others rules which are more complicated such as: "can't be the same password as the last 5".
Those 'password rules', for example, cannot be encapsulated in the schema of the DB and yet must be applied in order for the data to be 'valid' from the perspective of the app, or 'overall system'.
The DB may only care that it's UTF and max 20 chars. But the system requires more validation than that.
Re: Your statement about 'one app writing data, and the other app not knowing what to do with it'. This is not true, because all apps operating on such data must understand it data in the context of business/logic context in which it was designed. Even 3rd party users of such data, via API's, must understand this data from the level of business logic - not merely 'schema validation'.
When you query data from Google Geolocation, the 'city' field may be a valid string of a certain length, but that's not very useful: it must actually be the name of a city! Any 'app' using this data must operate with the explicit understanding that this is in fact the name of a city - and not just a string that met a DB schema validation requirement.
Postgres actually lets you run triggers and similar that can validate data arbitrarily. You can even do web requests with the right extension.
If that is not enough, you can run Python code in your database instead and do the same thing with a slightly more powerful language for general purpose computation.
You could write the entire logic of any business app in a PG database and only use the app as a shiny view layer.
Are you familiar with SQL constraints, triggers, user-defined functions, stored procedures?..
Data always outlives the application. You could argue that some app + data lives on together, but then you have just poorly reimplemented what an RDBMS does for you up front.
Applications generally can't recreate ACID properties and specifically, they shouldn't be trying to.
ACID and 'data validation' are generally separate issues.
Data generally has to be validated as it enters the business logic, before it gets stored in a DB. While a DB may in some cases ensure that data adheres to a schema, this usually does not fulfill all of the validation requirements.
You can do that yourself. Or let the database do it. For things where you can't express it in a database schema, sure. But you'd be surprised how far it gets you.
To be fair you can do use schema validators in Mongo. Not sure it's widespread in practice. And there are other distributed databases that aren't document stores that have schemas and various subsets of SQL implemented.