Most RDBMSs can do key-value stores very well now. Most applications also care more about consistency over availability, which is what RDBMSs do (CAP theorem). Many NoSQL data stores choose availability and partitioning and sacrifice consistency (i.e., "eventual consistency"). There's a lot of applications that you can't sacrifice consistency for. Electronic health records, financial records, student records, employee records, etc. You care that the data are accurate and up to date, and you want the system to error if it can't provide that. Wrong answers and "close enough" answers aren't good enough.
Now, if you're running Reddit or Wikipedia or Facebook or HN... do you really care if a user doesn't get the absolute latest version of a document or comment? No, not really. If the content is hours old it's a problem, but it's not a big deal if it's a few minutes out of date. You care more that your users get a version of the document more than you care that they get the latest version of the document.
But "applications" are built by development teams.
So: Does an "RDBMS makes sense for most applications"?
EDIT: If you downvote, please explain why. You can't disagree with the truth.
Although I kind of got used to RDBMS crowd not understanding consistency, it's just another technology cult.
https://docs.microsoft.com/en-us/sql/t-sql/language-elements...
> If the transaction committed was a Transact-SQL distributed transaction, COMMIT TRANSACTION triggers MS DTC to use a two-phase commit protocol to commit all of the servers involved in the transaction. If a local transaction spans two or more databases on the same instance of the Database Engine, the instance uses an internal two-phase commit to commit all of the databases involved in the transaction.
I'm only versed in SQL-Server but I'm pretty sure other RDBMS vendors provide similar functionality.
At the single server level (which is how I think others here are interpreting your comment)? No, they all do, with the exception of some configurations of MySQL (especially older editions, which is why it's often maligned by DBAs). That's what transaction logs do. They're literally a write ahead log (WAL). You commit a transaction, and the DB first obtains an exclusive lock on the affected rows (or page, or table). Any other transaction attempting to read or update those rows will be blocked (with exceptions). It then writes the change to the transaction log and flushes the change to disk. Then it writes the changes to the database file and flushes the change to disk. Then it returns the results of the query to the user. Many RDBMSs let you control how tightly the locks are and the degree that the data are isolated during a transaction.
At the distributed network server level? Then I guess I kind of agree with you, sure. RDBMSs let you "get around" the problems of distributed scaling by not letting you do it easily. SQL servers often only have master/slave or publisher/subscriber setups or otherwise partition the data between instances with sharding. There's no need for raft or paxos type algorithms because they don't attempt to implement a true multi-master environment. There's either a fixed overall master, or each server is the deterministic master of it's own little world, so you avoid consistency problems with distributed data. However, in doing so you sacrifice availability, since if a shard goes down so does all that data, or if the master is busy then you can't always submit queries to the slaves. Replication is used for redundancy, not scaling or load balancing. The solution RDBMSs had was sharding + master/slave replication for redundancy, which can get messy fast and has issues like hot spots or limited queries or variant performance. It's just a lot harder to do than it feels like it should be, and with storage as cheap as it is it feels like a waste of effort.
That said, some RDBMSs do allow you to use multimaster, bidirectional, or peer-to-peer replication, but most of those configurations basically warn you that you're sacrificing consistency by doing it and all of them that I've seen are a huge pain in the ass that makes shard + replicate look like child's play. They also have schema requirements that make life difficult, and they're somewhat notorious for being difficult both to administer and develop for. You have to design the whole thing from the ground up to work with this type of replication, it still feels like a house of cards, and it's this exact level of pain in the ass that encouraged the partitioning and availability focused NoSQL data stores.
However... most applications don't need that kind of scaling. They don't need a database in every time zone for single millisecond response times globally. They don't have the users to demand it, or don't have the quantity of data to require it, or have other requirements that make a traditional RDBMS desirable where you can't accept a system that allows for out-of-date data (which is when PACELC theorem kicks in because NoSQL typically doesn't have locking like an RDBMS does to mitigate this particular problem).
Google’s Cloud SQL is a good example of that, by using TrueTime as transaction id, and an MVCC implementation, they are able to provide consistency, while also being good enough on the other metrics.
Some NewSQL implementations copy that concept, but unless you run GPS clocks yourself, you’ll get slightly worse results.
Yep, all of MongoDB is just one bullet point on Postgres's list of features. Anyone spending on money on it ought to be hauled before the shareholders and given a talking to on fiduciary responsibility...
Tell me again how Postgres can seamlessly do horizontal scaling and synchronous replication?
[0]: https://www.2ndquadrant.com/en/resources/pglogical/ [1]: https://www.2ndquadrant.com/en/resources/bdr/
In general Postgres was not designed at its core for a distributed world. Even now, replication feels like an afterthought in the grand scheme of things, and sharding nonexistent without extensions.
> MongoDB’s version 0 replication protocol is inherently unsafe.
Tell me again how MongoDB took 8 years to get to the point where its replication is kind of OK.
I'm very curious about what people think about the future of Mongo, independently and particularly in comparison to Postgres. However every time that comes up, people keep bringing up that Mongo was a buggy piece of crap in some irrelevant past. So what?
8 years ago PG did have replication though, so not sure why it not having feature X 8 years ago makes my point moot.
People keep bringing up that it was a buggy piece of crap because its the icing on the cake and pretty much something you never want your database to be, past or present. Not that software configured by default to eat your data and not persist it can be called a database mind you.
Of course I would prefer Postgres when I can use it, and I can generally use it basically all the time, but NoSQL still has its use cases.
I mean... do you? I often come back a few minutes after posting to add something I forgot or rephrase something for clarity. I hate when I am tweaking a Reddit comment a couple times during a period of high server load and I get served an old version of the comment and end up losing something I added in a previous edit.
With something like Wikipedia it would be quite frustrating to lose revisions.
Obviously it is what it is, I can't change their codebase, and I'm sure it's necessary as currently engineered, but is there really no other way to cluster their data except "one big table"? Maybe like shard subreddits to specific servers ala Hyperdex?
But yeah, most places that Mongo is applied aren't exactly Facebook or Reddit either, in terms of total data throughput.
Data stores like Cassandra and MongoDB don't lose revisions. That's not the kind of consistency we're talking about. CAP consistency is just getting the most recent version. You won't lose data -- data loss is a bug, not expected behavior, just like any other data store -- you just won't always get the most recent version of it. And, keep in mind, when we talk about eventual consistency here we generally mean "consistent on all nodes within a few minutes, but we're not blocking reads to write this data." It's not going to take hours.
That said, if you find you get an old version of your own comment, I'd be more willing to believe it's the fact that your request failed with a 503 error or otherwise timed out as much as it was a data store problem. Next time it happens, wait 5 minutes and try again.
> is there really no other way to cluster their data except "one big table"? Maybe like shard subreddits to specific servers ala Hyperdex?
The whole point of MongoDB or Cassandra is that you can get shards without all the headache that RDBMSs usually put you through. You configure your sharding function and let the system do the rest. You don't have to connect to the right shard or anything of the sort, which some RDBMSs do (or did, it's been awhile since I've looked) require with sharding.
Reddit has their code and architecture posted, though it's out-of-date now, it makes it clear that it's basically just two big tables:
https://github.com/reddit/reddit/wiki/Architecture-Overview
It's PostgreSQL, ThingDB, Cassandra, memcached, and RabbitMQ.
Why? An RDBMS has never been the best option for any application I have created and I have created standard business applications as well as consumer applications.
Wikipedia uses MariaDB (so, MySQL). https://meta.wikimedia.org/wiki/Wikimedia_servers#Software
That said, it's pretty hard to make sense of when you would want non-relational dbms these days, especially in an era where you can get 100core systems in AWS/GCP. Write-scaling is still a pretty obvious reason, though things like Citus might help here.
With non-default settings[0].
The Jepsen tests passed with the "linearizable" read concern. The default read concern is "local", which "Provides no guarantee that the data has been written to a majority of the replica set members (i.e. may be rolled back)."[1] This is like having "READ UNCOMMITTED" be the default read level in a traditional database system.
The Jepsen tests passed with the "majority" write concern. The default write concern is "1", which means only the "primary" in a replica set needs to acknowledge the write[2]. This does not guarantee safety in the face of network partitions.
It's still not safe out of the box.
[0] - https://jepsen.io/analyses/mongodb-3-4-0-rc3
"With the v1 protocol, majority writes, and linearizable reads, MongoDB 3.4.1 (and the current development release, 3.5.1) pass all MongoDB Jepsen tests:"
[1] - https://docs.mongodb.com/manual/reference/read-concern/
[2] - https://docs.mongodb.com/manual/reference/write-concern/
Same goes for example for PostgreSQL [1], that uses Read Committed rather than Serializable transaction isolation by default because for the majority of the people, this is fine, and the performance tradeoffs are worth it.
[1] https://www.postgresql.org/docs/9.5/static/transaction-iso.h...
It was a bug that was fixed many, many years ago and was only true if you didn't use any client libraries.
Cassandra as well doesn't require a full quorum to acknowledge writes. It just relies on the closest node. Likewise for Oracle.
http://docs.datastax.com/en/archived/cassandra/2.0/cassandra...
Personally, I find a non-relational database useful when my data model is non-relational.
This talk is a pretty nifty perf overview:
https://www.percona.com/live/e17/sessions/high-performance-j...
That said, if you know beforehand that horizontal scaling will be a crucial factor, probably postgres isn't the first choice. But with how fast CPUs are these days it's usually not important for a long time.
That’s a very naive statement to make.
Absolutely majority of the companies will do just fine, because the hardware improves faster than their demands. Starting out with a distributed system "because one day we might need it" is just silly, because chances are you'll never hit it, and you'll have to pay for the overhead of having a distributed system (which is non trivial).
Actually my company started using PG and had presentation and someone asked if we considered a distributed database so we can scale. The presenter nicely said it was evaluated and this solution worked best, but that was too nice.
1. It's only about 100GB of data
2. The hardware is barely utilized, we didn't tune it (except some standard memory settings), because there's no need yet.
3. Our data is relational (in fact most data from most companies is relational)
Which means you can't use it for any big data/analytics use cases. MongoDB has fantastic client libraries e.g. Spark, Java.
A tree is relational. Each child has a relation to its parent.
I can't imagine data that has no relation (no connection) to anything else. Maybe what you meant was heterogeneous (e.g. data elements that do not all have the same attributes) - but even then I can't readily come up with an example.
Yes, I agree that most types of data can be stored in tabular form (or an "n-ary relation" per Wikipedia). I'm just wondering what concrete types of data one would rather store in a document.
I don't think there is a good example. The decision to store some data outside of an RDBMS must have more to do with the processing model or something else.
What else other than the processing model and business requirements would determine how you model and store your data?
I'd recommend the paper What Goes Around Comes Around[1], the first paper in Readings in Database Systems[2]
[1] https://scholar.google.com/scholar?cluster=73661829057771494... [2]redbook.io
https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80...
EAVT is great as an intermediate format but it is absolutely useless to query for since most of the time you are trying to find a set of attributes for a given entity i.e. full table scan.
What you want is a "wide table". One entity column and all the attribute columns to the right. Often with most of the values set to null.
This is the dream use case for MongoDB since it you can ignore sparse values yet when you query it via their drivers it will appear as a wide table. You can't do this at all in PostgreSQL since you will hit a column limit.
JSONB is designed for exactly this, isn’t it?
The lack of a Spark driver alone renders PostgreSQL useless for most companies.
This is what indexes are for. An index on the entity id should avoid any full table scans.
And you want to build indexes on half the table ?
Good luck with that.
Your math is at odds with your own requirements, null values don't need a row.
Sparcity is an issue for the wide table not the EAVT form.
I still can't imagine what sparse heterogeneous data exists in the world that makes sense to store. Any type of querying or processing requires some kind of structure (even if implicit in the code) which you can just put in different table structures.
You have to make sense of data to process it and that kind of implies a structure, doesn't it? Am I missing some obvious example of heterogeneous data?
One customer column, tens of thousands of attribute columns.
If you need everything about a customer it is a single, O(1) fetch operation which makes it perfect for driving chat bots, call centres, websites, operational decisioning engines, dashboards etc. Almost every large company will have one of these.
You can't really do it in relational systems properly because (a) you hit the column limit, (b) often it is sparse i.e. lots of NULLs everywhere, (c) you need this system to be distributed since it often gets a lot of load.
But where you get into tens/hundreds of thousands is when you have machine learning models automatically selecting and storing important features from the data.
In years past, people called these "data warehouses" and essentially took snapshots of their production DBs and denormalized the hell out of them so that aggregations wouldn't crash the server.
Try modelling a cyclic graph in a relational way and you'll quickly tie yourself in knots trying to update and query it.
The point is, relational databases a great for storing data that you've decided to model relationally. If you decide not to, then you probably want some other sort of database.
Sure. But that database is not MongoDB.
The "you only need relational databases" mantra bugs me though, because it's so obviously not true.
I bet two years ago there was someone out there saying "if you're going to do NoSQL you better use RethinkDB over MongoDB"
How great the technology is, is absolutely not the only factor. Good thing people can take in many different factors when making their decisions.
https://www.hntrends.com/2017/september.html?compare1=Mongod...
While you're at it you could also elaborate on why Postgres is much less "solid" than a database that literally eats writes without any consensus as to if they are valid and/or actually written. After that you could explain why "post-cap" is a thing.
Until you do, your comment is pretty useless and it sounds like you could do with a nice shot of consistent, well designed database right to the heart.