What if you didn't need relational queries and was dealing with a large volume of data that needed to be sharded?
Honest question - perhaps I'm too used to think in SQL but even with e.g. a completely flat bunch of documents one usually needs at least access control (i.e. document owners, which is a relation).
When you have multiple types of objects with differently populated fields that you want to store in a single collection and query across them in currently unknown ways, and where you need flexibility for the unknown.
Eg. Dump all of the data to S3 and search it in parallel via AWS Athena.
If you want fast fuzzy search, then elasticsearch is built for this.
If the data is small enough that you want B-tree indexes (with the accompanying slower writes and lower query latencies), then any SQL database with a JSON column will work.
Too proprietary. Can't be run locally, on robots, etc. Can't be transferred to other cloud providers. Also S3 is horribly slow to write or delete thousands of records at once.
Every major cloud provider has a blob store. Most are S3-interface compatible, or can be fronted by something which is, like Minio.
> Can't be run locally, on robots, etc.
If the contents of your S3 keys are just line-delineated JSON files, you can easily download those files and run scripts or process them locally.
> Can't be transferred to other cloud providers.
Again, not true — a tool like rclone handles this case in a single command. Depends on the amount of data, but if anything it's easier to move a bunch of flat files between providers than copying a database backup around. You have to pay for egress bandwidth, though, of course.
> Also S3 is horribly slow to write or delete thousands of records at once.
If you're keeping one JSON record per S3 key, that is a blatant misuse of the tool and performance will be terrible. On the other hand, if you batch records into files of appropriate sizes, it's very cost-effective to store and query via Athena, Spark on EMR, etc.
If you need individual record-level access as identifiable by a key, then you'd probably be better off with Redis or Memcached. (though, those will not be as good for bulk offline processing) It's all about your access patterns.
I'll add that S3 is only slow to delete things sequentially. You can delete or write an effectively infinite amount of items in parallel.
Too much work. If you're a startup that's too much stuff to maintain. Much easier to get a hosted MongoDB and launch tomorrow.
And when you only have 2 months rent in the bank and investors want demo after demo in order to give you cash, tools like MongoDB do make a difference. And yes, I've been in that situation before.
That's what interface contracts and API versioning is for - and new fields shouldn't be any problem at all unless you're parsing the JSON by hand.
In basically every case you are querying all data for a symbol in a given timeframe and then process it further outside of the database.
I would think it is in fact very much the wrong tool for the job. Time series data screams structured storage, not unstructured.
select *
from prices
where symbol = any (?)
and timestamp >= ?
and timestamp < ?
The only way to beat a relational database here would be in performance. Because of course, for a specific use-case, you can always write a specialised data store that is faster than a general-purpose one. But the performance advantage might not be enough to justify the cost. If you're talking about storing every tick on an exchange, it might be, but if you're storing close prices, probably not.
It's why Data Lakes become so commonplace because it was an easy way to just get all of the data out of the silos into one place so that the business could attempt to join between them.
And it's why MongoDB (and other unstructured friendly storage systems) have remained so popular because the reality is that those datasets are not well governed, no-one knows the schemas and no-one knows how to join between them.
Not only missing or corrupted individual columns, but completely incoherent and arbitrary complex document structures. If "no-one knows the schemas and no-one knows how to join between them" the data lake project has already failed to provide value.
If you can't or won't do the work to understand and define the schema at the time of ingestion, you're asking consumers of that data to do the work for you later on, and then you're looking at an archeology project to try and figure out what the heck people were thinking in the past. It's only going to get worse as you keep shoving new data in there with its own incompatible poorly defined schema.
It probably ticks a compliance box but good luck making use of it...
Say you have a pile of garbage in silos: user metrics, analytics, usage stats, logs, sales, revenue across three different systems, etc.
You copy the garbage out of the silos and into a lake. It's a mess. No one could tell you anything real with that data.
So you write queries that ignore the errors, and don't show what you say they show, but look nice and you say they show what you know people want to hear. You know you have sales, right? You know things are going well, right? Or they will be soon, for sure. So you're making the numbers tell the story -- any problems in the data are just bumps in the measurement.
You show investors your fabrications, they show their investors your fabrications, and so on. You get another funding round for ~1.5x one year of salary for your employees + office costs. Life goes on, you have meetings. You make presentations.
Rinse. Repeat. You sit on boards, talk about innovation, sponsor a charity. Maybe make some investments! Be a local tech celebrity. 5 years go by, your business is huge and bleeding cash, and somehow... no one's willing to invest more.
So you sell what you can for what you can get. Some investors broke even, some lost the farm, some maybe even made 3-4x. No hard feelings overall! "Our Amazing Journey" blog post. The employees move on. You move on.
Rinse, and repeat.
Garbage in, garbage out.
It doesn't matter, because you were never really running a business.
I should be clear that I'm not speaking in particular about a present or past employer, nor do I really mean to denigrate businesses operating as I described. My rhetorical cynicism might be misread as just cynicism, so I'd like to try again, hopefully more charitably:
Not many top decision makers are (or closely employ) good stats people. So pretty optimistic non-stats folks try to figure out what to measure, how to measure it, what is meaningful, and how to extrapolate that into good forecasts -- not a great start for accurate measurement and forecasting.
Next, data is often bad: because of adblockers, brief app outages, known workarounds, bugs, gaps, measurement-method changes over time, differing cohort assumptions, etc. There always ends up being some hand waving. Another strike against accurate measurement and forecasting.
Finally, businesses must survive, and many are not cash-rich enough for safety or for growth, so in the near term they may be oriented more toward investment than revenue. Some good hand waving can then be the essential survival activity, and good hand-waving may require treating some fictions as real. The dreaded third strike falls against accurate measurement and forecasts.
So in a sense, these aren't traditional businesses. Their measurements and forecasts may not be very accurate, but for now it doesn't really matter. What matters more is usually the story.
A lot of applications can fail with an "eh, whatcha gonna do?" shrug when records fall out because their schemas drifted or two different developers interpreted them differently. So somebody's preferences stop working or their early "likes" get forgotten; so what? Move fast, break stuff, etc.
But when your data actually matters, I'd much rather pay the expense of having some discipline about it. The idea that my money might become the stuff that gets "broken" is kinda terrifying.
"Relations" are stored as an object, not a separate table.
See: AggregateRoot
Eg.
Document: { metadata: {}, owner: {}, sharedWith:{}}
https://en.wikipedia.org/wiki/Relation_(database)
If your data is completely flat, and fits in one table, that's still a relation!
This is anecdotal, of course, and it is possible that I simply didn't optimize Redis enough (though I did give it my best!). Let me also say that Redis is great otherwise, it was just in this specific case that PostgreSQL surprised me and fulfilled the need itself.
I also simply inserted and read the data, no complex queries or anything. Key-value store is all I needed and for this particular case PostgreSQL was as good at it as Redis.
And once you implement a global map which is accessible from multiple processes, you might as well use an existing solution, like Redis. Or, as I've learned, pgSQL - which gave me the same performance with less moving parts (because I was already using it) when I disabled WAL.
Thing is most people don’t have scale these days. You can get a single box with hundreds of logical cores and many hundreds of TiB of locally attached ssd. Until you exceed that you don’t necessarily have scale.
And then that box falls over because the entire region fails like just happened yesterday with OVH. Or it just randomly fails like has happened to me with AWS dozens of times.
Vertically scaling a database on your own cloud instances is amateur hour. Either use a cloud-managed database or one that is highly available.
Slack, GitHub both are stupidly shardable. I doubt it’s one RDBMS handling every customer, chat room, git repo. And instead they’ve segmented the workload across multiple instances.
That doesn’t work for every use case
As soon as you're back to needing complex queries (with or without relations, Postgres it is again.
- schema on write: You know what future queries you will ask. Use SQL.
- schema on read: You don't know what future queries you will ask. Use object storage.
Of course, in practice, part of your data will be schema on write and part on read.
With an object store you mostly have to have decided upfront which data is collocated together.
SQL essentially forces you to first define your data schema, then write data according to that schema. This works great for reading a user profile given an email; or read all comments of all users who are born before 1995. Since you have a schema on write, you can add indexes to optimise these queries.
In contrast, if you need to do "anomaly detection" over an opaque dataset (where the definition of "anomaly" moves faster than you can rewrite your schemas), then it's better to just do a full dataset scan and create a schema on read.
My 2 cents.
With a document store, you often find yourself de-normalising and copying data to new collections on-write to deal with the limited query language.
This means your writers often implicitly encode the possible queries.
As an example, in Redis if you need a 3 way join/filter, you will either a) have your application read all the data and filter it or b) maintain the filtered collection via writers copying the data.