In 2022, there are so many more mature, reliable, battle-tested NoSQL options that could have solved their problem more elegantly.
But even today, if I were to build Reddit as a startup, I'd start with a Postgres database and go as far as I can. Postgres allows me to use relations when I want to enforce data integrity and JSONB when I just want a key/value store.
This is accepted wisdom on HN but I don't see it:
- True HA is very difficult. Even just automatic failover takes a lot of manual sysadmin effort
- You can't autoscale, and while scaling is a nice problem to have, having to not just fundamentally rearchitect all your code when you need to shard, but also probably migrate all your data, is probably the worst version of it.
- Even if you're not using transactions, you still pay the price for them. For example your writes won't commit until all your secondary indices have been updated.
- If you're actually using JSONB then driver/mapper/etc. support is very limited. E.g. it's still quite fiddly to have an object where one field is a set of strings / enum values and have a standard ORM store that in a column rather than creating a join table.
>True HA is very difficult. Even just automatic failover takes a lot of manual sysadmin effort
Nearly all Postgres cloud providers such as RDS, CloudSQL, Upcloud, provide multi-region nodes and automatic failover.
Also, true HA databases are very expensive and have their own drawbacks. Not really designed for a Reddit-style startup.
>You can't autoscale, and while scaling is a nice problem to have, having to not just fundamentally rearchitect all your code when you need to shard, but also probably migrate all your data, is probably the worst version of it.
Hence why I said "as far as I can". I can keep upgrading the instance, creating material views, caching responses, etc.
Lastly, the most important reason is because I know Postgres and I can get started today. I don't want to learn a highly scalable, new type of database that I might need 5 years down the road.
From memory you have to specify a maintenance window and failover is not instant. And relying on your cloud provider comes with its own drawbacks.
> Also, true HA databases are very expensive and have their own drawbacks. Not really designed for a Reddit-style startup.
Plenty of free HA datastores out there, and if we're assuming it'll be managed by the cloud provider anyway then that's a wash.
> Lastly, the most important reason is because I know Postgres and I can get started today. I don't want to learn a highly scalable, new type of database that I might need 5 years down the road.
That's a good argument for using what you know; by the same token I'd use Cassandra. Postgres is a very complex system with lots of legacy and various edge cases.
It turns out that SQL is crucial to non-app-developers such as business analysts, data scientists. Trying to setup an ETL to pull data from your MongoDB datastore to a Postgres DB so the analysts could generate reports is such a waste of time and resources. Or worse, the analysts have to request devs to write Javascript code to query the data they need. For this reason alone, I will always start with a DB that supports SQL out of the box.
The entire database world has been working ever since trying to get back those RDBMS features into their horizontally scalable solutions, it's just taken a while because the problems are exponentially harder in a network. It's not like people are using eventual consistency and a loss of referential integrity because they prefer it.
I agree that this should have been the reason for choosing something like MongoDB over MySQL/Postgres.
In reality, I saw a lot about the companies switching/starting with because so their devs didn't have to learn SQL and they can just stay in Javascript/whatever language they were using.
Or because they wanted their data to be schemaless.
Now, a valid reason for wanting data to be schemaless is that you store free form documents, generic cache entries (Redis) or other such things where it really makes no sense to specify the shape of the document/entry, because at any point in time, different entries will have different shapes. Another can be scalability concerns or other reasons for why you consciously choose to have your data fully denormalised.
But invalid reasons, IMHO, include:
- "It's faster." It is, until you get data corruption and have to manually fix data on prod.
- "We don't know the schema yet." That's not "schemaless", because your application will still assume a schema, the schema is now just implicit. If you want to iterate on your schema, you can easily do so using DB migrations; this is actually easier because the DB will help you verify that you migrated your data properly when you changed its schema, as opposed to changing the schema, but still keeping data around that is invalid under the new schema.
- "Not all of our data has a fixed schema." This is now easily solvable in e.g. Postgres with JSONB columns, so you only need to fully specify the data that does have a clearly defined shape.
I once led a team creating a public-facing site based on its own custom forms/wizards. We had half a dozen internal departments wanting dozens of wildly varying forms defined, and decided we didn't want fixed schemas as future field requirements were unknown.
Rather than the corporate C#/SQL Server we went for Node/MongoDB, and we (successfully) ramped up a new system that transformed the power and cost-base of our public sector employer.
We were well chuffed, and it was deemed revolutionary (it also did cloud integration whilst pulling submissions, parsing/verifying, communicating, managing, and pushing into legacy IT systems, so not hyperbole).
And yet, despite the outcome, privately the team regretted the choice from about half-way through. I think for me the clue was when we introduced Mongoose to ... enforce a schema onto our schema-less Mongo collections and help out with the relational querying.
(Edit: I'm not saying schema-less is bad, I'm saying you need to be clear about when it's good.)
> I think for me the clue was when we introduced Mongoose to ... enforce a schema onto our schema-less Mongo collections and help out with the relational querying.
I agree that when you're doing things like that you're not really taking advantage of "schemaless", and that's probably a sign that that's not what you really want.
Hbase was a pretty solid NoSQL option by 2010, no?