https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... (2013 post had much less redundancy, but even their 2016 architecture is pretty undistributed in terms of being able to recreate everything from the single source of truth database)
The problem in MySQL we eventually ran into was the read replicas fell behind master during peak load (by 30-60 minutes depending upon the day). So we still had to hit the master for certain queries. I left before we took the next step to fix this issue.
- Switching from a car to a van to a lorry is fairly low cost. You don't need to recreate your product (probably).
- You don't need to run distributed databases in a cluster to start.
But I think most importantly the decrease in dev speed and performance is an investment in future scalability. And I only imagine that this different will shrink over time to where for example a 1 node "distributed" DB isn't that different to work with than a 1 node "traditional" DB. And that small difference pays off because adding a second node doesn't have huge tradeoffs.
I agree that right now it doesn't make sense. If I was starting a new product I would still fire up a PostgreSQL instance. However I think that the day will come where the difference is small enough and software support is complete enough that we will start with something distributed, much like people don't often start with SQLite today, even though it is quicker to get going (also not a great comparison because it has less features).
It makes sense only if you'll ever need this scalability. And you take a hit on other fronts too: Infra cost, Deployment complexity. And both deployment complexity and code complexity also increase QA cost, instability, product and company reputation.
>> much like people don't often start with SQLite today
Maybe they should start with SQLite by default
This CRDB engineer respectfully disagrees. This thread takes it as a given that a non-distributed DB is better if you don't need to scale up (i.e. if you run a single "node"). Let me offer a couterpoint: it's easier to embed CRDB into some software you're distributing than it is to embed Postgres. This is to say, we do try to compete at every scale (well, perhaps not at SQLite scale). CRDB doesn't have all the SQL features of PG, but it does have its pluses: CRDB does online schema changes, it is generally simpler to run, comes with a UI, comes with more observability, can backup to the cloud, can be more easily embedded into tests.
Online schema changes are a big deal; the other thing that I hope will help us win small-scale hearts and minds is the ever-improving observability story. I hope CRDB will develop world-class capabilities here. Other open-source databases traditionally have not had many capabilities out of the box.
You can run it either as an embedded DLL or as a standalone server, and changing between the two is often just a connection string change; It seems to be fairly popular in the POS space for this reason.
CockroachDB understandably wants you to use their Cloud or Enterprise products: the OSS version is quite limited. For example it doesn't support row-level partitioning (https://www.cockroachlabs.com/docs/stable/configure-replicat...). Which means, if I understand correctly, it is not of much help for scaling writes to a single big table.
If you are only looking to scale write throughput then manual partitioning is not be needed. This is because CRDB transparently performs range partitioning under-the-hood on all tables, so all tables scale in response to data size and load automatically. If you are interested in learning more, https://www.cockroachlabs.com/docs/stable/architecture/distr... discusses these concepts in depth.
There is literally nothing preventing distributed databases from having excellent scale-up performance too. Unfortunately, people who design distributed databases have a strong bias toward unnecessarily throwing hardware at performance and scalability problems. This is partly because very few people know how to design a modern scale-up database; making something (trivially) distributed is easier.