I Don't Want to Shard (MySQL)
hackmysql.com
hackmysql.com
Sadly, hardware has limits, especially if you use cheap(er) virtual hardware. OTOH you can see Stackoverflow pulling just this trick: a single colossal SQL server handles most of the load [1]. They, of course, run two DB servers because they need a hot spare.
[1]: http://highscalability.com/blog/2011/3/3/stack-overflow-arch...
But as the article mentioned, NewSQL is already there. DBs in this category has actually been awhile. I think Spanner's paper was 10 years ago. And it is ubiquitous across Google. So let's just accept that we should think things differently in 2022. Distributed RDBMS is already a thing, used in production, in many companies. Like TiDB(https://github.com/pingcap/tidb) mentioned in the article, Square already uses it to replace some MySQL's use cases (https://www.youtube.com/watch?v=TjqL50qzy3A).
Ok, but it still makes me wonder how far you can go today with a good old-fashioned RDBMS, running native on modern server hardware. I believe you can go a lot further than most developers seem to think, without having to go to more complex distributed setups.
In the environments I work in, it seems people often first check on scaling horizontally instead of checking how far they could scale vertically... my preference is to first scale vertically before trying to scale horizontally. (I am aware both have different advantages, but I'm mainly thinking from a performance point of view.)
Many services will not even approach the point where "unlimited cores and disks" becomes a concern.
> So let's just accept that we should think things differently in 2022. Distributed RDBMS is already a thing, used in production, in many companies.
I do not accept that. Why would I take a critical piece of infrastructure and layer on distributed systems problems if I don't absolutely have to?
Searching through the mentioned TiDB, it's quite interesting to find through its affiliated site https://ossinsight.io/analyze/pingcap/tidb, that the popularity (stargazer) seems pretty high in China. The US stars is not bad, comparable to Mongo's.
Honestly I've not heard about it much. How much of the load these databases can support up to, with a single cluster ?
"in this application, data increases by 8 billion records and 1.5 TB every day."
Imo the biggest challenge of rdbms in modern saas is hybrid (oltp + olap) query pattern. Any cloud managed service that is going to nail this in a costly efficient manner will become the next mainstream thing.