They claimed they successfully onboarded a customer with 1 PB of data at KubeCon NA.
I have strong doubts about distributed DBs in general, but I also can’t see them blatantly lying in a talk.
I have strong doubts about distributed DBs in general, but I also can’t see them blatantly lying in a talk.
I think people’s idea of scale and operating at scale is limited to their experience.
You can get MySQL to run at any scale, look at Meta and Shopify. Operational complexity at that scale is a different story.
Distributed databases reduce a lot of the operational complexity. To take one example:
Try a DDL on a 5 TB table in any replicated MySQL topology of choice and compare it with TiDB’s Distributed execution framework.
That’s a new one to me; never have I ever heard someone claim that making a system distributed reduces complexity.
I’ve operated EDB’s Distributed Postgres at a decent scale (< 100 TB of unique data); in no way did it enable reduced complexity. For the devs, maybe – “chuck whatever data you want wherever you want, it’ll show up.” But for Ops? Absolutely not, it was a nightmare.
Most tech companies have poor knowledge of proper data modeling and SQL, leading to poor schema design, and suboptimal queries. Combine that with the fact that networked storage (e.g. EBS) is the norm, and it’s no wonder that people think they need another solution.
The amount of QPS you can get out of a single DB is staggering when it’s correctly designed, and on fast hardware with local NVMe disks (or has a faster distributed storage solution). Consider that a modern NVMe drive can quite easily deliver 1,000,000+ IOPS.
> Consider that a modern NVMe drive can quite easily deliver 1,000,000+ IOPS.
There could be other bottlenecks, for example I am consistently experiencing that linux kernel doesn't handle memory pages allocations fast enough once disk traffic hits few GB/s, because it does it in single thread.
Now go to OLAP, and a single query might be doing multiple table joins. It might be scouring billions of records. It might need to do aggregations. Suddenly "millions of ops" might be reduced to 100 QPS. If you're lucky.
And yes, that's even using fast local NVMe. It's just a different kind of query, with a different kind of result set. YMMV.
But yes, OLAP is of course its own beast, and most DBs are suited for one or the other.
The organization might say "Okay. Maybe you should do your ad hoc exploration on an OLAP system. Preferably our data warehouse where you can let your report run for hours and we won't see a production brownout while it's running."
So complexity of ad hoc joins in the warehouse generally can get more complex.
Which means interest in Postgres (specifically) is only 9.13% of overall interest in databases; MySQL another 14.04%. Combined 23.27%.
Is that a significant percentage of interest? Yes. Many others are a fraction of 1% of mindshare in the market.
Yet the reason there are 423 systems ranked in DB-Engines is because no one size fits all data, or data query patterns, or workloads, or SLAs, or use cases.
PostgreSQL and MySQL are, at the end of the day, oriented towards OLTP workloads. While you can stretch them to be used for OLAP, these are "unnatural acts." They were both designed in days long ago for far smaller datasets than typical for modern-day petabyte-scale, real-time (streaming) ingestion, cloud-native deployments. While many engineering teams have cobbled together PostgreSQL and MySQL frankenservers designed for petabyte-scale workloads, YMMV for your data ingest, and for p99s and QPS.
The dynamic at play here is that there are some projects that lend themselves to "general services" databases, where MySQL or PostgreSQL or anything else to hand is useful for them. And then there are specialized databases designed for purpose for certain types of workloads, data models, query patterns, use cases, and so on.
So long as "chaos" fights against "law" in the universe, you will see this desire to have "one" database standard rule them all, versus a Cambrian explosion of options for users and use cases.
I’m not saying it fixes everything, but knowing how a DB works, and applying proper data modeling and normalization could severely reduce the size of many datasets.