Announcing Vitess 12
vitess.io
vitess.io
With vanilla MySQL there are limits to how big a database can get. If your application/dataset is continually growing, eventually you will need to shard the database. With Vanilla MySQL that entails spinning up a second primary db and teaching the application which db to route queries too. Or putting it another way, the sharding logic lives in the application.
With vitess the sharding logic lives “in the database” so your application doesn’t need to be updated.
It comes with the overhead of some middle ware but in my experience that overhead is well worth keeping the sharding complexity out of the application.
Keep in mind I’m not talking about a database with hundreds of tables getting too big but rather a small set of tables that are large enough and have enough traffic that scaling the underlying instance becomes problematic.
It's always been unclear to me what these limits are in practice. Is it a certain combination of the number of rows for table size in GBs?
That is unfortunately not true in the majority of cases. Queries of medium complexity or higher will often just fail to run on vitess. You also still need a sharding key if you want queries to be efficient. Essentially you don't want the vtgates stitching tables from different shards together.
As you scale, scatter queries become more and more problematic. You need to have carefully considered lookups, and often alter your queries to include sharding keys everywhere.
Cross shard writes and two phase commits are also a recipe for deadlocks.
Finally, you need to keep in mind that every shard you connect to uses up a connection in its pool. If you have relatively long lived sessions (like... request length) that touch a ton of shards in the same query, you can exhaust your connection pool with 'idle' connections.
Vitess _is_ amazing. I am grateful for it every day. It is not a panacea. Neither are the alternatives. Switching to Vitess is a beautiful problem to have, because you have a problem of scale. The switch is harrowing.