lizztheblizz

Hi there, PS employee here. In AWS, the instance types backing our Metal class are currently in the following families: r6id, i4i, i3en and i7ie. We're deploying across multiple clouds, and our "Metal" product designation has no direct link to Amazon's bare-metal offerings.

Lambda was a means to an end for us here, and we're not specifically endorsing its use in _this_ way. Our goal was explicitly to test our ability to handle many parallel connections, and to observe what that looked like from different angles.

We're a DBaaS company, and we do need to be prepared for anything users may throw at us. Our Global Routing infrastructure has seen some major upgrades/changes recently to help support new features like PlanetScale Connect and our serverless drivers.

From our point of view, this was a sizing exercise with the interesting side benefit that many people do happen to use Serverless Functions similarly.

Article author here. I fully endorse Aaron's correction and appreciate the call-out.

For context: I initially wrote this paragraph to include more flavor and history around crash recovery challenges with relational databases, implying that while your data might be safe, it is still 100% preferable, even today, to avoid crashes by accurately sizing AND limiting the database to live within its means. Crash recovery can still take a certain amount of time, and when people are weighing whether to bring their app up faster or maintain their data integrity, taking a shortcut in a high pressure situation is sadly not unheard of.

Alas, in my editing, I opted to spend less time in the weeds there, and without the proper context, the use of the term "data corruption" lost all meaning, and no longer belonged in that sentence. Totally fair correction.

So, disclaimer, I am not an Amazon Billing wizard, but given that we ran the Lambdas from an isolated sub account, I can be particularly certain that I was able to filter this down accurately.

We hit the one million connections total probably between 10-20 times over the course of a couple of days, and probably spent at least another 20-30 runs working our way up to it, testing various things along the way. Keep in mind, these were all very short-lived test runs, lasting maybe up to 8 minutes at the most.

Our total bill for Lambda in the month of October came out to just over 50USD.

Absolutely agreed. As others have already pointed out, there is no underlying implication of preference to this kind of application architecture. Since we do run an actual DBaaS, one of our main internal goals in running these experiments was to specifically test our Global Routing Infrastructure, and construct a scenario that allowed us to help size specifically those components for capacity planning.

As long-time DBA's ourselves, we do as much as we can to educate and empower users to architect their applications wisely... but we still need to be prepared for the worst. As it turns out, Lambda was an easy way to accomplish that. :)

Article author here, interesting question! We didn't run into that issue, explicitly.

Our setup was effectively as follows: - AWS Lambda functions being spawned in us-east-1, from a separate AWS sub account. - Connections were all made to the public address provisioned for MySQL protocol access to PlanetScale, using port 3306. The infrastructure did also reside in us-east-1. - Between the Vitess components themselves, and once inside our own network boundaries, we use gRPC to communicate.

Since the goal we set was to hit one million, and realizing we were staying just barely within the limits of the Lambda default quotas, we didn't aggressively try to push beyond that. Some members of our infrastructure team did notice what appeared to be some kind of rate limiting when running the tests multiple times consecutively. Many tests before and after succeeded with no such issues, so we attributed it to a temporary load balancer quirk, but it might be worth going back to confirm if this is the behavior we saw.

I don't have nearly the experience in Postgres environments to have seen the same level of real-world impact there, but a quick search presents me with the following documentation, which seems to indicate mostly similar performance challenges related to the use of Foreign Key Constraints: https://www.postgresql.org/docs/13/populate.html#POPULATE-RM...

Depends. Have any examples of "normal" sharded MySQL databases?

EDIT: To clarify, sharding is not a standard feature included in Community Edition MySQL. Over the years, there have been various Oracle-initiated attempts at providing it as an enterprise scaling strategy through MySQL (NDB) Cluster, MySQL Fabric, etc., but these have either ended up having limited applicability outside very specific use cases and are not widely in use.

Most large MySQL users (e.g. Facebook or YouTube) ended up rolling their own frameworks, like Vitess, which has since been open sourced and adapted to more diverse environments. Until that became more accessible, though, the rest of the world mostly made do with wobbly multi-master setups relying on circular replication, behind some kind of proxy, or had to implement the sharding logic itself into their application code.

Give it a shot, because the current implementation specifically side steps the need for that full copy, and does let you test functionally against fully up-to-date production data.

You're right, though, it doesn't cover all of those uses cases. Luckily, Vitess does offer most of that out of the box already. Just need them exposed through the PlanetScale UI. :)

12+ years of massive scale production use for Vitess and 26+ years of hardening for MySQL and InnoDB. PlanetScale adds some (imho) great features on top of that, but it's standing on the shoulders of giants that have proven themselves over and over.

... Also, I guess it's MySQL-compatible rather than Postgres-compatible? :)

Already on it. :) https://vitess.io/docs/reference/vreplication/vreplication/

To be clear, this is not a Vitess/PlanetScale-specific opinion or choice. Foreign key constraints are a bit of a controversial topic in large-scale MySQL environments in general, which is the greater context in which this design decision was made by the Vitess team.

PlanetScale's (and Vitess') non-blocking schema changes rely on open source tools for MySQL like pt-online-schema-change and gh-ost, which are widely used in production environments everywhere, and neither of them are too comfortable supporting FK's, though pt-osc does accommodate them to some extent (https://www.percona.com/doc/percona-toolkit/3.0/pt-online-sc...). gh-ost's lack of support was discussed on HN previously here: https://news.ycombinator.com/item?id=16983620

A good collection of resources on why they're considered problematic and many companies designing large-scale MySQL schemas tend to drop them can also be found here: https://federico-razzoli.com/foreign-key-bugs-in-mysql-and-m...

Vitess' compatibility with MySQL has made major leaps in the past couple of versions and the team has started focusing on locking in ongoing compatibility with various popular development frameworks. You can find those here, and more are getting added regularly: https://github.com/planetscale/vitess-framework-testing/

The basics of MySQL compatibility are described in here, though it's important to keep in mind that just because something "works" doesn't always mean it's the best way to do things in a sharded environment: https://vitess.io/docs/reference/compatibility/mysql-compati...

That is exactly what we _do_ support. Our team includes the original developer of gh-ost for MySQL, which has been built to execute these kinds of massive changes at scale at GitHub. We've integrated it tightly with Vitess. Once your branch is ready to be merged with production, it executes that change in a completely non-blocking way.

The "ease" we used to refer to in Vitess primarily relates to its interaction with the application side, where it basically presents itself as "one big MySQL datastore". It uses a standard MySQL connector, and in general, once you have the infrastructure up and running with a compatible schema design, there's not too much to worry about from a coding standpoint. Sharding happens transparently to the application code, which generally translates to fewer code changes required.

Admittedly, that view left out the considerable challenge of actually deploying and running the infrastructure, designing and optimizing that schema, along with all the joys of managing large cluster environments.

That's what PlanetScale, the product, aims to solve. Dealing with clustering infrastructure IS a hurdle for most teams to overcome, and though Vitess' feature set and compatibility has expanded greatly to accommodate some of the most demanding use cases on the web today, a lot of its functionality can still be out of reach for a developer just trying to merge some code and a schema change.

Abstracting as much of that complexity away from the end user is the goal, as well as making their lives easier with a ton of the functionality we've always wanted to see built with Vitess. I can confirm that that is not an "easy" job on our end. :)

It's closer to your original assumption than your second. Branching relies on vreplication, and it does actually allow you to develop multiple versions of your schema against your full production dataset. The "magic" is a powerful materialization engine that allows every branch to maintain its own version, and it _does_ allow you to work with your actual data. This is just scratching the surface of what that can do, though. We have loads more features cooking. :)

This is monthly. :)