rcrowley

It felt rude to take so many r8gd instances away from our customers who really love those (and i8g and i8ge).

Yes, you can certainly use up your CPU allocation on an M-10 database (at which point we offer online resizing as large as you want to go, all the way up to 192 CPUs and 1.5TiB RAM). Even still, I've been able to coax more than 10,000 IOPS from an M-10. (Actually, out of dozens of M-10s colocated on the same hardware all hammering away.)

You can get a lot more out of that CPU allocation with the fast I/O of a local NVMe drive than from the slow I/O of an EBS volume.

This will be faster than an equivalent RDS instance and will handle more of the operational lifecycle around failover and high-availability with less downtime than RDS.

Within PlanetScale's product lineup, Metal refers to the use of local NVMe drives. Nothing more. These extremely affordable sizes are indeed slices of larger boxen, though no resources are overcommitted.

Soon!

These are all three-node clusters with PlanetScale's management handling backup, restore, failover, and replication.

PlanetScale's Postgres offering is as close to plain-old-Postgres as we could possibly build.

We run on the same instance types the larger PlanetScale Metal sizes offer as whole instances. For Intel that's r6id, i4i, i7i, i3en, and i7ie. For ARM that's r8gd, i8g, and i8ge. (Right now, at least. AWS is always cookin' up new instance types.) Same story will soon be true for GCP.

We've engineered in protections from noisy neighbors in both CPU and I/O usage and we do not over-commit resources.

If your or another customer's workload grows and needs to size up we launch three whole new database servers of the appropriate size (whether that's more CPU+RAM, more storage, or both), restore the most recent backups there, catch up on replication, and then orchestrate changing the primary.

Downtime when you resize typically amounts to needing to reconnect i.e. it's negligible.

PlanetScale operates databases in AWS and GCP. There's no network latency penalty for choosing PlanetScale if you're hosting your app in one of those cloud providers (and in one of the many regions we operate in).

Correct you are.

Just want to add that you don't necessarily need to invest in fancy disk-usage monitoring as we always display it in the app and we start emailing database owners at 60% full to make sure no one misses it.

RAID isn't the answer, either, for the record. In AWS and GCP, the CPU or RAM blowing up will cost you access to that local NVMe drive, too, no matter how much RAID you throw at it.

Yes, a single disk in a VPS or cloud provider has durability concerns. That's why EBS and products like it that pretend to be a single disk are actually several. Instead of relying on multiple block devices, though, we create that redundancy at a higher level by relying on multiple MySQL or Postgres servers for durability, each with a local NVMe drive for performance.

The question isn't how many orphaned rows do you have, it's whether it matters. Databases are wonderful but they cannot maintain every invariant and they cannot express a whole application. They're one tool in the belt.

I computed P(node failure within MTTR) = 0.00007 same as you. I extrapolated this to the outage scenario P(at least two node failures within MTTR) = P(node failure within MTTR)^2 * (1-P(node failure within MTTR)) + P(node failure within MTTR)^3 = 5.09 * 10^-9 which rounds to 0.0000001%.

The short answer is that RDS doesn't run on local NVMe disks, it runs on EBS.

A lagging replica, even one that just acknowledged a semi-sync write, will return stale results if you route a `SELECT` to it.

First and foremost, the extra copies of the data are for fault tolerance. In specific circumstances they may offer some slack capacity that you can use to serve (potentially stale) reads but they're never going to offer read-your-writes consistency.

The docs you quote are a bit obtuse but the "acknowledgement" is the row event being written to the binary log. "Fully executed and committed" is when it makes its way into InnoDB and becomes available for future reads.

I think we've got (1) covered elsewhere in the comment tree.

For (2), semi-synchronous replication is a MySQL term which we realize in Postgres is by using synchronous replication with ANY one of the available replicas acknowledging the write. This allows us to guarantee durability in two availability zones before acknowledging writes to clients.

In MySQL the _semi_ part of semi-synchronous replication refers to the write only needing to be written to the binary log on the replica and not (necessarily) applied to InnoDB. This is why a MySQL database might be both acknowledging semi-synchronous writes and reporting non-zero replication lag.

Good point, Max. I glossed over the "rather than" bit. We do, as you say, write to disks all over the place.

Even writing to one disk, though, isn't good enough. So we write to three and wait until two have acknowledged before we acknowledge that write to the client.

1. I don't know if there's a single name for this. I will point out that AWS EBS and Google Persistent Disk as industrial examples of distributed, replicated block devices are also providing durability via replication. They're just providing it at a lower level that ends up sacrificing performance. I'm struggling to come up with a citation but I think it's either Liskov or Lynch that offered a proof to the effect of achieving durability in a distributed system via replication.

2. The thinking laid out in the blog post you linked to is how we went about it. You can do the math with your own parameters by computing the probability of a second node failure within the time it takes to recover from a first node failure. These are independent failures, being on physically separate hardware in physically separate availability zones. It's only when they happen together that problems arise. The core is this: P(second node failure within MTTR for first node failure) = 1 - e^( -(MTTR node failure) / (MTBF for a node) )

3. This one's harder to test yourself. You can do all sorts of tests yourself (<https://rcrowley.org/2019/disasterpiece-theater.html>) and via AWS FIS but you kind of have to trust the cloud provider (or read their SOC 2 report) to learn how availability zones really work and really fail.

Fair to say 500GB is small, especially compared to some of the folks who've already migrated, but do note that it's 15x RAM on the benchmark machines, so we really were testing the whole database and not just the memory bandwidth of the CPUs.

You don't (typically) lose the data on the ephemeral drive across a reboot but you definitely can (and do!) when there are more permanent hardware failures. (They really happen!) That's why PlanetScale always maintains at least three copies of the data. We guarantee durability via replication, not by trusting the (slow, network-attached) block device.

I did an interview all about PlanetScale Metal a couple of months ago: <https://www.youtube.com/watch?v=3r9PsVwGkg4>

PlanetScale always maintains three (or more, if you want) copies of the data and only acknowledges writes after they've been written to at least two cloud provider availability zones. Durability is provided via replication at the database layer rather than hidden in the slow network-attached block device. Most of our largest customers use Metal for their most critical data and all of them saw their 99th percentile latency plummet when they migrated.

I did an interview all about PlanetScale Metal a couple of months ago: <https://www.youtube.com/watch?v=3r9PsVwGkg4>

All our list prices are monthly and our bills are actually even finer-grained - there's no commitment to pay for a database longer than you run it.

If you do decide to operate on PlanetScale long-term, check out <https://planetscale.com/pricing> for consumption commitment discounting and other options that might make sense for your company.

Many!

<https://planetscale.com/docs/postgres/extensions>

AWS, for one example, provide a feed of upcoming "events" in EC2 in which certain instances will need to be rebooted or terminated entirely due to whatever maintenance they're doing on the physical infrastructure.

If you miss a termination event you miss your chance to copy that data elsewhere. Of course, if you're _always_ copying the data elsewhere, you can rest easy.

Good points. PlanetScale's durability and reliability are built on replication - MySQL replication - and all the operational software we've written to maintain replication in the face of servers coming and going, network partitions, and all the rest of the weather one faces in the cloud.

Replicated network-attached storage that presents a "local" filesystem API is a powerful way to create durability in a system that doesn't build it in like we have.

Neat workaround! We only started working with GCP Local SSDs in 2024 and can report we haven't experienced read or write failures due to bad sectors in any of our testing.

That said, we're running a redundant system in which MySQL semi-sync replication ensures every write is durable to two machines, each in a different availability zone, before that write's acknowledged to the client. And our Kubernetes operator plus Vitess' vtorc process are working together to aggressively detect and replace failed or even suspicious replicas.

In GCP we find the best results on n2d-highmem machines. In AWS, though, we run on pretty much all the latest-generation types with instance storage.

Glad to find a kindred spirit. I proposed PlanetScale Metal (though our CEO Sam gets credit for the name) based on how we ran MySQL and Vitess at Slack for many, many years.

We earn our durability through MySQL replication and redundancy and reap the benefits in low, predictable latency.

That does work well for environments because typically you’d run exactly the same code, maybe with different cluster sizes or instance types, in each environment. But it doesn’t work well for isolating two services where the code is significantly or even entirely different.