PlanetScale – Database for Developers
planetscale.com
planetscale.com
Speak for yourself. I love my schemas.
This article is also 100% fluff, and zero actual information. Obviously I won’t sign up to anything without an ounce of information being provided up front.
That is: how can I pair a brand new frontend preview deploy, with a serverless database with the specific schema my new feature needs?
This technology makes the whole serverless stack feel complete.
More to the point, it probably cannot solve it efficiently at all, since it is not a graph database and thus cannot be paired with a generic GraphQL resolver (would generate join queries instead of lookups across edges) and a stack of generated static queries in the backend (no need to allow generic queries, just make it possible to write queries once and only once).
IMHO, this isn't trying to solve people looping queries. The fastest way to solve this is to not loop queries.
This solves auto-sharding (well vitness did). See slack architects comment below.
When I started using git, it broke my brain. I wasn't really sure I understood it, but the idea of cheap local branches soon became the most important thing to me in a vcs, and everything became easier and so much faster. You could just work on code the way real life happens, not in some methodical pre-planned release schedule that always rubbed harshly against the reality of bug fixes and ever changing minds.
Planetscale is a lot like that transformation. We've been thinking of databases as this specialist thing, where the arcane knowledge required to make it perform well is a specialist role, and something completely untouchable by mortal engineers. While we've learned about the importance of good data modeling, we've dealt with a layer in our stack that is essentially static. Shipping features that require changes to a database schema sit and languish, because the pain and coordination required can often be too much for a team to want to deal with.
What PlanetScale has done is solve two massive problems in one platform. First, since it's built on Vitess, it Just Works At Scale. You don't need to fiddle with knobs to get it to perform well. You most definitely are not even in the top 10% of what is already running on Vitess, so you don't really need to worry about ever outgrowing PlanetScale. But the BIG innovation here is what is possible now that a database works the way the SDLC has evolved with git. Make a branch, change the schema, roll it out with your code. Just like anything else, really. Use the PlanetScale CLI to actually USE your database. Change the schema because it will make your code better, and don't worry that you're somehow going to do it wrong.
PlanetScale just made databases useful for developers beyond a nearly-static data store. It's a high-scale database that you can change like code. It will break your brain a little at first, just like git did. And then you'll wonder why the hell we waited so long to have a database this good.
One question about pricing -- I can't tell if the "free" tier costs are for one month, or ongoing. That is, do you get the free tier amounts for one month and then pay from then on, or is that level of service free from then on?
or as long as a free tier exists ;-)
I think you just described why developers liked MongoDB. It does not get a lot of love on HN, but having the DB schema map to your object model is very convenient.
And then what does PlanetScale add on top of Vitess hosted anywhere else?
Sorry, the linked blog post is both very abstract and assumes a high level of preexisting knowledge about database scaling.
Rarely is it the case that someone's actually discovered e.g. novel math or something to make the hard part easier. Better tools (to do well-understood things more easily for this use case) and restrictions (so you don't use it in ways the tools can't handle) are the usual way.
Admittedly, that view left out the considerable challenge of actually deploying and running the infrastructure, designing and optimizing that schema, along with all the joys of managing large cluster environments.
That's what PlanetScale, the product, aims to solve. Dealing with clustering infrastructure IS a hurdle for most teams to overcome, and though Vitess' feature set and compatibility has expanded greatly to accommodate some of the most demanding use cases on the web today, a lot of its functionality can still be out of reach for a developer just trying to merge some code and a schema change.
Abstracting as much of that complexity away from the end user is the goal, as well as making their lives easier with a ton of the functionality we've always wanted to see built with Vitess. I can confirm that that is not an "easy" job on our end. :)
Isn’t that the same for a normal sharded MySQL database though?
EDIT: To clarify, sharding is not a standard feature included in Community Edition MySQL. Over the years, there have been various Oracle-initiated attempts at providing it as an enterprise scaling strategy through MySQL (NDB) Cluster, MySQL Fabric, etc., but these have either ended up having limited applicability outside very specific use cases and are not widely in use.
Most large MySQL users (e.g. Facebook or YouTube) ended up rolling their own frameworks, like Vitess, which has since been open sourced and adapted to more diverse environments. Until that became more accessible, though, the rest of the world mostly made do with wobbly multi-master setups relying on circular replication, behind some kind of proxy, or had to implement the sharding logic itself into their application code.
Thanks for the correction!
I wouldn't say there's too much "magic" in there, but it does a lot of known difficult things (schema management, sharding/resharding, connection pooling, query optimization, DB administration, monitoring, backup/failover) which are generally painful and expensive to do yourself.
The assumption that tenants are perfectly isolated is actually the original sin of early Slack infrastructure that we adopted Vitess to migrate away from. From some earlier features in the Enterprise product (which joins lots of "little Slacks" into a corporate-wide entity) to more post-modern features like Slack Connect (https://slack.com/help/articles/1500001422062-Start-a-direct...) or Network Shared Channels (https://slack.com/blog/news/shared-channels-growth-innovatio...), the idea that each tenant is fully isolated was increasingly false.
Vitess is a meta-layer on top of MySQL shards that asks, per table, which key to shard on. It then uses that information to maintain some distributed indexes of its own, and to plan the occasional scatter/gather query appropriately. In practice, simply migrating code from our application-sharded, per-tenant old way into the differently-sharded Vitess storage system was not a simple matter of pointing to a new database; we had to change data access patterns to avoid large fan-out reads and writes. The team did a great write-up about it here: https://slack.engineering/scaling-datastores-at-slack-with-v...
> Today, we serve 2.3 million QPS at peak. 2M of those queries are reads and 300K are writes.
I think the "today" QPS numbers are still doable with a properly tuned single-writer galera cluster running on machines with TBs of memory. Of course, with Slack workload, there would be too much historical data to fit into a single host, so I can see the reasons to shard into multiple clusters/hosts.
Still, the numbers seem a little off. Let's say back in fall 2016 there were already 200K write QPS at peak, with 200 sharded hosts accepting write. That's just 1K write QPS at peak per host on average, and let's say 20K write QPS at peak for a particularly hot shard. What could be the bottleneck? Replication lag? Data size? I don't think any of the articles from Slack has talked about this.
What Vitess provides is invaluable, especially the very solid implementation of secondary index. But sometimes I feel like it is being used/advocated as a sledgehammer ("just keep sharding") without looking at what could be done better at the lower MySQL/InnODB level, in exchange for a much more costly cloud bill.
CockroachDB (aka CRDB) is completely custom and compatible with Postgres wire/datatype protocols, which operates directly on its own key/value store called Pebble (but originally was also RocksDB): https://www.cockroachlabs.com/blog/distributed-sql-key-value...
Both systems are foundationally the same SQL-on-KV but implement it very differently.
You always need a lot of attention to sharding otherwise you'll have poor performance.
Implementations vary but there are the 2 major architectures: systems like Vitess/Proxy SQL/Citus/Timescale that act as a proxy layer on top of existing RDBMS running on multiple servers to make them look like a single database, and entirely custom projects like CockroachDB/TiDB/Yugabyte/Cloud Spanner which have their own native processing and data layers.
OLAP relational data warehouses like Vertica/Greenplum/MemSQL/Redshift/Bigquery are also natively distributed but focus on large-scale analytics with features like column-oriented storage and vectorized processing.
And it is the first truly serverless relational database offering that I am aware of. The cost scale to 0, so it is perfect for small projects, but that same instance will scale to support massive load when you need it.
So is this a wrapper around managing Schemas powered by Vitesse? (Btw, had to go to your github to figure that out)
If you want to be the database for developers you should know that developers do care about how you do this scaling.
> PlanetScale's Non-Blocking Schema Changes' workflow doesn't support FOREIGN KEYs in users' databases.
> PlanetScale determined that the production safety that Non-Blocking Schema Changes provide are worth this technical tradeoff. Learn more.
https://docs.planetscale.com/concepts/nonblocking-schema-cha...
PlanetScale's (and Vitess') non-blocking schema changes rely on open source tools for MySQL like pt-online-schema-change and gh-ost, which are widely used in production environments everywhere, and neither of them are too comfortable supporting FK's, though pt-osc does accommodate them to some extent (https://www.percona.com/doc/percona-toolkit/3.0/pt-online-sc...). gh-ost's lack of support was discussed on HN previously here: https://news.ycombinator.com/item?id=16983620
A good collection of resources on why they're considered problematic and many companies designing large-scale MySQL schemas tend to drop them can also be found here: https://federico-razzoli.com/foreign-key-bugs-in-mysql-and-m...
> Developers want the durability, stability, and scalability of a SQL database but do not want to be constrained by managing a schema. It has been our goal to give both, not compromising on the power of your datastore but making changes feel as easy as deploying code.
Sorry. There’s always a schema, and learning to manage it is a _good_ thing. If you’re not ready to manage and migrate, then you’re not in production. You’re in something else.
In my past experience as a production engineer, and in our many discussions within the community, we've seen many approaches developers take just to avoid the hassle of running yet another ALTER TABLE; whether because it requires going through a different team, or because this poses an availability risk, etc. PlanetScale offers to bring ownership back in the developer's hands.
Hope this clarifies. We'll publish more soon, and I'll be able to provide more links.
(I'm an engineer at PlanetScale)
(Edit for grammar/typos)
Given the number of people—on HN, no less—who have war stories about having accidentally dropped the production database because of insufficient protections and the desire of many businesses to know that their vendors are following SOC2 and similar safety and security protocols, I completely get that.
In the latter case (availability risk), there’s no sympathy I can provide. Developers need to be aware of what their changes mean. Sometimes this means that you make a separate table. Sometimes it means you have a downtime event which means an overnight deploy. Sometimes it means you have to figure out how to do a zero-downtime series of migrations.
Step 0: make sure you need the migration. Step 1: modify the app to conditionally query or update the new column based on whether the change is present or not, and without requiring a value in the column. (Since it’s not, this will work as an `if false` case for now.) Deploy. Step 2: modify the database to add the column with no constraints (safe and trivial on most SQL databases). Deploy. Step 3: modify the app to start filling in fields that should not be null. (That is, provide constraint validation at the application level; or, if dropping a column, start setting the value to a least damaging value or NULL if you could have made a required field no longer required.) Deploy. Step 4: After some time, run a backfill to fill in new required values (or clear old values) that haven’t been modified yet. If you’ve done this right, it should be a small number. Optionally, do this periodically for small subsets of data. Step 5: Run a final backfill and deploy a schema migration that adds your constraint/drops your column (dropping a column is optional if you’ve got a hard zero downtime requirement; just make sure no one tries to use it and make sure it’s not part of the application schema; schema comments are your friend). Step 6: Deploy a version of the app that doesn’t act conditionally.
Yes, it takes longer. I’ve run migrations this way for the last seven years and we’ve had essentially _no_ downtime because of migrations. I think we’ve had ~4 hours of preventative downtime across ~10 databases in that time. Usually, when we’re ready for the piece that can cause downtime…there’s about a five minute hiccup.
Developers who can’t reason through safe table alterations shouldn’t be making those changes, and PlanetScale is going to make some business people very unhappy when their developer changes something that they didn’t actually understand and causes downtime regardless of PS’s “guarantees”.
With sharding, where this stuff eventually gets you is when your assumptions about shard keys no longer hold.
Eg, you might have user initiated traffic to start with, so you can easily and automatically shard everything by user id or whatever. Then one day, those assumptions change because you might have to accommodate event driven traffic, ie not request/response, and the user’s id can’t be assumed to always be present. For example let’s say something in the real world causes an event to be pushed onto your queue. That event could correspond to a real user, but since it originated somewhere in the real world, there’s probably a separate id for how that user is represented. So you can’t rely on that user ID being present to shard things by.
Not sure, if that makes sense, but sharding can be hard. It’s not like free, and I still think it’s important for engineers to understand the mental model they’re using, even with a tool like vitess.
Also, I saw claim either on planetscale or vitess that MySQL has no native support for horizontal scaling with automatic sharding, but I think they do? I think you just have to pay for that though.
Also, cross-shard transactions were mentioned as another difficulty with sharding. They can be done with either sagas (depends on the context but it’s a design pattern), or 2PC which is available in MySQL > 5.8 I believe in the form of XA Transactions.
Ironic coming from the infinitely scalable database, isn't it?
Basically a Hosted Vitess with MySQL? Where is DC? Backup included? Redundancy? Spending Cap? Own Infrastructure or on top of other Cloud ? Support Level? Uptime Guarantee? etc etc.
All of these are basic info required from a SaaS, and they are missing. Not even a FAQ.
As for the rest? Very good questions. Just coming out of beta, so perhaps they're still filling out the website.
Thanks for the info on Amazon. I guess I will have to check it out again when they announce it is out of beta.
Doesn’t look like there’s Postgres compatibility.
Is it yet another example of vendor lock-in? Yes.
For small/medium businesses, sometimes it's better just to let the experts do their thing and build on top of it...
It's more like a clever trap, first hit is free etc.
Will be interesting to follow this. I'm not db savvy so I don't mind leaving that job to someone else.
"Vendor lock-in" means that you're tied at a business-critical level to some proprietary technology _that you can't get anywhere else_.
If you rent Linux servers, Postgres databases, or even k8s setups from Azure, there's no shortage of alternative vendors willing to sell you compatible product should you tire of MS, and ideally you won't need to change much more than a few endpoints. However, if your entire user base lives in Azure AD, it's a very different story.
Am I stupid or misunderstanding something fundamentally here? Is just a hosted DB or something like that?
[EDIT] on reading further in their docs, my suspicion is that their "branching" concept is a hell of a lot more limited than I believed at first. I initially took it to mean you could have multiple active schemas working on your data at once—instead, I think it's more like exporting just the schema of your DB and importing it to a fresh DB, which is nothing new and doesn't run into all kinds of operational and security issues the other workflow would. I'm fairly sure all the actual magic is in the schema diffing, and the docs make me think even that isn't as fully-magical as one might hope.
Where are you hosted? What latency should we expect from AWS, GCP, DigitalOcean and fly.io?
And also what degree of compatibility should we expect with MySQL? The doc is quite sparse on this.
The basics of MySQL compatibility are described in here, though it's important to keep in mind that just because something "works" doesn't always mean it's the best way to do things in a sharded environment: https://vitess.io/docs/reference/compatibility/mysql-compati...
I like the cool things but I can’t migrate to MySQL just because of this.
On the one hand, making light of how big an undertaking a _database engine migration_ would be makes you come off sounding like the "mongodb is web scale" guy. That's a pretty terrible look for a company proposing to take on critical production infrastructure.
On the other hand, adding postgresql support to vitess is probably a _massive_ undertaking. Sure, it might make sense for you to contribute to in the long run to open up that customer segment, but at your current stage postgresql customers are probably more of a product-distraction than anything. In that light, driving us away for now is probably the best policy.
I'm gonna give you the benefit of the doubt. Well played, sir. Well played.
EDIT: nevermind, all your other comments so far in this thread are variations on "Yes, we can do that!". You're promising the moon, and that's fishy.
Bit of unsolicited advice (you know what they say about unsolicited advice, but here goes...): You'd be better off admitting some weaknesses and discussing tradeoffs. For examples, take a look at other HN threads where execs have hopped on. What qualities do you seen in posts that get the most positive responses? Now compare that to the confusion and skepticism in this thread. Your messaging has landed off-target.
You're proposing to take over a piece of people's critical production infrastructure. As a potential buyer considering an infrastructure provider, seeing "buy our product and we'll solve all your problems" just makes me skeptical. It makes you sound like a used car salesman. Your landing page delivers that basic message, and your comments in this thread reinforce the resulting impression.
Conversely, if in these comments you soberly acknowledged your limitations, transitioned to a detailed treatment about tradeoffs, and _then_ used that save to toot your horn some more, that would give the impression you've considered this problem deeply, help me figure out how closely your value prop is aligned to my goals, and make me more inclined to trust you, your company, and your product.
As it is...
I see your comment and it makes me think your company isn't ready to take on my infrastructure (this is my database we're talking about here... if you're not taking something as basic as migration costs seriously, what other nasty surprises are waiting for me down the line?).
Then, I take a look at your about page (maybe this guy's just an entry-level marketer in the people-pleaser phase), and I see that you, Sam Lambert, are the Chief Product Officer. That gives me some serious doubts about the future viability of your company, because now I'm worried the CPO is fundamentally out of touch with the userbase and can't grapple with difficult details.
You will not be touching my data any time soon, but I'll be watching for changes, and I honestly wish you luck. I like your vision, but your messaging execution needs work and your product execution is yet-unproven.
EDIT: kudos to lizztheblizz for giving more detailed answers that inspire confidence. You've significantly repaired my impressions of your company.
We concern ourselves with the productivity of our users. We also solve hard problems at scale, for many large companies. We have so many exciting developer focussed features that I think people will love, all the while being on a viable database. Vitess is by far the highest scale open source database solution out there.
Migration cost is a serious thing. We believe the immense leap in productivity you will gain over time from our product makes it more than worth it.
It looks like the "branching schema" is essentially:
CREATE DATABASE branch_name WITH TEMPLATE main;
I might actually have to try this - I hadn't really thought template dbs would be all that useful in pg - but this branch and test för dev use-case is interesting.
Then there's COPY (for data) and rename to "promote" a branch with data (one would probably want to run DDL on main db though:
ALTER DATABASE branch_name RENAME TO main; -- would have to move old main out of the way first - but might be possible in same transaction?
Very impressive tech! But the upside is not worth the vendor risk for me.
Edit: or maybe this is not an issue at all - but the website copy does not make it clear, so take it as a suggestion please.
$1.25/mo per 10GB storage
$15/mo per 100 Million rows read
$15/mo per 10 Million rows written
But I won't lie, super excited to give this a try.
It seems to be the case with all the NewSQL databases that they're still just not economical to run unless you are FAANG. Get a Bigtable / HBase / etc and build a custom system and save $50-350K per month. For a sub 100 person company, it's a no-brainer.
Scaling down to 0 opens up a world of opportunities when it comes to team development workflows. The idea of quick environments per pull request is within reach.
Not to mention that it scales with you.
The one thing I'm curious about is how it compares with CockroachDB
... Also, I guess it's MySQL-compatible rather than Postgres-compatible? :)
I already played around with Upstash and Fauna.
Somehow the signup shows an 422 error, then I get an confirmation email that leads to a blank page.
I want it for my next project, because I don't want to pay for what I don't use, but also don't want to provision or maintain additional instances manually.
You're right, though, it doesn't cover all of those uses cases. Luckily, Vitess does offer most of that out of the box already. Just need them exposed through the PlanetScale UI. :)
Does a full table scan count as a read? Is there planetscale specific tooling for managing indexes or primary keys?
Using databases outside the same availability zone can be slower...