Making 768 servers look like 1
planetscale.com
planetscale.com
What about foreign keys? Do they all have to live on the same shard? How do you do distributed transactions?
On cross-shard reads: how do you do sorting? And cross-shard joins?
I’d love to be proven wrong, but I suspect the 768 servers look like 1 only on the very surface, and you’ll get wildly different characteristics from cross-shard and single-shard queries.
I personally would prefer if they _didn’t_ look like 1 if they can’t behave like 1.
Global services using relational dbs typically severely restrict queries that run against the cluster. So no joins, no intervals, no grouping, etc.
Transactional queries are usually limited to something like "get a single record, preferably from cache". For many typical web services this can go VERY FAR. Only a handful of global services needs more than a few dozen database servers and a caching cluster. In fact, i have seen major businesses running off a pair of very big postgres instances.
Analytical stuff is extracted into dedicated storages optimized for throughput, like Snowflake or Redshift or BigQuery.
At what point does hiding the sharding become counterproductive? I imagine teams still need a fairly deep understanding of shard keys, query routing, and failure modes to avoid accidentally expensive cross-shard operations.
Distributed systems introduce severe restrictions on what can be reasonably done at scale. Having a single connection string is one thing, being able to do a massive JOIN is another (and should only be ever done in analytical databases).
The question is not "when sharding becomes counter-productive" but "when it starts making sense".
With sharding something somewhere has to know how to route queries to subsets of data. So it is a complexity price paid for being able to scale. If one can avoid paying this cost then he should.
And USUALLY cross-shard queries are not just expensive but simply impossible in operational clusters. Like, if you do COUNT on a table, you only count within a single db shard table.
The point is that it’s a leaky abstraction. It should not be at the database proxy level.
Instead, the application should be responsible to route queries to the correct shard. That way it can’t make cross-shard queries or cross-shard transactions accidentally, it must do so explicitly.
simply wrong
"Get a row" "Get ten rows"
Etc, etc. All constant, all predictable, all not requiring unpredictable time to run. Also caches nicely.
What is "simply wrong" here?
https://vitess.io/docs/faq/sharding/advanced/
Looks like it's not drop-in, you need to heavily modify your data and indexes structurally to make sharding performant. Cross-shard joins are possible but need to be designed for explicitly (to be fair, indexes also need to be designed explicitly in regular SQL). IDK if a drop-in solution is possible or if this is an information-theoretical limitation.
Why not divide ID range (63bit?) by the maximum planned number of shards and then set on each shard it's min/max value so ranges will not overlap?
This might be the perfect case to use an UUIDv7 as a key instead of a number.
But the use-case for this is mostly if you do not need, or have very limited use for anything cross-shard.
https://planetscale.com/blog/making-768-servers-look-like-1#...
This decision matters a ton. I drilled into this specific point on sharding in one of my articles from awhile back:
https://planetscale.com/blog/database-sharding
This is something you face any time you shard data, no matter the system. Single-shard queries are preferred to cross-shard ones. We want 99% of what we do to be single shard, but occasionally cross-shard work is unavoidable.
Why optimize for the 1% and put the logic in a database proxy, then? It’s a leaky abstraction.
Require the application to explicitly and loudly make a query for each shard and collate the results.
In another usecase where joins are not needed realtime, you can build another system where results are precomputed and cached. I consider sharding as methodology for scaling instead of onefit all solution. Overall it depends on usecase and sound sharding strategy for performance.
> A single database server cannot handle such demand, so we must spread the queries and data out across many servers with database sharding
Did you max out the capacity of the best server you can buy?
Such a database can serve millions of customers (the numbers given).
You always want to scale up the other parts first, request handlers, caching, etc. The day you can no longer inspect the essential state of your system is the day your company better be included in NASDAQ and ready to pay a few hundred engineers 300k salaries.
There are truly customers that bigger machine no longer cuts.
Disclaimer: I'm an Engineer at PS.
I worked at a startup that had _money_ - a lot of it - flowing in. Stakeholders wanted to see growth, which meant features. They did not want to see us making platform improvements; those didn't show well to clients. Best throw money at the problem, rather than sacrifice the all-important features.
We had a number of architectural quirks that reflected this. One was that we ran our databases in AWS on ephemeral instance store. Yes, the kind that doesn't persist when you stop or lose the instance. The kind that you should never run a database on. We did it so that we could join them eight of them together in RAID 0 on the biggest instance types, and pull 400,000 IOPS to the disk that way. We paired this with hot spares of each shard, as well as hourly backups.
We did this to support a Rails / ActiveRecord data model, for data that should never have been part of a Rails data model. We had exactly one person working part-time on query optimization.
The maintenance burden was immense. The AWS bill, even more so. But again, we are hypothesizing that we could host all of this data on a laptop.
To an extent, maybe up until our thousandth customer or so, I actually think we could have. But we wouldn't have spent most of our time working on _features_ - we'd have instead spent most of our time trying to performance optimize. That would've meant the investors wouldn't have seen growth, which in turn would've highly impacted our success. We had a well defined domain, and optimizing the queries for performance was not it.
It's unrealistic to expect a startup to build a product (or at least it was prior to LLMs generating a lot of boilerplate) with best practices in place.
You seem to have misunderstanding on what scales some things run at, and expense curves of getting them to run at those levels.
A poorly built app running on hundreds of servers can be massively expensive. A well built app running on a single server can also be massively expensive because people that can build apps like that tend to command whatever price they want under the banner of server cost savings.
Can you build software this efficiently? If you can't, then being an armchair expert about it helps noone. If you can, you're undercharging your clients.
But I hadn't considered this, so thanks for pushing back. Good to keep in mind their incentives.
I will say, since their product is a proxy whose interface is a single SQL connection, you should in theory be able to do dev queries through that black box, much the same as application queries? What is so scary here that it would require a hundred engineers?
Because it’s a leaky abstraction which is trying to make guarantees over network connections which are extremely difficult to make within the same kernel.
A few questions I would start with:
- is the system even ACID compliant?
In my reading of this article, no.
- is my sql feature set limited? Will it enforce all constraints? Or are their cross-shard limitations?
For example the article doesn’t discuss transactions or how they would roll back, or how they guarantee a consistent view of data.
Next:
- how does it respond to error?
- how does it respond to load?
This is a complex system and complexity breeds bugs. But now you don’t have the tools or procedures to investigate those bugs because you can’t poke the system at desk with tools. You can’t run experiments; you can’t even see all the data.
It looks like it does support transactions, but they basically destroy the performance benefits: https://vitess.io/docs/faq/sharding/advanced/can-i-use-vites...
The more I read the more I'm struggling to understand the benefit of a router like this that sits on top of a monolithic SQL, vs a truly distributed DB like cockroach.
Like you I'd love to learn more about the internals of their actual SQL engine, which is just barely touched on in the article. The idea of ripping out the layer of a SQL interpreter that does just enough to route it to a real server tickles my brain in the same way as when I learned how node.js ripped the js interpreter out of a browser.
Unfortunately the SQL language is the simplest part of the database. The concurrency and consistency guarantees are the key technology,
Even then I would probably use those resources to optimize software instead.
In the physical world of trucks and cranes no company would make that mistake to try to save 30-80k.
When you start calculating things that are not just the server, the single server looks cheaper and cheaper. How do you get a consistent backup? How do you do DR? How do you tune queries when it could go to this node or that node? Now writes are going to be significantly slower if you need multi-node commit because no matter what you are racing the speed of light on the network.
It's a scale that transforms a 10 year MTBF to a ~5 day MTBF.
€84.70 max. per month €0.1357 per hour CPU Intel Xeon E5-1650V3 RAM 256 GB Drives 2 × 6.0 TB Enterprise HDD Location #FSN1-DC1 Information IPv4 ECC iNIC
cashapp: https://code.cash.app/planetscale-metal github: https://github.blog/engineering/infrastructure/partitioning-... etsy: https://www.etsy.com/codeascraft/migrating-etsyas-database-s...
These companies could not realistically operate off a single database server.
I want to see it fail first. I suggest their org chart has more to do with their architecture than database performance.
Even so you’re in the category I said. Hundreds of expensive engineers maintain this stuff.
- loss of atomicity and transactions.
- tons of custom tooling to work around limitation of custom sql (knowing which operations are allowed and consistent).
- GitHub was able to operate for almost a decade on a single MySQL instance, and it’s not clear they really hard to change.
Etsy said this was a huge problem and took 5 years and thousands of PRs to resolve.
your lack of experience at scale doesn't really invalidate this architecture
Your sense of scale is entirely artificial and cargo cult. And you’re committed to selling that problem.
Have you tried maxing out the server yet? Got a write up about it?
While cashapp may actually need sharding, there are so many companies who are overpaying for shitty performing network storage databases.
Not to mention you are subject to cloud provider networking and compute allocation code. They change it. Big slow network storage DB suddenly gets even worse and you don’t have leverage to have it fixed on their end.
I am not suggesting you do this but some businesses can actually be down a little bit, like my non-IT one. I've had issues in the past with email. If the issue happens after business hours and resolves before start of business next day, nobody will even notice.
p.sI thought it was a GIF, but it's an iframe. That was a nice little surprise.
It really feels like the best way to learn is by studying other people's code.
As for caching strategy, cache invalidation issues do pop up from time to time. Since the filenames usually include a content hash, I'm guessing they're using a Vite style caching strategy. Cache invalidation might almost never be a problem, but you still plan for it just in case.
Technically, they are using js + gsap + svg embedded i the article with iframes.
Process-wise, I drafted most of them as static images in excalidraw, passed the images along to cursor for a first draft, applied styling rules, and then did a bunch of fine-tuning.
Err, do they? For what percent of real world use cases?
The database can scale to handle more traffic by adding replicas. An extreme example of this is OpenAI's use of 50 replicas on a single Primary.
So an extreme example is OpenAI needing 50 replicas, but we're doing five blades ... err, we're doing 768 servers because the need arose "pretty quickly"?
When we needed to store a petabyte of data (one million gigabytes), we'd need many more shards
For who? The United States government? How many end-users are running 1PB Postgres database on DBaaS?
If you read the OpenAI article, you'll see that they actually used sharding to offload a bunch of work from their "1 primary 50 replicas setup"
>>> "To mitigate these limitations and reduce write pressure, we’ve migrated, and continue to migrate, shardable (i.e. workloads that can be horizontally partitioned), write-heavy workloads to sharded systems such as Azure Cosmos DB..."
The 768 servers and 1PB example is just one of many configurations. A business with 10TB may choose to go from a monolithic database to a 8-shard setup to improve backup times, eliminate single-point-of-failure, have more breathing room for scaling.
I'm old enough that this is not true.
Technically, they're powered by js + gsap + svg.
Process wise (for most of them) I sketched them out in advance in excalidraw for figure out layout, then passed these along to cursor to have it build out an initial draft from the image, then used some styling rules to get all the styles inline, then did a bunch of fine-tuning.
there's a reason why postgres writing is (mostly) serialised (asterisk) to a single writer (asterisk asterisk). something something ACID, but in short by having multiple writers improves availability, but weakens integrity.
The author - certainly not deliberately! - says some untrue things about relational databases. The most important one is this:
> To understand why sharding is a necessary part of scaling relational databases...
But sharding isn't a necessary part of scaling relational databases. It's really just a requirement of simple databases like Postgres and MySQL.
If you want a relational database cluster that really does make 768 servers look like one, then you want what Oracle calls RAC ("real application clusters"). Your cloud will probably rent you access to one under a name like Autonomous Database. Self hosted it may be called ExaData, which is a unified hardware/software "rent a rack" style offering. You may be surprised to discover that it's not much more expensive than many managed Postgres offerings.
RAC can scale a non-sharded relational database horizontally. That means all queries can access all data, all SQL features like sequences and joins work, any server can take part in transactions with any other and in general it looks exactly like a really big single machine would. In other words, it's a synchronous multi-write-master system.
RAC scales very well. 768 servers is well within reach as long as your query patterns scale too i.e. don't all contend on writing to one row. Behind the scenes it uses a dedicated high speed RDMA network with lock coordination to transfer data blocks directly between nodes, never hitting disk for memory that's already in the buffer cache.
Additionally RAC is fully HA and supports rolling upgrades of the cluster whilst live. You don't need any NLBs or routers either. The client drivers automatically discover and load balance between nodes without needing intermediaries, transactions can start on one node and fail over to another without applications noticing, and so on. There's plenty of opportunity for caching and replication. You can run asynchronously replicated failover clusters, run multiple clusters in a Raft-driven globally coordinated super-cluster and can deploy coherent read-through caches anywhere; the main clusters will inform them the moment data in them becomes stale.
In other words, it can do a lot.
If for some reason you do need sharding then that's also supported with features like automatic sharding key distribution to client drivers that transparently route queries correctly, but most apps don't need this.
In case you're wondering why I say all this, firstly, obviously, I have a financial conflict of interest. But the database hasn't driven Oracle's stock price for a long time, so it's not a big one. These days it's all about cloud and AI.
No, the main reason is that HN fills up every month with blog posts where engineers talk about the incredible pain involved with scaling and running Postgres. And almost always, it's clear that they don't realize there's any alternative to that pain. It's not that they considered the options and then explain why they picked this one, it's that they think - as claimed in this article - that it's almost some fundamental limitation of reality itself, imposed by the laws of computer science.
There are lots of startups that lose time and money due to database problems they simply don't need to have. And that sucks. If they'd prefer to spend that time, pain and money to avoid using a DB from Oracle, fine, so be it. I won't argue with random devs about lawnmower memes. But if it's because they don't realize what's possible.... well, maybe someone will be helped by being aware of this. Database scaling problems are a choice, not an inevitability.
But anyway, do you have benchmarks from anybody not related to Oracle showing that performance? Because Oracle forbids talking about it...
The question is what you'd want to benchmark. For example, imagine testing Postgres with a write load that goes well beyond what a single machine can do. It would collapse and query latencies would go through the roof. A horizontally scaled DB would keep going and process all those queries. Would you accept this as evidence that Postgres is slow or would you say it's not valid to benchmark at traffic loads Postgres physically cannot handle, given it never claimed to scale horizontally? Stuff like this is where benchmarking gets complicated.
I used to work at a different company that sold a kind of database system. It was much faster than our nearest competitor, so we were surprised when that competitor claimed to an important customer they were just as fast as us. Their benchmark counted transactions that failed and rolled back due to overload (optimistic concurrency) as "successful".
That's why you won't see many serious comments bragging about the performance of SQL Server either.
Even though is has way fewer performance land-mines that you must design your entire architecture around than Oracle.
Anyway, you are correct that RAC is a very impressive piece of software. If Postgres had something like it, it would be a beast. It's not enough to save Oracle, though.
Still it's not just SQL Server and Oracle. PlanetScale - the author of this blog post - also had a DeWitt clause until literally just days ago. They replaced it with a slightly different clause but I wouldn't be willing to benchmark under their new rules.
https://planetscale.com/blog/transparency-in-benchmarking
It says benchmarking is allowed only if "the Benchmark is conducted in good faith and uses a fair and transparent methodology". What does good faith mean, or fair and transparent? You still effectively need their permission because otherwise they might decide you weren't benchmarking in "good faith" retroactively.
It's no surprise. The stuff that makes database benchmarking useless is the variety of features. You can always design a benchmark that makes a specific vendor look good or bad.
Like, Oracle can add missing indexes on the fly. If someone designed a benchmark in which the core table was huge but missing an index, PostgreSQL would die and Oracle would recover then fly. But how many would accept that as a fair test?
For example the Oracle's incapacity of reordering tests of different kinds (as in constants, functions, rows...) when optimizing a query; or the quadratic run time of triggers that change data in large updates.
There are some cloud databases that scale horizontally but they often only support a subset of SQL, or require careful design to avoid creating hot tablets due to using a shared nothing design e.g. due to hot read nodes.