Zero downtime migrations at petabyte scale (2024)
planetscale.com
planetscale.com
"Real world" being something that covers max what, 10 hours of a day? What about things that are used by the entire world? I think there is more than you realize of those sort of services underpinning the entire internet and the web, serving a global user base.
Nobody is using slopwork’s new CrudX at a global scale.
??? I’ve worked in this software game for over 20 years. I’m yet to experience this “no need to worry about the globe”. I think you have the fallacy of thinking local experience is general experience.
There is a very large amount of b2b software out there that is serving multi-nationals of all types. Perhaps it is surprising, but there’s a large number of software solutions that aren’t that big, but still have customers in all the 4 corners.
- How will you maintain multiple deployments across multiple regions in the world? Backups, security patches will start to take a toll. - How granular is the right split? Not every country has a cloud provider. Then you need to start thinking about regions and office timings and then it starts to get all blurry.
The same way as always - by automating the crap out of it.
> How granular is the right split? Not every country has a cloud provider.
Doesn't have to be one deployment for one country, does it? Having like 3 or 4 deployments across the globe already gives you (at least) 3-4 hours of inactivity window, let's say 1 am - 4 am or something.
Well, for the remaining 0.1% - go ahead and use the fancy hot replication thingy. Sometimes there is no choice, and that's fine. Although that might mean, that the system architecture is busted.
In this age, many smaller companies serve customers across the globe. There is no common “asleep”.
Like B is slightly out of date ( replication wise ) the service modify something, then A comes with change that modify the same data that you just wrote.
How do you ensure that B is up to date without stopping write to A ( no downtime ).
Have old database be master. Let new be a slave. Load in latest db dump, may take as long as it wants.
Then start replication and catch up on the delay.
You would need, depending on the db type, a load balancer/failover manager. PgBouncer and PgPoolII come to mind, but MySQL has some as well. Let that connect to the master and slave, connect the application to the database through that layer.
Then trigger a failover. That should be it.
400TB its about a week+ ?
> Then start replication and catch up on the delay.
Then u have a changes in the delay about +- 1TB. It means a changes syncing about few days more while changes still coming.
They said "current requests are buffered" which is impossible, especial for long distributed (optional) transactions which in a progress (it can spend a hours, days (for analitycs)).
Overwall this article is a BS or some super custom case which irrelevant for common systems. You can't migrate w/o downtime, it's a physical impossible.
"Take snapshot and begin streaming replication"... like to where? The snapshot isn't even prepared fully yet and definitely hasn't reached the target. Where are you dumping/keeping those replication logs for the time being?
Secondly, how are you managing database state changes due to realtime update queries? They are definitely going in source table at this point.
I don't get this. Im still stuck on point 1... have read it twice already.
https://github.com/vitessio/vitess
For my own sake, I'm not sure what is so surprising here. "Turn up a hot second replica and fail over to it intentionally behind a global load balancer." Is pretty well trodden ground.
Thank you! :D
> Is pretty well trodden ground.
YES!! But the article point us to it's a 400TB+ w/o downtime migration. This is impossible. That why is looks like clickbait and advertising of a product.
This work has been and is being used by some of the largest sites / apps in the world including Uber, Slack, GitHub, Square... But sure, "it's BS, super custom, and irrelevant". Gee, yer super smart! Thank you for the amazing insights. 5 stars.
https://github.com/vitessio/vitess
These larger ones are fully using the PlanetScale SaaS, but they are using Managed -- meaning that there are resources dedicated to and owned by them. You can read more about that here: https://planetscale.com/docs/vitess/managed
All of the PlanetScale features, including imports and online schema migrations or deployment requests (https://planetscale.com/docs/vitess/schema-changes/deploy-re...) are fully supported with PlaneScale Managed.
VDiff (v2) only compares the source and destination at a specific point in time with resume only comparing rows with PK higher than the last one compared before it was paused. I assume this means:
1. VDiff doesn't catch updates to rows with PK lower than the point it was paused which could have become corrupt, and
2. VDiff doesn't continuously validate cdc changes meaning (unless you enforce extra downtime to run / resume a vdiff) you can never be 100% sure if your data is valid before SwitchTraffic
I'm curious if this is something customers even care about, or is point in time data validation sufficient enough to catch any issues that could occur during migrations?
But there's also nothing stopping you from doing a new VDiff to cover all data at that later point in time.
https://github.com/vitessio/vitess
https://vitess.io/docs/reference/vreplication/
"isn't this just pushing the same issue forward in time?" I don't understand what you are trying to say here. You can only compare the two sides / databases at the same logical point in time. While you are doing this comparison at that point in time, the timeline continues to progress. Unless you want to stop the world and prevent writes for the full duration of the diff (which can be days or even weeks).
I think it's still the same issue where data modified after the VDiff point in time isn't validated before SwitchTraffic. I'm mostly curious how vitess users handle this case, or if any users even care about about this case in the first place?
Is there no demand for continuous data validation similar to what TiDB offers?
Do people who care about 100% correct data validation just accept the downtime required to run a full VDiff before SwitchTraffic?
Going through their pricing (https://planetscale.com/pricing?engine=vitess&cluster=M-5120...), for just 15TB storage with RF=3, the pricing comes to around 24000 USD/MONTH, not year. Adjusted for 400TB and per year, this becomes 7.6 million usd. Of course, you also get a lot more, but, the difference is just insane.
That said at $24K you could pay off an entire server like that from Dell in 4 months despite Dell charging something stupid like $2000/TB.
The issue is your budget is for 400TB of data but minimal requests per second. That's a valid thing to consider, but it's extremely apples and oranges to a fleet of 75 high powered servers.
To put it a different way, their prices are pretty high but the calculation of powerful servers costing 40x as much as raw storage isn't "insane".
4.0 Freeze old system
4.1 Cut over application traffic to the new system.
4.2 merge any diff that happened between snapshot 1. and cutover 4.1
4.3 go live
to me, the above reduces the pressure on downtime because the merge is significantly smaller between freeze and go live, than trying to go live with entire environment. If timed well, the diff could be minuscule.
What they are describing is basically, live mirror the resource. Okay, that is fancy nice. Love to be able to do that. Some of us have a mildly chewed bubble gum, a foot of duct tape, and a shoestring.
Lots of systems can tolerate a lot more downtime than the armchair VPs want them to have.
If people don't access to Instagram for 6 hours, the world won't end. Gmail or AWS S3 is a different story. Therefore Instagram should give their engineers a break and permit a migration with downtime. It makes the job a lot easier, requires fewer engineers and cost, and is much less likely to have bugs.