Moving 6 Billion Messages Without Being Noticed
dt.deviantart.com
dt.deviantart.com
What they really needed is ChronicDB http://chronicdb.com
ChronicDB is a combination of a software engineer and a hosted database: You run your database in our datacenters -- http://chronicdb.com/how_it_works
Your database will be hosted in our datacenters -- http://chronicdb.com/features
The author of the 6b article kindly provided more information about the time it took for the migration:
http://news.ycombinator.com/item?id=3621561
A couple of limitations of the Percona tool are related to changing data in tables with ON DELETE and ON UPDATE foreign key constraints and possible lock-ups of tables. It is a bi-directional replication tool, so it has to deal with the master-master replication case and as such does not guarantee data consistency.
Still the tool is helpful in many cases and congrats to Percona for developing it. Replication safety is a non-trivial problem in general.
Percona toolkit has some useful stuff. Thanks for the link.
In this case our solution ended up being pretty much the same as pt-table-sync except that it made use of our distributed task queuing system so was MUCH faster than if we had used a single threaded perl script. (Up to a few hundred chunks of rows being checksummed/synced in parallel.) I didn't talk about this aspect directly in the article to keep it focused.
Not sure if it would be possible to use pt-table-sync in that way, certainly would have taken a lot more work to get a third-party perl script to distribute the work through our job queue.