Moving 6 Billion Messages Without Being Noticed

dt.deviantart.com

17 points·kemayo··3 comments

Moving databases between clusters is no easy feat. Can you share more details on the man-hours it took to oversee and verify the migration?

I see you mention several days for the data to be copied, but how about the time to put your solution together and verify it? Also, did recomputing checksums take less time between passes, proportional to the size of the remaining changes?

We didn't find much published material about the best way to do this,

You may want to have a look at our live migration solution: http://chronicdb.com

It goes beyond checksums: it guarantees the data move is atomic and data consistent.

It took a while. cant put a number cause a lot of it was done in spurts between other projects. altogether the project took about four months, but that included refactoring a lot of old code to play nice with mirrored writes and developing some framework level components. and waiting for hardware. and waiting for innodb conversion. and building setting up proper monitoring and alerting. future projects like that will take fraction of the time. Migration and verification itself took about 2 weeks. Yes, passes got proportionally quicker. first took almost a week, last few hours. we did not crank the speed to the max though as we wanted to keep the load low and we had time.

thanks for the link, will check it out

The first comment on the article expresses surprise that nothing broke when we did this. Normally I might think that's a vote against our expected competence, but honestly I was a bit surprised that nothing went wrong as well.

Reply on news.ycombinator.com