Real-World Engineering Challenges: Migrations | Hacker News Reader

Real-World Engineering Challenges: Migrations

newsletter.pragmaticengineer.com

72 points0x54MUR4119 comments

Mostly good stuff, but:

> You’ll notice that with backend-only data migrations – like the ones Box or Pinterest executed – there was no need for rollback plans, as the rollout moved forward only after the team had validated that everything worked fine.

Speaking from direct experience: That is outright bullshit. Ideally your testing will catch problems, but if you're migrating to a new database architecture, it's an architecture you don't know well, and by the way, your sys admin never even read the installation requirements. This is yet another reason dual-write is ideal, so that you can roll back late in the game when the new DB suddenly seizes up and goes catatonic (esp. Mongo/Cassandra/ElasticSearch). Be that Negative Ned and plan for the worst.

Also: Alpinism is for romantics; engineers seige-climb. Incrementalism is everything. Figure out how to break it down into smaller, safer parts. Big-banging that migration is the almost certain path to failure.

Also-also: Institutional knowledge is bullshit. Old-timers will tell you "Don't worry, the system never needs to do that," and they will be dead wrong. Verify architectural impact at the source.

Check out GitHub scientist if you are doing a migration with a ruby based system: https://github.com/github/scientist

Great support and functionality for testing differences between two systems of record.

I remember when we redesigned Fotolog.com to a modern stack it took a huge challenge to write PHP code to migrate trillions of comments, posts, to the new stack written in Javascript. As far I can remember, migrating the photos only (~30TB) took one month, and the data around 2 weeks.

Is this site gone because of the migration? I get a spam blog on that domain.

The company has been sold, so far I know. I don't know what happen later.

Migrations are a huge issue, and many companies don't bother. They just wipe the slate clean, dust off the erasers, and start over.

I don't have the data (but I'm sure it's there), but I suspect more than one company ended up committing seppuku, over a badly-done migration.

I just got finished migrating a fairly crucial SDK, in the app I'm working on. It was one the fundamental "infrastructure" server connectors, and affected everything.

It came out great. Far fewer bugs than I expected.

In my experience, migrations take planning and strategy. Often, they need to happen over a long time period, in steps.

Many companies and engineering teams don't have the patience to manage that.

You can't just wipe the slate clean and start over. This is data that needs to be preserved and is a core part of the business. Not even sure how an SDK is related to database migrations?

It isn't. The title was "migrations." I tend to write my systems in "layers," which makes "migrations" easier. With layers, you can plan a migration in steps. With monoliths, you'd be best advised to pray.

We have a "data" migration, coming up; but probably not for a few months. It will be handled in a similar manner.

And I have watched "wipe the slate clean" migrations, done, several times. In some cases, they were not deliberate, but the migration screwed up the data so badly, it had to be binned.

> The title was "migrations."

Are any of the migrations in the article non-data migrations?

Fair point.

I guess the only "migrations" that matter, are data ones.

I'm not a "big data" chap, but I have written distributed (franchised) systems, and have had to do many upgrades and transitions, affecting many folks.

It's taught me to be forward-thinking in my architecture.

Okay?

You don't need to be a "big data chap" to run data migrations. Regardless of your infrastructure and system designs, your business will continue to evolve and the data can outgrow your original database requirements, so there's always a need for a migration even if one is "forward thinking in their architecture"

Hey, listen. I just read the article, and said to myself:

"Self, y'know, this chap is talking about database migrations, which isn't really my cup of tea, but his point is just as valid for codebase and dependency migrations."

So I made a point, based on that. It was not my intention to pee in anybody's punch bowl.

So I leave the field to you.

I yield.

I surrender.

You win.

Have a great day.

One thing that caught my eye in the Spotify screenshot is that there's what is presumably a Python 2 -> Python 3 migration monitor, which is also the only red one. I would typically assume that a code migration would be much more difficult than a data migration, which sort of feeds into that "Python would be the only red migration monitor" idea, even if data migrations are more complex from a making-sure-everything-works perspective.

This has been recognized for a long time.

Those change-over things are really severe. Really severe problems. - Joseph Henry Condon, Bell Labs ... via https://github.com/globalcitizen/taoup

However, we have much better tools now, such as end to end / black box testing with mature client automation stacks, high availability clustering, CI/CD, distributed architectures, middleware proxies, protocol-level support for temporary failures and retries, mature caching systems and fault tolerant hardware and software infrastructure. So it's practically not as hard as it once was, because if you've got your ducks in a row you can use a battle-proven method and/or get fallbacks for free.

Yes but all of that is a good idea to implement on a new project starting today with lots of money and a big competent team. Migrations are hard when you have to fight with the project management triangle and then realize that the migration you're doing is on a system which is old, bespoke or not supported. Or all of the above. I like your optimism, but it's always hard work. The most important part of all processes is knowing when you're past a point at which you cannot turn back from.

Years ago, I was responsible for a small scale migration (thousands of records) and it was really painful. All the edge cases, datatype mismatches, and undocumented functionality really made it a sticky problem.

Especially as I was a consultant, billing hourly and trying to provide a reasonable estimate and visibility for the client.

Wrote up some more here: https://www.mooreds.com/wordpress/archives/876

I do a lot of control systems migrations for brownfields, where we would be often partially migrating a system both hardware and software.

First rule of brownfields cutover - every interface is an opportunity to fail.

Reply on news.ycombinator.com