Hacker News Reader

Comment by freedomben | Hacker News Reader

I very much disagree. If your bad deploy included migrations that can't be reversed (drop an old table for example) then rolling back just gave you two problems.

As long as the dev wrote (and tested!) two-way migrations and they are possible, then yes you are correct.

View on HN

The long-term viable approach is to not make forward-backward incompatible migrations in short period of time.

If your system is truly distributed (multiple machines hosting it), then if one server performs migration that deletes the table, then the other server stops working.

You must have 2 checkins and 2 rollouts: Create a new table while maintaingin the old one -- and let it bake in; and then delete the old table a week later (or whatever is your cadence/release cycle).

IMHO the answer here is to just never do irreversible migrations. (In other words: just leave old tables/columns around--if not indefinitely, then for a few weeks after they stop being used.)

I have done that in the past and I agree, it's a decent solution. The risk is that the old stuff never gets cleaned up and before long your DB is full of all kinds of cruft and nobody knows which part is needed and which isn't.

I saw a heinous bug once because a newer person was using a column that had stopped getting updated years earlier (because it was replaced). This person saw the column and it was exactly what they needed. Then customers started getting billed on old accounts that they had either closed or changed with us. They were really mad.

My solution to this is to rename that table/column to “table_deprecated” or “column_deprecated”. This has the nice property of being reversible, causes nice visible errors if something unanticipated is actually using the column/table, makes it obvious that it shouldn’t be used (well, one would hope), and makes it easy to find for permanent deletion later.

That's a great idea! Although I did work with a person once who would use anything that looked helpful regardless of "deprecated" being all over the place (Eclipse would even yell at the method calls but he didn't care). Granted that's Java and at that point in time APIs were being deprecated without workable replacements, so it's hard to fault him.

At that point you might as well just delete them.

I think you're missing the point. If your migration renames a table or column, then the migration can be rolled back, by simply reversing the rename. If you delete the thing, then there is no way of rolling back other than restoring from backup.

I can see it being useful to rename these things, as long as there is some process in place to delete the renamed items after a period of time has passed.

Would be good if you could schedule a future migration that will happen only after a certain date has passed, which deletes the renamed tables/columns.

This sounds so much like something that I'd do, that I had to stop and think about my personal hall-of-fame screwups to make sure that wasn't one of them.

In terms of things that scare me as a developer, stale data is WAAAAYYY less concerning then having people try to shotgun fix an issue on a production server after an irreversible update took down the entire system... Especially now that the developer and the entire organization is likely now in a panic state and not thinking clearly.

That bug you describe, maybe stale data is part of the problem, but ... why are you having a new person working on the billing system presumably without any code review? It sounds like you have a lot of process issues.

Just create an issue in your issue tracker with the highest priority to delete a column (and why) with a deadline in 4 weeks. Stupid simple solution but works perfectly.

"Clearly the answer for any software problem is to just not have any bugs."

You see how that isn't an actual solution?

An irreversible migration isn't a bug, nor is anyone saying the answer for any software problem is to not have bugs.

They're saying the solution to the specific problem of being unable to rollback due to irreversible migrations is to not write irreversible migrations, which is a completely valid solution and indeed the correct one. The whole point of migrations is to track db changes so that undoing them is easy.

That misses the point.

Code and the DB always need to be compatible one version forward and back. That's required engineering discipline.

creating a table is an irreversible migration. Once you have the data, you can't just ... reverse the migration by deleting the whole table.

I would never ever EVER want to do a deploy with migrations that can't be rolled back. That sounds like professional malpractice TBH.

Two way migrations are both possible and SHOULD be done for any real data.

Reply on news.ycombinator.com