Hacker News Reader

Comment by nathankunicki | Hacker News Reader

Fun to read, but there's a lack of detail here that I'd like to see. For example, this talks purely about code changes. However times a code change requires a database schema change (as mentioned above), different API's to be used, etc. In the percentage based rollout where multiple versions are in use at once, how are these differences handled?

View on HN

For database schema changes, here is the standard practice: - You have version 1 of the software, supporting schema A. - You deploy a version 2 supporting both schema A and new schema B. Both versions coexist until the deployment iis complete and all version 1 instances are stopped. During all this time the database is still on schema A, this is fine because your instances, both version 1 and 2, support schema A. - Now you do the schema upgrade. This is fine because your instances, now all runnning version 2, support schema B - At last, if you wish you can now deploy a version 3, dropping the support for schema A.

We do it the other way (and I’ve always seen it done this way): database change is compatible with current code and new code. So deploy the database change, then deploy the code change. It usually allows you to rollback code changes.

This is generally harder to pull off though unless you do things like force all DB access to go through stored procedures.

And then you're really still pursuing the same strategy described above, except for your stored procedures instead of your app code.

My company uses HBase currently for things on premise and we're moving to a mix of psql and BigTable in GCP. This is how we do things except all of our "schemas" are defined by the client so we just have to make sure that serialization/deserialization works correctly. With psql we might have to figure out a migration strategy, but for now we'll just be using it to store raw bytes.

Easy: don't do that.

Always make your code compatible with the old and new schema. Migrate the database separately. Then after the migration, remove the code that supports the old schema.

I think every DB change should be done like you suggest. An example I worked on recently:

- migrate DB and create new field

- deploy code for writing into such field (not read yet), in parallel with old field

- backfill data migration for older records

- deploy code with feature flag to read new field in workflows, but still write to both fields

- switch read feature flag on

- make sure everything works for a few weeks

- switch write feature flag to only use new field

I'm more curious about how DB rollbacks occur in situations where a PR changes DB and is then reverted.

It would be a good practice to first make a DB change alone, which is compatible with both and new code, so you don't need rollbacks. Then separately deploy a code change.

Edit: also suggested by Martin Fowler https://www.martinfowler.com/bliki/BlueGreenDeployment.html

Reply on news.ycombinator.com