Hacker News Reader

Why not canary releases? You can load balance for example 1% of the traffic to the new deployment and see if you experience any issues. If you do - you just change the loadbalancer to use the known good pods.

View on HN

Honestly, we haven't considered that. Breakage rarely reaches production, rollbacks are quick, a process orchestrator retries most of the stuff and because of that we do not have a problem worth solving with "canary deployments". Note that we do sometimes carry out "canary releases" via toggles.

How you take care of DB updates when using cannary deployments ? For example those which are not backwards compatible ?

Ps. Releases are about building new versions of code packages. Deployments about pushing them out to environments.

Best way is to not write forwards / backwards incompatible database changes.

This depends a lot on the databases used and flexibility of the code that's accessing the data. One method is to deploy in multiple stages and to use views. In pre-deploy you create a view with schema/data needed for the release. In deploy stage you roll-out the canary code. In post-deploy stage remove either the old data/schema on success or new data/schema on failure. It's quite an overhead to implement and maintain this process.

Reply on news.ycombinator.com