Hacker News Reader

Comment by atemerev | Hacker News Reader

Blue-green deployment is the only way to fly: https://martinfowler.com/bliki/BlueGreenDeployment.html

There are two identical prod servers/cloud configurations/datacenters: blue and green. Each new version is deployed intermittently on blue and green areas: if version N is on blue, version N-1 is on green, and vice versa. If some critical issue happens, rolling back is just switching the front router/balancer to the other area, which can be done instantly.

View on HN

Yup, can't beat blue/green. A big part of the reason I recommend it is because unlike most other rollback schemes, the actual mechanism for rolling back is the same as for a normal release (a load balancer flip), so it's continually exercised and validated. To roll back, you literally do a deploy, but you just skip the step where you alter the bits on the dark cluster. This is highly unlikely to fail, since the only thing that has to not go wrong is the part where the software update is skipped, which basically boils down to a conditional.

Any mechanism for rollbacks that isn't tested continuously is likely to fail during incident response. It's a huge anti-pattern to have 'dark' processes only used during incident response -- same thinking behind why you should also be continually testing your backups, continuously killing servers to verify recovery, etc.

They you just have to worry about rolling back the changed database schema that you have rolled out to both green and bluie variants - I presume. That bit feels tricky

Yep, that's a fair point. In scenarios where you have made non-backwards compatible database changes, this approach cannot be used. Typically the dark cluster is accessible for pre-flight testing, so you can run tests there to determine if the service is still operating as expected despite the database being one migration step ahead. In your response runbook, you should have a contingency for the scenarios where that happens. (For many projects, the % of changes that have this characteristic are small enough that it doesn't move the needle on the risk profile, but Murphy's law can mess with you :))

Cost can be a killer here though. If you're flipping from blue to green and vice versa you either have to have capacity in stand-by(expensive) or spin up new capacity before flipping(time-consuming).

We use a blue/green strategy but our infrastructure can be created on demand with Pulumi. https://www.pulumi.com/

It takes Pulumi about 15 minutes to create our kubernetes cluster with all pods and monitoring in place.

After a succesful rollout we can pulumi down one of the clusters and reduce costs (We're on azure.)

Reply on news.ycombinator.com