How release canaries can save your bacon

You are a CTO sitting atop very expensive hardware and software. Would you start removing deployment and runtime safety guards (such as a consumer-facing staging environment) because you want to "discipline coders and devops"?

On the one hand, rollbacks need to be culturally normal. Finding out that an essential part of your process (essential because it keeps your mean-time-to-repair low) is undependable because the last time you ran it was a half a year so, and right now is precisely when you need it to be dependable, well, that royally sucks.

On the other hand, what you're talking about shouldn't be that hard to implement. Just hook your rollback system into your issue tracker to create a post-mortem issue whenever a rollback is necessary, and assign it to whoever initiated the deployment (or their manager). Easy.

On the third hand, you might end up finding that a lot of your post-mortems end up looking something like "we don't have reproduceable builds and we made a managerial decision not to invest in that now". And now you just created a ton of recurring paperwork for everyone with little benefit.

It is not that hard to imagine that every failed request/action in your application costs some real money to the client (cannot complete sale, must record data locally and then reenter, etc). If that can be regressed (fines, compensations, lost sales) to vendor that is sort of increased operation costs for the duration of outage. Can you get that sure of your pre-release process as not to have ability to 'rollback' to cheaper (earlier) state? I guess not.

While I agree on your points that it is better to catch errors earlier in the pipeline and the necessity of mini-postmortems, I personally think that rollbacks are inevitable and compare them to backups. Of course it is better not to restore from backups, it is easy to rationalise good processes over a few metric tons of never been read backup tapes, but a single accidental drop of production database may quickly pay for all the effort put into ensuring backing up works.

If it was an automated rollback, a system that automatically generated either test data or other heatmap of the system when the rollback was called would be awesome.

It is amusing, because we are basically saying it would be awesome to have a coredump when the crash happens. Which... used to be standard behavior but was essentially lost in most modern development environments. (Not to mention the tooling lagged in two directions. One to help you with the coredumps, and the other to make usable core dumps.)

> They signify something didn't go quite right with your unit/integration/qa process

Staged rollouts are part of the QA strategy (whether this is something to be aspired is another question).

[Deleted because I'm being a dickhead, and can see that]

That's not a drawback so much as it's a fact of life. In any large-scale (read: distributed) system trying to provide a high degree of availability, rolling upgrades are the only way code goes out and individual components need to deal with interacting with newer/older dependencies. You can constrain the matrix by only allowing current version plus one back running in production, or forcing deployment orders, and so on, but in modern systems (read: ones where you can't just say "We're taking everything down for 3 hours on Sunday to upgrade.") you can't escape non-atomic upgrades.

At my job the DB is backwards compatible by one release. This means we roll back code, not data. It takes a little longer to do something like delete a column, but it's worth it.

Rainforest QA had a blog post on strategies a while ago.

They touched on this in their SRE post last week: https://cloudplatform.googleblog.com/2017/03/reliable-releas...

Probably depends on your workplace, but at Google canary has meant a subset of production running at a newer version at least since '07.

I think your definition comes from Amazon/AWS... haven't heard it much from outside Amazon.

Well, you could definitely front two versions of your application lambda function with a traffic splitter lambda function that would send 99% of traffic to the production alias and 1% to the canary (or whatever the number you wanted was. See how to call one lambda from another: http://stackoverflow.com/questions/31714788/can-an-aws-lambd... and aliases: http://docs.aws.amazon.com/lambda/latest/dg/versioning-alias...

This post might also be of interest: https://blog.jayway.com/2016/09/07/continuous-deployment-aws...

Note, I have never done this, this is just how is approach it.

Maybe I misunderstand, but don't you want to invalidate the cache when a new version comes along? Isn't the risk of version skew worse?

Clearly everyone should consider their delivery costs and error acceptability when determining their development & release process.

Continuous automated deployments might not be a great fit for satlellite control software but that doesn't mean SaaS apps should switch to the same process as satellite software teams.

We don't just throw stuff at prod. Google has a culture of testing and code review, so code review, unit testing and integration tests are the first line of defense. Then we do QA testing, where a lot of problems not caught by tests are found, which then inform us where tests may be lacking. We also roll out to internal dogfood first, given Google's number of employees this is kind of like a pre-canary.

The point of the canary is it gives you one last real world test where you can limit the damage if anything goes wrong. Without it, you'll have to assume your tests are perfect, and I doubt that's ever achieved in practice in a real environment, except maybe in avionics and space control systems.

I think the author of that sentence agrees with you. It's saying you should think of a canary as something to do only after testing is complete, not as a means of testing.

>but not an excuse to throw stuff at prod and see what happens

The problem with production traffic is you will never be able to simulate it perfectly. At the end of the day you are required to flip a switch and test things out in the wild.

What's a high traffic customer of Google?