Deployinator: Etsy's deploy tool open sourced

While it doesn't handle every case, using a database migration framework with Up/Down style methods as part of your deploy process can work pretty effectively.

Another option is to use a VM for your database and do a snapshot before deploying.

A number of techniques help here. These are useful whether or not you're using continuous deployment.

A simple approach is not to delete data, but rather simply set a flag which marks the data as deleted. The application storage layer acts as if this data were deleted. A separate process (changed separately) is responsible for actually purging data which has remained deleted for a certain period of time.

In any case, do you really wish to delete the data? Keep it around -- it might be useful. The only reason to delete data, outside legal concerns, is to control the cost to retain it. Given that you mostly never really want to delete data, you probably want to orient your schema toward hiding it instead. In that case, if you need to roll back application changes, you can do so by backfilling the data to restore it to a visibile status.

Keeping detailed programmatic logs is also helpful for making the backfill easy to execute. For example, in addition to your main textual log file, you keep something like a log consisting of JSON objects, one per line, each which represent the essential details of some application action.

Soft-delete (+ action logs) for data recovery. Also, code review and lots and lots of tests.

For schema changes, there are a bunch of approaches. Just a few:

* Treat (non-purely additive) schema changes as a special case and take extra care deploying them. (This sounds like a cop-out, but it might be the right effort tradeoff for many startups.)

* Write forwards and reverse migrations for each change, and avoid data-losing schema changes.

* Implement all (or all non-purely additive) schema changes by shadowing data operations onto a new version of the schema (asynchronously migrate all previous data onto the new schema in some back-end process.) Write tests to verify the migration/shadowed data is correct. Switch data reads to be on the new schema. Eventually delete the old schema in a subsequent deploy after an appropriate stabilization/verification period.

I find #3 to be the most general, but there's plenty of overhead (computational, storage and man-hour) that may not be appropriate for your case.

The problem is pretty universally solved:

1. Use some sort of schema migration tool to keep track of and version all changes you've made to the database since your very first commit.

2. Never do a backwards-incompatible schema change. Never. Time and time again these are the ones that will completely bite you in the ass when the day comes that you broke production with a deployment and desperately need the old version back up and running. You're going to end up losing data and shit will hit the fan. It doesn't take much code-wise to support the old scheme, and it will ensure that you have zero downtime while rolling out your updates.

3. Never delete data. The only exception being if some user-submitted content violates a law, like child pornography. In which case, it shouldn't be part of your migration/update process to begin with. Just use deleted flags / boolean fields. Note in your code when certain fields are no longer used anymore because they're only there to support legacy versions.

You are correct, at Etsy we deploy schema changes once per week. The code always works against both versions of the schema. We never take downtime for schema changes.

We avoid data loss by doing soft deletes as much as we can. Sometimes we do want to do real deletes, especially for data that is heavily denormalized, but in those cases we keep an audit trail so that the data can be reconstituted in the event of a mistake.

I was at the talk and both of these were addressed. For rollback, the answer was they don't support it, only roll forward.

Databases are handled differently from this tool. Iirc he said Thursdays are database day and all changes are done/initiated then.

They're not mutually exclusive: you want basic correctness/design review of course (my favored approach is now pre-integration code review + authoritative master/master-always-deployable), but a proper feature-flagging system will let you beta or A/B test potentially incorrect (in a code-correctness, performance, or UX sense) changes gradually in the real world. Binary (on/off for everybody) flags are more the exception than the rule.

Server affinity + staged rollouts are another approach, but for long timeframes (e.g. A/B UX tests, feature prototypes) I'd think you'd want to go with flagging anyway from a codebase management point of view.

Some things do take more effort to make dynamically flaggable than it would take cost to delay integration, to be fair. Judgement call.

Config flags and how we commit code are two completely separate issues. Very simply, you need config flags to operate a site that degrades gracefully when something is going wrong. They cannot be avoided. Is there a query in the forums suddenly hitting the database too hard? Turn off the forums while we fix it so the rest of the site isn't affected. Etc.

When a feature is in development, its config flag is off or enabled for admins. We deploy it as we develop it, in pieces that are generally not more than a few dozen lines of code long. This is another critical idea: we don't push out all of the code for an entire sizable feature all at once, ever. Doing that is a recipe for having a problem and having to review thousands of lines of code to try to figure out what it is.

For new features, we generally do not remove the config flag once a feature is live because we want to be able to disable it if anything goes wrong.

If we are replacing a feature with another one, we generally want to keep the old one around for a little while (we generally do A/B testing or ramp up new versions whenever we do this). After that, we do delete the old code and reduce the config flag to an on/off switch which was probably there in the first place.

If you are specifically interested in this kind of continuous deploy process then "Continuous Delivery[1]" is a decent book on the subject. The IMVU post "Doing The Impossible 50 Times A Day[2]" is another good place to start reading.

[1] http://www.amazon.com/gp/product/0321601912

[2] http://timothyfitz.wordpress.com/2009/02/10/continuous-deplo...

PDF format: http://bit.ly/qRhsj4