Hacker News Reader

Comment by danialtz | Hacker News Reader

Nice approach! "reload" was a nice touch to learn.

What I miss usually are two parts of deploys that are often ignored, and are critical in production environments:

1. Revert/rollback to older versions. We're humans and despite having all the processes sometimes murphy's law applies on moderate to complex server setups.

2. There is git/svn for code tracking, but even more important is the database consistency. Versioned backups and restores should be also part of the whole setup.

I'm currently looking into having the full setup with no downtime with open ears.

View on HN

Any advice about versioned database backups?

In my opinion, this is one of the major unsolved (or under-solved) issues in web development. You have to write and test migrations, and then you have to make sure the migrations are tied to your deployment/rollback process.

If there's a service/tool that automates a lot of this or makes it safer, I'd be really happy to learn about it!

When I used to use django ages ago, there used to be Django south. I think its built in now.

Yes, its built in now: https://docs.djangoproject.com/en/1.8/topics/migrations/

Of course a database rollback may loose information from any newly created fields.

So if you're aiming for zero downtime rollbacks with Django, here's how adding a new field to a model might work:

  1) Add a Migration that adds the new field, but allows null. Deploy.
  2) Add the field to the model. Also make sure it sets it's default on write. Deploy.
  3) Execute a background task to set the field to whatever it's default value should be.
  4) Add migrations to enforce integrity and add indexes. Deploy.
  5) Actually deploy code that needs the new field.

Yes, this is super annoying, no, most people don't do this. Separating out #1 and #2 means you can always roll back all the way to right after #1 without losing any data. An extra, nullable field on a model with no indices on it shouldn't hurt anything.

Totally agree. I've been advocating this for months but the ease/simplicity of simply restarting the server with new code has won out for now.

I'm planning to do something like this workflow most probably reinventing the wheel along software history: - Every release is already tagged, and merged into master (git-flow). - There is a post-commit hook to backup the database with the tag and time. So I know which database tag is newer. - Store the encrypted zip on S3, since each developer can deploy. - Deploy, like cuu508 did.

The problem is with restore. If there is no change in database it is more or less straight forward to download and import the db with the n-1 code tag. But if there are any inserts in database then a migration back should be applied to the latest stand of the database. I can imagine it's going to be a hefty workflow to do.

How about .. have one db with log of events (cqrs style), the other one is transactional and transactions are being performed based on ordered events.

When deploying new version - backup the transactional db and keep a pointer to the last event in events db. Keep the events db running.

If restoring - restore transactional db backend based on backup and "play back" all events from the events db.

The distinction can also be made using one db and backing up / restoring subset of the tables.

So basically kafka + another layer for database migrations?

In docker-land you've got Flocker. I guess you could version the container with tags...

https://clusterhq.com/

Reply on news.ycombinator.com