Hacker News Reader

Comment by polskibus | Hacker News Reader

Is there a recommended way to handle database migrations in kubernetes? Is there a best practice or a tool for that?

This depends highly on the desired availability of your service and how you're shipping migrations. Some people are using init containers[0] to execute migrations before bringing up Pods in their Deployments. The deployment for Quay.io, for example, requires zero downtime, so for that a series of container images are rolled into production in order. Basically, you want different phases of the migration to have different read/write policies to new and old code-paths as you copy data into a new table. The tedious part of this is actually just writing the migration into these phases, not deploying them.

[0]: https://kubernetes.io/docs/concepts/workloads/pods/init-cont...

We've been packaging migrations into a container and just shipping it as a Kubernetes Job. It executes once, and if doesn't success (non-zero exit code), Kubernetes will reschedule it.

So if you want to migrate a particular env from schema version 3 to version 6, do you deploying 3 different init containers manually, or is there a single migration container that always contains all migrations and it by itself understands from which level to which level should the migration be run? Schema migration failure is a stop the world a scenario for our entire stack.

> Schema migration failure is a stop the world a scenario for our entire stack.

That's the big problem, and I don't believe that Kubernetes can help with that. It's too intimately tied to your application internals. The recommended practice for migrations is to do them in multiple zero-downtime steps, e.g. first deploy code that handles both old and new schema, then migrate, then deploy to remove transitional code.

Sorry, I forgot to mention that it is acceptable to stop the world. We do in-house deployments and a lot can fail on the customer's side (IT disabling the DB Server, network failure, etc.). We'd rather have the stop and be focused on fixing it ASAP, than be tricked into thinking that the issue may resolve itself.

What I need is a recipe on how to handle db migrations, which are quite frequent, every customer has different current version of the app and therefore is on different db schema. We have our own tool to do that, I suppose we could wrap into a kubernetes-something, the only thing special is that it should run : after old version containers are brought down, before new version of containers are brought up, run only single time, include backing up the db and stopping the entire process on failure.

If you're using Helm charts you can add hooks to a few points in the deployment process to give you database migrations. Currently I'm using an install hook to create the DB and upgrade hooks to migrate on deploy.

This is also something I've been wondering about. I've been trying to nudge my workplace to start moving to a containerized infrastructure, but this has been one of my nagging questions I've had. Admittedly I haven't done much research on the subject though.

I currently have a swarm cluster on DigitalOcean that I use for self hosting apps. And, only one of those required migrating the database before it could be deployed, so to get it running I just spun it up on one of the nodes and only connecting the database container to the data volume I was going to have it connect to in the swarm, migrated the db, then killed the containers and then deployed it to the swarm. But, as I was doing it I realized this would hardly be maintainable/enjoyable method of handling this in any non-personal/production environment.

We "kubectl run" the migration in CI as a simple bare pod. It will shutdown itself after migrations are done.

For getting the resource configuration, we render the "helm template", take out the workload's podspec and supply additional parameters to invoke the migration.

What do you mean when you say 'database migration'?

In my mind, there could be a few things:

1) Migrating the database from server A to server B where the server is on Kubernetes

A1) Don't do this. Don't run a (traditional) database server in Kubernetes. Sure, you can do this - there are all kinds of volume support for all kinds of things. Everything I've ever heard and read has told me that containers aren't a good for for this type of long-term service that gets refreshed infrequently.

2) Migrating the database from server A to server B where you have to tell all of the clients what the database is

A2) This should probably be done via service discovery or even just by a short TTL on a CNAME in DNS.

3) Something else?

Database migrations typically refer to schema/ddl changes to an application along side deployment of a new version.

Everything I've ever heard and read has told me that containers aren't a good for for this type of long-term service that gets refreshed infrequently.

That was a safe rule of thumb 5 years ago. Since then support for scheduling a persistent volume to be available along side your long running container has become a bog standard and boring feature.

> Don't run a (traditional) database server in Kubernetes.

With node affinities and persistent volumes this might not be A Bad Thing, per se... For something mission critical I'd keep the DB on a VM, though.

One interesting development in the Kubernetes space are plugins which expose VMs outside of the cluster to Kubernetes as though it were just another container. That may provide the best of both worlds: a traditional VM and automated scheduling.

Spring boot microservices using liquibase as schema evolver, we are using helm hooks to run a job pre-deploy that leverages a spring boot profile which runs the evolutions. If it fails, return non-zero exit code and the deploy doesn't continue. The only pain point we've run into with this model is that the configmaps that the service depends on (we use the same docker image in both the deployment and evolution job) aren't created before the hook is called, so we need to duplicate part of the config with -D startup params.

On premise I wouldn't suggest to run your DB inside kubernetes. It's ugly. Because on prem storage is not really solved yet (and google has no interest in changing that).

And in the cloud what would be a migration scenario? (serious question, never did it in the cloud but would suspect there are fewer scenarios in which migration is necessary)

Reply on news.ycombinator.com