Hacker News Reader

Comment by gaius | Hacker News Reader

What about running the DB process(es) in a container mounting non-containerized storage from outside? All the "state" is then external to the container? This is very do-able with Docker and it's where my thoughts are heading for a "best of both worlds".

View on HN

That's what you do. If you're on GCP or AWS, you typically mount a network volume (Persistent Disk on GCP, EBS on AWS).

Kubernetes ensures that the container always has this volume mounted, and of course only one container at a time can claim the volume for itself.

What you should avoid doing is to use a host mount and pin a pod to run on a specific node, because then that pod can only run on that node, and you have no way of migrating without manually moving the mount and unpinning the pod. With Kubernetes, you really want to avoid thinking about nodes at all. State follows pods around; pods don't follow state around.

You also have the option of using local storage and combine it with a consensus protocol to keep the data distributed. You can actually achieve better durability than mainframes.

Spanner uses cross-datacenter Paxos. Your data won't be lost even if an entire datacenter goes dark.

For Vitess (http://vitess.io), we use semi-sync replication that always ensures that at least one other machine has the data.

In K8s that's called "Stateful Sets" (beta in v1.5).

https://kubernetes.io/docs/concepts/abstractions/controllers...

You can solve this without StatefulSets, it's just a manual process, and requries that you (1) don't use a replication controller (edit: rather, you use a controller with "replicas: 1"), and (2) can ensure that a single pod claims the data volume.

StatefulSets are more geared towards apps that manage their own redundancy, such as Cassandra or Aerospike, where adding another instance is a matter of just starting it. One of the things that a StatefulSet permits is to preserve the network identity of a pod. For example, if you wanted deploy Cassandra without StatefulSets, you'd deploy each instance as a separate Deployment + Service pair, called, let's say, cassandra-1, cassandra-2 and so on. You would not be able to use Kubernetes' toolsets to scale the cluster. Each instance would use a persistent volume, so effectively it would be almost exactly like a StatefulSet, except Kubernetes would not be handling the pod replication.

In the case of something like Postgres, you'd probably not get any benefit from using a StatefulSet for the master (since only one instance can run), but you can use a StatefulSet to run read-only replicas.

Without Stateful Sets, or a replication controller, doesn't it mean that if my host dies, the DB won't get started anywhere? Who's managing where that pod should spawn next (with its associated storage)?

Right, I simplified a bit there: You would use a replication controller, but it would have "replicas: 1". This way, the pod is rescheduled and the service repointed, and the volume management ensures that the database gets mounted.

Got it, thanks!

Note that replicas: 1 does not actually guarantee "at most 1". If you have block storage with locks (AWS/gce/ceph/cinder), then he second replica won't start until the first is gone. If you try to use "replicas: 1" with a shared filesystem you can have 2 pods running against that filesystem at once.

StatefulSets guarantees "at most one"

That's solved by setting "strategy.type" to "Recreate", isn't it? This will disable rolling deploys. The replication controller wouldn't then attempt to have two pods running at the same time.

Yep. Even works in Docker Swarm (1.12 an up) via Volumes.

Reply on news.ycombinator.com