Comment by rsanders | Hacker News Reader

Comment by rsanders | Hacker News Reader

Parent Full thread

There are many solutions for network block device style persistent storage in K8S now. In the cloud, you have solutions like Amazon's EBS and GCE's persistent disks. On VMware you have local VMDKs. On your own equipment, there's support for iSCSI, Ceph RBD, OpenStack Cinder, Portworx, ScaleIO, Quobyte, and just about anything else if you write a few scripts to format and mount volumes on the host (FlexVolume). Except for FlexVolume, K8S is able to dynamically provision volumes of a requested size for you.

If you're feeling adventurous there's support for network filesystems like NFS, GlusterFS, CephFS, but I wouldn't recommend those for Postgres.

https://kubernetes.io/docs/concepts/storage/persistent-volum...

In one of his demos, Josh Berkus recommends just using the K8S ephemeral "emptyDir" storage type on each node, and counting on Patroni to maintain data persistence through replication. e.g. any one Postgres node may die and lose its local data, but when its replacement is created, it will be populated with a fresh copy of the database wherever it is scheduled on the K8S cluster. That sounds workable for smaller databases, though it gives every single person I've suggested it to nightmares.

It's a bit scary when you're using K8S ephemeral storage and depending on a basebackup from another pod, but you can also set it up to restore from your usual backup system (WAL-E, pghoard, barman, pgBackrest). That way any time you lose a pod, you get a test of your backup system on restore.

There is currently some work being done for improving local storage usage in k8s. https://github.com/kubernetes/community/pull/306

Additionally k8s can be configured to restart pods on the same node so you can use the local node storage for persistence and rely on your DB for replication.

you can also set it up to restore from your usual backup system (WAL-E, pghoard, barman, pgBackrest)

That sounds interesting, has anyone documented that kind of setup?

I'm working on one. I am waiting for a PR to land and a release to get generated for a bug that I found.

If the master pod in a StatefulSet is deleted, and kubernetes sets up a new one too fast, it currently keeps the master lock and failover never happens (it also tries to bootstrap from itself).

Once those land I'll work on a writeup of that setup

I've got a few Postgres instances running as single node Deployments and StatefulSets right now that backup and restore using pghoard for simple services that I've documented here http://alexkerney.com/2016/10/pghoard-kubernetes.html I do need to add a note about using livenessprobes to make sure the restore completed successfully. Sometimes these pods are cycled every day as half my K8S cluster is preemptable.

Reply on news.ycombinator.com