The main reason for keeping dbs out of k8s was that the storage (persistence) solutions in k8s were not up to the task yet. Now I wonder how is Patroni doing persistence? I cannot find anything on it in the Patroni docs. Maybe Patroni is so "self healing" that a proper storage solution is not an issue -- dunno, just guessing here.
If you're feeling adventurous there's support for network filesystems like NFS, GlusterFS, CephFS, but I wouldn't recommend those for Postgres.
https://kubernetes.io/docs/concepts/storage/persistent-volum...
In one of his demos, Josh Berkus recommends just using the K8S ephemeral "emptyDir" storage type on each node, and counting on Patroni to maintain data persistence through replication. e.g. any one Postgres node may die and lose its local data, but when its replacement is created, it will be populated with a fresh copy of the database wherever it is scheduled on the K8S cluster. That sounds workable for smaller databases, though it gives every single person I've suggested it to nightmares.
Additionally k8s can be configured to restart pods on the same node so you can use the local node storage for persistence and rely on your DB for replication.
That sounds interesting, has anyone documented that kind of setup?
If the master pod in a StatefulSet is deleted, and kubernetes sets up a new one too fast, it currently keeps the master lock and failover never happens (it also tries to bootstrap from itself).
Once those land I'll work on a writeup of that setup
I've got a few Postgres instances running as single node Deployments and StatefulSets right now that backup and restore using pghoard for simple services that I've documented here http://alexkerney.com/2016/10/pghoard-kubernetes.html I do need to add a note about using livenessprobes to make sure the restore completed successfully. Sometimes these pods are cycled every day as half my K8S cluster is preemptable.
And that's a great talk, made Patroni much clearer for me.
Patroni with Spilo e.g. relies on EBS or local discs and can also ship WAL to S3. On Google it uses their equivalent solutions.
On Kubernetes earlier posts are correct that you may rely on Kubernetes bringing your pod back with the same underlying volume thus providing some kind of availability, but for others this is not good enough, e.g. remember EBS is single AZ only. Patroni can help here, running and orchestrating either slaves or giving you automated recovery from S3.
I personally wouldn't place a database in container for that reason. Essentially when you do so any durability guarantees that a database provides are going out of the window.
Things might seem to work fine, but when container get abruptly terminated in a wrong moment you might learn that all data is gone.
The danger is that some bugs in a filesystem, manifests themselves only when there is abrupt termination, otherwise things appear normal. Stuff like data is being written in incorrect order, or data is synced to disk at wrong time, or not at all.
On top of that user also expects performance, so some shortcuts can be taken that compromises the above (especially if the criteria is that docker is intended for stateless applications).
When there bugs like that and things are abruptly terminated, the data on the disk can be heavily mangled to a point that can't be recovered so essentially it would be lost.