Experiences with running PostgreSQL on Kubernetes

"That sounds like a shitty excuse to just dismiss the whole thing as too hard"

Disclosure: I work at the company that published this post.

I read it differently (albeit, I have much more context). I read it as a cautionary tail that Kubernetes makes it easier to get in trouble if you don't know what you are doing - so you better have a deep knowledge of you stateful workloads and technology. Perhaps obvious to some, but still a good reminder when dealing with a well-hyped technology like Kubernetes.

Sasha's experience running Postgres under Kubernetes started before StatefulSets were a thing, back in the days of PetSets. At the time of his tech diligence the best option appeared to be pg/stolon https://github.com/sorintlab/stolon. The gist of the interview is that there are many more edge cases with HA Postgres + leader election than you can reasonably imagine at first glance.

Our experience building and running 24x7x365 HA Kubernetes clusters for clients is almost exclusively in air-gapped or on-premise environments where EBS (or even an IOPS QoS guarantee) does not exist. Most SaaS and managed service provider teams we encounter would much prefer to leverage a cloud provider's primitives than DIY.

I run a bunch of Kubernetes microservices which use Postgres, and I treat Postgres in the same way as etcd - it runs on its own box, or RDS or CloudSQL, and the containers talk to it there. When you do it like this, you can use all your favorite postgres management tools, and not have to worry about Kubernetes doing something unexpected.

If you want any semblance of DB performance from Pg as a container, you've got to give it its own data volumes, mount that into its container, etc. Not sure what the benefit of containerizing it would be. Shared tenancy on kube workers makes your performance unpredictable as well, and if you use affinity to run it on a dedicated worker, what's the point?

I think he makes some really great points here, and obviously has some pretty good experience at running data stores in the real world, at scale.

I think his comment about stateful sets is more focused about running data stores on distributed filesystems, or network attached storage. This isn't a good idea generally, and I am not going to go into why, but Cassandra and Etcd advise against this for a reason, its well documented. It wouldn't just be using a 'better' storage system, it would be designing and implementing a very complex operator as he mentions.

The real take away which I think we should all realise is that, this is shiny tech to run mission critical data stores on, unless you know what your doing and want to invest a lot of time and effort in to it. Right now it's probably better to use RDS/Cloud SQL, or just run a traditional postgres setup with a SAN and decent failover that is tried and tested.

> network storage like EBS volumes where AWS guarantees me that I won't have data corruption issues in exchange for money.

Source? I've never heard that claim, and in fact we've lost data on EBS (though it has been a couple years)

Yes they should. It's nice to have a (ultimately) data-center-wide scheduling platform with unified role-based access control, monitoring, utilization optimization and reporting, cost control and even exotic features like hardware validation. Just because earlier Kubernetes versions weren't good at everything at once doesn't mean it's doomed forever to be unsuitable for something.

Moreover, the use case he's discussing is quite unusual: Gravitational takes a snapshot of an existing Kubernetes cluster (including all applications inside of it) and gives you a single-file installer you can sell into on-premise private environments, basically it's InstallShield + live updating for cloud software. So, running everything on Kubernetes opens entirely new markets for a SaaS company to sell to.

This level of zero-effort application introspection hasn't been possible prior to Kubernetes, so that's another reason to use it for everything: it promises true infrastructure independence (i.e. developers do not have to even touch AWS APIs) that actually works.

I wouldn't say that you need deep knowledge at this point. However, you do need at least journeyman level knowledge. Lots of folks (including me, on Patroni) are working to make more things automatic and lower the knowledge barrier, but we're not there yet.

A big part of the obstacle is that preserving state in a distributed environment is just hard, no matter what your technology, and the failure cases are generally catastrophic (lose all data, everywhere). This is true both for the new distributed databases, and for the retrofits of the older databases. So building DBMSes which can be flawlessly deployed by junior admins on random Kubernetes clusters requires a lot of plumbing and hundreds of test cases, which are hard to construct if you don't have a $big budget for cloud time in order to test things like multi-zone netsplits and other Chaos Monkey operations.

Making distributed databases simple and reliable is a lot like writing software for airplanes, but clearly that's possible, it's just hard and will take a while.

They're not kubernetes-specific, but apparently there are people who think deploying PG in it will magically Just Work(tm).

What I took away from TFA is two-fold: clustered-DB administration is hard to automate (e.g., there's a reason DBAs exist), and a lot of the tooling DBAs use now has to be rebuilt to function in a k8 environment, or replaced.

The first struck me as blindingly obvious; the second I find highly relevant, personally.

It's not Kubernetes specific and that is actually addressed in the article!

It's not postgres specific either. It will happen whenever you have a cluster or HA solution which is not intimately cognizant of the app it is supporting.

Shit, it is hard enough to get any database extremely reliable even with their own solutions (Oracle RAC, Postgres SLONY, mysql replication).

Adding Kubernetes / Docker to that mix makes for an interesting life. Caution is advised.

I think it's not K8s-specific but specific to anything when there is an external automated orchestration that can mess with the software at random (schedule and reschedule deployments etc etc).

If you have N servers that are deployed by hand or, better say, with some automation ran by hand - you can be sure things are not moving at random. If you schedule a payload in K8s (or Docker Swarm or whatever) - the scheduler will only ask you for preferences, but will otherwise make decisions on its own. Like deciding to drain a node because disk is getting full or whatever. Or just schedule an updated service to another node because it felt like doing so. And most of software - Postgres included - doesn't have any idea about all of this.

Some of the points raised (like leader election) are not specific to orchestration, but some (like persistent storage for containers) are sort of unique to containerized world. Unless there is a crazy sysadmin who can walk into server room and randomly shuffle drives. :)

You will probably still encounter data loss.

There are two main things to understand. First is that your application is running in a distributed environment and will encounter data loss if it is not designed for it. Second, even if it is properly designed to run in a distributed environment it's also has to be aware that it's running on kubernetes, configured specifically for it.

I think I'm just echoing rb808's comment, below, who is more informative than I can be, but I originally thought to comment only that my impression without any kubernetes experience is that the likelihood is, based upon my impressions of the article alone, is that in all probability this article is recording a attempt to coerce the inappropriate solution to deliver a much more difficult to achieve result than at least the scope of the article indicates is appreciated.

I could have just said that even knowing nothing much beyond cursory reading about kubernetes, the article comes across as a excercise in attaining disappointment thru hurried assumptions about what constitutes both a silver bullet and the daemon to be dispatched from unruliness.

The part that is disconcerting is the introduction to the article as a interview with the CTO, but it only takes a turn for the worse almost immediately by admitting to I'm production deployment of the solution, to which the subsequent admission to encountered difficulties is not compounding the sin do much as burying this entire excercise beneath condemnation, if I simply put down the impression conveyed. This has to be at the very least terrible PR. I'm increasingly concerned too, about the abundance of misapprehension of not only the capabilities of file systems but just fundamental design constraints, at s level of understanding that I would have expected to be fired for from a operations position in any of my customers. Have I missed the redeeming features in my haste to comment? It just feels so imbalanced and insecure to be so forthright about the level of accomplishment that's claimed.

Josh Berkus gave a really good overview of Patroni at KubeCon Austin: https://youtu.be/Zn1vd7sQ_bc

Yes, and (1) no longer applies because service locators which locate the master are now easy to do.

The thing about stuff that is hard is that it's usually hard no matter which way you look at it. I don't really know anything about CockroachDB, but I use Couch on a daily basis. I like the way it works, but it's a massive foot-gun if you don't realise that you have to do things completely differently. In our shop we a variety of DBs (Couch, Postgres, MySQL and even MSSQL). If my only concern was scaling, I would not choose Couch. While it makes scaling and replication easier, it does so by making a whole ton of things harder (and basically unscalable, ironically). You still need to choose the tool most appropriate for your problem. (In case you are wondering, Couch is awesome for problems where you need versioned, immutable data -- for example financial systems. But there are a lot of trade offs with respect to querying).

Pg streaming replication is asynchronous by default and there are some big caveats besides performance when it comes to synchronous use cases, esp around automatic fail-over / recovery. The pg/stolon FAQ hints at some of these issues:

1. https://github.com/sorintlab/stolon/blob/master/doc/faq.md#h...

in the video referenced by @aberoham , a the end there is a question to the presenter (Josh Berkus), about the 2 types of replication their deployment tool is supporting.

this was on about 33rd minute of the presentation

He suggested , that if you need to use , synchronous replication, to use PostgreSQL 10 with synchronous quorum replication, so that it will not block transactions if one of the replicas fail.

The synchronous replication is a difficult from a performance point of view.

No, but Kubernetes is at peak hype right now. It’s a floor wax and a dessert topping.