It's not enough to intercept `set search_path`, in fact any arbitrary function could change it, but Postgres doesn't report it back, unlike some other gucs.
18 karma · joined February 27, 2017
The Spilo docker container which packages PostgreSQL + Patroni was also always publicly available from the first day. I agree, the Spilo is a bit opinionated due to the way how it is used at Zalando.
These two projects have quite a long history and originally weren't even targeted to be deployed on K8s because back in 2015 K8s wasn't absolutely suited for running stateful workloads.
The most opinionated one is Zalando Postgres-Operator, and yes, first it was the internal tooling and was solving our specific problems. Now I would also argue that with such an amount of external contributors it already became a way more general solution.
After all, Patroni is so general, that we call it a template for PostgreSQL HA. You can take it and build something that you need/want without relying on Spilo and Zalando Postgres-Operator. Speaking of Patroni@K8s, there are already two very nice examples: Crunchy Data PostgreSQL Operator and StackGres are both relying on Patroni for running PostgreSQL HA on K8s.
If patroni can't update leader lock in Etcd, it will restart postgres in read-only mode. No writes will happen.
> like the patroni default haproxy config doesn't even seem to kill live connections which seems kind of risky.
That's not true: https://github.com/zalando/patroni/blob/master/haproxy.cfg#L...
On network partition Patroni will not be able to update leader key in Etcd and therefore restart postgres in read-only mode (create recovery.conf and restart). No writes will be possible.
Yeah, crunchy was a little faster with releasing it than Zalando, but try you look closer how they deploy postgres on kubernetes. Somehow it feels that they are trying to map 1 to 1 the same approach how folks used to run postgres on bare metal. That is: deploy master pod, wait until it's up and running, deploy a replica pod, and so on...
It doesn't really look cloud-k8s-native.
In my opinion such deployment should look absolutely different. You just need to deploy k8s manifest, which will create Secrets, StatefulSet and Service which will be used to connect to the master. The rest should happen automatically: * StatefulSet will start N pods with postgres * pods (Patroni) will elect leader. * elected leader will initialize (initdb) a new cluster * all other pods will get basebackup from the leader and become replicas * if the master(leader) pod die - other pods will elect a new leader * StatefulSet will start a replacement of failing pod and it will join the cluster as a new replica
And more important all it should happen without connection to Etcd, ZooKeeper or Consul. It should just use Kubernetes API.
Basically all this "magic" is already supported by Patroni https://github.com/zalando/patroni/pull/500
We are planning to merge this PR today or tomorrow, and it will make Patroni+PostgreSQL first class citizen on Kubernetes.
Sounds like a holy-war topic :) But lets be serious. How statically typed language helps you to avoid bugs in algorithms you implement? The rest is about proper testing.
> and because the proxy it brings out of the box makes it convenient: On any machine I connect to localhost:5432 to get to postgres
It seems like you are running a single database cluster. When you'll have to run and support hundreds of them you will change your mind.
> if the Postgres fails over, it ensures to disconnect me so that I'm not accidentally connected to a replica.
HAProxy will do absolutely the same.
> Bugs. While Stolon works great most of the time, every couple months I get some weird failure. In one case it was that a stolon-keeper would refuse to come back up with an error message, in another that a failover didn't happen, in a third that Consul stopped working (I suspect a Consul bug, the create-session endpoint hung even when used via plain curl) and as a result some stale Stolon state accidentally accumulated in the Consul KV store, with entries existing that should not be there and thus Stolon refusing to start correctly.
Yeah, it proves one more time: * don't reinvent wheel: HAProxy vs stolon-proxy * using statically typed language doesn't really help you to have less bugs
> I suspect that, as with other distributed systems that are intrinsically hard to get right, the best way to get rid of these bugs is if more people use Stolon.
As I've already told before. We are running a few hundred Patroni clusters with etcd and a few dozen with ZooKeeper. Never had such strange problems.
Talk was also recorded but video is not yet published.
And one more thing, you can try to run a Live-Demo the same way as I did it during presentation: https://cyberdemn.blogspot.de/2017/04/patroni-ha-postgresql-...
In the HAProxy config file you just need to list all Patroni nodes, specify a health-check and it will do a load-balancing for you.
The same approach works for example with AWS ELB.
If you read a further comments to this news you can get familiar with the part of new functionality of Patroni comparing to Governor.
should do the trick
patroni:8008/master will return http status code 200 only if the node running as elected master
patroni:8008/replica will return http status code 200 if node running as replica.
And final missing bit is a automation of generation of haproxy.cfg - it could be done with confd: https://github.com/kelseyhightower/confd
And here is an example of template file: https://github.com/zalando/patroni/blob/master/extras/confd/...
Let me as a question, what feature Stolon has and Patroni doesn't? Can you give an example?
From my side I can provide list of features available in Patroni, but not in Stolon. For example:
* there is no way to do a controlled failover (switchover) in Stolon.
* in Patroni it's possible to exclude some nodes from a leader race.
* Patroni supports cascading replication
* Patroni can take basebackup from replicas if they are marked with a special tag
* it's even possible to configure Patroni to use a custom backup/recovery solution instead of pg_basebackup
* Patroni can give you a hint that postgres must be restarted to apply some configuration changes
* with Patroni it's even possible to schedule switchover or restart restart of postges on some specific time (for example 04:00 am, when traffic is minimal)
* with Patroni you can control High-Availability / Durability ratio. I.e.: what to do if master postgres has crashed? Either you will try to start it back or failover to a replica. But start of a crashed postgres may take a long time. With Patroni you can configure that if postgres didn't started in some amount of seconds - please do failover.
Stolon provides cloud-native deployment and Patroni doesn't? This is not really true. The main idea of Patroni is to be not dependent on some specific technologies. It can work as on bare-metal with the same success as on Kubernetes. It's very easy to build a custom solution with Patroni for your use-case. For example there is a Spilo project: https://github.com/zalando/spilo, a docker image build with Patroni and wal-e for a cloud deployments.
// This implementation does not guarantee that only one client is acting as a // leader (a.k.a. fencing). A client observes timestamps captured locally to // infer the state of the leader election. Thus the implementation is tolerant // to arbitrary clock skew, but is not tolerant to arbitrary clock skew rate.