Running Databases on Kubernetes
questdb.io
questdb.io
You can spend a lot of time getting databases and other stateful workloads to work -- mess around with StatefulSet and PVC on top of all the normal Kubernetes concepts, and what do you get in the end? Are you really better off than you would have been if you ran the database in EC2?
Plus, "herds not pets" kind of breaks down once you start using StatefulSets and PVCs. Those things exist to make Kubernetes more like a static environment for workloads that can't handle being run like ephemeral cattle. So why not just keep using your static environment?
If Kubernetes is the only workload management control plane you have, then I guess this makes sense. But if you are already able to deploy your databases with existing tools, and those existing tools don't really suck, it's probably not worth migrating. It would take a lot of time and introduce significant new risks and operational complexity without a compensating payoff.
I simply wanted to highlight that there are many ways to skin a cat, and "no shared tenancy" is not in itself a valid argument for hosting your DBs in k8s, even though there may be other good reasons.
There's always a point at which running things on Kubernetes becomes worth it. RDS is an expensive service, and if you want to enable every developer to boot up a development environment, or several even, if becomes prohibitivly expensive to use one process for everything. This is also true about running any workloads on Kubernetes to begin with though, and many companies definitely would be better off with a service like ECS/FarGate/CloudRun/Fly.io. Especially if they don't need the flexibility to build their own addon metrics/secrets/logging stacks.
Where I was the tooling was very focused on disposable api servers.
I've tried other Postgres operators and been disappointed and it did require a little learning, but it's not like getting replication, Patroni, etcd, PGBouncer, HAProxy, and pgBackRest all running for a high-availability Postgres deployment is easy and wouldn't require learning.
As the author says, "[k8s's] operator model allows end users to programmatically manage their workloads by writing code against the core k8s APIs to automatically perform tasks that would previously have to be done manually." To me, that's the benefit. The operator can handle tasks like adding a replica or failing over the primary to one of the replicas. I could presumably do some of that with other tools on bare metal/VMs (I can always shell-script things), but I've had a good experience with CloudNativePG's operator. Likewise, as the author says, making day-2 operations easier is a big thing.
K8s does have some annoying amount of complexity, but it's been nice overall.
You need ha because k8s should run already with automatic node upgrades.
You need a pod disruption budget to make sure it is running and switching over when a node fails or gets upgraded.
You want to either totally Oberprovision on memory or look into keep 2400 to make sure to fine-tune memory before k8s starts to throw your database out constantly.
K8s is not a VM.
If you use k8s and still don't take care of application migration strategies you still don't understand what cloud native means.
There are still other things missing here but still...
Of course excluding hobby people playing with k8s.
Memory and upgrading nodes are the two single most issues will see which disrupts service.
Otherwise k8s is a dream come true.
I still would try to use a db managed if it's critical.
Additional points: Zalando postgres operator is great and shows the real magic of k8s and operator.
Use a helm chart and just bring your own little database for dev test and e2e tests.
You can easily use Auto scaling for node profiles. No noisy neighbors. If your db is too small for normal nodes you don't have a problem anyway.
dev, test, and e2e tests should be done against full-size db clones
Real customer/sensitive data should not exist outside prod (and backup). So generally no, not full-size clones. I'd argue instrumentation in prod should give information on performance - for some tests/development you might need prod-size fake data.
Monitoring prod closely and understanding what could constitute a costly workload/query which would in turn require a temporary test env with similar sized dataset is the correct approach.
Imagine a system that could do this in CI/CD pipelines:
1) first, run all the tests on a tiny DB as usual 2) extract queries 3) run them against full-size DB branch/thin clone (thin provisioning, CoW; PII is not there of course, wiped out for security/compliance) -- auto-guessing parms, that's the trickiest part, but assume it's solved 4) collect all the details about performance, focusing on IO numbers (rows, block read/writes) 5) if some queries are off – say, you forgot LIMIT – post a warning, block the change, do not allow deploying it, letting backend dev fix it.
This would be proactive. And it's becoming possible with modern tools.
that's cute, what is your "full-size"? I don't have 2 days to run a test, and I'm pretty sure every single compliance requirements we are following would get obliterated the second someone hears about us doing that
Although, cannot agree with the former part. If your tests are running 2 days on a full size clone, and it's an OLTP case, what about users, do they suffer from long query duration too? It sounds like it's time to optimize queries and/or redesign test sets (or both).
If it's bad in testing, it will be bad in prod. That's the idea of testing.
Example: how do you check schema changes?
This is difficult to impossible to do with databases; even if your database has a built-in recovery method for when a primary is taken offline, in such a way that allows for zero-downtime in theory, the reality is that such mechanisms depend on the secondary staying online until the failover mechanism is complete. If you turn over control of node upgrades to the cluster provider, the node under the secondary will get rebooted in the middle of the failover process, and you will get downtime at best, data loss at worst. What kubernetes teaches us is that databases aren't tied to the literal VM they're running on (which is now cattle), but rather on the availability of that node. If you run databases on kubernetes, you need to have a mechanism to slow down node upgrades.
Source: helped run hundreds of Elasticsearch and Kafka nodes on kubernetes in production at one point in my career
I've done it with Cassandra...and yeah Kafka can do it I've heard.
But those can be 30 hour operations even with you ducks in a row, and you better have backup strategies ready.
Fun story, Amazon said rds would be always be zero downtime upgrades. But then came a major version upgrade and .... Surprise it wasn't.
Cloudnative PG rebuilds the secondary during failiover from the streamed WAL to an S3 endpoint. No primary needed.
1. No way to signal that the workload is ready to accept traffic but not ready for disruption (such as an async background refill)
2. No way to provide backpressure over a time window, vs just instantaneous (ie during a rolling node upgrade PDB doesn’t prevent you from being moved multiple times)
3. No way for an admin and workload owner to coordinate on the amount of disruption a particular disruption will cause and discriminate between light and heavy (restart on node, recreate on node, move across node)
But yes, everyone with a nominally HA workload should have a PDB.
K8s, if anything, is an API. An API that allows you to interact with compute, storage and networks in a way that is abstracted from the actual underlying infrastructure. This is incredibly powerful. You can, essentially, code and automate all your infrastructure.
But this goes beyond deployment, something you could achieve (more or less) with tools like Terraform or Pulumi. Enter "Day 2 operations".
Day 2 operations are essential for any database. And cloud services have done a good job at automating them. Speaking of Postgres, my daily job, things like HA, backups but also minor and major version upgrades are table stakes day 2 operations.
If you want to build these day 2 operations in the cloud (say on VMs), even though you have APIs do to so, a) they don't implement a pattern like Kubernete's reconciliation cycle; and b) you have a distinct API per cloud. K8s solves both problems, making it way "cheaper" to build such an automation. On K8s, a given operator can code these day 2 operations against K8s APIs. Therefore, if you want to build such automation, either you are a cloud provider (and potentially do this only for your own cloud) or you do it on Kubernetes.
This is so much true, that existing operators have already gone beyond what DBaaS do. Speaking of StackGres [0] (disclaimer: founder), we have implemented day 2 operations (other than the "table stakes" ones that I mentioned before) that no other DBaaS offers as of today, such as vacuums, repacks and even benchmarks (and more day 2 operations will be developed). See [1] for the CRD specs of SGDbOps, our "Day 2 operations" if you are interested.
[0] https://stackgres.io [1] https://stackgres.io/doc/latest/reference/crd/sgdbops/
I won't disagree with others that RDS is probably worth it until you need something very specific or have reached a certain scale.
Happy to share tips or pointers for anyone going down this path specifically with MySql or database workloads in general.
ps. And how you deal with migrations? ps. Forgive me if I'm asking for too much!
There is a main operator responsible for all the databases. It handles configuration changes, provisioning pods and slowly rolling out changes. In kube we model this with a custom resource we've defined called a KeyspaceShard which represents a named set of database instances that should participate in replication together. Once provisioned, the pods know how to hook up and detach from Vitess without requiring further involvement from the operator. Vitess handles backups and maintains the replication topology. "Complicated" is an apt description of what it does but not "complex". Evicting a database pod and letting the system reschedule and converge is a routine operation that doesn't cause much concern.
Migrations are done with gh-ost, which has its own custom operator that manages the lifecycle of the migration and ties into self service tooling we provide that is integrated with our build and deploy system.
On latest conference their CEO said it's all about serverless and virtualization. Having kubernetes doing all the work [1] - "we are doing it automatically for you with our management which is based on using multi-tenant kuebrentes deployment". Even more surprising is that instead of using NVMEs they want to use s3 for backend storage and NVME is only cache [2] :o I am not database expert, but this is very interesting.
The only thing I can think of is cost. My usage probably isn't high enough where there is any financial benefit to an alternative... but if it was, maybe I'd be considering this.
When you start to have dozens or hundreds of databases in production, and developers asking "I need Postgres in production, why can't I just click a button and get a Postgres instance for my service in production?" then scaling the monitoring and firewalling gets a little more complicated. Hooking into standard Kubernetes monitoring and service meshes can really help to simplify things.
In 2019, every operator had crazy bugs, we inherited all of them. You have to solve not just databases level error but also errors popping from operators. If you can avoid databases on kubernetes, you should just do it.
It seems to get worse the further down the stack you go. I’ve seen tons of problems with operators, monitoring tools, and CNIs.
It’s somewhat better now but there is still a lot of stuff you can’t depend on. The CNCF seems to endorse pretty much anything even if it’s crap.
Does anyone have any specific recommendations on what to use (like which operator) when setting up a postgres cluster on k8s, specifically for standby replication?
We had a couple unusual requirements that the operator wasn't really suited for, so we ultimately ended up writing our own helm chart and forgoing the operator route altogether
We've been running Zalando Postgres Operator for all our prod and dev clusters (around 100 in total) and couldn't be happier.
- Can't set up two separate clusters in the same kubernetes instance because some cluster specific configuration is inexplicably set globally in the operator.
- Documentation and error messages are cryptic. Have to do a lot of trial and error to compensate for that. Maybe the issue here is a lack of experience with the stack used. Like Spilo and Patroni.
That phrase is inaccurate. With the cloud and K8, the pets move from being software that is tightly tied to the hardware to being a collection of configurations and software that are tightly tied to themselves.
We just make the actual physical hardware anonymous. But from the perspective of the actual stack, there is still a server with its cpus, filesystem, i/o and everything.
"Pets that you can carry" is more like it.
I have seen it fail way too many times. Inspecting a failing deployment that now has some magic Go code someone wrote running on this cluster. I can see using the basic kube building blocks: deployments, pods, config maps, etc.; there are enough guides and tools to help you out. As soon as you start writing code that runs in there, you're now dealing with two problems: your actual thing you're deploying, and now the operator.
Well, and then you need a mesh, and a way to manage certificates. and if it's a database to manage all the volumes. Everything looks good at the architect level - all the boxes and arrows line up, but when it breaks in production it's a nightmare to debug.
I worked in this problem space extensively until 2020, and I think that there are paths forward but they require changes in K8S that none of the folks involved seem motivated to make. Realistically to make databases in K8S work well today you need a database built for K8S rather than one adapted for K8S.
The building blocks present today are not fundamentally capable of building a positive UX for adapting existing databases to K8S, but this is something that is worth making possible and I hope the community gets there some day.
Is https://kubernetes.io/docs/concepts/workloads/controllers/st... unsuitable for that?
This will be a problem for any database where clustering is synchronous and a specific primary node must start first on a full cluster restart. There are other out of band hacks you can do with reassigning PVCs, but it’s never elegant in the current primitives provided.
During my work in this problem space I became convinced that primitives for stateful applications in K8S were built specifically without considering databases as a valid use case. Everything else is just hacks after the fact to make it “work”.
If a specific primary must start first, that’s partially what ordinals were intended to allow (0 is your primary, the others are always 1-N, and kube is responsible for ensuring the primary is never reassigned). I’d love to take feedback about places where the primitives are unusable, or ways they can be improved, because there are always new tools to add.
Ia there any chance you might be able to provide a link to these? I would be curious to take a look.
To handle this case, some teams may have a manual runbook, some teams may have some automation with ansible, some teams may have nothing. So, if someone can come up with some hacks and package that into a k8s operator, it is still a win. It seems to be the best primitives we have at the moment.
Stateful Sets were designed to standardize two hard problems: being able to identify all the valid members (pods identified by number that are running at most once on any node) and give admins the button to decide a member was never coming back (the force delete pod / force delete pv action). That was true black magic before - everyone did it their own way. So we worked with the ecosystem to enable vendors/communities/individuals to map those primitives into specific solutions, but did it somewhat deliberately as “we have to build this together”.
What I think the gap has been is that there is significant friction in between the three realms of expertise - knowing what kube is providing, knowing how to map that to a specific problem like translating the Galera runbook into operator/script logic, and then communicating that to the teams that will be accountable for reacting. Vendors have incentives to make you pay for that expertise (or may not have it), large organizations hire people to provide it (most large db on kube deployments are also large tech companies), and in between you have a lot of uncertainty and knowledge gaps that doesn’t necessarily transfer, and that is what drives “Kube isn’t great for stateful”.
It’s ironic to me because StatefulSets were intended to take advantage of those incentives to help the ecosystem scale, and the result is “worse is better” in that many more people can do state on HA DBs than were successful on VMs or metal, but it doesn’t mean they’re completely successful and when people hit the rough edges it hurts. We can do better (that’s partially my day job), but there is a lot of pain people have taken so far that was probably unnecessary. You should use managed DBs if you can - and when you don’t, Kube should be the best alternative it can be (which isn’t far - many DB SaaS uses some kube), and that’s what we need to focus on.
Can you say what you see as those possible paths forward and what changes they would require?
After switching to the Chrunchy Data pg operator v5 on k8s, we've had close to zero problems - one or two times a year the log shipping / HA replication fails and we have to restart it, but it's really neat! I can *warmly* recommend it; it really is CloudSQL in K8S.