This is what I was wondering, in the article it looks like kubernetes is just used to launch the node containers - why is the database and rabbitmq outside of kubernetes? This architecture looks like it’s been cobbled together by a junior
Is that still true?
I wouldn't call the parent comment charitable enough, because there definitely can be some reasons for running stateful workloads even outside of containers altogether (familiarity included), but at the same time it feels like a lot of effort has been invested into making that a non-issue.
For example, how many database Operators are now available for Kubernetes: https://operatorhub.io/?category=Database&capabilityLevel=%5...
Honestly, as long as you have storage and config setup correctly, it's not like you even need an Operator, that's for more advanced setups. I've been running databases in containers (even without Kubernetes) for years, haven't had that many issues at small/medium scale.
I run https://atomictessellator.com solo, using kubernetes, and my database, Minio object store, application servers, quantum workers, everything is all on kubernetes, it’s self healing and much simpler to run all the infrastructure the same.
Recently I had a node failure while I was sleeping and the whole system healed itself while I slept, the monitoring system didn’t even alarm me because the small blip of increased latency while the pods rebalanced wasn’t above the alert threshold so it didn’t even wake me up.
What happens in the article infra when the rabbitmq or database nodes fail? The whole system goes offline, which seems very silly setup when you have kubernetes sitting right there, who’s primary function is to handle all of this.
We run all of our stateful and stateless workloads on 10+ kubernetes clusters at work in multiple datacenters in multiple continents, and we serve 500 million users a month with it.
I wrote the first BORG version of DFP backend systems at Google, where we served billions of users billions of ads a day, and we used stateful infrastructure management on some of the first container runtime systems that inspired k8s during it's development.
Using rabbit and "most databases" native fallover strategy is fine for toy projects, but when you're operating at this scale, you need automated infrastructure provisioning and all of the automated tooling around it.
Then you get a bit worried that the Postgres Helm chart, while good, doesn't do what you want. So you update to use a dedicated clustered Postgres, using some Postgres clustering tech.
Finally, you're at so much scale you can throw giant wads of advertising cash at the problem, and you can use anything you like and it'll work. You just need to choose the best thing for your particular problem.
I’m not claiming that it’s totally bullet proof, I never said that - I’m saying that if you had a kubernetes cluster anyway why not benefit from its abilities? Especially when the alternative is single node, single points of failure, which is clearly inferior.
The "what if the storage detaches" argument could easily apply to the single node VMs too, in which case the outcome would be a total system failure.
We are discussing the contrast between the articles architecture and running everything on K8s ... and I'm saying that running everything on K8s is clearly better
> What happens in the article infra when the rabbitmq or database nodes fail?
I agree; I was replying to this invented problem:
> What happens in the article infra when the rabbitmq or database nodes fail?
It makes sense if you read the reply as a reply.