I'm very happy to see you better support for docker compose. I think about 50% of the time I find Kubernetes used in production/development when it could have just used Compose along with a simple Terraform or Pulumi deploy script.
Yup thanks for noticing, that's why I use it
However with Kubernetes your infrastructure will be ready to scale. You need to expose both front-end and back-end services under the same host? No need to tinker with nginx configs, you just create two ingresses and do it in a standardized way. You need to run your service with two replicas and rolling zero-downtime update? Kubernetes has it out of the box. Good luck implementing all the necessary dances with custom docker-compose setup without dropped requests. You want to customize healthcheck? Kubernetes got you covered with all the imaginable scenarios.
The only missing block in Kubernetes is support for building containers. This is implemented with docker-compose extremely elegantly and simple. That I admit.
That is true if you already have Kubernetes. If you don't, then you still need to run and configure the Kubernetes' control plane (e.g. kube-apiserver, etcd, scheduler, etc). Doing that alone may exceed the complexity of a simple setup.
I say this as someone who has looked at Kubernetes a lot and wanted to use it, but could never justify it. I have concluded multiple times that docker compose is a better fit for my use case.
> However with Kubernetes your infrastructure will be ready to scale.
True, but for many purposes you aren't gonna need it.
I hope you're not putting something like that in production.
Every time you hear yourself saying Kubernetes, "only", and "just" on the same sentence, please pause for a moment.
This Cult of Kubernetes has to go.
90% of the projects won't need scale. You're paying forward for something you won't use most of the time.
Speaking as a k8s admin.
"Simple setups" in Kubernetes will have massive amounts of complexity and overhead, they're just buried in a different abstraction layer.
If you have "free" k8s and don't need to touch it operationally? Sure, do everything with it!
But somebody is going to have to run that k8s environment. If it's you, and you're not a k8s expert, then you'd better buckle up, because all that hidden complexity is now your problem.
In the world where you need to deploy "an app" to "a VM", the simplicity of docker-compose is a sweet spot that any developer can grok without needing an encyclopedic knowledge of how to manage it.
https://github.com/operator-framework/awesome-operators archived since 2021 and now https://operatorhub.io/
I hadn't heard of this, interesting. Layers and layers of abstractions. What an interesting way to solve things. "It's YAML all the way down"?
- Prometheus operator
Lets us spin up a prometheus monitoring stack on our kubernetes cluster for monitoring metrics and scraping metrics from our services pretty much automatically.
- Thanos Operator
Spins up prometheus monitoring aggregations across multiple clusters with object storage backends, basically you can store metrics for years and query them with grafana.
- Strimzi Operator (Kafka)
Orchestrates a kafka cluster for kafka connect or even full blown kafka brokers with zookeeper.
- Istio Operator
Builds a service mesh on the cluster, offering MTLS and an envoy load balancer with some incredible flexibility
- Cert-manager operator
Lets us auto-generate certificates for resources via letsencrypt. This one is so nice for internal services and keeps certs valid without any hand holding
- Argo CD operator
We use this operator to deploy argocd pipelines and ship new services and updates via github actions in a very standardized way, and gives our users a UI to see what's going on.
- Actions runner controller
Using this actions runner controller we can offload github actions runners to self-hosted runners that automatically scale for pipeline workloads. Another set up and forget system and saves us from buying more github actions minutes.
Ultimately we use a ton of operators because it gives us an out of the box framework for tooling we need to deploy without reinventing the wheel. Of course some operators aren't worth using and it's definitely worth checking them out to see if they worth your time. A lot of them are really open to updates and changes also so you can help evolve them as you grow into other cases.
I will say, it's very important to understand what you are abstracting with these, you can absolutely blindly deploy services with operators and have it blow up in your face if you aren't sure what's going on behind the scenes or lack distributed service knowledge in production.
Ultimately, I would love for something to exist out there that had the opinions and scope of deployment like Swarm in terms of simplicity, but didn't conflate other tools out there so that ergonomics and documentation were better and less confusing. Kubernetes gives you so much, but I do feel like it's a misstep that the only reasonable way we've simplified Kubernetes for production workflows is to pay a large PaaS to manage it for us.
I will say Kubernetes being as open as it is and all the overlapping tooling would seem incredibly overwhelming for someone trying to enter that space.
Ingress is also more complicated/bespoke - the best I've found is traefik with labels for routing/config.
My advice today would be to scale Docker compose vertically (eg: on a dedicated server) - then move to Kubernetes.
The swarm middle ground isn't really worth it IMNHO.
One way round that is to use an NFS volume. However, I've hit problems with too many client NFS connections on a docker swarm and so found it better to mount the NFS volume on each host and use a bind mount instead.
Also the volume plugin spec is so simple that it is possible to maintain your own plugin (even without csi).
Fwiw I found swarm lovely and just so much easier to work with than anything else solving the same problems.
For a new app, one generally should and can embrace 12-facors, and delegate state to stateful services (managed databases, key-value stores, s3 etc).
Do note that for simple services, local disk can be very hard to beat for low complexity, extremely high performance - with the caveat that horizontal scaling might be tricky.
Ed: also depending on privacy requirements - self- hosted s3 (eg minio) might be tricky for a production load. OTOH self-hosted Kubernetes is no walk in the park either!
the inevitable "you could have used something simpler than Kubernetes!" comments that appear every time it's mentioned neglect to note that you're more likely to find a Kubernetes example for whatever you're doing readily available in the wild.
The A and B group can be implemented with docker compose and deployed to each cluster with pulumi/Terraform. Two or a hundred, it doesn't really matter how many cluster groupings you have.
Very much unlike Kubernetes, where even a small deployment can sometimes be maddening to debug. Great for larger deployments where complexity is basically guaranteed, but anything that could run on one server with a hot spare is a great fit for Compose!
It almost necessitates deeper monitoring and/or extra tools to give a dashboard/'single pane of glass', whereas I can run Compose and just log into the individual server and jump around logs.
Sidecars do not exist in kubernetes, unless you install something that adds them.
Containers are... containers? The same containers you'd see in docker.
Kubernetes does not need deeper monitoring compared to docker. You can jump around kubernetes logs just like you do with docker. Except you don't even need to log into individual server.
I don't use any kind of dashboard for my kubernetes cluster. I installed kubernetes-dashboard, so developers and managers can enjoy their dashboards but I've found no value with it. kubectl is more than enough.