I Stopped Using Kubernetes. Our DevOps Team Is Happier Than Ever
blog.stackademic.com
blog.stackademic.com
Literally every single decision they listed was to use any of the given tools in the absolute worst, incompetent way possible. I wouldn't trust them with a Lego toy set with this record.
The people who quit didn't quit merely out of burnout. They quit the stupidity of the managers running this s##tshow.
yes, it’s complex, but it’s simpler than running true high availability setups without something like it to standardize the processes and components needed. what i want to see is a before and after postmortem on teams that dropped it and compare their numbers like outages to get at the whole truth of their experience.
A new leader arrives and says, “we could optimize it by…”
I think the bottleneck is networking and how much crosstalk your control nodes can take, but that’s your architecture team’s job?
It makes sense that getting off k8s helped them if they were using it incorrectly.
If your api gets hosed, you can create a new cluster and tear down the old one and call it a day.
With a stateful cluster, you can’t do that. As such you put in a lot more care with eg k8s upgrades, or introducing new controllers or admissions/mutating webhooks.
Are these not covered by taint/toleration? I guess maybe isolation depending on what exactly they're demanding but even then I'd think it could work.
Can you elaborate on that? What does it do?
But this is also a caveat with "managing it yourself", you get a lot of people having ideas and shooting themselves in the foot with it
I'll take this opportunity to once again bitch and moan that Kubernetes just fucking refuses to allow the KV store to be pluggable, unlike damn near everything else in their world, because they think that's funny or something
https://www.perfectscale.io/blog/etcd-8gb
https://github.com/etcd-io/etcd/issues/9771
And yes, I agree not allowing a pluggable replacement is really stupid.
> stale bot marked this as completed (by fucking closing it)
Ah, yes, what would a Kubernetes-adjacent project be without a fucking stale bot to close issues willy nilly
Other than that, the experience I have is that using a managed solution like EKS and one cluster per env (dev, staging, prod) with namespaces to isolate projects takes you a long way. Having used k8s for years now, I'm probably biased, but in general I disagree with many of the k8s-related posts that are frequently upvoted on the front page. I find it gives me freedom, I can iterate fast on services, change structure easily without worrying too much about isolation, networking and resources. In general I feel more nimble than I used to before k8s.
After all Kubernetes provides all the primitives you need to enforce separation. You wouldn't create separate VMWare production and test clusters either unless you have a good reason.
Edit: To expand on it a little bit. I think there is always a real, theoretical risk that must be taken into account when you design your infrastructure. But when experience tells you that accounting for this potential risk may not be worth it in practice, you might get away with discarding it and keeping your infra lean. (Yes, I am starting to sweat just writing this).
I've actually asked for a task to be reassigned to somebody else before now on the grounds that I knew it deserved to be done the simple way but could not for the life of me bring myself to implement that.
(the trick is to find a colleague with a task you *can* do that they hate more and arrange a mutually beneficial swap)
Also I have yet to experience that an outage of any kind had any negative consequences for me personally. As long as you stand by the decisions you made in the past and show a path forward, people (even the higher-ups) are going to respect that.
Anticipating every possible issue that might or might not occur during the lifetime of an application just leads to over-engineering.
I think rationalizing it a little bit may also help with the paranoia.
Edit: And even when you "brick" it and cannot roll back, there is still a way to bring your control plane back by using an etcd backup, right?
I completely agree with keeping everything as simple as possible though. No extra clusters if not absolutely necessary, and also no extra namespaces if not absolutely necessary.
The thing with Kubernetes is that it was designed to support every complex situation imaginable. All these features make you feel as though you should make use of them, but you shouldn't. This complexity leaked into systems like Helm, which why in my opinion it's better to roll your own deployment scripts rather than to use Helm.
I agree that k8s offers many features that most users probably don't need and may not even know of. I found that I liked k8s best when we used only a few, stable features (only daemonsets and deployments for workloads, no statefulsets) and simple helm charts. Although we could have probably ditched helm altogether.
And operators/helm charts/CRDs use APIs which can and are deprecated, which can cause outages. It pays to make sure your infrastructure is automated with Got apps, CICD, and thorough testing so you can identify the potential hurdles before your cluster upgrade causes unplanned service downtime.
It is a huge effort just to “run in place” with the current EKS LTS versions if your company has lots of 3rd party tooling (like K8s operators) installed and there isn’t sufficient CICD+testing to validate potential upgrades as soon after they are released.
3rd party tooling is frequently run by open source teams, so they don’t always have resources or desire/alignment to stay compatible with the newest version of K8s. Also, when the project goes idle/disbands/fractures into rival projects, that can cause infra/ops teams time to evaluate the replacement/ substitute projects which are going to be a better solution going forward. We recently ran into this with the operator we had originally installed to run Cassandra.
Then we have test environments, around 20 of them: 20 clusters.
Then there are 10s of clusters installed on customers' infra.
So 47 clusters isn't really a huge/strange number.
I've never seen that, but yes 47 seems like a lot. Often you'd need production, staging, test, development, something like that. Then you'd add an additional cluster for running auxiliary service, this is services that has special network access or are not related to you "main product". Maybe a few of these. Still there's a long way to 47.
Out in the real world I've frequently seen companies build a cluster per service, or group of services, to better control load and scaling and again to control network access. It could also be as simple as not all staff being allowed to access the same cluster, due to regulatory concerns. Also you might not want internal tooling to run on the public facing production cluster.
You also don't want one service, either do to misconfiguration or design flaws, taking down everything, because you placed all your infrastructure in one cluster. I've seen Kubernetes crash because some service spun out of control and then causing the networking pods to crash, taking out the entire cluster. You don't really want that.
Kubernetes doesn't really provide the same type of isolation as something like VmWare, or at least it's not trusted to the same extend.
Why couldn't you do that with a dedicated node pool, namespaces, taints and affinities? This is how we run our simulators and analytics within the same k8s cluster.
In my experience companies do not trust Kubernetes to the same extend as they'd trust VLANs and VMs. That's probably not entirely fair, but as you can see from many of the other comments, people find managing Kubernetes extremely difficult to get right.
For some special cases you also have regulatory requirements that maybe could be fulfilled by some Kubernetes combination of node pools, namespacing and so on, but it's not really worth the risk.
From dealing with clients wanting hosted Kubernetes, I can only say that 100% of them have been running multiple clusters. Sometimes for good reason, other times because hosting costs where per project and it's just easier to price out a cluster, compared to buying X% of the capacity on an existing cluster.
One customer I've worked with even ran an entire cluster for a single container, but that was done because no one told the developers to not use that service as an excuse to play with Kubernetes. That was its own kind of stupid.
Network Policies have solved that at least for ingress traffic.
Egress traffic is another beast, you can't allow egress traffic to a service, only to pods or IP ranges.
I can see security teams getting uppity about that.
Also budgetary and org boundaries, cloud providers, disaster recovery/hot spares/redundancy/AB hotswap, avoid single tank point of failure.
single YAML point of failure
mobile autocorrect is super "helpful"
So sure, I can kinda see your point, but it feels rather moot. In the cluster, there isnt much that is a single point of failure that also wouldn’t be a point of failure in multiple clusters.
Normally in K8s, segregating environments is done via namespaces, not clusters (unless there are some very specific resource constraints).
To be clear I'm not defending the 47 Cluster setup of the OP, just the practice of separating Development/Production.
If you say they share the same control plane is commingling… then what do you think a cloud console is? And if you are using different accounts there… then I hope you are using dedicated resources for absolutely everything in prod (can’t imagine what you’d pay for dedicated s3, sqs) because god forbid those two accounts end up on the same machine. Heh, you are probably violating compliance and didn’t even know it!
Sigh. I digress.
It does make some sense though. People are not perfect, especially in large organizations, so there is value in just following the masses rather than doing everything your own way.
Which for separating dev and prod, you absolutely should be.
(Separate accounts for AWS; separate projects would suffice for GCP.)
That said it sounds like these people just made a cluster per service which adds a ton of complexity and loses all the benefits of k8s.
One cluster per country in prod, one cluster per team in staging, plus individual clusters for some important customers.
A DevOps engineer famously pointed that it was stupid since they could access everything with the same SSO user anyway, and the CISO demanded individual accounts with separate passwords and separate SSO keys.
In a cloud environment the backup cluster can be scaled up quickly if it has to take over, so while it's idling it only requires a few smaller nodes.
Many workloads don't need to be multi-region as a requirement. So they might run just on one cluster with the option to fail over to another region in case of an emergency. Running a workload on one cluster at a time (even with some downtime for a manual failover) makes a lot of things much easier. Many workloads don't need 99,99% availability, and nothing awful happens if they are down for a few hours.
k8s, and the go runtime and network stack have been heavily optimized by armies of engineers at Google and big tech, so I am very suspicious of these claims without evidence. Show me the resource usage from k8s component overhead, and the 15 minute to 3 minute deploys and then I'll believe you. And the 200 file YAML or Helm charts so I can understand why in gods name you're doing it that way.
This post just needs a lot more details. What are the typical services/workloads running on k8s? What's the end user application?
I taught myself k8s in the first month of my first job, and it felt like having super powers. The core concepts are very beautiful, like processes on Linux or JSON APIs over HTTP. And its not too hard to build a CustomResourceDefinition or dive into the various high performance disk and network IO components if you need to.
I do hate Helm to some degree but there are alternatives like Kustomize, Jsonnet/Tanka, or Cue https://github.com/cue-labs/cue-by-example/tree/main/003_kub.... You can even manage k8s resources via Terraform or Pulumi
I feel you. I learned K8s with an employer where some well intentioned but misguided back end developers decided that their YAML deployments should ALL be templated and moved into Helm charts. It was bittersweet to say the least, learning all the power of K8s but having to argue and feel like an alien for saying that templating everything was definitely not going to make everything easier on the long term.
Then again they had like 6 developers and 30 services deployed and wanted to go "micro front end" on top of it. So they clearly had misunderstood the whole thing. CTO had a whole spiel on how "microservices" were a silver bullet and all.
I didn't last long there but they paid me to learn some amazing stuff. Which, in retrospect, they also taught me a bunch of lessons on how not to do things.
That's how I see most of these projects. You create a massively expensive infra because webscale, then 3 years down the road you (or someone else) gets to rebuild it 10x cheaper. You get to write two blog posts, one for using $tech and one for migrating off $tech. A line in the cv and a promotion.
But kudos for them for managing to stop the snowball and actually reverting course. Most places wouldn't dare because of sunken costs.
It makes sense to use whatever $tech helps you get an MVP out asap and iterate on it. once you're sure you found gold, then it makes sense to optimize for scale. The only thing I guess one has to worry about when developing something like that, is to make sure good scalability is possible with some tinkering and efforts and not totally impossible.
So those savings are possible only if your devs are willing to leave the lock in of comfort
They saved $100k in the move or 2% of their engineering costs. And they're still very much on the cloud.
If you tell most organizations that they need to change everything to save 2% they'll tell you to go away. This place benefited because their previous system was badly designed and not because it's cloud.
By the way, 8 DevOps engineers with $150k/year cloud bill deserves to be highlighted here. This is a very high amount of staff dedicated to a relatively small infrastructure setup in an industry that keeps saying "cloud will manage that for you."
Honestly they could have picked any stack as next one because the key win here was starting from scratch
My suspicion is the original environment (47 Kubernetes clusters, 200 YAML files, unreliable deployments, using 3 clouds, etc.) was not planned out. You probably had multiple teams provisioning infrastructure, half completed projects, and even dead clusters (clusters which were used once but were not destroyed when they were no longer used).
I give the DevOps team in the article a lot of credit for increasing reliability, reducing costs, and increasing efficiency. They did good work.
Out of curiosity, in what case would you _not_ see 200 files for 200 services? Even with Helm, you'd write a chart per app wouldn't you?
We could have spent a tiny fraction of that 10-dev-months to deploy something to production on a bare VM on any cloud platform in a secure and probably very-scalable way.
These days I cringe and shudder everytime I hear someone mentions writing "helm charts", or using the word "workloads".
I see this happening all the time and unless you really have DevOps or SysAdmins who are feel acting like 'assholes' enforcing rules it is going to be like that.
Of course 'assholes' is in quotes because they have to be firm and deny a lot fo crap to keep setup clean - but then also they will be assholes for some that "just want to do stuff".
You'll get the whole team helping you when replacing the tool and setting up a better solution.
If you blame anyone, people will start to be extra cautious and won't take any initiative.
But don't overuse it, if you always blame the tool, you'll end up like my ex colleague "Steve" where every failure was Microsoft's fault.
This does require management who won't punish you for recording it as your fault, though. I've been fairly lucky in that regard.
Steve usually said that Microsoft should ask for a confirmation before deleting anything, even with the "-Force" parameter.
It was Microsoft's fault when the whole test environment, that he spent two days setting up, was deleted with the "-Force" parameter. He said something along the lines of "Microsoft shouldn't let me do this".
Kubernetes is not an easy to use technology and sometimes its documentation is lacking. My gut feeling is Kubernetes is great if you have team members who are willing to learn how to use it, and you have a LOT of different containers to run. It probably is not the best solution for small to medium sized teams because of its complexity and cost.
> In short, organizational decisions and an overly cautious approach to resource isolation led to an unsustainable number of clusters.
And while I emphasize with how they could end up in this situation, it feels like a lot of words were spent blaming the tool choice vs being a cautionary tail about for example planning and communication.
Since you can't run the Fargate control plane that indirectly solved that problem for them
> As the number of microservices grew, each service often got its own dedicated cluster.
Wow. Just wow.
I've worked on both ECS and K8s, and K8s is much better. All of the problems they listed were poor design decisions, not k8s limitations.
- 47 Clusters: This is insane. They ack it in the post, but they could've reworked this.
- Multi-cloud: It's now not possible with ECS, but they could've limited complexity with just single-cloud k8s.
I thought for sure you were going to link to https://news.ycombinator.com/item?id=42226005 (Dear friend, you have built a Kubernetes; Nov 24, 2024; 267 comments)
Each cluster is wasting tons of cpu and i/o bandwidth just to be idle. I was told that it is etcd doing thousands i/o per second and this is normal.
For a few monolith
Infrastructure is a lost art. Nobody knows what they're doing. We've entered an evolutionary spandrel where "more tools = better" meaning the candidate for an IT role who swears by 10 k8s tools is always better than the one who can fix your infra, but will also remove k8s because it's not helping you at all.
Vendor lock in does not come about by relying only on one cloud, but by adopting non-standard technology and interfaces. I do agree that running on multiple providers is the best way of checking if there is lock-in.
Lowering the level of sharing further by running per-service and per-stage clusters, as mentioned in the piece was likewise at best an uninformed decision.
Naturally moving to AWS and letting dedicated teams handle workload orchestration at much higher scale will yield better efficiencies. Ideally without giving up vendor-agnostic deployments by continuing the use of IaC.
-replicates OS services poorly
-OS is likely running on a hypervisor divvying up hardware resources into VPS's
-wastes ram and cpu cycles
-forces kubectl onto everything
-destroys integrity of kernel networking basic principles
-takes advantage of developer ignorance of OS and enhances it
I get it, it's a handy hack, for non production services or oneoff installs, but then its basically just a glorified VM
To get to this point, someone must have fucked up way earlier by not knowing what they were doing. Don't do k8s kids !
Then I read someone saying that K8s should never be used for teams <20 FTE and will require 3 people to learn it for redundancy (in case used to self-host a SaaS product). This seemed really good advice.
Our team is smaller than 20FTE, so we use AWS/Fargate now. Works like a charm.
As of now, it seems like my only alternative is to run k8s on a 2GB of RAM system, so I'm considering moving to Hetzner just to run k3s or k0s.
ie nginx has the public facing ipv4 & ipv6 address(es), with docker swarm behind that communicating to the nginx proxies over ipv4
https://news.ycombinator.com/item?id=28838053 is where I learned about that, with the top comment showing a bazillion bookmarklet fixes. I'd bet dollars to donuts someone has made a scribe.rip extension for your browser, too
“Hey, look under all that DevOps cloud infrastructure! There’s some business logic! It’s been squashed flat by the weight of all the containers and orchestration and serverless functions and observability and IAM.”
() Even here, a lot of it comes from ignorance and human nature. 99.9% of people do not set out to fail. They usually do the best they can and sometimes they make mistakes.
Performance isn’t the only reason to use container orchestration tools: they’re handy for just application lifecycle management.
> Stateless services → Moved to AWS ECS/Fargate.
> Stateful services → Deployed on EC2 instances with Docker, reducing the need for complex orchestration.
> Batch jobs → Handled with AWS Batch.
> Event-driven workflows → Shifted to AWS Lambda.
And I would have gotten away with it too if only someone would rid me of that turbulent meddling cluster orchestration tooling!
I also hope they have a good support contract with AWS, otherwise they are going to be in for a fun surprise when ECS has some weird behavior or bug.
Smelly points from the article:
- "147 false positive alerts" - alert and monitoring hygiene helps. Anything will have a low signal-to-noise ratio if not properly taken care of. Been there, done that.
- "$25,000/month just for control planes / 47 clusters across 3 cloud providers" - multiple questions here. Why so many clusters? Were they provider-managed(EKS, GKE, AKS, etc.) or self-managed? $500 per control plane per month is too much. Cost breakdown would be great.
- "23 emergency deployments / 4 major outages" - what was the nature of emergency and outages? Post mortem RCA summary? lessons learnt?..
- "40% of our nodes running Kubernetes components" - a potential indicator of a huge number of small worker nodes. Cluster autoscaler been used? descheduler been used? what were those components?
- "3x redundancy for high availability" - depends on your SLO, risk appetite and budget. it is fine to have 2x with 3 redundancy zones and stay lean on resource and budget usage, and it is not mandatory for *everything* to be highly available 24/7/365.
- "60% of DevOps time spent on maintenance" - https://sre.google/workbook/eliminating-toil/
- "30% increase in on-call incidents" - Postmortems, RCA, lessons learnt? on-call incidents do not increase just because of the specific tool or technology being used.
- "200+ YAML files for basic deployments" - There are multiple ways to organise and optimise configuration management. How was it done in the first place?
- "5 different monitoring tools / 3 separate logging solutions" - should be at most one for each case. 3 different cloud providers? So come up with a cloud-agnostic solution.
- "Constant version compatibility issues" - if due diligence is not properly done. Also, Kubernetes API is fairly stable(existing APIs preserve backwards compatibility) and predictable in terms of deprecating existing APIs.
That being said, glad to know the team has benefited from ditching Kubernetes. Just keep in mind that this "you don't need ${TECHNOLOGY_NAME} and here is why" is oftentimes an emotional generalisation of someone's particular experience and cannot be applied as the universal rule.
Or course they are. The original value proposition of cloud providers managing your infra (and moreso with k8s) was that you could fire your ops team (now called "DevOps" because the whole idea didn't pan out) and the developers could manage their services directly.
In any case, your DevOps team has job security now.
The original value proposition is false, and more and more are realising this.
But I agree that moving your own infra or outsourcing operations when you have managed to do it on your own for a while is most likely misguided. Speaking from experience it introduces costs that cannot possibly be calculdated before the fact and thus always end up more complicated and costlier than the suits imagined.
In the past, when similar decicions were made, I always thought to myself: You could have just hired one more person bringing their own, fresh perspective on what we are doing in order to improve our ops game.
I think an operations background gives you a strong ability to smell nonsense and insecurity. The DevOps team seems to be people who want to be 'developers' rather than people who care about 'ops'. Yaml slinging without thinking about what the yaml actually means.
also : post paywall mirror https://archive.is/x9tB6
Either your pre profit or some other bogus entity, or your company streamlined moving to k8s and then further streamlined by cutting away things you don't need.
I'm frankly just alarmed at the thought of wasting that much revenue, I could bring up a fleet of in house racks for that money!
What a doozy of a start to this article. How do you even reach this point?
> "Create an account to read the full story."
If they had told me that from the start, I would not bother with creating an account at all. The message implies that a free account would be enough
It would be nice if there was a common practice of including a link to the alternative version as the first line of a Medium article.