https://en.wikipedia.org/wiki/Principle_of_least_astonishmen...
Though now AI slop is upon us so we'll probably be even worse off for a while.
Kubernetes is promoting Gateway API for a while now. It's in GA for 2 years already (while Ingress was in GA quite late, 2020/K8s 1.19?).
Sun-setting ingress-nginx was not exactly a secret.
The whole Ingress in k8s is marked in docs as "frozen" for a while as well. There are no radical steps yet, but it's clear that Gateway API is something to get interested in.
Meanwhile Nginx Gateway Fabric [1] (which implements gateway API) is there, still uses nginx under the hood and remains opensource. They even have a "migration tool" to convert objects [3].
There are still a few months of support and time to move on to a different controller. Kubernetes still continues support for ingress so if you want to switch and keep using Ingress, there are other controllers [2].
[1] https://gateway-api.sigs.k8s.io/implementations/#nginx-gatew...
[2] https://gateway-api.sigs.k8s.io/implementations/#gateway-con...
[3] https://docs.nginx.com/nginx-gateway-fabric/install/ingress-...
But the point is this, it worked, it does work and will, if given developer time continue to work.
I now need to schedual in time to test the changes, then adjust the metrics and alerting that we have.
For no gain.
It just feels like kuberenetes is carbon fibre programming.
well that's the root of the problem, no? there's no one who wants to maintain the complex lua written to make nginx cloud native. they were looking for maintainers for quite some time with no one stepping up
and I'm not surprised, their issue tracker always was full of very entitled people, so you would be doing a stressful/thankless job... for what exactly?
A deal with the devil was made. The C suite gets to tell a story that k8s practices let you suck every penny out of the compute you already paid for. Modern devs get to do constant busy work adding complexity everywhere, creating job security and opportunities to use fun new toys. "Here's how we're using AI to right size our pods! Never mind the actual costs and reliability compared to traditional infrastructure, we only ever need to talk about the happy path/best case scenarios."
Kubernetes is incredibly reliable compared to traditional infrastructure. It eliminates a ton of the configuration management dependency hellscape and inconsistent application deployments that traditional infrastructure entails.
Immutable containers provide a major benefit to development velocity and deployment reliability. They are far faster to pull and start than deploying to VMs, which end up needing some kind of annoying deployment pipeline involving building images or having some kind of complex and failure-prone deployment system.
Does Kubernetes have its downsides? Yeah, it’s complex overkill for small deployments or monolithic applications. But to be honest, there’s a lot of complexity to configuration management on traditional VMs with a lot of bad, not-so-gracefully aging tooling (cough…Chef Software)
And who is really working for a company that has a small deployment? I’d say that most medium-sized tech companies can easily justify the complexity of running a kubernetes cluster.
Networking can be complex with Kubernetes, but it’s only as complex as your service architecture.
These days there are more solutions than ever that remove a lot of the management burden but leave you with all the benefits of having a cluster, e.g., Talos Linux.
This is not always a problem of Kubernetes itself though, but of teams always chasing after the latest shiny thing.
Automatically created for me: - Ingress, TLS, Domain name, Deployment strategy, Dev/Prod environments through helm, Single repo configuration for source code, reproducible dev/prod build+run (Docker)...
If a company sets this up correctly developers can create tooling incredibly fast without any tickets from a core infra team. It's all stable and very performant.
I'd never go back to the old way of deploying applications after seeing it work well.
How long would you estimate that deployment would have taken with more a „classic“ approach? (e.g. deploying to a Java application server)
With a well managed Kubernetes, around 5-15 minutes. Not a theoretical time, I have personally had thousands of devs launch that quickly on clusters I ran.
I find that it has its place in companies with lots of micro services. But I think that because it is made "easy" it encourages unnecessary fragmentation and one ends up with a distributed monolith.
In my opinion, unless you actually have separate products or a large engineering team, a monolith is the way to go. And in that case you get far with a standard CI/CD pipeline and "old school" deployments
But of course I will never voice my opinion in my current company to avoid the "boomer" comments behind my back. I want to stay employable and am happy to waste company resources to pad my resume. If the CTO doesn't care about reducing complexity and costs, why should I?
Also a release is just a PR merge + helm upgrade.
If one has a small project where a few seconds of downtime is acceptable, you can just setup a simple Github action triggered on commit/merge. It can scp the file to the server and run "sysctl restart" automatically. I have used this approach for small side projects (even with external paying users)
And if you need a "no downtime" release, a proper CI/CD pipeline can handle a blue/green switch. I don't think you would spend much more time setting that up, than Kubernetes from scratch unless you have extensive experience with Kubernetes.
Typically, a k8s installation is looked after by a cross-functional Platform team, who look after not just the k8s cluster but also the gateways, service mesh, secrets management, observability and other common services, shared container images, CI/CD tooling, as well as platform security and governance.
These platform services then get consumed by the feature dev teams (of which there could be anywhere between half a dozen and multiple thousands). To deploy a new app, those dev teams need only create a repo and a helm chart, and the platform's self-service tooling will do the rest automatically. It really shouldn't take more than a few minutes for a team with some experience.
Yes, it's optimised for a very different scale of operation than a single server at a managed hosting provider. But there are plenty of situations in which that scale is required, and it's there that k8s shines.
I have a small application running under single-node k3s. It's slightly (but not hugely) easier to work with then the prior version that I had running under IIS.
The fuck it is.
> It eliminates a ton of the configuration management
Have you used k8s recently? to get it secure and sane is a lot of work. Even if you buy in sensible defaults, its a huge amount of work to get a safe, low blast radius deployment pipeline working reliably
Like if you want vaguely secure secrets, thats an add on. if you want decent non-stupid networking, thats an addon, Everything is split horizon DNS.
Thats before we get to state management, trying to play the pvc lottery, is not fun. which means its easier to use a clustered filesystem. Thats how fucked it is.
> there’s a lot of complexity to configuration management on traditional VMs
Not really, you need at least terraform to spin up your k8s cluster in the first place, its not that much harder to extend it to use real machines instead.
It is more expensive, unless you're binpacking with docker.
> cough…Chef
Chef can also fuck off. Although facebook use it on something like 8 million servers, somehow.
> Networking can be complex with Kubernetes
try making it use ipv6.
Look what the industry needs is a simple orchestration layer that places docker containers according to a DAG. You can have dependencies, and if you want a plugin system to allow you to paint yourself into a corner.
Have some hooks so we can trigger actions based on backlog
Leave the networking to the network, because DHCP and DNS are a solved problem.
What I'm describing is basically ECS, but without the horrid config language.
If you were working in the orgs targeted by k8s, I think it was generally more of a mess. Think about managing a park of 100~200 servers with home made bash scripts and crappy monitoring tools and a modicum of dashboards.
Now, k8s has engulfed a lot more than the primary target, but smaller shops go for it because they'r also hoping to hit it big someday I guess. Otherwise, there will be far easier solutions at lower scale.
E.g., Chef Software, especially after its acquisition, is just a dumpster fire of weird anti-patterns and seemingly incomplete, buggy implementations.
Ansible is more of the gold standard but I actually moved to Chef to gain a little more capability. But now I hate both of them.
When I just threw this all in the trash in my HomeLab and went to containerization it was a major breath of fresh air and resulted in getting a lot of time back.
For organizations, of the best parts about Kubernetes is that it’s so agnostic so that you can drop in replacements with a level of ease that is just about unheard of in the Ops world.
If you are a small shop you can just start with something simpler and more manageable like k3s or Talos Linux and basically get all the benefits without the full blown k8s management burden.
Would it be simpler to use plain Docker, Docker Swarm, Portainer, something like that? Yeah, but the amount of effort saved versus your ability to adapt in the future seems to favor just choosing Kubernetes as a default option.
It's also basically a standard API that every cloud provider is forced to implement, meaning it's really easy to onboard new compute from almost anyone. Each K8s cloud provider has its own little quirks, but it's much simpler than the massive sea of difference that each cloud's unique API for VM management was (and the tools to paper over that were generally very leaky abstractions in the pre-K8s world).
If you are in the position to pick a config management system, the best you can do is to chart out your current and known upcoming use cases. Then choose the tool that sucks the least for your particular needs.
And three years down the line, pray that you made the right choice.
Yes, kube is hideously complex. Yes, it comes with enormous selection of footguns. But what it does do well, is to allow decoupling host behaviour from service/container behaviour more than 98% of the time. Combined with immutable infrastructure, it is possible to isolate host configuration management to the image pre-bake stage. Leave just the absolute minimum of post-launch config to the boot/provisioning logic, and you have at least a hope of running something solid.
Distributed systems are inherently complex. And the fundamental truth is that inherent complexity can never be eliminated, only moved around.
So instead of an ansible playbook/role that installs, say, nginx from the distro package repository, and then pushes some specific configuration, I have a dockerfile that does the same thing? Woohoo?
Other issues like secrets and environment management is something I find way more annoying using a tool like Chef.
Try doing a chef policyfile bootstrap that gets some secrets using its own built in chef vault. You can’t do it without wild workarounds because the node isn’t granted access to secrets until it becomes a registered node, and it doesn’t register until a chef client run completes successfully. It’s a really dumb catch-22 design.
The solution is “just use a big 3 cloud secrets vault or Hashicorp vault” and that’s fine but it’s really strange that the tool can’t handle something so simple on its own.
Which solutions do you have in mind?
- VPS with software installed on the host
- VPS(s) with Docker (or similar) running containers built on-host
- Server(s) with Docker Swarm running containers in a registry
- Something Kubernetes like k3s?
In a way there's two problems to solve for small organisations (often 1 server per app, but up to say 3): the server, monitoring it and keeping it up to date, and the app(s) running on each server and deploying and updating them. The app side has more solutions, so I'd rather focus on the server side here.
Like the sibling commenter I strongly dislike the configuration management landscape (with particular dislike of Ansible and maintaining it - my takeaway is never use 3rd party playbooks, always write your own). As often for me these servers are set up, run for a bit and then a new one is set up and the app redeployed to that (easier than an OS upgrade in production) I've gone back to a bash provisioning script, slightly templated config files and copying them into place. It sucks, but not as much as debugging Ansible has.
yes the possix shell is not a good language which is why thinks like perl, python and even php or C got widely used but there is a intermediate layer with tools like fabric(https://www.fabfile.org/) solving a lot of the problems with the fully homegrown without locking you into the "Infrastructure as(manually edited) Data" paradigm that only really works for problems of big scale and low complexity which is exactly the opposite of what you see in many enterprise environments.
Kubernetes is for rather special case environments. I am coming around to the idea of using Kubernetes more, but I still think that if you're not provisioning bare-metal worker nodes, then don't bother with Kubernetes.
The problem is that Kubernetes provides orchestration which is missing, or at least limited, in the VM and bare-metal world, so I can understand reaching for Kubernetes, because it is providing a relatively uniform interface for your infrastructure. It just comes at the cost of additional complexity.
Generally speaking I think people need to be more comfortable with build packages for their operating system of choice and install applications that way. Then it's mostly configuration that needs to be pushed and that simplifies things somewhat.
> Generally speaking I think people need to be more comfortable with build packages for their operating system of choice and install applications that way. The it's mostly configuration that needs
why, it’s 2025, docker / container makes life so easy
They should understand CS/CE core fundamentals but they don't need to know how to admin.
Oh wow, so uh... I'm managing around 1000 nodes over 6 clusters, alone. There's others able to handle things when I'm not around or on leave and meticulously updated docs for them to do so but in general am the only one touching our infra.
I also do dev work the other half of the week for our company.
Ask your boss if he needs a hand :)
How many different applications/services are you running?
In any case, absolutely amazing what one person can manage with modern infrastructure.
At one job I was the only IT person and we had ~250 plain boring VMs on some bare metal Linux/KVM hosts. No config management. No Kubernetes. I fixed that quickly. There was one other guy capable of taking a look at most of it.
I was also doing the software builds and client releases, client support, writing the documentation for the software, and fixing that software.
I suspect we would have had no problem scaling up with some better tooling. Imagine a team of 150? When people tell me things like that, it sounds more like the solution isn't much of a solution at all.
Hehe, you lack skill in empire building. You know "leading a team of highly motivated team of 50+ devops engineers". The kind of talent that postpones patching until you are back from vacation. Or deploying config change that needs at least two rollbacks before finally going in.
Because 100-150 for the devops would be crazy for a mid-sized system like that.
Unless you're managing Windows servers or something.
Have you been in a company with ~2000+ servers where devs install their apps on these OSs and building packages that refuse to upgrade to the latest OS? I mean even with LTS a 20 year old company may still have 3-4 LTS OSs because that last 5% refuse to or cannot upgrade their application to work with the new OS. Sure you could VM the entire thing, but Docker + K8s removes that completely.
Also, intricate linkage between an app and the host OS also means there’s more work involved with upgrading.
THats what ld_preload is for. But real talk, if you have 2k servers and you can't package your apps to run on your OS, then you need a different platform team.
We managed 36k servers using fucking salt and perl. We were packaging nvidia drivers and all sorts. One system that everyone used still needed the athena widget set.
But the main point is, if you're using old packages, then you're gonna get hacked. You either need to kill that app, fire that developer or virtualise it and fill out the risk register, and do monthly recovery tests.
Docker allows you to pack in CVEs like no tomorrow. so sure k8s can let you do that, and given the hardly anyone properly enforces namespace isolation, so they can have a service mesh, you can still steal loads of data from a compromised container.
I fully accept that there are sizes and complexities where k8s is a reasonable choice, and sometimes it's a reasonable choice because it's easier to hire for, but the bar should be a lot higher than what it currently is.
It's a reason why I'm putting together alternatives for those of my clients who wants to avoid the complexity.
We have Configuration Management systems like Puppet in mature enough state for over a decade now.
I haven't installed server manually or "with handmade scripts" in good 12 years by now.
We have park of around 100-200 servers and actually managing hardware is tiny part of it
> Now, k8s has engulfed a lot more than the primary target, but smaller shops go for it because they'r also hoping to hit it big someday I guess. Otherwise, there will be far easier solutions at lower scale.
K8S is popular because it gives developers a lot of power to deploy stuff, without caring much at underlying systems, without bothering ops people too much. Cloud-wise there is a bunch of native ways to just run a few containers that don't involve it but onprem it is nice way to get a bit faster iteration cycle on infrastructure, even if complexity cost is high.
It is overkill for I'd imagine most stuff deployed in K8S and half of deployments are probably motivated by resume padding rather than actual need.
Not even that. One repository I checked this week had some commits which messages were like "synchronize code with what is on production server". Awesome. And that's not counting the number of hidden adhoc cronjobs on multiple servers.
Also as a dev I like having a pool of "compute" where I can decide to start a new project whenever instead of having to ask some OPS team for servers, routing, DNS config.
That said, (a) the Gateway API supercedes Ingress and provides much more functionality without much more complexity, and (b) NGINX and HAproxy have Gateway controllers.
To generally answer your question, I use HN, /r/devops and /r/kubernetes to stay current. I'm also working on a weekly blog series wherein I'll be doing an overview and quick start guide for every CNCF project in their portfolio. There's hundreds (thousands?) of projects in the collection, so it will keep me busy until I retire, probably :)
I was one of those whose first reaction was surprise, because ingress was the most critical and hardest aspect of a kubernetes rollout to implement and get up and running on a vanilla deployment. It's what cloud providers offer out of the box as a major selling point to draw in customers.
But then I browsed through the Gateway API docs, and it is a world of difference. It turns a hard problem that requires so many tutorials and products to help anyone get something running into a trivially solvable problem. The improvements on their security model is undoubtedly better and alone clearly justifies getting rid of ingress.
Change might be inconvenient, but you need change to get rid of pain points.
Yet they are retiring a core Ingress that has been around for almost as long as Kubernetes has.
Kubernetes is a gift.
There is a wild-grow of 80% solved problems in the Kubernetes space though, and especially the DevOps landscape seems to be plagued by half-solutions at the moment.
I think part of the complexity arises from everything being interconnected services instead of simple stand-alone software binaries. Things talking with other things, not necessarily from the same maker or ecosystem.
I don't understand decisions such as these though, retiring de facto standards such as Ingress NGINX. I can't name a single of our customers at $WORKPLACE that's running something else.
Nonetheless, it was around a full decade before they finally decided to retire it. It's not like this is something they introduced, advertised as the ideal fit for all production use cases, and then promptly changed their minds. It's been over a decade.
Part of the problem here is the Kubernetes devs not really following their own advice, as annotations are supposed to be notes that don't implement functionality, but ingress-nginx allowed you to inject arbitrary configuration with them, which ended up being a terrible idea in the main use Kubernetes is really meant for, which is you're an organization running a multi-tenant platform offering application layer services to other organizations, which it is great for, but Hacker News with its "everything is either a week one startup or a solo indy dev" is blind to for whatever reason.
Nonetheless, they still kept it alive for over a decade. Hacker News also has the exact wrong idea about who does and should use Kubernetes. It's not FAANGs, which operate at a scale way too big for it and do this kind of thing using in-house tech they develop themselves. Even Google doesn't use it. It's more for the Home Depots and BMWs of the world, organizations which are large-scale but not primarily software companies, running thousands if not millions of applications in different physical locations run by different local teams, but not necessarily serving planet-scale web users. They can deal with changing providers once every ten years. I would invite everyone who thinks this is unmanageable complexity to try dipping their toes into the legal and accounting worlds that Fortune 500s have to deal with. They can handle some complexity.
Plus, the ops side has a lot of challenges that can really be a different beast compared to the application side. The breadth of knowledge needed for the job is staggering and yet you also need depth in terms of knowing how operating systems and networks work.
This is one of those explanations that sounds reasonable but when you actually experience it you realize the explanation makes no sense.
If you're "running behind of technical debt" you'll always feel understaffed no matter how much staffing you have. And adding more staffing will make your tech debt worse.
Plus, tech debt doesn't really exist. It's a metaphor for all the little annoyances in your system that add up, but the metaphor makes it sound like it's the problem of management or accounting to solve when it's actually created by developers and solved by developers.
> no matter how much staffing you have
That’s not what I said. I said that there tends to be not enough staff. Businesses are more willing to hire software engineers (shipping features = revenue) than hiring DevOps people (keeping the lights on).
> tech debt doesn’t really exist.
Well that’s news to me. I’m pretty sure it exists. It has an entire Wikipedia article, and that article doesn’t agree with your definition.
And yes, more staff would help. Hiring me literally helped my organization fix its lack of monitoring and alerting because nobody had time to address the problem during my team’s day to day responsibilities.
Your assertion that it’s not management’s fault is absurd. Management is by definition the bearer if ultimate responsibility. Every problem in any business is something where the buck stops at management.
If I ship something with long term problems because management told me to work faster and meet the deadline, that is directly management’s fault. Even me shipping something bad on my own volition is management‘s fault indirectly: they hired the wrong talent (me), or maybe they assigned me to the wrong project where my expertise wasn’t good enough, or they misjudged risks and didn’t leave enough contingency buffer or didn’t make a plan for what to do if we fail.
The way businesses view humans are as machine-like resources of labor (Human Resources), they don’t view you as an individual with emotions and thoughts and feelings. When they hire someone they have quantitative measures surrounding that person: how likely are they to perform well, burn out and quit, steal from the company, get run over by a bus, etc. The corporate system actually dictates that management is responsible for the way it arranges and commands its human machines.
That’s why we aim to call it DevOps, so that you can take all that into account from the start of the project?
Almost every company works with the “throw stuff over the wall to DevOps” mentality. The word “DevOps” is meaningless.
Replacing an ingress controller in Kubernetes is also a well documented practice, with minimum or even zero downtime if you want to.
Generally, if your engineering team can reasonably keep things simple, it's good. However, business needs to grow and infrastructure needs to scale out. Sometimes trying too hard to be simple is, in my experience, how things become unmanageably complex.
I find well-engineered complexity to be much more pleasant to work with.
One thing that I push for nowadays, after a few scars is managed platforms.
But hey, it keeps a lot of people busy, which means it also keeps a lot of managers and consultants and trainers busy.