All is fine and dandy and BSDs can solve many use cases. Unfortunately for the solution we are working on, which imply many microservices we need Kubernetes and no BSD equivalent to Kubernetes exists.
I don't really see Kubernetes as being a game changer. The biggest pro, it makes it easier to onboard both development and operations personnel having a quasi-standard for how a lot of things like scheduling and application networking work.
But it also seems to come with a magnitude of accidental and ornamental complexity. I would imply the same about microservices versus, say, figuring out your repository, language, and deployment pipelines to provide a smooth developer and operator experience. Too much of this industry is fashion and navel gazing instead of thinking about the core problems and standing behind a methodology that works for the business. Unless google moves its own infrastructure to Kubernetes, then maybe there's something to be had that couldn't reasonably be done otherwise :)
We went from a virtualized server model to managed Kubernetes and costs have escalated considerably. The additional complexity and maintenance overheads of Kubernetes are not trivial and required additional staff hires just to keep things ticking. I think the cost so far from moving from two cages in separate datacentres running blades to AWS is approximately a 6x multiplier including staff. This was all driven on the back of "we must have microservices to scale", something we have failed entirely to do. It's a complete own goal.
You could have easily run Kubernetes on your virtualised servers.
Also, k3s. https://k3s.io/
For an existing business not in a hyper-growth phase with lots of restrictions and policies and processes and legacy stuff it's just extra work with no gain. It's just a round peg in a square hole. Wrong tool for the job.
If you look at most enterprises today you will see it deployed everywhere.
And most of the complexity has been abstracted away by the cloud providers so all you're left with is a system that can handle all manner of different applications and deployment scenarios in a consistent way.
> If you look at most enterprises today you will see it deployed everywhere.
This doesn't mean it's a game changer. It just means it has a big cargo-cult and people keep using it regardless of whether they need it.
> most of the complexity has been abstracted away by the cloud providers
The abstractions that people add make things more complex, not less. Unless of course you don't care to understand what you're running, which is precisely the problem with the culture around most folks that use kubernetes (and more broadly the US cloud providers as an entity).
Obviously, there is exact opposite with stuff like fly.io but they can be extremely constraining.
Now I just dump Caddyserver and install things bare metals servers. I avoid all dependencies like the plague
The problem with Kubernetes and complexity is that because it simplifies a lot of things that are a huge PITA to accomplish on a "homemade" server/container setup is that there are a huge number of products and things you can run on kubernetes to "do stuff".
And it is hard for a lot of people and organizations to resist the "oh shiny" aspect. Stuff like "Oh, look I can do network policies and service meshes!" or "Lets create this really complicated and big thing so we can configure all our AWS infrastructure with kubernetes objects! Who cares that a bad commit can destroy the infrastructure the cluster depends on along with our ability to manage any of it!" or "Look we can have lots of namespaces for all our internal orgs and departments, lets make a gigantic centrally managed Kubernetes cluster that will be managed by IT and that everybody will be forced to use at the same time! Putting all our eggs in one basket is a awesome idea!".
K8s sorta removes the barrier of entry that world normally stop people from implementing those sorts of bad ideas.
Otherwise the core vanilla Kubernetes isn't really that complicated compared to most DIY solutions that try to manage large numbers of apps on clusters of systems. Most of the time it ends up a lot more robust and simpler in the right hands.
I guess I'm just really frustrated with seeing the people that get paid to use and promote it be so bad at what they do and I don't like cleaning up after them :)
Everything in our industry is built on abstractions.
By your logic we shouldn't be using operating systems, compilers, libraries etc and writing our own custom software to manually move applications between different hosts if they go down.
I don't really see how this is a particularly controversial statement and I will say I don't appreciate you trying to reduce it to some ridiculous extreme. You didn't add anything to the discussion and it's tragic, because you could have made a nice point about system boundaries, levels of abstraction and perhaps information overload.
That would have made a great point about why abstraction is sometimes necessary. Then we could have spoken about what makes good and bad abstractions. Perhaps someone who came upon this thread later could have learned something and we'd have made the site a better place together.
Kubernetes solves administration of a cluster of Linux machines, as opposed to administering a single Linux machine. It abstracts away the concept of a machine, because it automates application scheduling, scaling across different machines, rolling updates of applications, adding/removing machines to the cluster all at the same time. There are no instruments like that for applications, the closest to them are something like Spark and Hadoop for data engineering tasks (not general applications).
Microservices are also used to solve a very specific problem – independent deployments of parts of the system. You can dance with your repository and your code directories as much as you want, if you're not in a very specific runtime (e.g. BEAM VM), you will not achieve independent deployments of parts of your service. The ability to "scale independently" (which tbh is mostly bullshit) is an accidental consequence of using HTTP RPC for microservice communication, which is also not the only way, but it allows reuse of the HTTP ecosystem.