Hacker News Reader

Comment by spyremeown | Hacker News Reader

One of the things I hear the most about Kubernetes is the complexity. Honestly, it's not that complex from an user perspective. It's not much more complicated than Docker, really. It's also super nice to just write two yaml files and have everything running nicely with rollouts, auto-scaling, node pinning etc. It just works and I never have to even think about the internals (although I took the time to learn most of them).

Adding: heck, I'll run a single node k3s on my computer (which takes one (1) command) instead of using docker-compose just because I like it better than compose.

View on HN

I find consuming K8s to be like this - it's great, provides facilities for common everyday things

I find building K8s (well) to be wildly different. CNIs, storage classes/providers, yada yada. For a 'solution', you have to build a lot of it.

It's in a weird spot to me, I like/enjoy it - but only because I have other people helping me tend to it

edit: Think of a typical 'cloud' -- it's typically made of compute, storage, and networking.

My 'thing' about K8s is it's hyper-focused on the compute part, touted as complete, but (understandably) lacking in terms of storage/networking OOTB

Being able to easily build (development) clusters perpetuates this a bit more than I care for. They let you ignore the parts of production-worthy K8s, worrying only about the workload.

I still remember a recent company I worked at. They had a strict on prem datacenter ethos. Their procurement was so bad that you had to know your hardware requirements months ahead of time to get what you ask for (pre-covid mind you). If you under provisioned, you were boned. What naturally happened? Teams allocated a ton of resources that ended up sitting around in bitrot hell. I think someone took a sample of all system CPUs and found something like 6% averages. Pathetically underutilized. If this was k8s, it means the company can provision for their general needs, and teams can requisition what they needs easily through technological means instead of bureaucratic ones. Crap like storage classes, etc are sure more annoying than simple apps teams, but it's way easier than needing to manage the bare metal and provisioning / maintenance / migration that is often completely invisible to apps teams. Oh my app got restarted an hour ago, why was that? Oh it turned out a node had it's hard-drive started to fail and the node automatically drained it's load and moved them to a spare capacity node without you. Plus it often saves the need to page a human to run around frantically to figure out why all these services are crashing or misbehaving.

You're comparing it with "buy server(s) per project approach" but many companies went to "buy a bunch of servers and run VMs under some controller" long before k8s existed. Hell, that's whole VMware business model. Sure containers allow for more dynamic fitting, especially memory wise but that's not the biggest benefit.

The difference here is not between utilization but the fact that k8s allows developer to deploy "smaller unit" - they don't need to manage whole VM, they just need to push a container to it. And keep the deployment config near the app itself and not a document to throw across the wall to ops to realize.

That, and not having to pay for VMware's licensing. Big factor in the adoption.

This is why you should never run plain-old-Kubernetes in prod (or in dev, if you want my opinion). Stock Kubernetes basically ignores security and, then, as you said, it requires a bunch of additional work to config.

There are several companies that take Kubernetes and polish it up, the two biggest examples are Red Hat OpenShift[0] (which I'm most familiar with, because I manage it as my day job), and Rancher[1]. Also, note OpenShift is a paid product, but Red Hat also releases OKD[2] for free, which is OpenShift without support.

OpenShift is Kubernetes with great default security policies. Storage, networking, logging, and more are all configured out of the box. I would recommend anyone who is using or looking to use Kubernetes to strongly consider OpenShift or OKD.

[0] https://developers.redhat.com/products/openshift/overview

[2] https://www.okd.io/

[1] https://www.rancher.com/

> For a 'solution', you have to build a lot of it.

This is true even in the 'managed' k8s world (like EKS). They'll automatically configure EBS storage for you, but that's not sufficient for non-trivial workloads. Same with networking, logging, or metrics. I like that EKS means I don't have to worry about etcd backups and restores, but it's not a production ready platform by any stretch.

Yeah most of the (IMO) legitimate complaints about the difficulty of k8s are about how hard it is to correctly set up and administer. (Networking setup, making ingress work, cloud service integration, etc.) Especially when you get into permissions/namespaces and trying to get that right for a decent size organization and trying to configure things for least privilege. Simply using it isn’t that hard IMO.

Right but this is the irony I think.

If you need the scale provided by k8s, you are probably in a large org. If you are in a large org, you are probably intermediated from running k8s directly by some kind of devops/cloudops team who needs to sort out all the "hard administration stuff" you listed.

Setting up a science project at home in your toy AWS account I'm sure is no big deal.

But this is true of every tech- "works on my box" is always easier than "doing this in enterprise".

The measure of a technology is how easy it is to use in the real production environment.

Being in a large organization does not equal having a dedicated team for running Kubernetes clusters. For example Daimler (the company behind brands like Mercedes) has hundreds of Kubernetes clusters across hundreds of small product teams.

IMHO, the biggest annoyance and difficulty with K8s, is figuring out the helm chart ecosystem with the crazy templated YAML. Some of those documents can get quite convoluted. One family of charts (say, from Bitami) will have one set of conventions for values.yaml, and other families will have a slightly different way of organizing this.

You can just not use helm. Kustomize, which is built-in, does almost everything needed and is much simpler.

But then you have an entirely different challenge if you want to use a project that only provides a chart as a deployment package / docs.

Try checking out kpt, it does packages and hydration.

https://kpt.dev/

Kubernetes is complex if you don't need what it provides.

However if you do need what it provides in _some way_, Kubernetes provides a pretty nice, coherent way of doing it.

The problem is a lot of people read about k8s and decided stuff that could be done with a queue, 3 micro services, and 4 cores of CPU running on a RasPi somehow needs ... k8s.

This might be true, but one of the under-appreciated benefits of Kubernetes is the control around deployments.

3 services and a queue isn't much... unless you have high uptime requirements, need to maintain capacity during releases, want a careful rollout process, etc.

It's possible to orchestrate complex release flows without an orchestrator, but it's often easier with one.

It moves most/all of the complexity on ops side. or in "whatever automation thingy you use to deploy it". Yes, easy on dev, but compared to "just docker" or "a VM with some systemd service running your app" it is hideously complex

Could you share those 2 YAML files? I feel like it's not hard to do k8s YAML the wrong way and miss out on the auto-rollout, auto-scaling, etc. features

You are implying that he needs all these automations.

Reply on news.ycombinator.com