Hacker News Reader

Comment by debarshri | Hacker News Reader

It was year 2016, kubernetes did not have jobs, cronjobs, statefulsets. Pods would get stuck in terminating state or container creating state. Networking in kubernetes was wonky. AWS did not have support for EKS. It used be painful.

It is year 2021, 1000s of new startups around kubernetes, more features, more resource types. Pods would still get stuck in terminating state or container creating state. It still pretty painful.

View on HN

I haven't seen these failure modes in 2021. We do managed clusters at work and have created around 100,000 of them, and basically all the pods we intend to start start, and all the pods we intend to kill die -- even with autoscaling that provisions new instance types. Our biggest failure mode is TLS certificates failing to provision through Let's Encrypt, but that has nothing to do with Kubernetes (that is a layer above; what we run in Kubernetes).

EKS continues to be painful. It has gotten better over the years, but it is a chore compared to GKE. I like to imagine that Jeff Bezos walked into someone's office, said "you're all fired if I don't have Kubersomethings in two weeks", and that's what they launched.

What are the EKS pain points?

The biggest pain point is having to manually use cloudformation to create node pools. This is especially irritating when you just need to roll the Linux version on nodes -- takes half a day to do right. In GKE, it's just a button in the UI (or better, an easy-to-use API), and you can schedule maintenance windows for security updates (which are typically zero downtime anyway, assuming you have the right PodDisruptionBudgets). I think AWS fixed that. I remember when I used it, they said they had some new tool that would handle that, but you had to re-create the cluster from scratch. This was a couple years ago, and is probably decent nowadays.

There are other warts, like certain storage classes being unavailable by default (gp3), the whole ENI thing for Pod IPs, the supported version being way out of date, etc. EKS has always felt like "minimum viable product" to me -- they really want you to use their proprietary stuff like ECS/Fargate, CloudFormation, etc. If you're already on AWS and want Kubernetes, it's just what you need. If you could pick any cloud provider for mainly Kubernetes, it wouldn't be my first choice.

Having used EKS, GKE, and DOKS, I definitely prefer GKE. GKE is very feature-rich, and the API for managing clusters works well. The nodes are also cheaper than AWS. (I use DOKS for my personal stuff and I haven't had any problems, and it is free, but it's missing features like regional clusters that you probably want for things you make money off of.)

For what it's worth, there's an off-the-shelf terraform module for EKS that is far simpler to use than AWS' cloudformation tooling, which does allow you to pass in a custom AMI and multiple nodegroup configurations as input parameters.

https://registry.terraform.io/modules/terraform-aws-modules/...

EKS has actually come a long way in the last couple of years. They now have 'eksctl' that helps a lot with provisioning/de-provisioning the node pools. They're also pretty up to date with supporting new versions of k8s (1.21.2 now, I think). I definitely know what you mean about the 'minimum viable product' and their overall views on k8s though, as our account rep definitely tried to steer us away from EKS when we ran into problems with CronJob. They also don't seem to have any particular relationship with k8s development, so we ended up not able to use CronJob for more than six months while our issue (with k8s repo) just sat waiting for someone to look at it.

You have probably thought about this already but I must admit I’m curious: If you’re on AWS, can you not use Certificate Manager instead of Let’s Encrypt?

Certificate Manager pushes you (shoves you, really) in the direction of using AWS managed services. They make certificate installation/rotation really easy for their own services and unnecessarily difficult for any that you implement yourself.

(This may have changed in the last year or two, but it was certainly this way when I tried it.)

In my experience, it's difficult-to-impossible to use AWS' certificate management and LB termination in conjunction with Envoy-based networking like Istio or Ambassador.

Yeah, the AWS LB has more issues than that, too. I'm pretty sure it's just nginx under the hood but they won't tweak the simplest parameters for you, even if you make a colossal stink, even if your company spends seven figures a year. I wonder if it isn't a decade-old duct-tape-and-bailing wire solution that shares the same config across literally every customer or something. Rolling our own was almost a relief -- the pile of awkward workarounds had grown pretty high by the point we bit the bullet.

I'm actually not on AWS, just used EKS extensively at my last job (and we still manually test our software against it).

AWS burned me hard with forgetting to auto-renew certs at my last job. It just stopped working, the deadline passed, and only a support ticket and manual hacking on their side could make it work. cert-manager has been significantly more reliable and at least transparent. The mistake we make right now is asking for certificates on demand in the critical path of running our app -- but since we control the domain name, we could easily have a pool of domain names and certificates ready to go. Our mistake is having not done that yet.

This year, I have seen those issues popping up in statefulsets alot. I realised somebody in the team was force deleting it. It is actually well documented.

https://kubernetes.io/docs/tasks/run-application/force-delet...

I have seen few scenarios where people patching a statefulset actually screws up the volume mount. It is sometimes not evident where the error is for instance if it is the CSI or in the scheduler unless you deepdive into the issue.

I have used Kubernetes extensively over the past couple years and have never seen pods stuck in terminating or creating state that didn't have to do with errors in container creation (your Dockerfile/bootstrapping is messed up) or issues with healthchecks.

We spawn thousands of pods per day for jobs and never get those stuck and it was not the case in 2018 either. Not sure what is it you are doing causes this.

Around 2015, I was at Spotify, and we were using a container orchestrator build in-house named Helios. They didn't build it because Kubernetes wasn't invented there; they built it because Kubernetes didn't exist, yet.

kubernetes was released in 2014, in late 2015 I already had a cluster marked as released candidate that was put into production early 2016.

so im guessing here that spotify wrote their own becouse they had a spesific requirement?

nomad was also early days in 2015

I'm pretty sure the work started before Kubernetes was released, and even then, it wasn't clear that was going to be the de factor orchestrator.

> Networking in kubernetes was wonky.

Can you elaborate? Does Kubernetes add some thrills to the relatively simple Docker network configuration?

Yes, Kubernetes (actually its add-ons) provide a virtual network that unifies communication within the cluster so you don't need to care on which computer your service runs.

You can make it as complicated as you want it to be; part of setting up a cluster is picking the networking system ("CNI"). Cloud providers often have their own IPAM (i.e. on Amazon, you get this: https://docs.aws.amazon.com/eks/latest/userguide/pod-network...; each Pod gets an IP from your VPC, resulting in weird limits like 17 pods per instance because that's how many IP addresses you can have for that particular instance type throughout EC2).

Please take a seat and let the joys of k8s networking overwhelm your senses:

https://kubernetes.io/docs/concepts/cluster-administration/n...

And yes... Kubernetes network configuration is on a whole different level from docker networking.

To be fair, multi-node networking of any sort is on a different level than single-host docker networking.

If you ever tried to use Docker Swarm to network multiple nodes, god help you.

Also worth noting that almost all users of K8s don't actually need to operate a cluster, the hosted offerings handle all of that for you. You just need to understand the Service object, and maybe Ingress if you're trying to do some more advanced cert management or API gateway stuff.

It's a common meme around here to point in horror to the complexity that is abstracted away under the K8s cluster API, and claim that k8s is really hard to use. I think that's mostly misguided, the hosted offerings like GKE really do a good job of hiding away all that complexity from you.

Honestly I think that it's defensible to say that the k8s networking model is in most cases _simpler_ than what you'd end up configuring in AWS / GCP to route traffic from the internet to multiple VM nodes.

> Honestly I think that it's defensible to say that the k8s networking model is in most cases _simpler_ than what you'd end up configuring in AWS / GCP to route traffic from the internet to multiple VM nodes.

How is routing from the internet to multiple servers a problem?

usually, you have either one of these setups:

- you run a loadbalancer that distributes traffic across your nodes. (This loadbalancer could even be distributed thanks to BGP).

- you either run your own firewall or have a managed one, in which you either announce your IP prefix yourself, or they are announced for you by your uplink provider.

- you run an anycast setup (for, for example, globally distributed DNS). and announce multiples of the same prefix across the globe. Routing in the DFZ does the rest for you.

Streched L2 across the globe/internet is also possible (although not very performant) either by doing IPsec tunneling, or by buying/setting up L2VPN services. (either MPLS or VXLAN based).

I didn't say it was a problem. My claim was just that it's easier in GKE than in GCE/EC2.

I only mentioned multi-node because exposing a single VM to the internet is trivial -- just give it a public IP -- and thus is not an apples-to-apples comparison with the multi-node load balancing that you get from the entry-level k8s configuration of Service > Pod < Deployment.

the largest issue with kubernetes networking seems to be the lack of integration with modern datacenter networking technology.

Things like VXLAN-EVPN are supported on paper, but are no where near mature compared to offerings from normal networking vendors. Heck, even the BGP support inside kubernetes is lacking. Which is a great shame because it creates a barrier between pods and the physical world. (Getting a VXLAN VTEP mapped to a kubernetes node is a major PITA for instance).

Most major cloud providers seem to have fixed this by building even more overlay networks (with the included inefficiencies).

Reply on news.ycombinator.com