Kubernetes 1.19 | Hacker News Reader

Kubernetes 1.19

github.com

116 pointsgtirloni39 comments

Reminder it's time to get off 1.16 now. Only 3 versions (current-2) are supported upstream.

https://kubernetes.io/docs/setup/release/version-skew-policy...

well, gcloud gke regular is still on 1.16.

And GKE stable is on 1.15.

Although you can't launch new clusters on it anymore, Google hasn't even forced clusters on 1.14 to update to a newer version yet.

Tell that to Google. Their "regular" (recommended) release channel in their managed Kubernetes Engine product regularly falls off the back of the supported range, while their "rapid" channel barely stays within it.

I’d like to see those two teams eat lunch together

I was preparing to say "Tell that to GCP!" :) In their defense it's one of the services that you fire and forget. It seems to just work.

Ingress going GA and the support cycle bumping up from 9 months to 12 are my fav two things in this one.

Of course it'll probably be another few months (if not six) before we see this in gke/eks.

EKS is still on 1.17

AWS is really dragging their feet on supporting latest version of K8s

I don't think being 3 months behind upstream release cycle is 'dragging their feet', but more 'waiting for stabilization' and 'researching and implementing required changes'.

They started supporting 1.17 very recently, I think, because last time I checked when 1.17 was the latest stable kubernetes version, they were supporting 1.14 only. EKS is usually behind.

Edit: I stand corrected. 1.17 was officially supported on July, 1.16 on April and 1.15 on March. Those versions were released a long time ago.

True, on 17 July EKS announced k8s 1.17 GA, 1 May was k8s 1.16 GA, and 10 March for 1.15. They've been catching up.

But, being behind is expected from a SaaS offering of 3rd party software: it takes time to ensure you can deliver the quality of service your customers expect, and for 3rd party software you don't get a head start.

As someone new to k8s and EKS it blows my mind things are only supported for such a short period of time. I find it very difficult to build a product on a platform in such constant flux. I'm struggling to see the benefits of k8s when things are rapidly being replaced and EOLd.

If it helps, it may not be k8s, but more EKS. We have had terrible luck at my company with EKS, being more painful than just an on-prem k8s cluster in my experience, getting the networking working, load balancers, etc.

Could you indulge me with any details? Not AWS or someone who works on k8s, I'm just always interested in war stories, and cloud providers "not working" is an interest too.

We got an on-prem cluster working with Rancher and MetalLB with L2 configuration (MetalLB spoke ARP). It was great. We tried to do the same on EKS. we got it iniitally working, but then we needed to get the nodes on certain subnets so that HQ would be happy or something, that was hard. It felt like any time we tried to make any changes, such as adding worker nodes or deleting old ones, the ALB configuration got all messed up and we couldn't get it fixed. Super brittle. We tried with NLB too, but eventually just got tired of fighting with a tool that was supposed to make this stuff simple. I'll tell you what was simple: MetalLB on our own on-prem network.

Just stick to stable Kubernetes API resources[1] and they are guaranteed to continue to work through all 1.x versions.

[1] https://kubernetes.io/docs/concepts/overview/kubernetes-api/...

It's not quite as bad as it seems. We have had a vanishingly small number of breakages as we upgraded from early verions of Kubernetes. Mostly just tweaking a few resources where the API got promoted from beta to stable.

One plus side of such a rapid release cycle is the changes tend to not be earth shattering. Big changes are deployed slowly and methodically, and the release notes are pretty easy reading. Compared to something with a yearly cadence, you could find yourself drowning under massive changes.

k8s is for teams that already manage something in flux. it's for teams that would be writing their own scheduler/orchestrator/manager/control-plane/platform.

sure, it's useful at much smaller scales too, but then you are not likely to feel the upgrade crunch, because you'll likely only use (and will be exposed to) a fraction of all the moving parts of it.

Ingress going GA seems big. I just read through the release notes but didnt see as much about Ingress Controllers. I always found it strange that Ingress Controllers are external components provided by the CSPs or 3rd parties (if you are going multi-cloud support as I am).

Looking at NGINX, Kong, Traefik, and the loose change of community-contributed extensions to NGINX, Kong, Traefik, I always wonder why it isn't just part of the core. Anyone know if that is the eventual plan? If so, what happens to the small companies building solutions, do their products just become 2nd class citizens?

Ingress going GA refers to the Ingress API resource in the Kubernetes API becoming stable. There are no discussion (or Kubernetes Enhancement Proposals (KEPs) regarding providing a default ingress controller. There are two areas that Kubernetes has taken a hands of approach and it is ingress traffic (and external load balancing) and storage. Those are provided by external vendors through APIs.

This is basically why ingress isn't seen as a sexy way to integrate with Kube. Companies like F5 etc know they don't want to be a "swappable" ingress provider, so they create stickier solutions like custom operators. Additionally, most companies graduate from ingress to service mesh, or skip ingress entirely, so service mesh is the key business to capture and where most companies are targeting their efforts.

I've used Ingress for some personal projects, and have also felt the pull towards service mesh. I've tried to set up Istio and Linkerd a couple times each, and bailed every time. These products need to be deeply integrated into the cluster (e.g. Istio installs its own ingress controller), so I've had to tear down and re-build my cluster in order to properly set up the mesh. I wish they were more approachable. Or I wish I were smarter!

On an unrelated note, did anyone here try Skipper as their ingress controller? We are torn between Skipper and HAProxy.

Our use case is transferring large size files and NGINX is rumored to have bad performance in that regard.

Have you tried Ambassador? Built on Envoy Proxy. We have submitted as well to the CNCF as an official project too. (Disclosure: I work on Ambassador).

We will certainly give them a try!

I think the problem is that HTTP reverse proxies do so much, it's hard to specify what Kubernetes should do and what you should get an external solution to do. (Personally I am appalled at how org-wide SSO works right now, and almost started my own company to sell an open-source solution to the problem. But... I don't know how to start a company and then there was a pandemic. Someone else will sell you a not-as-good solution for mega-bucks.)

In my opinion, people are always going to want their special thing; compatibility with their legacy nginx.conf, support for HTTP/3, whatever. Kubernetes can never satisfy people with those requirements. For that reason, I think people need to agree on a service discovery protocol, a route discovery protocol, and then hook those into their frontend proxy of choice. For example, I never really liked Ingress (lots of unspecified landmines, like what order rules are evaluated in when matching a request to a route), and so just run Envoy inside Kubernetes. Envoy is then fed a static route table and a valid TLS certificate, and handles all traffic in an explicit and easy to debug way. I wrote something to glue Envoy's service discovery to Kubernetes so my static config file contains less boilerplate (https://github.com/jrockway/ekglue), and it's served me well. (I need to write something to add route table entries to DNS, though. I currently do that by editing my DNS records in a web interface, which is tedious and uninteresting.)

With that in mind, what people should really be focusing on is standardizing xDS (Envoy's config language basically), and hooking it into their container orchestration framework and web server of choice. Then you can run Nginx on Openshift or Apache on ECS and everyone is using the same tools to do the very boring gruntwork of figuring out what your backends are called, and what syntax you use to say "send all requests that match example.com/foobar to a backend my container orchestration framework calls foo-bar-v2". I am sure it will sort itself out eventually -- xDS has a lot of momentum in things that aren't Envoy (notably gRPC), so this could all be a solved problem outside of Kubernetes soon enough.

(I work at AWS, this is a personal opinion :) )

IMO, I'd like to see a broad standardization on xDS/UDP to all services where direct dataplane integration would be beneficial. Why can't I direct Envoy to autoscaling so they can tell me what endpoints are available and what certs to use? or to RDS to direct me to nearby replicas and assist with authentication/authorization? I think the typical solution would be to embed more magic into the infrastructure, but that breaks down as you split things across the cloud and on-prem. While some customers only want magic and will change whatever they have to to get it, other customers don't/won't buy into that. Having a general-purpose method for a control plane to configure an arbitrary dataplane would be a big benefit for them.

I generally agree, and would add though that the Service API work that is forming the basis of ingress v2 will help a lot here as well.

(I do think that xDS is a non trivial interface to learn and standardize on though and is probably overkill for most.)

I think a too-complex API is better than a too-simple API. If the underlying infrastructure is too simple, then people have proprietary workarounds to get the features they need (see Kubernetes Ingress and all the vendor-specific annotations). If the underlying infrastructure is too complex... then implementation is harder, but at least you can do what you need. Someone else can smooth it over and give you a simplified version, if the subset you need is the same subset they needed.

This is always a spectrum, of course, and xDS is probably missing crucial features... but at least if it becomes somewhat widely used, there will be an incentive to add the missing features.

The path going forward is to split L7 layer proxies and L3/L4 layer - https://kubernetes-sigs.github.io/service-apis/spec/ "Ingress" is too broad of a term and it does too much and too little depending on your use case

The rationale for the support cycle bump is a little surprising to me. I always thought it would happen with feature stabilisation, but tne reason given is simply “people weren’t upgrading”.

That's a pretty valid reason. It's the community taking care of itself.

Ingress is still a bit of a mess, IMO. The reliance on annotations to ensure functionality with a specific ingress controller feels strange in the context of other k8s API objects. I understand why Ingress works this way, but it doesn't live up to the portability expectation. I know there is a "v2" coming for Services, and I'm hopeful that Ingress continues to get love. Networking in k8s is a big problem to solve :)

Didn't IngressClass change that?

https://kubernetes.io/docs/concepts/services-networking/ingr...

I think the result is still mostly the same, just a bit more organized. That's definitely progress though, thanks for pointing it out!

The new Service APIs were supposed to solve it, but it's delayed I guess - https://kubernetes-sigs.github.io/service-apis/spec/

>"The reliance on annotations to ensure functionality with a specific ingress controller feels strange in the context of other k8s API objects. I understand why Ingress works this way, but it doesn't live up to the portability expectation."

I'm curious what are the reasons for the reliance on annotations for this object type?

The core Ingress object is super minimal, basically just "listen on ABC host+path and forward requests to XYZ service". But Ingress controllers can do all types of things, and might need more context to properly route the traffic. For example, you might want HTTP requests to be automatically upgraded to HTTPS. Or your ingress controller might need specific instructions if it's handling something like a gRPC connection. You might want one Ingress object handled by Traefik and another by nginx. These types of things have been addressed by adding annotations to the Ingress object. It's honestly a pretty elegant solution to a pretty hairy problem, but it ties your YAML to a specific cluster incarnation. Of course, if you use CRD's, you're in the same situation; but the dream of k8s is portability (for me).

Reply on news.ycombinator.com