Hacker News Reader

Comment by TuringNYC | Hacker News Reader

Ingress going GA seems big. I just read through the release notes but didnt see as much about Ingress Controllers. I always found it strange that Ingress Controllers are external components provided by the CSPs or 3rd parties (if you are going multi-cloud support as I am).

Looking at NGINX, Kong, Traefik, and the loose change of community-contributed extensions to NGINX, Kong, Traefik, I always wonder why it isn't just part of the core. Anyone know if that is the eventual plan? If so, what happens to the small companies building solutions, do their products just become 2nd class citizens?

View on HN

This is basically why ingress isn't seen as a sexy way to integrate with Kube. Companies like F5 etc know they don't want to be a "swappable" ingress provider, so they create stickier solutions like custom operators. Additionally, most companies graduate from ingress to service mesh, or skip ingress entirely, so service mesh is the key business to capture and where most companies are targeting their efforts.

I've used Ingress for some personal projects, and have also felt the pull towards service mesh. I've tried to set up Istio and Linkerd a couple times each, and bailed every time. These products need to be deeply integrated into the cluster (e.g. Istio installs its own ingress controller), so I've had to tear down and re-build my cluster in order to properly set up the mesh. I wish they were more approachable. Or I wish I were smarter!

Ingress going GA refers to the Ingress API resource in the Kubernetes API becoming stable. There are no discussion (or Kubernetes Enhancement Proposals (KEPs) regarding providing a default ingress controller. There are two areas that Kubernetes has taken a hands of approach and it is ingress traffic (and external load balancing) and storage. Those are provided by external vendors through APIs.

I think the problem is that HTTP reverse proxies do so much, it's hard to specify what Kubernetes should do and what you should get an external solution to do. (Personally I am appalled at how org-wide SSO works right now, and almost started my own company to sell an open-source solution to the problem. But... I don't know how to start a company and then there was a pandemic. Someone else will sell you a not-as-good solution for mega-bucks.)

In my opinion, people are always going to want their special thing; compatibility with their legacy nginx.conf, support for HTTP/3, whatever. Kubernetes can never satisfy people with those requirements. For that reason, I think people need to agree on a service discovery protocol, a route discovery protocol, and then hook those into their frontend proxy of choice. For example, I never really liked Ingress (lots of unspecified landmines, like what order rules are evaluated in when matching a request to a route), and so just run Envoy inside Kubernetes. Envoy is then fed a static route table and a valid TLS certificate, and handles all traffic in an explicit and easy to debug way. I wrote something to glue Envoy's service discovery to Kubernetes so my static config file contains less boilerplate (https://github.com/jrockway/ekglue), and it's served me well. (I need to write something to add route table entries to DNS, though. I currently do that by editing my DNS records in a web interface, which is tedious and uninteresting.)

With that in mind, what people should really be focusing on is standardizing xDS (Envoy's config language basically), and hooking it into their container orchestration framework and web server of choice. Then you can run Nginx on Openshift or Apache on ECS and everyone is using the same tools to do the very boring gruntwork of figuring out what your backends are called, and what syntax you use to say "send all requests that match example.com/foobar to a backend my container orchestration framework calls foo-bar-v2". I am sure it will sort itself out eventually -- xDS has a lot of momentum in things that aren't Envoy (notably gRPC), so this could all be a solved problem outside of Kubernetes soon enough.

(I work at AWS, this is a personal opinion :) )

IMO, I'd like to see a broad standardization on xDS/UDP to all services where direct dataplane integration would be beneficial. Why can't I direct Envoy to autoscaling so they can tell me what endpoints are available and what certs to use? or to RDS to direct me to nearby replicas and assist with authentication/authorization? I think the typical solution would be to embed more magic into the infrastructure, but that breaks down as you split things across the cloud and on-prem. While some customers only want magic and will change whatever they have to to get it, other customers don't/won't buy into that. Having a general-purpose method for a control plane to configure an arbitrary dataplane would be a big benefit for them.

I generally agree, and would add though that the Service API work that is forming the basis of ingress v2 will help a lot here as well.

(I do think that xDS is a non trivial interface to learn and standardize on though and is probably overkill for most.)

I think a too-complex API is better than a too-simple API. If the underlying infrastructure is too simple, then people have proprietary workarounds to get the features they need (see Kubernetes Ingress and all the vendor-specific annotations). If the underlying infrastructure is too complex... then implementation is harder, but at least you can do what you need. Someone else can smooth it over and give you a simplified version, if the subset you need is the same subset they needed.

This is always a spectrum, of course, and xDS is probably missing crucial features... but at least if it becomes somewhat widely used, there will be an incentive to add the missing features.

The path going forward is to split L7 layer proxies and L3/L4 layer - https://kubernetes-sigs.github.io/service-apis/spec/ "Ingress" is too broad of a term and it does too much and too little depending on your use case

On an unrelated note, did anyone here try Skipper as their ingress controller? We are torn between Skipper and HAProxy.

Our use case is transferring large size files and NGINX is rumored to have bad performance in that regard.

Have you tried Ambassador? Built on Envoy Proxy. We have submitted as well to the CNCF as an official project too. (Disclosure: I work on Ambassador).

We will certainly give them a try!

Reply on news.ycombinator.com