Everyone does this - because Kubernetes Achilles heel is its ingress. It is still built philosophically as a post-loadbalancing system .
This is the single biggest reason why using Docker Swarm is so pleasant.
The reason node ports are used in the Cloud today is because most Cloud load balancing solutions only target VMs, not arbitrary endpoints such as containers, a limitation that will go away over time.
[1] Envoy with Kubernetes Endpoints integration: https://github.com/kelseyhightower/kubernetes-envoy-sds
[2] https://kubernetes.io/docs/concepts/services-networking/dns-...
I am about to try using helm for packaging my Kubernetes configs to make use of its templating. Being able to include Kubernetes changes in the release makes it less common to forget some new env variable etc.
The only thing I don't like is that it replaces kubectl in a way. And some comments here speak of problems during deploy and rollback which makes me wary.
What are some best practices around that topic? I was also thinking of deploying with spinnaker but could not confirm it works with helm.
Any info is appreciated!
P.S. have you thought about making a redis-cluster example repo? Redis 4.0 has a new `cluster_announce_ip` setting with which I made it work, but I still don't like my setup 100%.
I know about this. However, ultimately the question is that Kubernetes is not a gradual scale up solution for most people. I have to be prepared to deal with building my own load balancer.
Basically, I cannot do an on-metal deployment very easily. Most of the questions on k8s slack for metal deployments were - how do I set this up with few tweaks like ssl pass through and source ip preservation.
It is not easy.
Either you build your own load balancer or you use a cloud provided one. Now, Ingresses are not pleasant. I'm not sure about the state of source ip preservation, but last I remember that the nginx ingress had still not surfaced ssl_preread_server_name to the ingress configuration.
Now, what would have been nice is if it was ingress-all-the-way-down : ingress with something like istio/linkerd, maybe it is possible.
Tl;Dr - I'm not github. I can't build my own load balancer. Give me something that works out of the box. Yes, I know it may go down - I'll survive. Docker Swarm does this.
https://www.amazon.com/Kubernetes-Running-Dive-Future-Infras...
Obviously that doesn't fly if there isn't an equivalent open solution, so we did what we could with the system to make it not terrible. We can do more.
The point about Swarm is interesting, and has been much on my mind. Some of Kubernetes' perceived complexity is because we go to great lengths to avoid ever having two users collide, with escape hatches for the people who really need "unfriendly" features. This is because, again, Kubernetes models Borg. Borg clusters are giant, shared, multi-user, multi-app animals, where the users are in different business units and chances of collisions are high.
Swarm, on the other hand, thinks of a cluster more as an application construct. Sharing is not a big problem, and coordination is easy and local. This allows them to make different tradeoffs. I doubt very much that you can run a large number of similar apps in a single swarm without having collisions on things like ports.
I still believe the large-shared-cluster model is right in the limit. There are so many efficiencies to be had. But there are legit reasons it is hard to achieve right now.
I'm very interested in ways to make Kubernetes easier to use, ESPECIALLY in this regard. Real user feedback is critical.
One very interesting tool that Docker makes available is https://store.docker.com/community/images/docker/docker-benc...
I think the issue with k8s is that it is competing with the "Ruby on Rails" of frameworks viz Docker Swarm. I think the pluggability of critical pieces like ingress and secrets was taken too far.
> I doubt very much that you can run a large number of similar apps in a single swarm without having collisions on things like ports.
I dont think that is true, it does manage its overlay networks pretty well. Which is FWIW, another place where k8s took the non-opinionatedness too far. I think the number of bugs on "my stuff doesnt work with flannel but works with calico" should tell you that.
To be honest, Docker Swarm has some of these issues as well - https://github.com/moby/moby/issues/25526. But the fixes are included in the "batteries". On kuberenetes, I have to run behind upstream projects with heterogenous configuration (nginx vs haproxy ingress. or flannel vs calico configuration) to try and fix it.
I don't think that's true. Kube secrets were introduced 2015-02-17 and was considered GA in Kubernetes v1.0
> I think the pluggability of critical pieces like ingress and secrets was taken too far.
I think the pluggability is not the concern but the lack of an included solution. Part of the problem is that SOME platforms have an included solution - e.g. Google Cloud, and some need 3rd party code like nginx.
> I dont think that is true, it does manage its overlay networks pretty well
Overlays are a waste for most people. I get that making it simple is attractive, but it's (IMO) not something everyone wants or needs. Again, we could/should have had a built-in option.
Last I looked (admittedly a while ago) Swarm had a pretty deeply rooted notion of exposing ports on all nodes in the swarm, which means that if you have multiple containers that need to expose the same port, it was a problem. Kube takes extra complexity here, to make it possible to share arbitrarily.
Anyway, it's not my intent to bad-mouth Swarm or try to convince you that you're wrong. Different trade-offs were chosen for the two systems. Your feedback is noted and appreciated. :)
https://github.com/unibet/ext_nginx
Basically just handling nginx.conf from information in k8.
We run in production with ECMP in our routers to load balance stateless over any number of nodes. Easy to understand and very scalable.
We could have just left our haproxies outside of kubernetes, and may eventually end up doing so if the network performance doesn't meet our needs. As it is, it all works but there are a ton of sidecar services all over the place.
It sounds like you've already got it nailed down, but maybe like to have a look at this: https://github.com/deis/router
(It's probably tightly coupled to Deis Controller, but something to look at anyway!)
https://docs.docker.com/engine/swarm/ingress/#publish-a-port...
What it means is that when you create a docker swarm - it starts working.
Docker Swarm's inbuilt ingress is now trying to build in proxy protocol and ipip mode for default usage.
Fwiw, you can use Swarm's inbuilt ingress with an external load balancer as well.
Functionally, we have this in NodePort, but because it is exclusively managed, you're very unlikely to have a meaningful conflict. BUT it depends on traffic ingress being managed.
If you just want to map port 8080 on every node into your kube Service, it's more complicated than Swarm. Granted. That's because we don't think you should do that - it doesn't scale.
If you just want to map a port on every node (and you don't care which port), kube Services have you covered. Swarm's model is a nearly-direct clone of this.
Swarm has an ingress mode that is "built-in". It works like the kuberenetes ingress (with currently the same limitation of source-ip mapping). Let me re-emphasize built-in.
Swarm also has a load balanced nodeport mapping (called host mode). That is also built-in.
In fact, what I'm trying to say is the opposite of what you are perceiving - swarm is not superior. It is actually similar to kuberenetes.
However, swarm has saner built-in defaults - both ingress and overlay networks (no more flannel vs calico vs weave incompatibility issues). Kubernetes has an escape hatch for defaults - kubeadm+kompose. I keep praying that kubeadm+kompose becomes the keras for kubernetes tensorflow. Lots of defaults that lets you get started quickly.