End of an era...
Are there any viable alternatives to Kubernetes and Nomad?
Are there any viable alternatives to Kubernetes and Nomad?
A good alternative to kubernetes is LXD [1] or just stick with docker compose. Kubernetes except for managed services from cloud providers is more difficult than an average application to manage and a huge cost in itself to run and maintain.
I'd imagine most small to medium companies would be running things on a cloud service using managed kubernetes. It seems mostly larger companies that are sticking with non-cloud hardware and services.
The advantage of kubernetes in that case is that there is a large ecosystem of helm charts, guides, documentation, etc. Deploying something new from scratch is fairly easy since someone else has done all the leg work and packaged it all up.
This is the same situation as Javascript circa 2008: "Learning this absolute dogshit language and using it widely will give you the ability to write universal applications that will run in browsers everywhere and make you very employable."
You're not wrong about k8s today, and wouldn't have been wrong about JS in the past, but boy is it a sad indictment of our industry that these things are true.
But also, k8s "secrets" are not, in fact, secret, you can't actually accept traffic from the web without some extra magic cloud load balancer (cf https://v0-3-0--metallb.netlify.app/ maybe eventually) or properly run anything stateful like a database (maybe soon).
Forget covering "everyone's" use cases: From where I'm sitting, k8s is an insanely complicated system that does a miserable job of covering the 95% use cases that Heroku solved in 2008.
It's great that k8s (maybe) solves hard problems that nobody except Google has, but it doesn't solve the easy problems that most people have.
edit: Kubernetes secrets are also either good enough (ie: on par with Heroku) or your cloud provider has an actual proper KMS.
Also ITT: Oh, but of course k8s is totally unusable unless you buy it from a Big Cloud provider.
It's all covered quite well here: https://kubernetes.github.io/ingress-nginx/deploy/baremetal/ and we use it in production.
But it does though. Your dissent is that you don’t understand what secrets are and think load balancers are “magic”?
Come on. Also see this[1]
If you want a picture of the future of computing, imagine everything-over-HTTPS-on-k8s-on-AWS stomping on a human face forever.
Then you’d expose a service, not an ingress. You can do this in a variety of ways depending on your environment.
I’m going to go out on a limb here and say you’ve never really used k8s and haven’t really grokked even the documentation.
It’s complicated, some parts more than others, but if you’re still at the “wow guys secrets are not really secret!1!” level I’m not sure how much you can really bring to the table in a discussion about k8s.
That only works to access the service from other pods inside k8s, it doesn't help you make that service accessible to the outside world. Tell me how you'd run Dovecot on k8s?
> if you’re still at the “wow guys secrets are not really secret!1!” level
I'm just pointing out one (of many) extremely obvious warts on k8s. You act like this is some misconception on my part, but it's not that silly to assume that a secret would be.
But to answer your smarm, yes, I've used Kubernetes in anger: my last company was all-in on k8s because it's so trendy, and it was (IMHO) an absolute nightmare. All kinds of Stockholm-syndrome engineers claiming that writing thousands of lines of YAML is a great experience, unable to use current features because even mighty Amazon can't safely upgrade a k8s cluster....
Quite the opposite. I’d re-read the docs[1]. Specifically this page[2]. If you’re on AWS you’d probably wire this up with a NLB and a bit of terraform if you’re allergic to YAML. Seems like a 5 minute job assuming you have Dovecot correctly containerized.
> I'm just pointing out one (of many) extremely obvious warts on k8s. You act like this is some misconception on my part
It’s hard not to point out misconceptions like the one above.
1. https://kubernetes.io/docs/concepts/services-networking/
2. https://kubernetes.io/docs/concepts/services-networking/serv...
> ClusterIP: Exposes the Service on a cluster-internal IP. Choosing this value makes the Service only reachable from within the cluster. This is the default ServiceType.
Internal only.
> NodePort: Exposes the Service on each Node's IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You'll be able to contact the NodePort Service, from outside the cluster, by requesting <NodeIP>:<NodePort>.
Nobody uses NodePort to expose external services directly, and I think you know that.
> LoadBalancer: Exposes the Service externally using a cloud provider's load balancer. NodePort and ClusterIP Services, to which the external load balancer routes, are automatically created.
As I mentioned above in this thread, requires cloud provider Load Balancer.
> ExternalName: Maps the Service to the contents of the externalName field (e.g. foo.bar.example.com), by returning a CNAME record with its value. No proxying of any kind is set up.
> Note: You need either kube-dns version 1.7 or CoreDNS version 0.0.8 or higher to use the ExternalName type.
This one's a new one to me, and apparently relies on some special new widgets.
Anyway, if you love k8s, I'm sure you'll have a profitable few years writing YAML. Enjoy.
ExternalName has been around since 2016.
> Nobody uses NodePort to expose external services directly, and I think you know that.
Sure they do. Anyone using a LoadBalancer does this implicitly. If you don’t want k8s to manage the allocated port or want to use something bespoke that k8s doesn’t offer out of the box then using a NodePort is perfectly fine. You can also create a custom resource type if you’ve got some bespoke setup that can be driven by an internal API.
The happy path is using a cloud load balancer, because that’s what you’d use if you are using a cloud provider and you’re comfortable with k8s wiring it all up for you.
Has your criticism of k8s evolved from “I’m unclear about services” to “well yes it supports everything I want out of the box but uhh nobody does it that way and therefore it can’t do it”?
Nomad is vastly superior in every way except for mindshare; Elastic Beanstalk or Dokku is superior for most normal-people use cases.
[0] https://kubernetes.io/docs/concepts/overview/components/
I do. It provides a convenient way to integrate our non-k8s load balancers (TCP haproxy tier with a lot of customization) with services on kube. This is good for reusability and predictability while we slowly migrate services from our prior deployment targets to k8s.
I get the impression that this is not uncommon.
It's not. Take any simple web app and deploy it into managed GKE with Anthos and you automatically get SLI's and the ability to define SLO's for availability and latency with a nice wizard. Takes a few minutes to setup.
The amount of engineering needed to achieve good SLO monitoring dwarfs the engineering needed to run a simple app so it just never happened. That's no longer the case.
> Only when company is reaching google scale kubernetes make sense.
Also obviously not true given the number of companies deriving value from kubernetes.
Note, I'm a Google Cloud partner.
Obviously for a Google Cloud Partner, more people are tied to gcp and kubernetes, higher the revenue. Its secondary if it's really necessary for an application to require k8s.
Granted, we use Nomad, not k8s, but we definitely need a scheduler and definitely are not reaching Google-scale.
If I did it how the usual "k8s is only for FAANG scale" people tell me to do, I'd go bankrupt ;)
I hear people repeating this truism all day, but from practical experience, it doesn't seem to be the case - Kubernetes is paying dividends even in small single-node setups.
I hated Kubernetes ops complexity at first, but there really isn't that much to it, and if it's too much, a service like GKE takes 70%+ of it away from you
Long before Google scale. Kubernetes won't handle clusters anywhere near that large.