Comment by TomBombadildoze | Hacker News Reader

Comment by TomBombadildoze | Hacker News Reader

TomBombadildoze

This starts with a lot of "you don't need Kubernetes" and then concludes with a pretty compelling argument in favor of using Kubernetes.

From the "The Nomad ecosystem of loosely coupled components" section:

> It integrates very well with other - completely optional - products like Consul (a key-value store) or Vault (for secrets handling).

> At trivago, we tag all services, which expose metrics, with trv-metrics. This way, Prometheus finds the services via Consul and periodically scrapes the /metrics endpoint for new data.

> The same can be done for logs by integrating Loki for example.

> Trigger a Jenkins job using a webhook and Consul watches to redeploy your Nomad job on service config changes.

shudder

> Use Ceph to add a distributed file system to Nomad.

> Use fabio for load balancing.

And the icing on the cake:

> All of this allowed us to grow our infrastructure organically without too much up-front commitment.

So if I understand correctly, the author (and his team) preferred to do all the work of integrating/testing/debugging those components, rather than using a tool that provides every single on of those features, and more, out of the box.

Kubernetes isn't a trivial lift but it's a damn sight easier than trying to roll a cheap imitation yourself.

The point is that Nomad allows to add more components as they are needed while Kubernetes is an all-or-nothing solution. Also, we used tools like Jenkins and Ceph before. They have been tested and we trust them so there's no point in replacing them. Quite the contrary: a lot of internal processes depend on them so it would have been painful to migrate away from them.

Nomad is not a cheap imitation of Kubernetes, it is a simple orchestrator which favors composability over an all-in-one approach.

> Nomad is not a cheap imitation of Kubernetes

Nomad isn't a cheap imitation of Kubernetes but all these components taped together are. Kubernetes is no less composable than a system built on Nomad, it just includes more functionality from day one.

If you're running containers with a orchestrator like Nomad, at some point, you'll need DNS. So you do it yourself and then you're managing Nomad and DNS. Then (we'll assume, because you're using a container orchestrator to manage multiple services) you'll need service discovery, so you write some jobs and event handlers to glue together Nomad and your DNS solution. And then, because you're a team of responsible people, you'll want to store secrets securely, so you graft in Vault. And then, you realize zero-downtime config changes would be great, so you slap Consul in there and write some sidecars or library code to handle config updates. Then metrics. Then logs. Then rolling deployments. And it continues, indefinitely, as you add features.

If you started doing this five years ago, fine. If you start doing this today, you're out of your mind. You're just doing work for the sake of "but it's composable". There's a reason why teams still build applications on frameworks like Rails and Django even though they don't need half the features--it's more important to them to get something functional up and running than it is to satisfy delicate sensibilities about only using what's needed. Kubernetes is the analogue in the world of systems and operations.

your description of "grafting in" consul and vault _is_ exactly what it feels like to do that with kubernetes, and it happens rather often.

Using industry standard components like consul and vault is not really a second thought for k8s in production, leaving you with duplicated hunks of infrastructure to step around, where the idea of tacking on kube-dns and kubernetes ~secrets~ onto something else is rather laughable. This, again, is the point which was being made -- you're forced to bear the brunt of kubernetes' NIH.

I'll assume by the contrived situations of inventing some wacky custom mousetrap to bind nomad to dns rather than using the "slapped in" (https://www.nomadproject.io/docs/configuration/consul.html) consul integration for dns, or writing config update/logs/rolling deployment code rather than using the core nomad scheduler features to do that, that you don't actually know, and this is FUD?

We use nomad because it's flexibility in this regard lets us slowly migrate away from our old infrastructure. Moving fully to Kubernetes would not be politically feasible in the organization.

This is a post about why I don't need kubernetes. Not a post about why kubernetes isn't politically feasible in not my organisation.

Reply on news.ycombinator.com