Hacker News Reader

Comment by sofixa | Hacker News Reader

Disclaimer: I work at Hashicorp but I have been saying what I'm about to say for years before joining (check my blog, linked in bio, for reference).

There is a way to get all of those necessary things without having the massive complexity of k8s - use Nomad, a vastly simpler orchestrator that does many things better than k8s - e.g. integrated templating instead of go templates over yaml with Helm or other similar abominations; not using yaml for configuration; self-contained configuration and handling of updates (e.g. if your ConfigMap or Secret in k8s change, you have to manually restart/reload your pods; with Nomad you can configure it to handle that directly). It doesn't have nearly the same ecosystem, but there's enough of it through CSI, CNI, various load balancer and autoscaling integrations that you get very far with a production environment.

View on HN

Why are people using Terraform, a relatively horrible and inconsistent tool to deploy their software as code ? As it is better than the alternatives and everyone is using it. It is the same with K8s. Yes, it is complex, however, as everyone is using it it is super easy to find documentation or example for pretty much anything. Finding the same documentation for Nomad is much harder.

Career wise all major platforms (Azure, GCP, and AWS) offer K8s, and therefore it is much better to learn the principles of K8s instead of focussing on the relatively obscure Nomad platform.

My issue with nomad is I also have to handle a second cluster, Consul, and that's it's own overhead. I'd rather just stay with k8s if the alternative is Nomad. Id totally jump to Nomad if all I had to do was deploy nomad, and consul was embedded in all nodes. I'm not an expert in consul, and really don't want another time sink.

Nomad 1.3 has builtin service discovery and 1.4 is around the corner with features for secrets so you can get started without vault too.

(I work for HashiCorp on Nomad)

At the core of K8s is an Etcd cluster, so if you’re spinning your own it’s the same thing. However, part of the value-add of cloud providers in the K8s space is that they hide it from you.

Etcd has come a long way and generally doesn't behave strangely or cause big issues these days.

The cloud providers generally aren't actually doing any significant amount of work on your behalf on the etcd front.

this happened very recently

Etcd v3.5.[0-2] is not recommended for production => https://groups.google.com/a/kubernetes.io/g/dev/c/B7gJs88XtQ...

https://github.com/etcd-io/etcd/issues/13766#issuecomment-10...

I understand that, but I already invested the time to work with etcd and can handle a few of k8s failure modes. Really do not want to throw that away for something that has a similar dependency, and that I do not now know well. Nomad might be simpler to use, but not to setup or manage due to Consul dependency.

Also, a lot of these products are missing a lot of documentation. I often find the tutorials are inflexible and focus on only 2 use cases out of the 4 most common. For example, focus on cloud when lots of engineers would rather deploy on bare metal. I would love for there so be one simple Nomad ansible role that would help setup nomad on bare metal with wireguard networking. This would probably be easy if consul was not required and it did not need so much babysitting.

Just throwing this out there:

Since Nomad 1.3, Consul is no longer required for Service Discovery, and since 1.4 Vault will no longer be needed for KV secrets.

And for what it's worth, i used to run Nomad+Consul in production on bare metal and it was easy to setup (2x 3 servers with a binary to start with some basic conf) and maintain (an update every month or so, which consisted of upgrading the binary on each node and restarting the service after 5 mins of reading the release notes).

Last week, when that same discussion popped up, "schmichael" (who works on Nomad I understand) said "Nomad supports CSI like Kubernetes. Unfortunately many CSI storage providers assume kubernetes or adhere to kubernetes behavior even when it deviates from the spec, so there are gotchas."

https://news.ycombinator.com/item?id=32588931

That makes be a bit wary to use it because I would like my cloud storage to be reliable.

Yep, some CSI providers do that, but as far as i know it's more niche ones (like VMware's), not any of the big cloud providers.

I tried using a CSI driver with Nomad (AWS) and AFAIK you have to pre-create the EBS volumes? I couldn't figure out how to spin them up on demand.

That was probably before the Nomad 1.3 release which added a bunch of new CSI features, including dynamic volume creation IIRC.

I wouldn't call vmware a niche provider, they own a large share of on prem virualization market.

Which is a dying breed, and furthermore running an orchestrator on top of another one (Nomad or Kubernetes on top of vSphere) is a terrible anti-pattern that wastes CPU and makes performance issues hard to debug due to the two layers.

My only problem with nomad is that I have to manage the infrastructure underneath as well. Give me nomad-on-fargate and I'll happily jump ship.

But Only problem of nomad is all examples are running on Vagrant which is designed for VMs . Everyone who is looking these days are containerization. There may be non-standard ways to run Nomad with containers but , putting Vagrant as main example is not a good idea.

Vagrant supports containers since v1.6 (released in 2014).

yeah but examples uses vagrant vm, i am just checking nomad and use of vm discourages me / not sure how much compatible if i to placed them with nomad/lxd/dockers.

Any good tutorials with Noman + Containers would be nice for CI/CD/Executor setup.

I'm not sure i follow. Nomad is an orchestrator which handles containers and other things (firecracker microVMs, LXC, etc.). The Vagrant examples are to get started and test it out, because running a container orchestrator in containers has some non-obvious pitfalls. You can just as well download the binary anywhere to play with it.

That doesn't mean Nomad can't orchestrate containers (in fact that's the main use case for it).

All the examples I’ve found are Docker…

Most use of Vagrant I saw was in demoing how to set up and deploy a cluster. So deploying 3 Nomad nodes within 3 VMs created in Vagrant.

I was amazed how easy it was to setup Nomad. I have a couple of VMs running on a VPS and Nomad works great for this use case. Super easy to maintain and works really well. Networking is my favourite part.

Reply on news.ycombinator.com