Hacker News Reader

Comment by marvinblum | Hacker News Reader

I've been running a production-grade Nomad cluster on Hetzner for the past 1 1/2 years and it's fantastic. It was amazingly easy to set up compared to Kubernetes (which I also did), the UI is awesome, updates haven't broken anything yet (as long as you follow the changelog) and it's stable. I really like the separation of concerns the HashiStack offers. You can start out just using Consul for your service meshing, and then add Nomad + Vault to get a similar experience to Kubernetes.

Yes, it doesn't cover as many features as Kubernetes, but it should be good enough for most software and you can still make the switch later. I would never go back.

You can read more on our blog if you're interested: https://pirsch.io/blog/techstack/

View on HN

Hi, I see you mention the tiniest nodes in Hetzner there, whereas the Nomad documentation [0] talks about 3-5 server nodes in the 2-digit GiB memory range, which is what has kept me from trying Nomad as I find it insane. How much truth is there in the docs?

[0] https://www.nomadproject.io/docs/install/production/requirem...

This very much depends on your workload, number of jobs and complexity of scheduling. Our Nomad servers have 4GB of memory in the VM and are using about 0.5 - 1G at a low three-digit number of jobs.

Hashicorp is doing a smart but normal thing for on-prem sizing there - they are recommending specs which ensure you have no problems for a lot of workload sizes. And "workload" can grow very large there, since a single DC can handle up to 5k clients and such.

I think the minimum number of nodes is high because they are recommending the minimum requirements for a fault tolerant setup. It is entirely possible to install either k8s or nomad on a single node, but clearly that is not a fault tolerant situation.

IIRC both k8s and nomad rely on the Raft algorithm for consensus, and so both of them inherit the requirement of a minimum of 3 nodes from that algorithm.

If you want something to run some containers in the cloud, and you aren't concerned by the occasional failure, then there is no issue running a single Nomad (or k8s) on a single machine and later adding more if and when you require it.

Yes, I did not even consider that angle.

And yes, nomad recommends 3 - 9 nodes in prod, maybe 11 or 13. The 3 are necessary to be able to tolerate one broken/maintained node and maintain the ability to schedule workloads.

You can increase the number of tolerated nodes by increasing the number of nodes - 5 tolerate 2, 9 tolerate 3, 11 tolerate 5, 13 tolerate 6, but raft becomes slower with more nodes, because raft is a synchronous protocol.

However, it is entirely possible to run a single node nomad system, if the downtime in scheduling is fine.

PSA: In practice, choose 3 or 5 nodes. Going higher exposes system to increased communications overhead. See also: all of the literature on this.

I am currently running over 13000 nodes with a 5 server cluster. These are large boxes (90gb memory, 64 cores) but cpu usage is under 20% and memory usage is under 30%.

I'm sorry, what is the exact benefit of running multiple nodes on one piece of hardware? Just software failure resilience?

I assume they meant a 5 server control plane supporting 13k worker nodes, not that they partitioned 5 large hosts into 13k smaller ones. It's a counterpoint to GP's "raft gets slow with more servers", I think.

This is correct.

We run a Raft cluster of 5 voting members with 13,000 physical Nomad nodes mixed across OS's with a bunch of random workloads using docker, exec, raw_exec, java, and some in-house drivers. I'll clarify that (and I can't... because I can't edit my post anymore :( ).

I don't see where you see multiple nodes. I'd read the parent post as: There are 5 identically sized, beefy VMs. Each of these 5 beefy VMs run 1 instance of the nomad server. And then, there is 13k other VMs or physical systems, which run nomad in client mode. These clients connect to the server to register and get allocated allocations / task groups / practically containers or vms or binaries to run.

We're nowhere near that size, but the architecture is probably identical.

Sorry for the delayed response.

We started out with a three-node cluster using the smallest (1 vCPU/2GB RAM) VM you can get. Our initial requirement was to be able to do zero-downtime deployments and have a nice web UI to perform them (alongside the other advantages you get from a distributed system). We have now rescaled them to the next tier (2 vCPU/4GB RAM).

The hardware requirements depend on your workload. We process about 200 requests/sec right now and 600-700 per second on Saturdays (due to a larger client) and the nodes handle this load perfectly fine. All our services are written in Go and there is a single central Redis instance caching sessions.

Our database server is not part of the cluster and has 32 physical cores (64 threads) and 256GB RAM.

I say start out small and scale as you go and try to estimate the RAM usage of your workloads. The HashiStack itself basically runs on a calculator.

For ~20 jobs in my local network for the longest time i was running single VM with 2cpu 1gb ram. Opening htop was taking more resources than rescheduling a job. For local development just use `nomad agent -dev`.

More anecdata:

It, of course, depends entirely on your use case and workload.

I have a few dozen jobs running with what I think is a pretty decent mix/coverage. I've got a bunch of services, some system jobs, half dozen scheduled jobs, etc, etc.

I'm running my cluster with 3x "master" nodes which act as the control plane for nomad and consul as well as run vault and fabio (for ingress/load balancing). I have three worker nodes running against this that are running consul and nomad agents as well as my workloads.

The control plane nodes are currently sitting on AWS's t3.nano instances. `free` shows 462MB total RAM with 100MB available. Load average and AWS monitoring show CPU usage within a rounding error of 0%.

If this were in a professional capacity I probably wouldn't run it this close to the wire, but for personal use this is fine--it will be quite easy to upgrade the control plane when or if the time comes.

Their (stated) minimum requirements are what stopped me from using it for projects too. My projects were pretty simple, so it was total overkill, but for any bigger projects, it was too risky to try something new. So I ended up never trying Nomad out, even though it always looked really nice.

I'm really curious about the local development story for Nomad.

I'm using Docker Swarm in production, which let's me easily spin up the same services locally as I do for production - it's really nice to have one set of yaml files that can be used across all environments, and of course it gives more confidence before shipping to prod.

How do you handle local development? Do you have Docker Compose files specifically for that, or something else?

Sorry, didn't see your reply because... well HN has no notification thing I think? Anyways, we don't run Nomad locally. All of our services can be started locally without additional tools other than Go.

I've been getting into Ansible and Terraform lately. It seems like Nomad would fit better in an environment where you weren't going to be putting everything in K8s, so a mixture where you have Consul and Vault handling standard VMs alongside your Nomad cluster would make a lot of sense.

Great read, thank you.

I'm glad you like it :)

Reply on news.ycombinator.com