Hacker News Reader

Comment by q3k | Hacker News Reader

The free version of Nomad is missing, IIUC:

- no quotas for teams/projects/organizations

- no preemption (ie higher priority job preempts lower priority job)

- no namespacing

So generally it's somewhat useless in organizations where there are multiple different teams that should be able to coexist on a cluster without stepping on eachothers' toes, or even where you want a CI system to access the cluster in a safe manner.

View on HN

Preemption is going OSS! https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md

We fully intend to migrate more features to OSS in the future -- especially as we build out more enterprise features. As you can imagine building a sustainable business is quite the balancing act, and there's constant internal discussion.

(I'm the Nomad Team Lead at HashiCorp)

This is going to make our AI team very happy because they can just dump experiments into the cluster at low-priority so those'll be done when those are done.

It's also going to make the operators very unhappy because it'll be harder to monitor actual memory utilization (allocated memory vs memory really in use) in order to plan cluster extensions. Are there some tools around or work planned to make this kind of scaling and utilization easier?

I was going to say "yes!" because we do have some telemetry/metrics improvements coming up, but then I realized those won't address your use case. It seems like aggregating cluster resource usage by job priority is what we need for that. Our UI might be able to calculate that, but I don't think our metrics include job priority so there's no way for your existing tooling to display it.

Please file an issue or link me to an existing issue of you have time. This seems really compelling!

Enterprise version has all of those things ;)

For my personal use I don't need them and for a business Enterprise won't break the bank. You'd be surprised how much your management might be interested in having an escalation path if everything goes south and you need a hot fix. I can vouch that Enterprise support is worth it if you're on a small team that can't spend all day on this stuff

We run everything in Nomad for all our teams, but we don’t use any of the features you mention, and it’s not causing issues. We wrote some metrics for how much memory all the allocations are claiming, and we throw another EC2 instance in the cluster when we’re running out. Works pretty well. Preemption seems like a nice feature, but I imagine getting teams to coordinate their preemption values would be a political nightmare.

For me the ability to squeeze in some jobs in between the cracks at best effort priority is crucial (ie. any sort of high latency batch processing / experiments). I wouldn't want these extremely low priority jobs to compete with, I don't know, an actual customer facing service.

Also, we run on bare metal, so there really isn't a way to request extra capacity within seconds.

Preemption is coming in the next release in OSS :)

Reply on news.ycombinator.com