Hacker News Reader

Comment by 4636760295 | Hacker News Reader

I worked at a hedge fund that also used nomad. The problem, however, is not how well it scales or whatever, but the fact that all the accompanying deployment info and literature is for kubernetes, and k8s has far more features.

I like the quality of products from HashiCorp, but k8s is far, far, ahead of where nomad is.

What I really want is better integration for Terraform and kubernetes. The current TF k8s support leaves much to be desired, too many things are missing or broken and I find there are several bugs that result in deployment flapping (i.e., constantly re-deploying the same thing when there are no changes).

View on HN

"Kubernetes but with less stuff" is a valuable niche that I'm glad someone is targeting.

That's a fair point, I guess it depends on your use case. The risk, however, is that the powers that be at HashiCorp one day decide to abandon Nomad once they realize it will never be a profit centre for them.

Nomad is open source (or, at least, a significant subset of it is). Anyone is able to continue to improve it, even if Hashicorp is no longer paying people to work on it.

I used to do sales for an enterprise "open source" software product, so I get it, but the truth is as soon as someone stops paying people to keep the project going it will die.

Who pays for k8s development?

Google. Commoditizing the cloud is their strategy to make GCP relevant.

Question was rhetorical, I still appreciate the effort though.

Not at all. Hashicorp literally pays for all development of nomad. They're the only commits short of a small number of PRs. Kubernetes commits are from a wide array of companies, and Google is only one.

And just as with Kubernetes and Google, Nomad development can continue outside of Hashicorp if Hashicorp no longer decides to support it. Which org has a longer track record of deprecating almost everything they release? Not Hashicorp, and frankly, I’ll always trust Hashicorp versus Google based on the historical behavior and forward incentives of both.

It's not quite the same. Hashicorp controls whether PRs get merged. Google does not control whether PRs get merged into kubernetes. There's a long list of companies that do, including IBM, redhat, Huawei, etc. Sure, you can fork it, but now you have a separate repo that requires people to know about it.

I’m not trying to convince you. If k8s works for you, that’s great! I choose to work at places that pick Nomad instead. Best tool for the job.

Agreed, thanks!

Seems like some projects survive their parent company abandoning them like illumos (successor to opensolaris). It's not common, but also not impossible.

its hard to say illumos is in good shape compared to when opensolaris was being worked on by Sun, or Joyent however.

That's not enough, Basho (creators of Riak database) also made it open source. In fact after they went out of business Bet365 even purchased their proprietary code and made it open source, but the database is still considered dead.

The CEO addressed a similar comment on Twitter recently. From what he's saying, it seems Hashicorp is standing by Nomad for the foreseeable future. https://twitter.com/mitchellh/status/1247581788706197504

Nit: Mitchell is their CTO. Dave McJannet is their CEO. (source: https://www.hashicorp.com/about/ )

Micro-nit: Mitchell is Co-CTO w/ Armon being the other

It would be very odd if he said the opposite, since customers could be spooked. But saying it's going to be around is very different from saying it will always have a lot of resources on it going forward, especially compared to their cash cows of consul and vault.

"standing by" until they're not. If their VCs start applying pressure to dump the unprofitable projects and seek more profits, their tune will change in an instant. At the end of the day these are for-profit entities, and the only thing that matters is profit.

The only reason they're pursuing this strategy is because they think they can get a piece of the kubernetes market. If that doesn't pan out, they will dump nomad like a bad habit.

For-profit companies aren't open source charities.

Crossplane https://crossplane.io might be what you’re looking for with its bunch of controllers and a nice composition API.

One of the best features is that you can bundle a CR to request a MySQL database and it will be satisfied with whatever config is in your cluster so that app only declares the need but not care how it’s done.

Disclaimer: I’m one of the maintainers of Crossplane.

Any chance you plan to integrate with Google's Config Connector? It's very similar to crossplane (but gcp specific).

https://cloud.google.com/config-connector/docs/overview

I think that'd be a bit challenging because Config Connector is highly opinionated, for example, Kubernetes Namespace corresponds to GCP project. Though it might be enabled to be used as part of a Composition when we support namespaced CRs to be used as composition member.

Every config connector resource can be annotated with the project for GCP resource to be created in. Namespece to GCP project mapping is encouraged, but not enforced, you are still free to create resources in multiple projects from a single namespace as well as in a single project from multiple namespaces.

Oh, thanks for letting me know!

Why would you want TF in k8s? That hurts my soul.

Isn't exactly declarative, nor really imperative, mostly but not entirely idempotent, and worst of all, it has faux-state that may or may not reflect the actual state of the services.

I used to love Terraform, but it broke my heart. Now we're done.

Which features is Nomad missing? Feature count comparisons are meaningless unless the features are tied to actual important use cases. Lots of software is encrusted with rarely used features that just add complexity.

The main missing feature is that it's not kubernetes.

You have an extra word there.

For us it's not a feature missing, but the mindshare is low, and there's not really any prebuilt examples of how to run and maintain services long term reliably.

Does it have "easy" side-cars/init/post? Last time I checked those were missing for example.

It does! Here is the relevant documentation and examples:

https://www.nomadproject.io/docs/job-specification/lifecycle...

https://learn.hashicorp.com/nomad/task-deps/interjob

Ill check that out, I really think K8s is becoming a bit too complex for many customers. Thanks!

The free version of Nomad is missing, IIUC:

- no quotas for teams/projects/organizations

- no preemption (ie higher priority job preempts lower priority job)

- no namespacing

So generally it's somewhat useless in organizations where there are multiple different teams that should be able to coexist on a cluster without stepping on eachothers' toes, or even where you want a CI system to access the cluster in a safe manner.

Preemption is coming in the next release in OSS :)

We run everything in Nomad for all our teams, but we don’t use any of the features you mention, and it’s not causing issues. We wrote some metrics for how much memory all the allocations are claiming, and we throw another EC2 instance in the cluster when we’re running out. Works pretty well. Preemption seems like a nice feature, but I imagine getting teams to coordinate their preemption values would be a political nightmare.

For me the ability to squeeze in some jobs in between the cracks at best effort priority is crucial (ie. any sort of high latency batch processing / experiments). I wouldn't want these extremely low priority jobs to compete with, I don't know, an actual customer facing service.

Also, we run on bare metal, so there really isn't a way to request extra capacity within seconds.

Preemption is going OSS! https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md

We fully intend to migrate more features to OSS in the future -- especially as we build out more enterprise features. As you can imagine building a sustainable business is quite the balancing act, and there's constant internal discussion.

(I'm the Nomad Team Lead at HashiCorp)

This is going to make our AI team very happy because they can just dump experiments into the cluster at low-priority so those'll be done when those are done.

It's also going to make the operators very unhappy because it'll be harder to monitor actual memory utilization (allocated memory vs memory really in use) in order to plan cluster extensions. Are there some tools around or work planned to make this kind of scaling and utilization easier?

I was going to say "yes!" because we do have some telemetry/metrics improvements coming up, but then I realized those won't address your use case. It seems like aggregating cluster resource usage by job priority is what we need for that. Our UI might be able to calculate that, but I don't think our metrics include job priority so there's no way for your existing tooling to display it.

Please file an issue or link me to an existing issue of you have time. This seems really compelling!

Enterprise version has all of those things ;)

For my personal use I don't need them and for a business Enterprise won't break the bank. You'd be surprised how much your management might be interested in having an escalation path if everything goes south and you need a hot fix. I can vouch that Enterprise support is worth it if you're on a small team that can't spend all day on this stuff

A few long-standing feature requests I've noticed a lot:

* No autoscaling

* Can't reserve entire CPU core: https://github.com/hashicorp/nomad/issues/289

* No way to run jobs sequentially: https://github.com/hashicorp/nomad/issues/419

Autoscaling is in Tech Preview as of March: https://www.hashicorp.com/blog/hashicorp-nomad-autoscaling-t...

Disclaimer: I work at HashiCorp in Product Management.

Nice! It's definitely been one of the bigger holes.

I hope you'll also consider sequential tasks in a single jobfile. Without it it's kind of awkward because you have to be passing shell scripts around just to run a setup step before your actual workload.

We’ve also started to implement this with a new task dependencies feature in the latest release. Check out: https://www.hashicorp.com/resources/preview-of-nomad-0-11-ta...

https://learn.hashicorp.com/nomad/task-deps/interjob

There’s more work going on there to further improve the feature in upcoming releases.

Disclaimer: I’m an engineer on the Nomad team.

Looked at https://www.hashicorp.com/blog/deploy-any-resource-with-the-... ?

Reply on news.ycombinator.com