Hacker News Reader

Comment by chucky_z | Hacker News Reader

I've started heavily using Nomad recently and it's a real joy. Adding things in is just so easy because it doesn't do much and (mostly) everything is very well defined. I cannot suggest it highly enough.

An example is I setup a log pipeline recently spanning multiple data centers with full mTLS. It wasn't that hard and thanks to Vault all the certs are refreshed at regular intervals all over the place. Pretty great!

View on HN

Came here to say this. Nomad is the perfect entry system for getting started with a small cluster for getting to know how container orchestration works and what things to keep in mind when migrating all legacy applications to fit into this type of deployment flow. Nomad allowed us to spin up a 3-worker-node "cluster" with consul in a matter of hours, and it has not needed any maintenance since. With Kubernetes we couldn't even agree on which of the many bootstrapping approaches and scripts-de-jour to use, much less any of the other many decisions that have to be made and will overwhelm you.

I'll name drop CircleCI here as a nomad user in case it peaks anyone's interest. Need to look into it more myself, but have always been impressed with Hashicorp tools.

+1 for Nomad, I'm extremely impressed with all of the Hashicorp suite.

Single binary deployments and upgrades are helpful.

even Terraform?

I manage everything soup to nuts with Terraform. If you know something vastly better please let me know, especially considering the changes to 0.12. It's a very well integrated system.

what kind of infrastructure are you managing? if you’re in the aws cloud, CloudFormation is the gold standard for “infrastructure as code”.

says you

Terraform does not automatically rollback in the face of errors. Instead, your Terraform state file has been partially updated with any resources that successfully completed.

do you know who can rollback and leaves your infra in a consistent state? can you guess?

yea that is surprising that you can't roll back when you first start with terraform but as you gain more experience with it, you realize that not rolling back means you can resume instead. And if you need to roll back, you can do so by just running destroy. It's actually a feature, not a bug.

no it’s not. i want to leave my infrastructure in a consistent state. i am in state A and want to move to state B. I want it to work. I don’t want a half-assed attempt to make it work.

what does terraform bring to the table? I have to use HCL to describe my infrastructure in terms that are NOT cloud agnostic (therefore introducing another layer) and in the face of adversity it throws its hands in the air and now you’ve got to figure out what went wrong, manually, by yourself. This is what I call True Devops (TM).

I have seen Terraform crap out and it cannot recover. It cannot move forward, it cannot rollback, it cannot destroy. It’s stuck. At that point you start praying that someone really understands the underlying cloud + knows the shenanigans terraform plays to fix it now and also make terraform happy moving forward.

we’re talking basic stuff here.

i don’t want to go into more advanced issues like: losing network connectivity, terraform process crashing (think oom conditions) or being killed or non-responsive cloud apis.

not to mention that destroying infrastructure you’ve created almost never works (unless it’s trivial infrastructure).

based on what I’ve seen up until now I would not use terraform in a production environment.

If I had experienced what you just described, I would probably have the same opinion - but after the initial learning curve, I haven't really had any of the problems you've listed. The only times I've had to go manually modify cloud resources to fix something was always because I was doing it wrong in the first place.

On the other hand, CloudFormation is not perfect either. The rollback does not work 100% of the time and I've had it roll back a set of templates that took 45 minutes to deploy because there was some inconsequential timeout that could have been ignored. I've also had pre-built templates developed by AWS outright fail, which is strange considering AWS themselves built it.

Use what works best for you and your team.

so I ran into the issues I’m mentioning while test driving it.

i have never experienced unpredictable behavior from CloudFormation, but it’s possible YMMV.

canonical maas, ns1, aws, nomad, consul, vault, and a handful of other providers

Please, please stop using the single binary argument against kubernetes. Kubernetes has hyperkube, which is a single binary. It's not magically better by having a file with a giant switch statement trying to figure out what you want, rather than a few small ones with small switch statements.

And the reason nomad's single binary is so small is because it doesn't do nearly as much. I'd rather have a platform that I can do things later that I don't know I need now.

Can you recommend any up to date best practices or guides for the ecosystem? I'm looking at the Hashicorp ecosystem for our new infrastructure.

This should get you started: https://learn.hashicorp.com

It does but leaves critical questions unanswered without already having an in-depth knowledge of the system.

If I want to use Vault and Nomad, should they share the same Consul cluster?

Should I deploy my Vault and Consul servers via Nomad?

The guidance on server/cluster size is hard to use. They only have 'small' and 'large' with no reference for what constitutes those sizes.

I have more questions as I'm going through this right now, but those are just a few off the top of my head.

I’m no expert but I would say Vault and Nomad should share the same consul cluster.

Don’t deploy your Consul servers with Nomad as that will create cyclic dependency if I’m not mistaken. The same should be true for Vault.

Reply on news.ycombinator.com