Let me know if you have any questions. I plan to write an article/kind-of-tutorial about the setup.
Let me know if you have any questions. I plan to write an article/kind-of-tutorial about the setup.
EDIT: Downvoters, I'm really curious what you're objecting to above, specifically.
IMO the few lines of yaml to set the path/host for an Ingress definition seems cleaner to me than using consul-template to spit out some LB config (as in the post's example).
For simplicity, a few years ago I preferred Traefik + Swarm. Add a label or two and you're done. But Swarm died :/
No one is arguing that they did.
> load balancer provider
consul connect handles this, how you get traffic to the ingresses is still DIY... kinda. you can also use consul catalog + traefik (I've actually put in some PRs myself to make traefik work with a really huge consul catalog so you can scale it to fronting thousands of services at once). there's also fabio. you can also get bgp ip injection with consul via https://github.com/mayuresh82/gocast run as a system job to get traffic to any LB (or any workload) if that's an option.
i've also ran haproxy and openresty without any problems getting stuff from consul catalog via nomad's template stanza and just signaling them on catalog changes.
> storage provider
anything CSI that doesn't have a 100% reliance on k8s works. if you're also just running docker underneath you can use anything compatible with docker volumes, like Portworx.
> ingress controller
consul connect ingress! or traefik, both kinda serve double duty here.
> external DNS
no good story here -- with one exception, if by "external" you mean "in the same DC but not the same host," consul provides a full DNS interface that we get a lot of mileage out of.
if you're managing everything with terraform though there's no reason you can't tie tf applies to route53/ns1/dyn or anything else though!
> monitoring
open up consul/nomad's prometheus settings and schedule vmagent on each node as a system job to scrape and dump somewhere. :)
we also use/have used/will use telegraf in some situations -- victoriametrics outright accepts influx protocol so you can do telegraf/vector => victoriametrics if you want to do that instead.
> secret encryption
this is all vault. don't be afraid of vault! vault is probably hashicorp's best product and it seems heavy but it's really not.
there's a lot here that doesn't really compare at all, like the exec/raw_exec drivers. we use those today to run some exotic workloads that do really poorly in containers or that have special networking needs that can map into containers but require a lot of extra operational effort, e.g.: glb-director and haproxy running GUE tunnels.
something interesting about the above is i'm testing putting the above in the same network namespace, so you can have containerized and non-containerized workloads in the same network cgroup namespace so you can share local networking across different task runners.
Bare metal nomad - use with consul and hook up traefik with consul backend. This would be the simplest, most ”zero conf”, way to go.
I’ve used this setup for a few years heavy production use (e-commerce & 50 devs)
As consul presents SRV records you can hook up a LB using those, or use nomad/consul templating to configure one.
Service mesh with mTLS is actually rather approachable and we’ve deployed it on selected services where we need to track access and have stricter security. (This however had us move off traefik and in to nginx + openresty)
Now if you want secrets management on steroids you’ll want vault. It’s really in many ways at the heart of things. It raises complexity, but the things that you can do with the nomad/consul/vault stack is fantastic.
Currently we use vault for ssl pki, secrets management for services & ci/cd, and ssh cert pki.
These things really form a coherent whole and each component is useful on it own.
Compared to k8s it’s a much more versatile stack although not as much of a “framework” and more like individual “libs”.
I always come back to the description: “more in line with the unix philosophy”.
In a mixed environment where you have some legacy and/or servers to manage I think using the hashicorp stack is a no brainer - consul and vault are tools I wouldn’t want to be without.
Consul is nice and easy to use.
Nomad has been a painful experience: the default UI is confusing (people accidentally killed live containers), we have some small bits and pieces that don't quite behave as we expect and have no idea how to fix them. Error rate is too low to care and there are more pressing issues so likely WONTFIX. We often found ourselves looking into github issues for edge cases or over-allocating resources to overcome scheduling problems.
We considered just switching to their paid offering, just not to have to worry about this.
It kind of feels like that's their business model: attract engineers with OSS software and then upsell the paid version without all the warts.
Please post it here when you do. :)
I'm in for an industrywide rename :D.
Bear Metal Semiconductor. Bear Metal Fabrication. Bear Metal Labs. You could get so creative with the branding and logo.