1,215 karma · joined June 16, 2015
I am mostly all-in on Apples ecosystem because I've been so continuously disappointed by the fragmentation on the Android ecosystem, and this is something that's seriously giving me a real hit of excitement!
I also try to make damn well sure everything we use is contributed too financially and we open source as much as possible.
We have several thousand hosts in a single cluster across Windows/Linux, all hooked up to Nomad/Consul/Vault and it takes some effort but our team isn't that big and we manage it. With the most recent releases of Nomad and Consul they truly shine at both ends of the spectrum, small/simple and extremely large/extremely complex.
within prom:
cpu{hostname,metadata,cpu_socket} 5.0
cpu{hostname,metadata,cpu_socket} 6.0
cpu{hostname,metadata,cpu_socket} 5.0
cpu{hostname,metadata,cpu_socket} 4.0
version{hostname,version_number} 1.0
instead of baking version_number in everywhere. just as an example, then you can know "oh, hostname has version number" on a chart, or even do some inversions and say "count of version number" to find stragglers.Also if you run Windows, Nomad is your absolute best choice. I understand k8s work on Windows, however I don't know a single individual running Windows containers.
https://www.nomadproject.io/docs/drivers/external/singularit...
Nomad is pretty darn easy to run and scales far. I help run about 20 clusters in day job from 100 to 1800 nodes, we’re going to push one of them to over 8500 nodes here soon too.
We don’t use the Singularity driver but we wrote two of our own and they work well and are easy to maintain.
Have you looked into nomad, consul and vault; along with everything they provide?
> load balancer provider
consul connect handles this, how you get traffic to the ingresses is still DIY... kinda. you can also use consul catalog + traefik (I've actually put in some PRs myself to make traefik work with a really huge consul catalog so you can scale it to fronting thousands of services at once). there's also fabio. you can also get bgp ip injection with consul via https://github.com/mayuresh82/gocast run as a system job to get traffic to any LB (or any workload) if that's an option.
i've also ran haproxy and openresty without any problems getting stuff from consul catalog via nomad's template stanza and just signaling them on catalog changes.
> storage provider
anything CSI that doesn't have a 100% reliance on k8s works. if you're also just running docker underneath you can use anything compatible with docker volumes, like Portworx.
> ingress controller
consul connect ingress! or traefik, both kinda serve double duty here.
> external DNS
no good story here -- with one exception, if by "external" you mean "in the same DC but not the same host," consul provides a full DNS interface that we get a lot of mileage out of.
if you're managing everything with terraform though there's no reason you can't tie tf applies to route53/ns1/dyn or anything else though!
> monitoring
open up consul/nomad's prometheus settings and schedule vmagent on each node as a system job to scrape and dump somewhere. :)
we also use/have used/will use telegraf in some situations -- victoriametrics outright accepts influx protocol so you can do telegraf/vector => victoriametrics if you want to do that instead.
> secret encryption
this is all vault. don't be afraid of vault! vault is probably hashicorp's best product and it seems heavy but it's really not.
there's a lot here that doesn't really compare at all, like the exec/raw_exec drivers. we use those today to run some exotic workloads that do really poorly in containers or that have special networking needs that can map into containers but require a lot of extra operational effort, e.g.: glb-director and haproxy running GUE tunnels.
something interesting about the above is i'm testing putting the above in the same network namespace, so you can have containerized and non-containerized workloads in the same network cgroup namespace so you can share local networking across different task runners.
I think as an industry everyone is pushed towards software engineering and the large clouds have vacuumed up everyone worth their weight for operations. We make it happen but it’s so stressful because our team and workload balance is so far out of whack.
It's not Kubernetes but it doesn't try to be at all.