I'm currently looking for container orchestration tools in my current company. We have our applications across multiple data centers (GCP, AWS, Alicloud).
From my research, I found out that the most common ones are Nomad and Kubernetes. Anyone has experience running either Nomad or Kubernetes in production? Furthermore, maybe the drawback and benefit from your experiences?
Thanks
Right now, we're seeing something similar to the OS wars in the 80's, 90's and 2000's. Back then there was Solaris, Linux, Unix, BSD, etc. In the 2000's, Linux won. After that, the sheer force of community made it easy to deploy and use.
The community around Linux mitigated its complexity by making distributions -- opinionated choices around how to set up linux, including what package manager to use, what init process to use, etc.
Kubernetes is, in my opinion, far more complicated than Nomad, but it's also more feature rich. To mitigate that complexity, quasi-distributions of kubernetes are emerging, like kops ( https://github.com/kubernetes/kops ) and kubespray ( https://github.com/kubernetes-incubator/kubespray ). My personal favorite is Typhoon ( https://typhoon.psdn.io/ ), because it uses a "package manager" (software delivery mechanism) that allows you to declare dependencies between packages and also allows you to deploy the entire cluster from the ground up -- Hashicorp's Terraform ( https://www.terraform.io/ ).
The kubernetes package manager is helm ( https://helm.sh/ ), but if I were you I'd wrap helm charts in terraform modules using the helm provider ( https://github.com/mcuadros/terraform-provider-helm ).
By contrast, Nomad is much simpler to set up and has a much better modular design. But then, so did the OS GNU Hurd. And no one uses that anymore. Kubernetes is much more monolithic in design (huh, so is Linux).
Therefore, I would choose kubernetes, but treat it like an OS. It's complicated. Don't choose what ingress you will use. Instead choose what distribution you will use, and let the distribution guide your decisions, unless you have a good reason.
Realize that you're learning an OS (resource scheduler = os in my head). Kubernetes has more complexity than Nomad up front, but at least it's not accidental complexity. K8s calls the resource scheduler problem what it is -- hard, and treats it appropriately.
The other safe bet is Terraform as the package manager. Choose a kubernetes distro that uses it (Typhoon, Kops). It is getting huge.
Kubernetes' extensible API is what really makes it cool. Having a prometheus operator ( https://github.com/coreos/prometheus-operator ) automatically monitor your services for you is pretty nice.
Finally, don't expect the stability you see in the Linux OS. Kubernetes is still coming out of the primordial ooze, though it's way more mature than Nomad in its community. Still, when a google search for how to do something on a cluster mostly turns up github issues, you know we're still in the early stages.
That said, Nomad is way easier to set up, and supports things like raw execution, JVM direct execution, and chrooted exec, so if you can't use containers at least you can still use a cluster. You'll be rigging everything yourself, but raw execution actually gives you a lot of rope ( https://github.com/hashicorp/nomad/issues/150 ). So if you need to do something really custom, Nomad might not be so bad. Kind of like embedded Linux.
For more comparison between the two schedulers coming from the point of view of someone who chose Nomad, look at this article https://medium.com/@copyconstruct/schedulers-kubernetes-and-... . I especially found his idea of incremental infrastructure improvement interesting, near the bottom if of the article.