Nomad vs. Kubernetes
nomadproject.io
nomadproject.io
The fundamental abstractions are as simple as they can be, representing concepts that you'd already be familiar with in a datacenter environment. A cluster has nodes (machines), and you can run multiple pods (which is the smallest deployable unit on the cluster) on each node. A pod runs various types of workloads such as web services, daemons, jobs and recurring jobs which are made available (to the cluster) as docker/container images. You can attach various types of storage to pods, front your services with load-balancers etc.
All of the nouns in the previous paragraph are available as building blocks in Kubernetes. You build your complex system declaratively from these simpler parts.
When I look at Nomad's documentation, I see complexity. It mixes these basic blocks with those from HashiCorp's product suite.
k8s = 10x the components & difficult to install.
Nomad = single binary, works with but doesn't require containers.
Nomad is the single binary one, however this is a little disingenuous as Nomad alone has far fewer features than Kubernetes. You would need to install Nomad+Consul+Vault to match the featureset of Kubernetes, at which point there is less of a difference. Notwithstanding that, Kubernetes is very much harder to install on bare metal than Nomad, and realistically almost everyone without a dedicated operations team using Kubernetes does so via a managed Kubernetes service from a cloud provider.
I've said this before. Kubernetes gives you a lot more too. For example in Nomad you don't have secrets management, so you need to set up Vault. Both Nomad and Vault need Consul for Enterprise set ups, of which Vault needs 2 Consul clusters for Enterprise setups. So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did you gain really?
This also dodges the crux of GP's argument -- instead of running 1 cluster with 10 components, you now need a half dozen clusters with 1 component each, but oops they all need to talk to each other with all the same fun TLS/authn/authz setup as k8s components.
> So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did you gain really?
GP's point was already talking about running Vault clusters, not sure you realized we aren't only talking about nomad.
If you want to have the Enterprise versions( which aren't required), you just need 1 each of Nomad, Consul, Vault. Considering many people use Vault with Kubernetes anyway(due to the joke that is Kubernetes "secrets"), and Consul provides some very nice features and is quite popular itself, that's okay IMHO. Unix philosophy and all.
Sure Kubernetes's secrets is a joke by default, it's easily substituted by something that one actually considers a secret store.
It's new but I think is quickly becoming preferred. I found trying to setup nomad/consul/vault as described on the hashi docs creates some circular dependencies tbh (e.g. the steps to setup nomad reference a consul setup, the steps for vault mention nomad integration, but there's no clear path outside the dev server examples of getting there without reading ALL the docs/references). There's little good docs in the way of bootstrapping everything 1 shot from scratch in the way most Kubernetes bootstrapping tools do.
Setting up an HA Vault/Consul/Nomad setup from scratch isn't crazy, but I'd say it's comparable level to bootstrapping k8s in many ways.
For example, you mention starting with consul. But here's a doc on using Vault to bootstrap the Consul CA and server certificates: https://learn.hashicorp.com/tutorials/consul/vault-pki-consu...
So I need vault first. Which, oops, the recommended storage until recently for that was Consul. So you need to decide how you're going to bootstrap.
Vault's integrated Raft storage makes this a lot nicer, because you can start there and bootstrap Consul and Nomad after, and rely on Vault for production secret management, if you desire.
I think Nomad's biggest selling point is that it can run more than just containers. I'm still not convince that's it's much better. At best it's equal.
I don't really follow this. etcd uses raft for consensus, yes, and it's built in. Kubernetes components don't use raft across independent services. Etcd is the only component that requires consensus through raft. In hashi stack, vault and nomad (at least) both require consensus through raft. So the effect is much bigger in that sense.
> I think Nomad's biggest selling point is that it can run more than just containers. I'm still not convince that's it's much better. At best it's equal.
Totally agree. The driver model was very forward looking compared to k8s. CRDs help, but it's putting a square peg in a round hole when you want to swap out Pods/containers.
No it isn’t.
> I've run Vault in an Enterprise
At this point I am starting to doubt that claim.
And after so many years of independent development, I see no reason to believe that K8s ressemble borg any more than superficially.
This seems to be very much assumed by kubernetes authors. Current borg users please correct me if I'm wrong.
This is incorrect. You don’t need consul for enterprise. Vault doesn’t need two consul clusters (it doesn’t need consul at all, if you don’t want it)
It's the same reason I still like to use postgres when I can versus NoSQL until I know I need the one feature I may not be able to achieve with postgres: automatic sharding for massive global scale. The rest of the features postgres (and friends) give easily (ACID etc) are very tricky to get right in a distributed system.
It's also the same reason bash is great for tasks and small automated programs, but kind of terrible for more complex needs. The primitives are super simple, easy to grok, and crazy productive at that scale, but other languages give tools to more easily handle the complexities that start to emerge when things get more complicated.
But that's just basically a calculated tradeoff of Postgres (and several CP databases) trading Availability for Consistency.
If you want a transactional, consistent datastore you are gonna have to put a lock on something while writes happen. And if you want consistency it means those locks need to be on all systems in the cluster. And the entire cluster needs to hold that lock until the transaction ends. If your DB’s are 100ms apart… that is a pretty large, non negotiable overhead on all transactions.
If you toss out being fully consistent as a requirement, things get much easier in replication-land. In that case you just fucking write locally and let that change propagate out. The complexity then becomes sorting out what happens when writes to the same record on different nodes conflict… but that is a solvable problem. There will be trade offs in the solution, but it isn’t going against the laws of physics.
Even if I won $100 in the lotto today and had the money in hand, I wouldn't describe my choice which house I bought years ago as a calculated trade off between what I bought and some $10 million dollar mansion. That wasn't a feasible choice at that time. Neither was making a distributed RDBMS as an open source project decades ago, IMO.
More closely matching would be MySQL's NDB clustered storage engine, which was released in late 2004.[1] Given that Postgres and MySQL both started ab out 1996, that's quite a time after initial release.
I spent a while in the early to mid 2000's researching and implementing dual master/slave or DRBD backed MySQL HA systems as a consultant, and the options available were very limited from what I remember. There's also probably a lot better tooling these days for developers to make use of separated read/write environments, whereas is seemed fairly limited back then.
FWIW, it's not as bad as that sounds. There are traditional locks, and there is optimistic locking. If a there are two conflicting transactions, a traditional lock detects this before it happens (by insisting a lock is obtained before any updates are done) and if there is any chance of conflict the updates are run serially (meaning one is stopped while the other runs).
Optimistic locks let updates run with lock or blocking at all, but then at the end they check if the data they depended on (ie, data that would have been locked by the traditional mechanism) has changed. If it has they throw it all away. (Well, perhaps not quite - they may apply one of the conflicting updates to ensure forward progress is made.) The upside of this is there is if there are no conflicting updates everything runs at full speed - because there is no expensive communication about why has what lock going. The downside is a lot of work may be thrown away by what amounts to speculative execution.
Most monolithic databases use traditional locking. Two CPU's in the same data centre (or more likely on the same board) can rapidly decide who owns what lock, but cycles and I/O on a high end server are precious. Distributed ACID databases like spanner, cockroachdb and yugabytedb favour opportunistic because sending messages half way across the planet to decide who owns what lock before allowing things to proceed takes a lot of time, whereas the CPU cycles and I/O's on the low end replicated hardware are cheap.
While opportunistic locks allow an almost unlimited number of non-conflicting updates to happen concurrently, their clients still have to pay a time penalty. The decision about whether there was a conflicting update still has to be made, and it still requires packets to cross the planet, and while all this happens the client can't be sure if their data has been committed. But unlike the traditional model, they are never blocked by what any other client is doing - providing it doesn't conflict.
Citus is specifically designed for it.
Timescale appears to have the use case covered too.
Ultimately though, sharding is one of the easier ways to scale a database out. NoSQL just does it by eliminating joins. You can do that pretty easily with almost any solution, PG or otherwise.
We already had something that just ran containers and that was Mesos. They had the opinion that all the stuff like service discovery could be implemented and handled by other services like Marathon. But it did not work well. Most people that was to deploy containers in an orchestrated manner want service discovery.
At least these parts are being provided by the same company (Hashicorp) so it probably won't suffer the same lack of coordination between separate projects that Mesos did.
The benefit to the kitchen sink opinionated framework that K8s does is that your deployments and descriptors are not very difficult to understand and can be shared widely. I do not think the comparison of massively sharded NoSQL to Postgres is the same because most people will not need massive sharding, but almost everyone is going to need the service discovery and other things like secrets management that K8s provides.
From the documentation:
> Nomad HCL is parsed in the command line and sent to Nomad in JSON format via the HTTP API.
So why not just JSON or even JSON at all and not MsgPack or just straight up HCL because that's over and over introduced as being machine readable and human friendly both at the same time?
HCL requires a lot more code to parse and many more resources to keep in memory vs JSON. I think it completely makes sense to do it this way. K8s is the same. On the server it does everything in JSON. Your YAML gets converted prior to sending to K8s.
JSON is fine for serialization, but I hate typing it out. There are too many quotes and commas - all keys and values have to be quoted. The last item in a list or object can't have a trailing comma, which makes re-ordering items a pain. Comments aren't supported (IMO the biggest issue).
YAML is too whitespace dependent. This is fiddly and makes copy-pasting a pain. I'm also not sure how it affects serialization, like if you want to send yaml over the network, do you also have to send all of the whitespace? That sounds like a pain. OTOH I like that quotes are usually optional and that all valid json is also valid YAML. Comments are a godsend.
HCL has the same basic structure of objects and lists, but it's much more user friendly. Keys don't need to be quoted. Commas aren't usually needed unless you compress items onto the same line. It supports functions, list/object comprehensions, interpolations, and variable references which all lead to more powerful and DRY configuration. Granted I'm not sure if these are specific to TF's HCL implementation, but I hope not.
For serialization, HCL doesn't have any advantage over JSON. Sure it's machine-readable but probably much harder to write code that works on HCL specifically than to convert to JSON and use one of the zillions of JSON libraries out there.
This means you get to build upon what you already have, instead of doing everything from scratch again because your new infrastructure-layer service needs to integrate with 5 different APIs that have slightly different structure.
I’ve been doing software for 20 years in one form or another. One of the things I’ve learned is the simpler and more polished something seems to the user, it is almost always because there is a hell of a lot of complexity under the covers to make it that way.
Making something that handles the 80% is easy. Every step closer to 100% becomes non-linear in a hurry. All that polish and ease of use took months of “fit & finish” timeboxes. It took tons of input from UX and product. It involved making very hard decisions so you, the user, don’t have to.
A good example is TurboTax online. People love to hate on their business practices (for good cause) but their UX handles like 98% of your common tax scenarios in an incredibly easy to use, highly polished way. Robinhood does a pretty good job too, in my opinion—there is a lot of polish in that app that abstracts away some pretty complex stuff.
This has some great account of war stories - https://k8s.af/
I'd manage services in Kubernetes before I start deploying VMs again, that's for sure.
I keep Nomad alive part-time. It's operationally simple and easy to wrap ones head around it and understand how it works.
Either way, as another comment points out, Rancher and many other solutions make the orchestration of creating your own Kubernetes cluster really boring.
We run a few kubernetes clusters on premise, and for the longest time it was just 1 person running some Kubernetes clusters.. we even have other teams in QA/Engineering running their own with it.
I.e. you don't run k8s, Google does it for you. I'm talking about where YOU run k8s, not run ON k8s. I agree running ON k8s is pretty easy.
This has so many assumptions rolled up in it. When we move from niche assumptions to the masses of app operators and cluster operators (two different roles) it tends to fall apart.
Which ultimately leads to things like...
> This has some great account of war stories - https://k8s.af/
We shouldn't assume other people are like us. It's worth understanding who folks are. Who is that ops person who wants to run Kubernetes at that insurance company or bank? Who is the person setting up apps to run for that auto company? What do they know, need, and expect?
When I answer these questions I find someone far different from myself and often someone who finds Kubernetes to be complicated.
I'm not sure if all of this is worth it, if you're running a small footprint. You're better of using more managed solutions available today.
Are you referring to actually spinning up and operating your own clusters here or utilizing managed k8s (e.g. GKE/EKS)?
In my understanding, sure - using the managed services and deploying simple use cases on it might not be that big of a deal, but running and maintaining your own k8s cluster is likely far more of a challenge [1] than Nomad as I understand it.
[1] https://github.com/kelseyhightower/kubernetes-the-hard-way
> Kubernetes The Hard Way is optimized for learning, which means taking the long route to ensure you understand each task required to bootstrap a Kubernetes cluster.
> The results of this tutorial should not be viewed as production ready, and may receive limited support from the community, but don't let that stop you from learning!
If you want to spin up cluster that you actually want to use you'll pick one of the many available free or paid distros, and spinning up something like k3s with rancher, microk8s or even the pretty vanilla option of kubeadmin is pretty simple.
- Nomad is a scheduler. Clean and focused. It is very fast. I was an early user and encountered a number of bumps, but that's software. The people at Hashicorp are super sharp and lovely.
- K8s is a lot more. It includes a scheduler, but in the simplest sense, it is a complete control-plane based on the control-loop pattern. You have an API, a scheduler, a db, various controllers, etc. Forget for a moment that most people use it to orchestrate containers -- it's really designed to orchestrate anything. Its API is extensible, you can add and compose controllers -- there are many possibilities once you wrap your head around it.
This is all opinionated and includes a lot of capability. It's just very different.
You can stitch together nomad, consul, vault, and various glue to create a container orchestration system... but when you start wanting to manage the control-planes as though they are the "kind" (the container for example) with meta-control-planes, and you start wanting to orchestrate network, storage, and other dependencies... all while doing this in a multi-tenant environment, then things get interesting.
-charles.
Such a monoculture has the same issues as MS Windows had, even for the same reasons.
The "Unix Way" of simple tools interacting seems more reasonable. Especially when it comes to lock-in effects.
It's very much built the UNIX way.
Runtimes (CRI) - Docker shim (deprecated), containerd, CRI-O, kata
Networking (CNI) - Flannel, Calico, Cilium, cloud specific ones, many more
Storage (CSI) - way too many device plugins - GPU, TPU, RDMA/SRIOV NICs
Data store - etcd, dqlite in microk8s, SQLite/postgresql/mysql in k3s
DNS - kube-dns (somewhat deprecated), CoreDNS
Ingress - NGINX (multiple), HAProxy, Envoy (many), etc
Kubelet -https://github.com/virtual-kubelet/virtual-kubelet
Kube-proxy - Cilium can act as a replacement
Cloud-controller-manager for each cloud provider
kube-scheduler - https://kubernetes.io/docs/tasks/extend-kubernetes/configure...
kube-apiserver and kube-controller-manager are two parts where I’m not aware of any other implementations but a) they are kind of the heart of k8s and b) can be easily extended with CRDs/operators.
Imagine if you wanted to use a Unix operating system, but instead of choosing a Linux distro, you just read the POSIX standard and went out and found every required utility, plus a kernel, and had to figure out on your own how to get those to work together and create a system that can run application-level software. If you just go to kubernetes.io and follow the instructions on how to get up and running with the reference implementation, that is what you're doing. It makes no decisions at all for you. You can run external etcd, or use kubeadm to set it up for you. You can run it HA or on a single node. You can add whatever overlay network you want. You can use whatever container runtime engine you want. You can use whatever ingress controller you want, or none at all, and not have any external networking, just as you can install Linux From Scratch and not even bother to include networking if you want a disconnected system for some reason.
You have pretty much complete user freedom, and that is, in fact, the source and reason for a whole lot of complaints. Application developers and even most system administrators don't want to have to make that many decisions before they can even get to hello world. I believe Kelsey Hightower commented on this a while back, saying something to the effect that Kubernetes is not meant to be a developer platform. It's a framework for creating platforms.
Application developers, startups, and small business should almost never be using Kubernetes directly unless they're actually developing a platform product. Whether you use a "distro" like RKE2 or k3s or a managed service from a cloud provider, building out your own cluster using the reference kubernetes is the modern day equivalent of deploying a LAMP stack but doing it on top of Linux From Scratch.
Anything but the most trivial workloads will also lead you into questions of how to mount volumes, cijfure) use configmaps and secrets, etc.
And that's not even touching the cluster configuration, which you can skip over if you are using a cloud provider that can provision it for you.
Yes, it doesn't cover as many features as Kubernetes, but it should be good enough for most software and you can still make the switch later. I would never go back.
You can read more on our blog if you're interested: https://pirsch.io/blog/techstack/
[0] https://www.nomadproject.io/docs/install/production/requirem...
Hashicorp is doing a smart but normal thing for on-prem sizing there - they are recommending specs which ensure you have no problems for a lot of workload sizes. And "workload" can grow very large there, since a single DC can handle up to 5k clients and such.
IIRC both k8s and nomad rely on the Raft algorithm for consensus, and so both of them inherit the requirement of a minimum of 3 nodes from that algorithm.
If you want something to run some containers in the cloud, and you aren't concerned by the occasional failure, then there is no issue running a single Nomad (or k8s) on a single machine and later adding more if and when you require it.
And yes, nomad recommends 3 - 9 nodes in prod, maybe 11 or 13. The 3 are necessary to be able to tolerate one broken/maintained node and maintain the ability to schedule workloads.
You can increase the number of tolerated nodes by increasing the number of nodes - 5 tolerate 2, 9 tolerate 3, 11 tolerate 5, 13 tolerate 6, but raft becomes slower with more nodes, because raft is a synchronous protocol.
However, it is entirely possible to run a single node nomad system, if the downtime in scheduling is fine.
We run a Raft cluster of 5 voting members with 13,000 physical Nomad nodes mixed across OS's with a bunch of random workloads using docker, exec, raw_exec, java, and some in-house drivers. I'll clarify that (and I can't... because I can't edit my post anymore :( ).
We're nowhere near that size, but the architecture is probably identical.
It, of course, depends entirely on your use case and workload.
I have a few dozen jobs running with what I think is a pretty decent mix/coverage. I've got a bunch of services, some system jobs, half dozen scheduled jobs, etc, etc.
I'm running my cluster with 3x "master" nodes which act as the control plane for nomad and consul as well as run vault and fabio (for ingress/load balancing). I have three worker nodes running against this that are running consul and nomad agents as well as my workloads.
The control plane nodes are currently sitting on AWS's t3.nano instances. `free` shows 462MB total RAM with 100MB available. Load average and AWS monitoring show CPU usage within a rounding error of 0%.
If this were in a professional capacity I probably wouldn't run it this close to the wire, but for personal use this is fine--it will be quite easy to upgrade the control plane when or if the time comes.
We started out with a three-node cluster using the smallest (1 vCPU/2GB RAM) VM you can get. Our initial requirement was to be able to do zero-downtime deployments and have a nice web UI to perform them (alongside the other advantages you get from a distributed system). We have now rescaled them to the next tier (2 vCPU/4GB RAM).
The hardware requirements depend on your workload. We process about 200 requests/sec right now and 600-700 per second on Saturdays (due to a larger client) and the nodes handle this load perfectly fine. All our services are written in Go and there is a single central Redis instance caching sessions.
Our database server is not part of the cluster and has 32 physical cores (64 threads) and 256GB RAM.
I say start out small and scale as you go and try to estimate the RAM usage of your workloads. The HashiStack itself basically runs on a calculator.
I'm using Docker Swarm in production, which let's me easily spin up the same services locally as I do for production - it's really nice to have one set of yaml files that can be used across all environments, and of course it gives more confidence before shipping to prod.
How do you handle local development? Do you have Docker Compose files specifically for that, or something else?
https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...
This is Nomad's most underrated feature, IMHO. You don't have to use containers for everything if you don't want to. For example, if you're a golang shop you can run everything as native binaries and cut out docker completely.
Nomad has much simpler networking, i.e. no web of iptables rules to figure out. You can add Consul connect as a service mesh if you need it, but if you don't, you can keep things very simple. Simple = easy to understand, run, and debug.
The main downside for me is a lack of plug and play pieces, e.g. a helm chart for Grafana or Prometheus. You'll have to write your own job for those, though it's very easy to learn. I'd love to see a series of Nomad recipes that people could use.
I think it's the ideal choice for on-prem, bare-metal, or 'weird' deployments where you need a bit more control. You can build the exact stack you need with the different HashiCorp projects with minimal overhead.
I can't recommend it enough! I help people move to Nomad, my email is in my profile if you want to chat :)
Good tools have very specific use-cases so as soon as you start talking about IIS on windows, VMs, containers etc. it just sounds like lots of the dependencies you are trying to remove in a scalable system.
Containers are relatively restrictive but they also enforce good discipline and isolation of problems which is a good idea imho. I would not want to continue to support our Windows servers running IIS, just too much to sort out.
Many people still have mission-critical Windows applications. Windows has nominal container support, but the footprint of a basic Windows container image is extremely high and the limitations of how you can use them are pretty onerous.
It's not going to force your hand. You can disable all the task drivers except the docker driver if you want a container-only cluster. The drivers themselves are lightweight.
In an ideal world, every company is two years old and only produces software in perfect docker containers, but in reality there's always some service that doesn't work in a container but could benefit from job scheduling.
I think it's great that we can add scheduling to different runtimes. Some folks want or need to use those different runtimes, and I like that Nomad lets you do that.
Ironically you can also just deploy a go executable into an empty Docker container and it's basically the same as the raw executable, but with all the config abstracted to be the same as other containers'.
f.e. We're exploring cortexmetrics currently, and spinning the full stack version up on k8s (openshift) was straightforward. Porting all that spaghetti onto nomad would be a huge job, though part of the frustration is knowing someone, tucked away on a private network, has already done this.
Hard agree. I know this person and have been this person before.
I've toyed with the idea of writing a book of Nomad recipes and tips, I wonder if anyone would read it?
Also, watch this space, helm for Nomad may be coming soon: https://github.com/hashicorp/nomad-pack
The big value of sites (mostly github master repos) that offer recipes for saltstack/chef/ansible/puppet, or the helm collective, etc - are that they're continually being tweaked as new versions of (upstream) software are released.
They usually all require a fair bit of localisation before they 'just work', at least in my experience, but the template taken from a proven functioning system, and then abstracted & shared, is worth its weight in gold.
I've set up a few nomad jobs, but nothing anywhere as complex as, say, this cortexmetrics monstrosity. Even our k8s & nomad guru baulks at such an undertaking.
Hashicorp just announced Nomad Packs filling that precise niche. Still in beta, and it was a long time coming but IMHO it was the main thing missing and is honestly awesome.
How do you deploy a thing to run on k8s?
One would think you deploy a manifest to it and that's it. Like yaml or json or hcl or whatever.
No. The built in thing is not good so someone wrote Helm.
So you deploy helm manifests on there?
No. Helm is not good either, you need helmsman or helmfile to further abstract things.
How do you do networking?
Layer a service mesh with custom overlay networking that abstract upon k8s clusterip.
jeesh. why?
kubectrl apply -f deploy.yaml
There is no reason to use it for your own software if you just have a single cluster.
People have got to stop just blindly running stuff off the internet.
Well, until now, when I'm using Terraform (particularly helpful relative to kubectl when I need to provision cloud resources directly), but I've still never used Helm.
I don't use helm at all, and I manage a large scale platform built on Kubernetes. Everything is either declared directly in YAMLs and deployed with `kubectl apply -f <YAMLs or directory containing them>`, or rendered using Kustomize and, again, deployed using `kubectl apply -f -`.
Kustomize can be rough around the edges but it's predictable and I can easily render the YAMLs locally for troubleshooting and investigating.
I had crypto miners using my k8s cluster within a couple of hours.
That was also the last time I used helm.
This was legitimate a bug that they immediately fixed when I reported it... There is no legitimate reason to expose the master JNLP port to the internet, ever. The chart did not have a configuration option for this, it just exposed the JNLP port as part of the same k8s Service that exposed the web port. (The JNLP port is for jenkins workers to phone back home to the main instance, it's not for external use.)
"Just read the docs" is not an answer to a chart setup which is just plain broken.
As I started using it more, i eventually moved up to using helm. I've been running production k8s for a few years now and haven't used helmsman or anything but helm yet.
But that's the problem right there. Shop after shop has invested in kubernetes with a team that was just learning it. And then they layered tool after tool to help massage pain points -- of which kubernetes has plenty. That leaves us at today where every shop effectively has a custom stack of kubernetes infrastructure cobbled together of various tools and philosophies.
It's the same problem as JavaScript. There's nothing terrible about the technology -- on the contrary it's amazing stuff! But the context of how it gets used leads to a type of gridlock of opinionated tooling published just last week.
The comment about development and packaging tools can indeed present a problem. I tend to favor Helm for applications that need to be deployed more than once and Kustomize to differentiate between environments but I've definitely seen what I would consider horror stories. Even if you add a bit of GitOps, it's not too hard to understand. The horror stories seem to occur when teams try to map their previous workflows directly to k8s. Many times the worst k8s development workflows are created to enable teams that had broken workflows without k8s.
kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml}
One JSON/YAML file too uwieldy? Generate it using jsonnet/CUE/dhall/your favourite programming language. Or just talk directly to the Kubernetes API. You don't have to use Helm - in fact, you probably shouldn't be using Helm (as the whole idea of text templating YAML is... thoroughly ignorant in understanding what Kubernetes actually is).
> Layer a service mesh with custom overlay networking that abstract upon k8s clusterip.
You don't have to, and you probably don't need to. I'm happily running bare metal k8s production without anything more than Calico, Metallb and nginx-ingress-controller. That effectively gives me 'standard Kubernetes networking' ie. working Services and Ingresses which is plenty enough. And if you're using a cluster deployed by a cloud provider, you don't have to worry about any of this, all the hard decisions are made and the components are deployed and working.
Agreed 99%, but there is one useful thing that Helm provides over Kubectl+structured templating (Nix/dhall/whatever): pruning objects that are no longer defined in your manifest.
However, that's solvable without taking on all of Helm's complexity. For example, Thruster[0] (disclaimer: an old prototype of mine) provides a pruning variant of kubectl apply.
This is fine for the first deploy, but if you delete a resource from your manifests then kubectl doesn’t try to delete it from the cluster, so in practice you need something like Terraform (or perhaps ArgoCD) which actually tracks state. And of course as you mention, you probably want to generate these manifests to DRY up your YAML so you can actually maintain the thing.
I would love to hear from others how they solve these problems.
> as the whole idea of text templating YAML is... thoroughly ignorant
1000%
I did not think about a “delete”, and I usually don’t want it to be automatically deleted. But that’s a great idea for that convergence mechanism if I have something that explicitly adds a deleted resource line to make sure it stays deleted until otherwise specified.
The Ruby code can access anything Ruby to generate the manifests, so I have a Turing-complete language and I can use class inheritance, mixins, method overrides. I was also able to write a suite of helpers to take a raw yaml manifest from an upstream project, and use a functional programming style to transform the manifest to something else. If I ever have time, I’d write the ability to import templates from Helm.
This was written for a one-person devops role, intended to be able to manage almost-similar things across different environments (dev, staging, prod), which works great for small, early stage startups, for our level of complexity. At this level, I don’t need it to track state.
When our team size grows, I’ll have to think up of some other patterns to make it easier for multiple people to work on this.
The other thing is that for much more complex things, I would just write operator code using Elixir, and bootstrap that into k8s with the Ruby framework.
I've built these kinds of things too in Python (or Starlark). It's an improvement over YAML, but I often feel the pain of dynamic typing.
> The Ruby code can access anything Ruby to generate the manifests, so I have a Turing-complete language and I can use class inheritance, mixins, method overrides.
I actually don't want any of these things in a dynamic configuration language. I kind of just want something that can evaluate expressions (including functions, objects, arrays, scalars, and pattern matching). If it's Turing complete that's fine but unnecessary. I explicitly don't want I/O (not in your list) or inheritance, mixins, method overrides, etc--unnecessary complexity.
Dhall probably comes the closest, but it has a tragic syntax and I'm not a good enough salesman to persuade real software teams to learn it. I've also heard good things about Cue, but my experiences playing with it have only been painful (steep learning curve, don't really see the upsides).
I use classes and mixins to be able to generate similar manifests with slight differences across different clusters or environments. I sometimes use imports (I/O) for manifests provided by an upstream (such as from AWS docs), do transforms to get manifest I want.
I modelled the design off of Chef, which will also declaritively define and converge a set of systems towards desired state. Well-known paths and conventions helps keep things organized.
I havn't had a problem with dynamic typing, but I have also been using Ruby in application development for over 10 years. You might see it as unneeded complexity, but I have been able to use the flexibility for years now. This tooling was designed for a team that uses Ruby as the primary language, and uses designs and idom that would be familiar for a Ruby dev team. It is definitely opinionated and I don't expect it to be universally useful for everyone.
kubectl apply --kustomize manifests/ --prune --selector app=myapp
It cleans up old stuff from the cluster and also allows you to split your manifests across multiple files.
Funny that the thing you shouldn't use is what the entire Kubernetes ecosystem uses for deployment. It's almost like there's no good way to do it.
Could you expand on that? It sounds like an interesting position (bordering on the philosophical), but I don't know enough about Kubernetes to gauge its accuracy for myself.
Helm is just sort of dumb text manipulation with a TINY bit of deployment management built on top of it. There isn't really a whole lot that helm buys over extending k8s.
1) Text templating YAML is just bad. It's the serialized format of some structured data - instead of templating its text representation (and dealing with quoting, nindent, stringly-typed variables, and general YAML badness), just manipulate the structures directly before serializing them. For example, write some Python code that composes these structures in memory and then just serializes them to YAML before passing it over to kubectl. You can also use ready made tooling that uses domain-specific languages well suited to manipulating and emitting such structured configuration, like jsonnet/kubecfg (or CUE/Dhall/Nix combined with kubectl apply -f). Eschewing text templating is not only more productive by letting you build abstractions (eg. to deal with Kuberenetes' verbosity with labels/selectors, etc.), it also allows you to actually compose more complex deployments together, like 'this wordpress library uses this mariadb library' or 'this mariadb library can take a list of objects that will be upserted as users on startup'.
2) Those YAML/JSON manifests aren't even that native to Kubernetes. A lot of things go behind the scenes to actually upsert a manifest, as Kubernetes' resource/object model isn't nearly as straightforward as 'apply this YAML document full of stuff' would indicate (there's state to mix in, API/schema changes, optional/default fields, changes/annotations by other systems...). With k8s' Server-Side-Apply this can now be fairly transparent and you can pretend this is the case, but earlier tooling definitely had to be smarter in order to apply changes. Things like doing structure-level diffs between the previously applied intent and the current intent and the current state in order to build a set of mutations to apply. What this means is that Helm's entire 'serialized as YAML, manipulated at text level' stage is not only harmful and a pain in the neck to work with (see 1.) but also unnecessary (as 'flat YAML file' isn't any kind of canonical, ready-to-use representation that Helm had to use).
Such thing surely does exist: https://cdk8s.io. And here's the relevant announcement blog post: https://aws.amazon.com/blogs/containers/introducing-cdk-for-....
I'll make the philosophical case: text-level templating of a computer-readable format is almost always the wrong approach. It becomes unnecessarily hard for someone who later needs to read the code to ensure that it generates valid (both syntax and semantics) markup. It's also more hard for tooling to verify the validity, because the syntactic validity may depend on inputs to the program.
Compare the approaches of PHP and JSX. They both accomplish a similar goal (interleaved HTML and control flow), but JSX works at the element tree level and makes it impossible to generate poorly-formed HTML syntax -- it becomes a compile-time error (though it doesn't ensure semantics, e.g. a tbody outside of a table is legal). Compare with PHP, which very much allows you to generate invalid HTML, because it's just a text templating language.
(From what I can tell, Helm works more like PHP; if I'm wrong my philosophical position stands but might not apply here)
What I don't get is some people in my company thought it was a good idea to make a "docker helm chart"... That is, a helm chart capable of deploying arbitrary containers, ingresses, etc... Like, a true WTF, since the values file ends up looking awfully similar to a k8s manifest :D.
kapp deploy -a my-app -f ./examples/simple-app-example/config-1.yml
If you look at Google they build a whole PaaS (Cloud Run/Appengine) on top of k8 and I guess that's the way it's meant to be used.
IaaS -> K8 -> PaaS -> deploy.sh
Overlay network is easy if you don't need your pod and service IP to be routable from outside the cluster. It took me less than 1 afternoon to learn to use Kubespray to deploy an on-premise cluster with Calico for overlay network.
Plain manifests in a Git repository will do, and let something like Flux apply them for consistency. It really isn't harder than SSH and ad-hoc scripts or Ansible.
Poor design. It's been cobbled together slowly over the past 7 years. They built it to work only a certain way at first. But then somebody said, "oh, actually we forgot we need this other component too". But it didn't make sense with the current design. So they'd strap the new feature to the side with duct tape. Then they'd need another feature, and strap something else to the first strapped-on thing. Teams would ask for a new feature, and it would be shimmed in to the old design. All the while the design was led by an initial set of "opinionated" decisions that were inherently limiting. The end result is a mess that needs more and more abstraction. But this isn't a problem when you are already a billion-dollar company that can pay for other teams to build more abstractions and internal tools to deal with it.
This is business as usual in any quasi-monolithic Enterprise project. Typically the way you escape that scenario is to have major refactors where you can overhaul the design and turn your 50 abstractions into 5. But instead, K8s decided to have very small release and support windows. This way they can refactor things and obsolete features within a short time frame. The end result is you either stick with one version for eternity (a security/operability problem) or you will be upgrading and deal with breaking changes every year for eternity.
I would call this grass roots marketing. Some companies are very good at it. Maybe that ones have more know-how in the ad space than others?
We started by using it as a "systemd with a REST API" on a single server, and gradually evolved into multiple clusters with dozens of nodes each.
It has been mostly a smooth ride, even if bumpy at moments, but operationally speaking Nomad is closer to Docker Swarm in simplicity and Kubernetes in terms of the feature set.
We didn't find ourselves needing K8s as we're also leveraging components from the cloud provider to complete our infrastructure.
[0]: https://monitoro.co
This a question I still need to google but what features does Kubernetes have that Docker Swarm needs?
Because the perceived complexity of Kubernetes just blows my mind, where Docker Swarm seems a lot more simpler for the same benefits, but my its just abstracted away?
I will say upfront im naive when it comes to container tech.
I also think K8s is a reasonable cloud independent abstraction when you need to move your workloads to another provider. It prevents cloud lock-in. And I suppose Nomad would do that too.
So far my K8s experience has been very good. However, if I ever have to do more with it (on the administrative side) I may have a less positive experience. Not sure.
You can have AWS "organizations" managing multiple accounts. IAM will allow to set up any policies you want in your organization.
On the other hands side now you have a multitude of complexity added to your setup…
If I would be snarky I would see a case of "resume driven development". But let's just assume it was lack of time to find the simplest solution and it's than often preferred to just go with the flock as so many others can't be possibly wrong.
Ecs is pretty simple but does not have the large mindshare that kubernetes or nomad have.
All major cloud providers offer a managed kubernetes service at minimal added cost to running the worker VMs yourself.
With managed Kubernetes, the simplicity question is no longer obviously in Nomad's favour. As other comments allude to, Kubernetes as a user is pretty easy to master once you get used to its mental model.
Hopefully Hashicorp will have managed Nomad soon.
Some of the things that might still be needed in managed k8s instances: better ingress with ingress-nginx, cert-manager, monitoring/logging/alerting, tuning the alerts, integration with company SSO, security hardening.
If it's a multi-tenant cluster: LimitRanges/ResourceQuotas, NetworkPolicies, scripts to manage namespaces and roles, PodSecurityPolicies (or equivalent), onboarding offboarding procedures.
I'm sure you'd need similar things to have a proper production Nomad cluster too, so your point still stands. But at least for EKS/GKE clusters, they're pretty bare-bones.
Your developers aren't going to say that it's simple when Google force upgrades their cluster to a version that deprecates APIs in their yamls for a job they worked on 2 years ago and swiftly forgot about.
Then when you explain to them that Google insists on putting everyone on a force-upgrade treadmill, you can literally watch as the panic sets in on the faces of your engineering team managers/leads.
Nomad is a breeze in comparison to managed K8s.
Everyone that I've talked to that thinks Kubernetes is simple is barely using the thing and could likely save a lot of money and development effort using something like Nomad instead.
Yeah no thanks.
At this point k8s has "won" for all intents and purposes. It has gained critical mass, succeeding where other infrastructure management tools both open and closed source have failed.
Also API upgrades shouldn't be a problem, even if you were using beta APIs. They are only really troublesome if you decided to indulge yourself with some alpha APIs before they were fully baked. If you don't do that then you won't run into any problems.
First of all, those things you're talking about are installables in K8s. They don't come by default. Many people with (managed) kubernetes installations aren't even using them and get by just fine. Having them certainly isn't free and work certainly had to be done to build those for Kubernetes and likely are (or will be) trivial to implement elsewhere. I certainly was able to automate my certificate management infrastructure before I had Kubernetes.
The reality is that there are many companies out there that want like 10% of the features of an orchestrator like Kubernetes and don't need all of those features that make you think that it's a zero-sum game that Kubernetes has won.
The reality is that there are competing offers like Nomad that more-easily accomplish our technology goals and thus more attractive to enterprises with big budgets like mine.
Kuberenetes having the "critical-mass" that you speak of isn't to the exclusion of other competing tools having "critical-mass". Just like Oracle and DB2 aren't the dominant RDBMS of today.
As for calling people indulgent for using beta APIs, let's not forget that Deployment, StatefulSet, DaemonSet, ReplicaSet, NetworkPolicy and PodSecurityPolicy all started as beta APIs and were only removed in 1.16. And I imagine you wanted to use ingresses, but the old ingress.class annotation was deprecated and replaced with IngressClass in 1.18 and that was firm cutover for _everyone_. I can't imagine Kubernetes being very useful to many people without any of these...
I just pointed out that for everything you would need to build custom on Nomad there exists off-the-shelf components that will plug right into k8s. There is no way this would save money or development time, which was my main contention with your assertions.
You basically mischaracterized everything I said and then missed the entire point.
I don't disagree that there are some cases where Nomad would be superior. Off the top of my head if I wanted to build a modern rendering farm and I knew I wouldn't want to use the cluster for anything else then I would consider it over the HPC toolkits I have previously used for that (RIP Grid Engine) or Mesos which used to be king of that space.
The problem is these are incredibly niche cases and for 99.99% of companies they are better off swimming with the flow.
This is what I mean by "k8s has won". It's not that other things won't continue to exist and be created but most of them won't survive unless they commercialise within a niche or find a sufficiently large user that is willing to do the vast majority of the development (eg. Netflix Titan).
It's not a zero sum game but it's damn close. See here the corpses of Docker Swarm, Convox, Flynn ( :( ), original Deis, arguably ECS (it's a zombie at this point let's be honest).
The tide really turned when Azure and AWS were basically forced into offering managed k8s. In AWS case they also had to made deep modifications to VPC and IAM to properly support it. These investments wouldn't have been made unless forced which lends credence to the weight k8s has in the ecosystem. (Worth mentioning there is no hosted Nomad I'm aware of, their enterprise offering is install + support)
k8s is becoming the POSIX of distributed scheduling. The API you use to run diverse workloads over large numbers of machines that is relatively portable between companies. Right now there is still some vendor nonsense going on but over time it will be smoothed out.
The vast majority of distributed applications are going to be built targeting k8s as their runtime API. We can already see this happening but it will only increase over time.
To summarize I think it's fair to say things like "Nomad can sometimes be good if conditions x/y/z are met" but I think it's very dishonest to consider it as a viable competitor to k8s in the general case for most users because of the reasons outlined above.
I dare to disagree. The costs are in fact horrific!
There are orders of magnitude in costs between running your stuff yourself vs the highest level of managed services on the cloud which is usually managed k8s.
(That also explains why there is so much marketing fuss, and push of management, to use k8s: It's by far the most profitable offering for the cloud providers).
So not only are the masters relatively cheap, they cost is amortized over the cluster size.
AWS is probably very unhappy about this but they were forced into this by GKE pricing model. Google pushed people to adopt k8s not because they can make a lot of money from people using it but because they can take away a lot of money from AWS and increase workload portability, which benefits Google far more than AWS as they are in second place.
The most expensive managed services are things like RDS, MSK, Elasticache, etc. MSK in particular is pretty egregious with a 80% premium considering they haven't done any additional engineering like RDS.
Nowadays I don't have use-cases for either, but from playing around with Nomad it felt a lot more "Borg-y" than Kubernetes ever did to me. I rarely hear anything about the Hashicorp alternative for service discovery and such though, so it would be interesting how that compares.
I know that Borg was written in Java and Kubernetes in Go. Though the latter had a reputation in the beginning as a systems programming language, its purpose was actually to build large-scale cloud infrastructure projects with it and it proved formidably well suited for the task. It compiles fast, anyone can read it, good tooling and it is efficient for the layer on which it is meant to be deployed. Go, as Java is one of the most productive languages in use today, judging by the ecosystems they have spawned.
The community also has the attitude to pretend that these pitfalls don't actually exist, which is very different from C++ where most peculiar behaviours are well-understood and controlled.
> There is no real agreement on what "strongly typed" means, although the most widely used definition in the professional literature is that in a "strongly typed" language, it is not possible for the programmer to work around the restrictions imposed by the type system. This term is almost always used to describe statically typed languages.
(random definition found googling).
Who wants to know the details of those scaling issues can google for the Omega paper.
The "proper"™ solution to those design flaws was implemented in Mesos (and to my knowledge nowhere else until now).
And yet somehow Mesos failed... sigh.
It chooses the (perceived) cheapest thing with the best marketing. Always.
Ultimately it all boils down to two things:
- The projects being well written and well maintained from the outset
- Personal preference
I cannot overstate that second point.
It really is about time developers stopped pushing their own personal preferences as if it's some kind of fact.
Agreed, so let's all go to Rust or OCaml because they have very strong static typing systems. ;)
Both C++ and Go have plenty of warts, with C++ having way too many footguns and Go allowing you to use it as a dynamic language whenever you figure it's too much work to go through your problem with static types.
Seems very reasonable. Especially OCaml should get more of the praise it deserves.
If you need to stay on the JVM there's Scala which allows (with some discipline) to write "when it compiles it works" code.
Did it, though? Honest question. I get the feeling that it ended up competing with Java (and Python) more than it ended up replacing C++. The C++ folks seem to be way more into Rust than Go.
It's an interesting phenomenon I observe quite commonly.
I think in the devops space they see these VM based languages as basically introducing ten redundant layers of unnecessary assumptions and complexity on top of what is already a good foundation - the Unix OS layer. They know the unix os layer well but every time they deploy one of these VM based languages it creates headaches through unique and strange behaviour that violates their assumptions and knowledge and they have no way to learn it since you need years of experience as a developer to become comfortable with it all. From a developer perspective, we see the OS as this annoying ball of complexity and want VM languages to manage that and make it go away.
So it's all about what you know better and where your comfort zone is in the end. For a long time the developers had the upper hand because the OS story was a train wreck, but containers have turned that around in recent years, so now it is not such a crazy thing to pin down the exact OS and version and entire dependency chain you are going to deploy on.
I guess those people are just not educated enough. Those Ops people don't know much about programming in large usually. A lot of them for example think it's OK to write serous programs in Bash. This says it all, imho.
It's OK when someone looking after admin stuff isn't a full blown programmer. It's a different kind of job after all. But this needs to be taken into account when looking at that mentioned phenomenon.
Go binaries are statically compiled usually, no VM or managed. It's a simple language for simple solutions, which is often underrated.
Not that versed in Python, but there is CPython.
Bash is totally ok in the hands of someone who uses it for good ;) Agree it has too quirky syntax when you need complexity, so not good for large stuff.
Footguns are everywhere. You usually trade one in for another.
They're not VMs, but it's really a semantic difference, if you're comparing it to Java, especially now that GraalVM native compilation is gaining popularity.
My experience with Go has just been making small changes to small programs. So, I don't know what the normal experience is.
My experience with logging varies from:
print (works fine I guess)
import logging (this is pretty good - means I don't have to parse my logs before deciding where to send them)
import slf4j (6 logging frameworks and a logging framework disintermediation framework)
But I'm used to the JVM world. And when I first met slf4j, I was like, wtf is this crap, but I appreciate it now as the embodiment of the desire to standardise logging across the Java ecosystem.
While using slf4j does make it trivial to swap out logging frameworks in an app, in my 13 years at my last job, we only did that once, from log4j to Logback, so that's not so important.
But yeah, what I miss from Java land in Go logging is the common approach - loggers and appenders are (usually) configured outside of code, the user can provide their own configuration at runtime to override the config shipped in the jar to troubleshoot issues - especially when you can configure the logging lib to check the conf file every X seconds for changes - allows you to change logger levels on the fly without restarting the app (ditto Logback's JMX configurator).
And lastly, no matter the logging library, configuring them is near identical.
-- edit typo
Also the inability in the ones I've used for the person running the app to set a particular logger to a desired level easily.
I think klog can do this with the vmodules flag, if, and only if, the devs used klog.V() for their logging statements.
Logrus requires the dev to allow the user to configure the log level, common idiom I've encountered is doing so via an env var, but IIRC, that applies to all logging in the app, no way to limit it to particular files/modules.
It's been a real pain at times for me. Feels like the Go philosophy on logging focuses more on the dev controlling it, than the person running it.
Consul is used for service discovery. Fast, reliable and easily extensible. Yet to have a serious issue with it.
And yes, the error messages can be utterly cryptic.
One thing: Don't ever let junior devs try to create a cluster out of their own laptops (yes they will think it sounds like a good idea) as this will be a never-ending nightmare.
* in a legacy system, a server, perhaps with the leader node, filled up disk space and became unhealthy, the consul agent kept reporting it and itself as healthy, and failover and gossip generally wedged
* in a dev environment, after we replaced some servers in the cluster, the other nodes noted cert changes and refused to work with the new servers
* second-hand, in self-hosted installations, it caused a number of hard-to-troubleshoot outages
* something about circular dependencies and going by "wait _n_ seconds" rather than by healthiness
It was reliable enough that it could gather a really significant blast radius, and it had different gnarly failure modes, so documentation could be irrelevant from case to case.
The exceptions are the few cases where you know you will forever only need some strict set of features and your own solution can provide them by way more simple means than the in comparison "fat" off the shelf solution.
I'm not sure service discovery in a cluster is one of those cases.
How is Go not suited to those? I'm not seeing it - and are you comparing Go to Java or C++?
Even the original k8s codebase was notoriously horrible in Go, and that was "from the source".
Ironically, HashiCorp has a better Go codebase, and that's where I picked best practices from, not from Google.
The problem with Go is that the Googlers fail to challenge some of the design decisions. The giants of computer science working on it at the company cannot be questioned, and that just leads to a broken feedback process.
As someone said, Go is the new C, skipping decades of new language design concepts.
Well, they can be questioned but they're not very receptive to it, so most people don't bother. During my time at Alphabet I didn't see much use of Go anyways, other than in glue tooling (including a horrific large-scale incident due to non-obvious behaviour of Go code in such a glue tool).
Well, some are doomed to repeat their errors.
In fact C was even crappy for it's own time.
[0] https://www.hashicorp.com/blog/announcing-hashicorp-nomad-1-...
In my experience (having worked with Nomad, k8s, mesos, and homegrown schedulers), k8s is fantastic from an early developer experience, but rough from an operations perspective. It's definitely gotten better, but there's a reason lots of folks want to have someone else host and operate their cluster.
Nomad is the opposite. Less tooling and bit more complexity for the app developer, significantly easier to operate at the cluster level. Nomad is also less "all-in" in it's architecture, which gives operators more flexibility in how they design and support the cluster.
The fact that we can run on anything and don't _have_ to use containers is a serious advantage, particularly for game development.
Which is really great, because we've basically built nomad-pack in terraform and we would really like nomad-pack to replace that, because .. it works, but it could be better than terraform wrangling for templating :)
Software: bunch of solaris/omnios vm's, torrent client, owncloud, mtproto proxies for family and friends, my site, minecraft, quake, openttd, trss, gitea, drone.io, jenkins.
There is other stuff, but it is usually shortlived, i check it out, test it and most often discard it.
We use Kubernetes instead of Nomad at work but we are also using Consul in the Kubernetes cluster.
Agree, Nomad is so easy to get started and because of the simplicity of the architecture, very easy to maintain as well.
> but Kubernetes allows you to do more ... We use Kubernetes instead of Nomad at work
Same here, Kubernetes at work, Nomad for personal projects. But I have yet to find anything I cannot do in Nomad that you normally do in Kubernetes. Could you provide some specific things you've tried to get working in Nomad but couldn't (or simply doesn't exists in Nomad), while it worked in Kubernetes? Would be useful to know before I get too deep into Nomad.
VM live migration. I was surprised that people use kubevirt for that, but apparently this is a valid usecase. Otherwise nomad can do relatively complex vm configurations.
Also I am using Nomad over baremetal to create VM's with Nomad clients onboard. Kind of a messy setup, but works really well for me.
[0] https://www.nomadproject.io/docs/drivers/qemu [1] https://kubevirt.io/
> Running Kubernetes is not free. It is expected that it will take about 0.5 to 0.75 of a single core on an embedded system like a Raspberry Pi just as a baseline. [1]
Is it simply a matter of Kubernetes being an open source project and Nomad being owned by HashiCorp?
It would also be interesting if Docker's Swarm was.also featured in this comparison, as it just works right out of the box and doesn't come with any bells and whistles.
Yes it does. Docker Swarm essentially provides an easy way to get a cluster of Docker instances running in multiple nodes that works right out of the box.
For example, with cert-manager running on Kubernetes you can request a TLS certificate by creating a Certificate resource (like you would any other Kubernetes resource). This is the same regardless of whether you want a self-signed certificate, an ACME-issued certificate (and whether that gets performed via HTTP01 or DNS01 or something else). Oh, and this fully ties into Kubernetes' RBAC system.
In Nomad the closest thing is annotating jobs with traefik-specific tags (and allowing Traefik to do cluster-wide discovery of all tags), but that only works for serving certificates that are managed by traefik, not if your application wants eg. to terminate the TLS connection itself, or if it wants some other PKI hierarchy (eg. a self-signed CA which then issues some other certificates for mutual TLS auth between application services).
Kubernetes also has better support for organization-wide multi-tenant clusters than Nomad seems to have (eg. nomad's policy engine, audit logging and resource quota system are gated behind their “enterprise” offering).
That one's a little weird, I suppose you're right, but all the clients our Kubernetes team works with all want separate clusters or testing, staging and preproduction. They certainly don't want a multi-tenant cluster and share resources with other clients.
And I think that's one of the biggest issues with how people use Kubernetes these days (another candidate being insisting on drive-by deploying a cluster from scratch instead of deferring to cloud providers or a dedicated platform team that can plan for long-term maintenance).
Kubernetes thrives in multi-tenant environments: you get huge resource savings and vastly simplified operations. Everyone in your organization gets access to all clusters, and they can just as easily deploy experimental best effort jobs or their development environment as they can deploy and/or inspect production jobs. Well set up quotas and priority classes mean that production jobs never run out of resources, while less important stuff (batch jobs, CI, someone's pet experiment) can continue to run on a best effort basis, just keeps getting preempted when production wants more resources.
You can even continue to have hardware separation between highly sensitive and fully untrusted jobs by using taints and tolerations, if you feel that's necessary. You still get one control plane instead of five different ones.
How much have you had that setup audited? It seems like a lot people aren’t comfortable saying that the internal boundaries are strong enough, which leads to the proliferation of separate clusters.
I don't really understand how you can say this and then...
> Kubernetes thrives in multi-tenant environments: you get huge resource savings and vastly simplified operations. Everyone in your organization gets access to all clusters, and they can just as easily deploy experimental best effort jobs or their development environment as they can deploy and/or inspect production jobs. Well set up quotas and priority classes mean that production jobs never run out of resources, while less important stuff (batch jobs, CI, someone's pet experiment) can continue to run on a best effort basis, just keeps getting preempted when production wants more resources.
... advocate for this. All what you are describing, which is basically what every hardcore k8s user/evangelist will tell you to do, it's reimplementing many, if not all, the features a Cloud provider is already giving you in their own resources. But you are taking the ownership and responsibility for this on your local platform/infra team. What if you screw something with CoreDNS? what if you break some RBAC roles used cluster-wide, while trying a change in the beta environment? I'm pretty sure there are (or will be) specific k8s tools to manage this but still, you are adding complexity and basically running another cloud provider inside a cloud provider for the sake of binpacking. For certain sizes of companies it might be worth the effort, but it is for sure not a silver bullet and probably applies to much less companies that many evangelists try to sell.
A well-designed KaaS offering from a cloud provider will do that by itself. GKE exposes GCP load balancers as an Ingress controller, IAM identities as Kubernetes RBAC identities, persistent disks as PVs, ... You just get them under a single declarative API.
> But you are taking the ownership and responsibility for this on your local platform/infra team.
With a platform team you're concentrating already existing responsibility into a team that can specialize in operational excellence - vs. that same responsibility being spread out across product teams that have to individually manage their own cloud resources, reinventing the wheel by writing the same terraform/{ansible,puppet,chef,...} boilerplate poorly. My experience is that these per-team bespoke AWS deployments are much more brittle than whatever a dedicated team can provide if given the responsibility and means to do things well.
> What if you screw something with CoreDNS? what if you break some RBAC roles used cluster-wide, while trying a change in the beta environment?
An outage is an outage, you roll back to stop the bleeding, investigate what happened and try to prevent whatever caused it from happening in the future. Neither of these examples are unsolvable in a multi-tenant environment, nor especially more likely to happen than similar screwups when using cloud provider resources.
My experience with EKS on AWS tells me that it's not that simple, there are still many things to be glued together. I understand AWS historical position on K8s, and they probably want to keep the k8s experience on AWS good but not awesome but I'm pretty sure that there are even in GCP still serious gaps between "native" GCP features and k8s ones, where you end up reimplementing them on both sides. But I'm no GCP expert so I might be totally wrong.
> With a platform team you're concentrating already existing responsibility into a team that can specialize in operational excellence - vs. that same responsibility being spread out across product teams that have to individually manage their own cloud resources, reinventing the wheel by writing the same terraform/{ansible,puppet,chef,...} boilerplate poorly. My experience is that these per-team bespoke AWS deployments are much more brittle than whatever a dedicated team can provide if given the responsibility and means to do things well.
I'm totally fine with this approach, and we are actually trying to implement it at $DAYJOB but I don't really get why you see thew AWS API as a different monster from the K8s API. With a complex enough system you will need many lines of YAML/charts/Terraform/whatever on the k8s just like CF/Terraform/Pulumi/whatever on AWS. And you can totally have a team that takes care of the quirks and details of AWS while exposing a usable and unified interface for services deployements to the the rest of the engineering organization. I understand if we were talking about bare metal vs Kubernetes (even on-prem), k8s would win hands-down. But in the cloud-native world, I don't really see that day vs night change. Everything has its tradeoffs and its quirks and bugs and corner cases.
I love Nomad and so do our devs, but K8s has a much larger ecosystem and more tooling options.
A few examples:
- If you want to run some third party software, there's a helm chart for that. In Nomad, you're likely going to figure things for yourself.
- K8s has a lot of cool stuff like Knative, KEDA and others for more sophisticated deployments.
- K8s has tons of operators to integrate with public clouds and external services. Nomad really lacks something similar.
- There's a ton of knowledge and guides related to K8s online, not so much for Nomad.
- There are many managed K8s solutions, which makes things easier. To this date Hashicorp still does not have managed Nomad.
That said, I think Nomad is a few years away from being something truly amazing. If they can solve the network security problem (beyond optimistic documentation), I think it'll be amazing. For now, it's just a good scheduler.
Any particular problems you've experienced that have been long standing an unresolved?
I occasionally like to kick the tires on Nomad, but so far haven't found it compelling enough to switch away from my current solution, GKE (mostly because of how integrated it is with everything else I am doing and, at this point, familiarity).
So, curious to hear from others where the rough edges are.
I’ve reported and participated in a few issues. Most got fixed within the next couple of minor versions, some went neglected.
I think the somewhat recent 1.0-release was pretty well-timed. There are still a few rough edges (though it’s gotten a lot better continuously over than last year). Specifically, Consul Connect and federation (especially together with ACLs and mTLS) are just about coming together now.
If you don’t rely on Connect and federation across regions, I’d say it’s reliable for production use. Other than that maybe wait 6-12 months and reassess.
I’d also advise to put ACLs and TLS in place before you start putting production loads there; it’s not great to introduce in a running cluster.
I can get so much value by plugging in other tools to this generic platform. Feature flags? just plug it in. On cluster builds? Just plug it in. Monitoring? Plug it in.
Interessting discussion about CapRover vs. Flynn, Dokku, Nomad on HN one year ago. [2]
[1] https://caprover.com/ [2] https://news.ycombinator.com/item?id=23465087
Probably, it could be end result of unconsistent design or bad technical choises. However, most likely it just means that there are multiple organizations and interest groups pushing changes and ideas to the project. This should be seen as a good thing. The downside is that there is no single source of best practices and this is confusing to newcomers. You just need to pick one distribution and trust the choises, or develop the competence and understanding.
And we could imagine that the userbase or the number of developers in single distribution (take OpenShift or Rancher) could be bigger that in Nomad itself.
Having said that, I still would like to see more stable Kubernetes landscape, and that has to happen eventually. The light distributions k3s and k0s are pushing things to nice direction.
OpenStack had similar, or maybe even worse, fragmentation and complexity issue when the hype was high. There were probably technically better alternatives (Eucalyptus?), but people (and companies) gathered around OpenStack and it won the round. However, comparing OpenStack to Kubernetes feels bad, as Kubernetes is technically far superior.
Even better if you're running Mac + BSD as well, it just runs everywhere.
A task definition is required, which is sort of a docker-compose.yml analogue, but it's also possible to use docker compose directly with ECS Fargate. So my 'get out of AWS' is pretty simple, I already have the docker-compose.yml ready.
> Kubernetes aims to provide all the features needed to run Linux container-based applications including cluster management, scheduling, service discovery, monitoring, secrets management and more.
> Nomad only aims to focus on cluster management and scheduling and is designed with the Unix philosophy of having a small scope while composing with tools like Consul for service discovery/service mesh and Vault for secret management.
This article just seems like "All of Nomad's best features phrased kindly vs all of K8s' worst features phrased critically".
Yes, but No. Kubernetes is a portability platform, that happen to -also- orchestrate containers.
Using kubernetes means you have complete reproducibility of the network setup, the deployment and the operation of any workload -no matter how complex-, on any kubernenetes cluster and cloud provider.
Nomad, is -well- just a glorified Airflow.
> Yes, but No. Kubernetes is a portability platform, that happen to -also- orchestrate containers.
The homepage of Kubernetes seems to disagree with you. Their headline there is "Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications." and also "Production-Grade Container Orchestration" so it feels safe to assume that Kubernetes is for orchestrating containers.
> Nomad, is -well- just a glorified Airflow.
I never used Airflow, but looking at the website it seems to be geared towards automating workflows, something like Zapier but self-hosted and open source? That's very different from what Nomad is.
Wouldn't you also use terraform for your kubernetes cluster?
People have trouble understanding what k8s is, and what to use it for. That's fine, it'll take a while, but they will eventually understand what "Production-Grade Container Orchestration" really means when they start working with it.
By just following this one tutorial (https://kubernetes.io/docs/tutorials/kubernetes-basics/deplo...), you already cover everything you used to do using docker-compose.
Now what ? You are going to learn about Services (https://kubernetes.io/docs/tutorials/kubernetes-basics/expos...), Ingresses (https://kubernetes.io/docs/concepts/services-networking/ingr...), Ingress-Controllers (https://kubernetes.io/docs/concepts/services-networking/ingr...), Persistent Volumes (https://kubernetes.io/docs/concepts/storage/persistent-volum...), Configuration, Pod Lifecycle, RBAC, Rolling Upgrades, Operator Pattern, ...
This is not about orchestrating containers anymore, it's a mix of network, configuration, storage APIs that just reunite everything you used to do with shell scripts under a fully declarative format. Then you realize, the _ACTUAL_ value of kubernetes isn't about the containers it can start-up, it's about being able to _MOVE_ those containers, their HTTP routing rules, their database, their backup schedule, their secrets, their configuration and everything else on totally different machines with different OS and different topology, just by running a kubectl apply.
For example, if your application uses a database the major lock-in concern is the flavor. If you're using Kubernetes to deploy MySQL or Postgres or paying e.g. Amazon/Google to provide it for you, there's relatively low cost of switching because it's a very tested, standard interface with well-defined semantics. On the other hand, if you're using something like AWS DynamoDB or GCP Datastore you probably do want to think carefully about an exit strategy because you'd be baking in assumptions about how those services work in a way which is non-trivial to replicate outside.
The important thing is remembering that this is a business decision, not a holy cause, and the right answers vary from project to project. For example, in a larger organization you might find that it's worth using the platform services not because they're cheaper or better tested than what you can build for yourself but simply because it's easier to be able to check various auditing boxes using the standard auditing tools than having to laboriously demonstrate that your implementation meets the same requirements.
I must say - that's exactly about it. No holy cause, technical decisions should not be made in vacuum and there's a lot more to it, always and if not, should be.
Having said that, mine is a qualified statement. If you have abstracted your workloads to the Kubernetes abstractions alone, you're good to go anywhere from Linode to GCP and anywhere in between.
This doesn't make sense. While in the end these tools might both run code in containers they serve a different purpose. Airflow is far more aware of ETL concepts and comes with a lot of batteries included for those use cases. Whereas Nomad is more a generic solution with more emphasis on infrastructure.
I've been thinking about building such a thing on Arch (btw) but haven't acquired enough time-energy to do it.
Kubernetes is a cloud operating system which lets you run modern apps in any environment. One important component of course container orchestration, but it went far beyond just orchestrator. Kubernetes has a very powerful ecosystem, and it managed to unite almost all infrastructure vendors and projects. Its amazing to see how so many competing companies could agree on something. And that become K8s.
Nomad is great when you're working with VMs, but I don't see it is much relevant in the modern era of K8s and cloud-native.
What's the simplest next step I can take? I'd like something with (close to) the simplicity of docker compose, but ideally being able to specify something like `instances: 2` and being able to route/load-balance to those instances from another service.
Honestly it sounds like you could be at the point where K8s is worthwhile.
Any suggested starting points for my research? DDG search for "ansible docker-compose" brings up some suggestions like [1] and [2] but I'm curious if you have other suggestions.
And just so I understand how these work together - I'd use Jenkins+ansible to push containers to my servers, I'd run consul within docker as well, and ... would Ansible register services with consul as it pushes them? Do the services need to be modified to speak with consul directly?
[1] https://docs.ansible.com/ansible/latest/collections/communit... [2] https://www.ansible.com/blog/six-ways-ansible-makes-docker-c...
[1] https://jinja.palletsprojects.com/en/3.0.x/
[2] https://docs.ansible.com/ansible/2.9/modules/template_module...
> Do the services need to be modified to speak with consul directly?.
I am not sure that i get you, but had no such need when migrating legacy stuff.
[0] https://docs.ansible.com/ansible/latest/collections/communit... [1] https://learn.hashicorp.com/tutorials/consul/get-started-ser... [2] https://github.com/gliderlabs/registrator
speaking just for myself, feels like I want a system that runs a mix of serverless (like knative), normal containers, raw linux binaries (easy w/ fat binary systems like golang), language-specific packages like heroku does, maybe firecracker VMs.
hard to get knative right if you don't have a reverse proxy tightly integrated with your tool
A natively distributed runtime providing similar capabilities to Docker + Nomad but running on "bare" WASM, providing a compilation target, distributed and clustering primitives right in your application code.
Imagine the nodejs cluster module, targeting nomad directly, or talking to an instance of a service running on another node, the communication and addressing would be handled by the runtime.
Similarly, cluster or instance level ports could be opened directly from your code and the distributed runtime will do the external/internal load balancing. Maybe taking it one step further and advertising services directly abstracting ports away.
Service mesh + Scheduling + Capability-based Runtime in one binary you could run on Linux. One can dream.
Nomad and kubernetes were released within months of each other in 2015. This isn't new.
I'm totally convinced by this product's marketing page!!
Install:
$ brew tap hashicorp/tap
$ brew install hashicorp/tap/nomad
Run:
$ nomad agent -dev
Start a job:
$ nomad job run example.nomad
There's a saying in Dutch: "Wij van Wc-eend adviseren Wc-eend"[1]. It basically boils down to pretending to give advice or information but you're just promoting your own interests.
On the other hand, the Rust project has purposefully avoided "Rust vs X" comparisons on its website. I can't find the HN comments to back this up, but people like steveklabnik have indicated that they don't find that sort of adversarial comparison to be useful. Rust has done an excellent job at community building, so I give a lot of credence to their approach.
There is a lot of nuance.
But actually, they're not competitors. Hashicorp supports K8s clusters for their customers. Nomad is just a product they built that is an alternative to K8s, and plugs into their whole product suite. Someone has probably asked them many times, "Why should I use Nomad instead of K8s?" So they have a page about it.