Ask HN: Which VM orchestrator do you use?
Looking into hashicorp nomad but I wonder if there is a good non commercial alternative?
The firecracker driver is a community one, however, and the offical Qemu driver is most likely more polished. Did not try that one since it did not match my needs.
Has anyone had success with the FC driver here?
Edit: typos
[1] https://www.nomadproject.io/plugins/drivers/community/firecr...
[2] https://github.com/cneira/firecracker-task-driver/issues/14
With Xen-Orchestra you manage one or more XCP-Ng Servers, combine them in pools, live migrate between hosts, patch your hosts and schedule VM backups to S3. It's a killer combination.
I feel nomad is a bit resource intensive for my use case,
Resource intensive? I run Nomad on a bunch of Raspberry PIs at home, not sure what orchestrator is more lightweight than Nomad honestly.
For raspberry pis - I would go with the following, https://k0sproject.io https://k3s.io
1. K3s may not be too heavy, but basically anything that runs on it is. For all but the most basic jobs, I was still looking at one, or maybe two, apps per SBC. Couple with 10/100 networking on many of these, and there's a lot of extra time/latency spent just on chattering across the network.
2. It's relatively rare that you're using _just_ the SBC - more likely is that you have some peripherals, or some local storage. That limits where your pods can schedule, but because of #1, you still end up with dedicated nodes for dedicated functions.
I ended up pulling the plug on that project early, and went back to scheduling workloads manually. For instance, I have Transmission and Squid running on the Pi with the big flash drive. InfluxDB and Grafana live side by side, and also live next to the SDR (I take in some metrics by radio). It's all still containerized, just with manual `docker run`'s and versioned config files, instead of with Kubernetes.
I've ended up writing basically this:
- Groovy for a lot of syntactic power, decent performance, ability to make workflow-ish scripts
- GPars for orchestration primitives to do stuff either serially or in parallel.
- a configuration service to have configuration scoped at the global/environment/cluster/datacenter/rack/individual node levels
- an access layer for executing remote commands and getting stdout/stderr back, and for sending, receiving files. It can be SSH, kubectl, AWS SSM, dockerrun, etc.
I've had to rewrite the core framework four times, but it has enabled me to do complicated orchestrations such as spawn clustered databases, spawn fleets of load generators, data loaders, or api servers, tear them down, etc.
I can code distributed backups and restores, and other cluster-wide "applications" with this as the basis.
I can run things from the IDE. I can run locally or from remote servers. I can CRON them or whatever. I can debug them from IDE. I can do adhoc/arbitrary cluster-wide operations/commands using the Groovy Console.
I will probably OSS it at some point, but it's not a finsihed product. But after three years I can't imagine doing what I do with it anywhere else.
It could be called by workflow engines, it can call out to terraform and other tools. It can do multicloud (with enough code :-), it can do mixed access / deployments (with enough code :-)
I guess the other limitation is the basic problem with a single orchestration node: probably won't easily scale to the 1000s of VM/nodes easily, or as efficiently as some other distributed coordinator could, but those things all come with the one-orchestration/integration/workflow-to-rule-them-all products. I don't want to be that.
However, for VM orchestration. OpenStack has been my choice for long time, but about to check KubeVirt and (old and underappreciated?) Apache CloudStack.
[0] https://www.nomadproject.io/docs/install/production/requirem...
I remember using Nomad back when it was still 0.x, and I don't remember ever reading about this feature. Awesome addition btw!
https://github.com/cneira/nomad
But if you want to implement something like zones driver for smartos forking/updating this would be your best option:
https://github.com/cneira/nomad-zones-driver https://github.com/Smithx10/nomad-driver-triton
My personal favorite is using the Firecracker driver for running Linux microvms.
In my case we use Ansible and Puppet to orchestrate VMs. Puppet for the stuff all VMs need, and Ansible for deploying custom software. Given that we at the most have 8 VMs that are completely identical, Nomad makes very little sense.
Here comes a long one, but tl;dr: make sure you know your goal before you select your tools.
Orchestration is a somewhat broad concept and can be interpreted at many levels. From a management perspective it is about what processes are acting to the benefit of the organisation to fulfil a business need, regardless of the amount of automation. Having a room full of humans vs. a couple of binaries and event logic doesn't matter if the orchestrated result is the same.
Now, when it comes to your extra information you seem to be using microservices, but deploying them as individual virtual machines instead of just containers. While I applaud the choice of better isolation, virtual machines isn't the only way to do this. Firecracker does this too but much more efficient with the same protections (https://firecracker-microvm.github.io). So before going all-in on virtual machines, think about your goal and check to see if this is really the best way to do it. General business applications really don't have this level of isolation as the main factor in safeguarding applications as there are many other more fragile layers where the actual problems occur. A well-isolated application with no AAA on its interfaces might as well not be isolated at all from a security perspective.
The selection of tools is probably going to be influenced by what else you are going to do. If you have automated infrastructure you'll probably want to tie in to that. Some automations have to trigger at deploy-time to (de-)provision at the right moment, while others have to simply react to the fact that something is needed and autonomously scale with the requirements.
If your (micro)service needs a database, it probably also needs configuration to access it, and the database needs to be configured as well. Perhaps there are some networking components, and storage if you have object or block persistence needs.
If you are running on AWS for example, your 'orchestrator' is EC2, but you still need to 'orchestrate the orchestrator' to start the correct configuration with all its dependencies prepared ahead of time. You'll probably want to prepare the virtual machine image so it actually contains everything it needs during build-time so it can just do the running at run-time. Packer comes to mind for such a job.
Most systems that are 'good' got there because they had a lot of usage, because that is what generates data (feedback) on how well it does the job it needs to do. With that larger amount of usage comes a higher load from stakeholders and with that there tends to be a growth of commercial incentive turning a lot of products into some collection of commercial side-options or main-options.
To get a bit closer to a possible answer, I'll sketch out two scenarios, a managed one and an unmanaged one (from a Layer 1 and 2 perspective, hardware).
Scenario 1: Mostly managed primitives.
Depending on what you already have you are likely to have an API that interfaces with the storage, networking and compute resources that are available to you. Depending on the systems, Terraform and Ansible can configure and prepare almost anything to your liking and make sure it stays in the desired state.
When deploying an application as a virtual machine image, you'll probably have the following events happen:
1. SCM change triggers application build (Git and some CI solution)
2. Successful build triggers next integration step (CI)
3. Integration takes your build output and wraps it in a virtual machine image (Packer)
4. Successful integration triggers deployment step (CI)
5. Deployment step contacts your API of choice to let it know about the new image (Salt/Ansible/Terraform/FaaS/Webhook)
6. API registers the new image and you can start rolling over instances until all the old ones are gone
Steps in there are essentially things you have to do before you can start thinking about scenario 1 which is what follows after this.
1. Make your hardware boot from the network
2. Setup a network boot server
3. Prepare self-starting hypervisor images of your choice (KVM-based, Xen-based, Hyper-V-Base, vmWare-based)
4. Make sure your DHCP and/or TFTP servers have logs you can access/parse
5. Once a server boots, it loads a self-starting image and depending on how you configured the images they will start out 'empty' with a well-known key or other credential
6. Use something like Ansible or SaltStack to discover any servers and select the 'empty'/'fresh' ones
7. Depending on your selected hypervisor, you'll now have to connect to a management API, control plane or simply write out a list of nodes somewhere (useful for small-scale SaltStack setups!)
Next, you'll need storage which is much too broad to cover here, but it generally ends up with some sort of redundant NFS, iSCSI or more modern setup like Ceph. Those are all configurable via SaltStack or Ansible.
You'll want networking after that, something like a vswitch with VLANs maybe. A very simple way to do this is start an OpenWRT or OPNSense VM with a bunch of pre-loaded virtual interfaces, subnets, and firewall rules. How this exactly works depends very much on what hypervisor and control plane you have. VMWare with NSX is very different from Proxmox with openvswitch (but they do the same thing).
When you now want to 'deploy' a VM, you use your Ansible/Salt/Terraform-enabled infrastructure where you can 'update' the base configuration of a VM-based application and let Ansible/Salt/Terraform create new VMs and destroy the old ones. Configurations might be flat files written out to the network-booted compute nodes, or some configuration database in a controlplane behind an API.
Now, all of this can be partially homelab-automated with things like Proxmox, Xen XCP-ng, VMWare etc. But as soon as you actually need more facilities like actual API-configued load balancers, databases, metrics, discovery you'll start home-cooking a complete OpenStack replacement. At that point, just get OpenStack or realise that the benefit of scale you get with Azure/GCP/AWS lets you do what makes you money instead of doing busywork that doesn't actually matter in the end in many cases.
- Find out what APIs/Configuration stores are available
- Put long-lived deployment-static things in Terraform/Salt
- Put show-lived deployment-bound things in CI (and then Ansible/Salt inside there optionally)
For me, if I were to run a small VMWare set for example, I would select Terraform as my orchestrator of choice. Then in my CI (GitLab-CI) I would make a step that once Packer has built a new VM image, I would update a file listing my to-be-deployed images and automatically commit it to a state repository. I would then also have Atlantis running which notices the change and runs terraform for me. It would add/remove all the resources needed for my VM to work with the correct network and storage. I would leave discovery and joining/leaving load balancers etc. to be the job of the VM image, perhaps via Cloud-init or Consul or something like that.
If I were to run a bit of a bigger setup, I might run Proxmox and Ceph, combine Kubevirt with that and run ArgoCD instead. Alternatively I might check out https://harvesterhci.io (via https://medium.com/@rmin.aminian/deploy-harvester-in-proxmox...).
The problem with applications-as-virtual-machines is that those are surrounded by tools and ideas that they live reasonably long lives and consume quite some resources. You won't really see someone run 1000 VMs with 0.15 CPU and 64MB RAM as a microservice 'instance' but you'd see that happen a billion times with containers. Managing the lifecycle of a virtual machine means you have to manage its storage and networking too, and responding to outages of any of the primitives (net, cpu, storage) is much more cumbersome than a container since the container doesn't need to know about any of that, it just receives it from the host.
A secondary problem is that whenever you end up in orchestration land, people tend to think it's complex tooling, or that containers are complex or that Kubernetes is complex. To a degree that is true, but when you start reinventing those wheels you just end up implementing 50% of those existing solutions with 0% documentation and nobody in the world will be able to help you because it contains so much nonstandard stuff. A lot of CI/CD and management processes are just 'state machines' or control loops. This is also what most services in Kubernetes are, and most services in public clouds. They just respond to the state of the world vs. the desired state. Whenever you're not using what already exists, you're essentially just re-implementing those exact same control loops/state machines in a less-robust and untested way.
https://registry.terraform.io/providers/hetznercloud/hcloud/...
The benefit of this is that you can put all the common items in place (network, volumes, keys, firewalls) ahead of time and then just add instances as needed.
Their official provider also has a lot of Datasources meaning you can automatically get all the data you need in other places and then use terraform to write it out wherever you need it. As an example, Terraform can use the hcloud provider to create or read resources and use the consul provider to then write it out to consul. Then, if your application or orchestration tools need some data they can just get it out of consul.
From the looks of it, you might have the following list of actions:
- Build the software from sources (whatever CI you are using)
- Take the built software and put it in an image using Packer (https://www.packer.io/plugins/builders/hetzner-cloud)
- Use terraform to make sure all the non-deployment-specific stuff is ready
- Use something else (Harvester, Kubevirt, Nomad etc.) to make sure the right amount of servers from the correct image are started/stopped
Using a ton of different tools might make it really complicated, but trying to do everything with 'one golden tool' is going to give more problems than benefits in general. One benefit of pulling in Kubernetes (but just for the Kubevirt part) is that there is a Kubernetes Autoscaler for Hetzner which means that you can specify "I want 10 virtual machines at all times, and they have to use this specific image" and Kubernetes and the Hetzner Autoscaler will make sure that there are always 10, if one crashes it gets replaced, and when you update the image it first adds 10 replacements and then removes the 10 old ones (or you can configure a different rollover method of course). The downside is that if you don't have Kubernetes experience yet it will be a very steep learning curve and in such a case I really would not recommend this (yet).
I do recommend keeping principles like GitOps (Git is the source of truth and operations have to flow through Git so you always have a record of change and can go back if needed) and disposable/cattle instances (never make an instance special, always be able to just nuke and replace it; this also means that if a system gets compromised you can just nuke/suspend it and know that you can start a new instance [with patches], and if there is an outage you know you'll be able to recover because that recovery is the same as your normal process).