Swarmlet: A self-hosted, open-source Platform as a Service
github.com
github.com
With Portainer and Swarm I've been forced to manually intervene with rollouts more times than I would have liked due to Swarm-specific errors and other quirky networking behaviour. At least it's simpler than Kubernetes!
Kubernetes solves a similar class of problems to Swarm, but in a much more complex way. Sometimes that complexity helps solve problems. For many organizations, though, Swarm would have been the better option.
Both Swarm and Kubernetes have their purposes. I'm sad that no option has popped up to replace Swarm in the "simple and easy container orchestration" space. Now it's either Kubernetes or... ECS, I guess.
I've had a hard time finding much via hn.algolia.com to back this up, other than some discussions in mid-2018, though.
I tried setting up nomad but couldn't figure out service discovery, external ingress / letsencrypt in the time I allotted myself...
Caddy for reverse proxy and tls handling running as a service, so all ingress goes through that, there wasn't too much exciting there. I was porting over a docker-compose workload, so most services have a static port and I just route to the port in the caddyfile (I passed my router as the DNS for the reverse proxy, so its by hostname). I'll get caddy to use nomad for service addresses at some point.
But for photoprism, I've got the service discovery set up to its database, it's kind of awkward since the nomad native discovery goes into env vars or a file through the `template` block [2], but it does work.
Nomad is pickier about setting resource limits, so I had to actually set those to something reasonable.
I've mostly got Docker based services, so this reference [3] has been useful. There's a couple ways to mount the volumes, which is annoying, and there's some gotchas around docker image handling (short shorty is don't use the `latest` tag).
I probably haven't organized my jobs/groups/tasks well (the analog to k8s pods hierarchy), but that's for later.
[0] https://developer.hashicorp.com/nomad/tutorials/get-started/... [1] https://developer.hashicorp.com/nomad/tutorials/enterprise/p... [2] https://www.hashicorp.com/blog/nomad-service-discovery [3] https://developer.hashicorp.com/nomad/docs/drivers/docker
Just so I understand - this means deploying a new service involves building a docker image, deploy it, then manually update the caddyfile (ie manual ingress)?
> There's a couple ways to mount the volumes, which is annoying,
Fwiw this is annoying on multi-node docker swarm too - i even consider proper volume support to be one of the strongest arguments for considering k8s even for somewhat simple setups.
And I'd agree that the volume story isn't great compared to what I know of k8s. There's info out there for setting up NFS volumes, or something like portworx or ceph, but that's going beyond what I want to do for 1-3 pet nodes on a home server, I can deal with volumes staying bound to a node.
[0] https://traefik.io/blog/traefik-proxy-fully-integrates-with-...
Or are you talking about swarm the product (versus docker swarm mode)
If you are using our Nomad or Kubernetes plugins, you can also run the apps on servers other than the one you are building on.
Dokku does support both Kubernetes and Nomad as deployment targets, so it's not strictly single-server (though app builds currently are).
https://pypi.org/project/docker-harbormaster
It's great if you want to run generic utilities at home (though I've used it at work in internal production and it was good), but it doesn't do ingress, so you have to bring your own.
It's basically a fancy/opinionated wrapper over "git pull && docker-compose up", with allowing you to specify all configuration in one file/repo.
Does anyone have suggestions?
- Single-server and non-secure (no mTLS; no ACLs) clusters are super easy to set up and a great way to try out things before committing.
- ...However, enabling especially ACLs but also TLS on running clusters is going to be more hassle than simply setting up a fresh, properly bootstrapped cluster.
- Their minimum and recommended resource requirements are hugely inflated, you can generally get good mileage out of way less
- ...However: Do follow their advice on keeping nodes single-responsibility (ie don't run a consul server and nomad server on the same node; generally keep your servers dedicated and not running jobs)
- Consul and Vault integrations are generally rock solid.
- ...However: Nomad native service discovery is not yet (wrt consistency and template rewrites). Consul Connect may or may not have edge (it's some time since last we seriously tried it).
- learn.hashicorp.com has material for most things you want to do.
- Put all your ACL configuration in Terraform (or whatever else you have for the same purpose)
It's just that people need to start planning more before they dive into k8s, there's a limit to where and when k8s makes sense in your app. And most people could get away with the more traditional setup of Frontend, App and DB Backend for a service oriented architecture and still be able to scale up into k8s when necessary.
But for mature companies K8s can make a ton of sense.
This is a pretty basic setup and with k3s one can be very productive at it. In fact, you will very likely hit quite a lot of k8s bottlenecks at Google scale if you throw everything at a single cluster.
I feel like we would actually be better off if people admit they'd rather manage all the complexity by hand all the tine, than spend a few hours reading the docs on k8s objects properties and lifecycles - that's really all there is to it for basic setups like k3s.