It does take a bit of effort to set up ”properly“ - ideally each of vault/consul/nomad server should run on 3 separate instance for HA quorum, for a total of 9 machines apart from the server (“client”) you’re actually running stuff on. For a casual homelab this is obviously severe overkill. You don’t need to. You can run everything perfectly fine as single instance on the same machine.
Personally I went a bit nuts with HA so I have 9 single-board-computers apart from my workload runners. You really don’t need to if OK with downtime in event of server restarts and maintenance, though.
I really want to like it, and for the most part I think it's a great experience, but if you're looking for a lightweight single-host PaaS-type thing, the minimum recommendation and default configuration for 3 servers is a bit wild.
I have Nomad running in single host mode on a DO box for personal stuff, and I get the distinct feeling I'm not really meant to be doing it.
3 instances minimum is a big ask IMO
Your single host mode DO box is most likely perfectly fine. You just won't have the availability benefits during downtime of that instance but, well, duh.
I guess the one practical catch is that if you feel strongly about security and noise neighbor issues, you really should at least separate out Consul Server/Vault Server/Nomad Server into separate machines.
Just whatever you do, make sure to run an odd number of servers (1 is better than 2, 3 is better than 4), or you may run into split-brain grief. Clients are fine to scale as you see fit as they don't partake in raft consensus. So you can have a single instance "client/server", but if you add just a single new instance to schedule jobs on, make it's a pure client.
I have a dozen Nomad clients and about 20 jobs at home. My active Nomad leader is yet to use over 300 MiB RAM, 10% CPU or 0.2 load on RPi-equivalents, even during leader rotation and restarts. It's only when you start to have very high throughput of evaluations and allocations and start to put pressure on raft that you may need to increase your raft_multiplier to make it more relaxed with timeouts.
Tl;dr take the minimum configuration advice with a heap of salt.
https://www.friendlyarm.com/index.php?route=product/product&...
And dokku is better if you want a PaaS.
There are probably dokku/heroku/deis-like things that run on kubernetes, but so far I haven't found 'it'.
Of course you can just run dokku on any k8s distribution. You can even use k8s as a backend for dokku instead of docker - https://github.com/dokku/dokku-scheduler-kubernetes
Let’s Encrypt integration not working properly without dokku-event-listener, which must be built & installed from source and is basically undocumented.
At the time I used it, blue/green deployments were not available. (I see this is now supported, great to see that.)
To be fair, I didn’t report any of these issues, so I may have done something wrong or there may have been solutions that I missed. This was also 6+ months ago.
I also think that with all the major cloud providers now supporting Kubernetes as a first-class primitive, and with there being an abundance of CI/CD tools that can trivially deploy to Kubernetes, the PaaS concept is no longer as attractive as it once was.
Interesting, would have been good to have these reported. Aside, unlocking an app should just be an `apps:unlock` call.
> Let’s Encrypt integration not working properly without dokku-event-listener, which must be built & installed from source and is basically undocumented.
That shouldn't be the case - the letsencrypt plugin was available long before the dokku-event-listener project was, and the latter has been built/released automatically since April[1] (though didn't have documentation on it's internals till recently, as that wasn't really necessary for end users as there isn't any configuration).
> At the time I used it, blue/green deployments were not available. (I see this is now supported, great to see that.)
Not sure what youre referencing here. We don't have anything specific for blue/green deployments, but we have had zero-downtime checks for a while. If you can point out to me where you found this (or what you were expecting) that would be awesome :)
> To be fair, I didn’t report any of these issues, so I may have done something wrong or there may have been solutions that I missed. This was also 6+ months ago.
Reporting issues is always appreciated as its almost certain there were bugs on our end, but it seems you've found your solution :)
> I also think that with all the major cloud providers now supporting Kubernetes as a first-class primitive, and with there being an abundance of CI/CD tools that can trivially deploy to Kubernetes, the PaaS concept is no longer as attractive as it once was.
I don't think this is quite true. Kubernetes is still a large system that doesn't come "batteries included", and every setup is at least subtly different from the next. I think a large number of teams spend a disproportionate amount of time configuring their preferred toolchain - and then maintaining it - once you get to a certain scale. Additionally, a kubernetes manifest is much lower-level than is necessary for the general app developer who just wants to get things done.
I'm personally hoping that Kubernetes distributions begin to mature more (Tanzu and Openshift are examples, but not the only ones) so that folks start worrying less about operating and maintaining clusters and more on either the underlying infra or the end developer experience.
If you want to go ultra-lean, and only need a single node RPi or VPS (many applications don't need massive scale-out) then take a look at what we've been building with "faasd" and containerd - https://github.com/openfaas/faasd
I’d also suggest Podman as a Docker alternative, especially for these scenarios. Images and Dockerfiles (and many commands) are 1-to-1 compatible but there are some nice things especially with this scenario.
I wouldn’t say NixOS is controversial, at all, but it’s a very steep learning curve.
https://docs.docker.com/config/containers/start-containers-a...
So if you have proper Docker installation, containers will restart on reboot.
Also, kubernetes load balancer is pretty easy to setup and plays well with multiple services and letsencrypt. On docker compose, I'll have to either use nginx and update it everytime I add more apps or use something like traefik if I want a load balancer that can read configuration from app's label. On kubernetes you simply specify several lines of ingress definition in your yaml file, which is not too complex for a typical app.
I like the fact that I can run the same stack on my laptop and the cloud
It's vastly simpler than Kubernetes if you just want to run some software on a cluster of machines (as opposed to single-node Dokku).