Hacker News Reader

Comment by KronisLV | Hacker News Reader

> it was using Docker Swarm, which fell out of favour (Kubernetes won after all)

As someone who is still running all of their homelab stuff (well, almost) on Docker Swarm, i dread the day when the eventual switch to Kubernetes will be inevitable.

Maybe Nomad will save me.

Or something like K3s or even Rancher - even though the former has some ingress weirdness with Traefik and default wildcard SSL/TLS certificates since not everyone uses Let's Encrypt or wants DNS challenge for private stuff and the latter needs a lot of resources to even run.

Still, when things go wrong with Docker Swarm, they're generally pretty easy to debug and solve - it's like adding just a little bit more on top of Docker Compose. Worst issues i've had were related to networking and kernel updates breaking something, though purging the node and carrying over all of the bind mount data as well as re-deploying the stacks solved that.

View on HN

Shit like this is why I never warmed up to Docker. The benefits for a single-man semi-hobbyist working on simple sites, often static, are heavily offset by the busywork these tools constantly generate. They move almost as fast as the JS world, add more and more things to track and update, and risk becoming another point of failure.

> The benefits for a single-man semi-hobbyist working on simple sites, often static, are heavily offset by the busywork these tools constantly generate. They move almost as fast as the JS world, add more and more things to track and update, and risk becoming another point of failure.

In regards to something like Kubernetes, i largely agree.

However, there is (currently) certainly some merit to just using Docker, Docker Compose or even Docker Swarm, all of which are relatively boring and stable.

Those essentially solved the problems of environments, configuration, as well as backups for me. Most *nix distros have humiliatingly failed at having a single directory for everything an application needs to run, which horribly complicates figuring out where your data actually lives and how you could re-create everything from 0 in case of a failure or migration. The same goes for different system packages and even sometimes pinning versions. The same goes for networking being weird and is especially applicable in situations where you would want different versions of the same package running on the same server (e.g. MySQL 5.7 for one project, MySQL 8 for another).

A lot of that was already solved by VMs and tools like Vagrant, but it was a bit heavy and never caught on - Docker essentially solved that problem by allowing you to reason about your applications like applications on a phone - just a package of stuff that you care about, adding some resiliency and load balancing capabilities on top.

And then things like Kubernetes came along and addressed more enterprise concerns that are applicable at scale and inadvertently pulled along a bunch of people for the ride, when they have neither the need nor the capacity to work on such complicated software.

I actually talked more about my experiences with this in my blog post, "My journey from ad hoc chaos to order (a tale of legacy code, services and containers)": https://blog.kronis.dev/articles/my-journey-from-ad-hoc-chao...

That said, it's nice if you don't need to solve the problems that containers do. Alternatively, if you have other ways of solving them (e.g. the ansible + systemd setup), then go ahead and use that until your needs expand!

> Those essentially solved the problems of environments, configuration, as well as backups for me. Most *nix distros have humiliatingly failed at having a single directory for everything an application needs to run, which horribly complicates figuring out where your data actually lives and how you could re-create everything from 0 in case of a failure or migration.

The Filesystem Hierarchy Standard actively militates against a single-directory setup. Instead it is designed so you can keep your static stuff on one read-only location, your interim application runtime stuff on another (/var), your user data on a third (/home)...

It's a high-end standard appropriate for serious computers where these directories can be on different disks, with different filesystems, security protections, performance implications, and backup strategies (no need to back up what you can simply reinstall!). As a small-time user it's only your choice because it's the standard choice.

I guess you can have /opt/xyz?

> As a small-time user it's only your choice because it's the standard choice.

Thanks to Docker i can mostly reject its conventions (for actually organizing my application data).

Instead, i might have something like the following file paths and container bind mounts:

  PATH ON FS --> PATH IN CONTAINER
  /docker/app_foo/data/container_x/var/www/html --> /var/www/html
  /docker/app_foo/data/container_y/var/lib/postgresql/data --> /var/lib/postgresql/data
  /docker/app_bar/data/container_w/app/data --> /app/data
  /docker/app_bar/data/container_z/var/lib/mysql --> /var/lib/mysql
  *the docker/app_X directories can contain deployment descriptors, instructions or other notes if need be

The containers can use FHS internally and not know about where or how things will actually be stored outside of the container (bind mount vs volume, local vs networked). Thankfully, usually there are 1 or 2 directories per container that i actually care about and want to persist, the rest is disposable (e.g. data vs runtime files and libs).

The hosts can have a non-standard folder at FS root under which all of the persistent data will live, thus backups become easy, and due to organizing stacks/apps in a tree structure, everything that's needed by a particular environment is also immediately obvious.

Furthermore, this doesn't get in the way of the actual OS operating, nor does the OS get in the way of my deployments - if anything ever goes wrong, i can just wipe and reinstall, setup with Ansible and carry over the /docker directory as necessary. Of course, depending on how everything is mounted, it might as well be just reconnecting another drive or a few drives and setting up fstab.

I think that treating your entire system as one large entity is a horrible idea - instead you should have a clear distinction between what's a part of "infrastructure software" and what's your "business software", those two should be kept as separate as possible. The former should be easy to wipe and update and even swap out altogether (within reason), the latter should be treated as critical and backed up thoroughly and often.

Edit: my point is that most existing software is really bad at this distinction and there are far too many opinions and as a consequence your Tomcat install is strewn across a dozen folders, same with many other pieces of software. So why should you live in fear of forgetting some important configuration file or some data directory?

Of course, YMMV, but things like Ansible and Docker (or Podman, or containerd) with the "infrastructure as code" and "cattle instead of pets" approaches is pretty serviceable.

We're using Swarm in production for a bunch of our small CMS projects and it's been great. "A little bit on top of Docker Compose" is a great way to describe it. It's just so easy to set up, and I think it's still a reasonable option when you want to dockerize stuff you've had running on a VPS.

Hey ! I'm the creator of SSLPing... yup, I'm the guy who got famous when he killed his dream.

Switch to K3S, you won't regret it. I was hesitant to pour more work into my project to switch to K8S (which I know well, it's part of my day job !)... K3S is an excellent way to get into Kubernetes. Or use Digital Ocean's or other hosted implementations...

Kubernetes has a lot to offer... peace of mind is one of them !

> Switch to K3S, you won't regret it.

K3s is pretty great, at least as long as you keep on the happy path.

In practice, that probably means a DEB based distro, quite possibly Ubuntu LTS and something like Rancher for management (if you prefer a UI of some sort and can afford the spare RAM), or Portainer, or even just Lens.

Sadly, i had a situation where i needed Kubernetes on a server at work, which had about 8 GB of RAM, so OpenShift was out of the question and most other "full" K8s distros also weren't viable, when that same server was also to run the actual containers. Moreso, i was stuck with Oracle Linux, on which getting K3s working properly was a bit problematic (though the same probably applies to most RPM distros).

Not only that, but instead of Let's Encrypt, i needed to use custom SSL/TLS certificates, installing which with the Traefik ingress and using them as the defaults instead of the self-signed ones was quite the mess, about which i wrote in another comment here on HN: https://news.ycombinator.com/item?id=30672765

In short, to get it working i needed:

  - a ConfigMap for Traefik, knowledge about the structure of the ConfigMap (tls.stores.default.defaultCertificate)
  - a TLSSecret for storing the actual certificate/key
  - a TLSStore (which i also needed to actually use the secret, spec.defaultCertificate.secretName)
  - a HelmChartConfig for Traefik to load the ConfigMap with the mounted secrets and config

none of which were documented as well as i'd like, because Traefik isn't necessarily aware of the intricacies of being used with K3s and K3s hasn't got instructions on such a setup because the happy path is using Let's Encrypt. Furthermore, attempting to use Nginx as the ingress instead failed and trying to uninstall all of Traefik resources hanged, something about Kubernetes waiting for the resources to do something so that they may be deleted, a callback that never seemed to happen.

Oh, attempting to use Rancher also failed due to recent changes with cgroups v2 and varying support for all of that, whereas it seemed to work properly on a throwaway Ubuntu LTS VM/VPS.

In short:

  - i still think that the goals of the K3s project are really nice, the current resource usage is surprisingly decent
  - if you have to use Kubernetes on RPM, go with OpenShift, if you don't have the resources for it, just use DEB distros
  - the struggle with niche setups, like i needed, is largely not worth your time, the documentation for these still isn't there yet (example of mature documentation: Apache, which has been around for decades)
  - ideally, just pay someone else to run a cluster for you, if you can (i can't, because mostly on-prem at work and am relatively poor in regards to personal homelab)

I'm still torn about migrating over to Nomad or K3s when i'll eventually retire my Docker Swarm clusters (maybe in the next 5 years), but for now i'm putting those plans on the backburner entirely to give both projects a few years to mature and become more stable/established. In regards to Rancher, they have RKE2 in the works as well.

I've been using k3s on Ubuntu (for development) for a while, and I switched it to ingress-nginx and cert-manager... It was always straightforward.

But I must admit it's easy/easier when someone else is managing K8S for you, be it DigitalOcean or Tencent (which I have experience with) or AWS / Google Cloud.

K3S is quite comparable to Docker Swarm though. I liked Swarm when I developed SSLPing because of its simplicity compared to K8S, but once you learn your way through K8S, there's no point sticking to Swarm I think...

I know you say you will let the projects mature, but I think it's a pretty good time to start picking up Kubernetes or at least getting your hands dirty with it. Nomad is a much simpler solution but not nearly as popular, in-depth or in demand as Kubernetes.

If you wanted an 'in' to Kubernetes and better automation of your homelab. I would suggest to check out this repo[1], it's nearly a A-Z guide on getting k3s running on Ubuntu (I am no distro-snob, Ubuntu just works well for most home workloads in Kubernetes). GitOps tools like Flux and Argo are really becoming popular to use with Kubernetes because all your configuration is stored in Git and the GitOps tools deploy manifests based on the Git repo state. My entire home cluster[2] is open source and there's many who are also doing something similar[3].

All of this comes at a cost of learning these tools which is not easy, but from my interactions with people who do take the plunge from docker-compose or docker-swarm most of them stay and once they see the benefits of GitOps and Kubernetes cannot go back to their previous ways. Automation is king and it is much easier using Kubernetes because most of the tools exist for it already.

1. https://github.com/k8s-at-home/template-cluster-k3s

2. https://github.com/onedr0p/home-ops

3. https://github.com/k8s-at-home/awesome-home-kubernetes

Reply on news.ycombinator.com