Comment by boloust | Hacker News Reader

Comment by boloust | Hacker News Reader

Running k3s on a small cluster of mini pcs and RPis.

Use Tailscale for MagicDNS and access from any network.

Have a custom wildcard domain pointing to my tailscale k3s node ips, and a traefik ingress controller. This means exposing a service from my cluster on a subdomain just requires creating an ingress object in k3s, and it's only accessible via tailscale. cert-manager and let's encrypt handle TLS.

All services are deployed via gitops using ArgoCD, so changes are auditable and can be easily rolled back. Replacing hardware is just a matter of installing k3s and joining the cluster, then everything automatically comes up.

Restic for backups to s3.

For home automation I use a USB zigbee controller, mosquitto, zigbee2mqtt, room assistant, and home assistant, all deployed on k3s. These control my lights, HVAC, and various garage doors and gates. Also have mains-powered zigbee switches bound directly to devices so everything still works even if network or home assistant goes down.

The RPis are used for Room Assistant, which can automatically control lights/HVAC based on presence detection via a smartwatch. More intrusive actions (e.g. making lights brighter when already turned on, opening blinds) are pushed to the smartwatch for confirmation.

Grafana/prometheus to monitor sensors.

For media, jellyfin and sonarr/ radarr. The native Jellyfin app works very well on modern LG TVs.

Pihole to block ads on any device connected to Tailscale. Works globally.

Right now it's zero maintenance, and changes are automatically synced after a git push, so I almost never SSH into the servers directly.

Always love seeing someone else create a similar solution as your own (albiet likely better!).

I have the same setup with K3S running on a couple PIs. You have a nice CI but I decided to use cdk8s[1] which lets you compile Typescript into K8 files. For access I did almost exactly the same but with CloudFlare Tunnels (might look into Tailscale). Stealing the zigbee2mqtt and room assistant ideas.

Where do you store volumes? I eventually just bought a NAS and mount persistent NFS volumes off it.

1. https://cdk8s.io/

cdk8s works really nicely with gitops.

> Where do you store volumes?

Back when it was a single node cluster, I just used hostFolder mounts with restic backups. I added Longhorn once the cluster grew, but there's still some local hostFolder mounts left around. For example, zigbee2mqtt needs to be on the node that has the zigbee controller plugged into it, so the node is tagged and zigbee2mqtt has a nodeSelector. This means the hostPath still works and I haven't needed to migrate it to Longhorn.

Longhorn initially scared me off with its relatively high listed resource requirements, but after configuring it to not run on the RPis it turned out to work quite well, most of the time just using a few percent CPU.

Thanks for writing almost exactly the post I was going to write. differences:

I don't use tailscale; I just port forward from my router to the k3s ingress IP, since that's fixed anyway. Accordingly k3s handles letsencrypt certificates. My router has a built in openvpn server.

I haven't moved to jellyfin... yet. Plex is super slick and runs nicely in the cluster. I've learned to keep it version locked though, to avoid regressions and unwanted new "features", which means jellyfin is only a matter of time.

I also run Nextcloud, and photoprism for my photo library.

Storage is on a built-from-scraps 16TB NAS which backs up to azure blob with duplicity, and longhorn for block-based storage (since lots of services nowadays prefer sqlite, which breaks on NFS). Yes I do need that space; I run an entertainment company and we store a LOT of video and audio. Not to mention media for plex!

I have many times considered moving most of this to a cloud system, but the cost is prohibitive. If anyone can find 13+TB of storage and transcode- and ML-capable hardware (for plex and photoprism face recognition) for less than $45/mo (my cost of electricity plus annual amortized hw cost), I'm interested.

This was basically my holiday project - Photo: https://i.imgur.com/2AnP4pu.jpeg.

I'm running Longhorn for storage but haven't figured out backup yet, and haven't got to grafana / prometheus yet.

I put up my work on GitHub as well: https://github.com/inssein/mainframe. I wanted to create a separate ingress controller for internal dashboards, but for now I just setup a separate nginx-ingress for internal, and using traefik for external, feels wrong.

> The native Jellyfin app works very well on modern LG TVs.

My experience on a CX OLED has been hit-and-miss. Freezes, crashes, and some times it just hangs when skipping and I have to force-close it.

I really want to like Jellyfin. I run it beside Plex and use both. I find Plex user-hostile but it still gives me a better video playing experience more consistently.

Interesting, I have a CX and a G1 and it works flawlessly. I didn't do anything special so might have just gotten lucky.

> changes are automatically synced after a git push, so I almost never SSH into the servers directly.

Can you elaborate how you're doing this?

I'll answer in his place, he said he's using ArgoCD and running everything on k3s. ArgoCD watches the files in a repo (kubernetes yaml manifests for example) and applies them in the cluster, so that the state of the running cluster (applications) is synchronized with the git repo.

Reply on news.ycombinator.com