Docker/containers used to not be hardened enough. Are they now?
Virtualization/VMs used to be the answer but it adds both performance and management overhead. Is there a good system here?
Or something else entirely? Like old school separate users.
The overhead on something like an RPi would be ridiculous, but on modern x86 hardware with an IOMMU (VT-d in Intel speak, AMD-Vi for AMD), the overhead of passing through HW is, for homelab purposes, essentially 0. A lot more expensive, but the organization and extensibility is well worth it.
I have anything that I expose directly to the internet on a separate VM from my "internal" services. If I were super paranoid, I'd expose them to separate VLANs, and then use my FW to control network traffic. The Intel 82599 can enforce different vlans on different VFs with SR-IOV.
I have a VM that runs flatcar for docker for things that are too hard to set up otherwise, but I vastly prefer NixOS for most things.
This is exactly what I did initially, but it was indeed a bit of a pain to manage. Eventually I went with something in between, by first compartmentalizing services and then putting them in separate VMs with separate VLANs:
0. Router / FW.
1. WireGuard / reverse proxy.
2. Personal, e.g. file storage, backups.
3. Hosting. My personal site is reverse proxied through Cloudflare and only their IP ranges are whitelisted.
4. Compute, i.e. stuff I want to compile / develop / run on my server. Handy if I want to run a heavy simulation overnight or need more disk space / RAM / CPU power than my M1 MB Air has available.
5. Services. This runs many small tools / services that don't need access to my RAID pool or anything like that. If this gets infected I wouldn't really care.
6. VPN. This VM can only access the internet through a VPN. Doesn't have anything installed ATM, but has been used in the past for urlwatch and torrenting.
7. Test. This is where I try out new software before actually installing it on the correct VM. Once I've concluded testing I rollback this VM to a clean install.
It takes a weekend to install Proxmox and set up the VMs / VLANs, but after that it easy to use.
and the best reason to use SR-IOV with networking is you completely avoid the awfulness that is the Linux bridging/firewalling stack
You can still use VMs, and some use that as an additional layer of isolation because they're virtualizing anyways (performance overhead is really negligible).
I've been self-hosting on my home server for at least 5 years now, and I think I've only seen two or three vulnerabilities across all the services I know about, none of which were ever really exploitable.
It’s your home environment. You want it to be easy. You want to use the tools you run not maintain them. If you want to learn k8 for professional growth, learn it separately from a home server.
Your home server can be more pet than cattle.
I've been moving workloads to an old gaming rig running NixOS with varying levels of isolation (some containers, but really just good user/group/permissions management), and it runs super well.
Of course, you could do the same with just Docker Compose and no Swarm, and I think you'd still be better off than using Swarm.
The main reasons swarm is better than other options for clustering IMO is networking. They can be set up to share the ports on all devices and map it back to the correct container on whatever host it’s on, so you can disconnect the target IP:Port from the container.
There’s only one and it changes manually as I need features to change. I download and install things as needed, from gui, with no version control or script to manage it. It’s a pet.
Yes absolutely. I can afford a new one, and I would immediately buy a new one (well I’m already waiting for the newly released one but still). I would still be quite upset and my life would be interrupted at least a little.
I took the pet/cattle analogy to be about how manual the setup is, and how replaceable it is. I think apple has smartly blurred that line with great backup tech, but I would still consider the “lovingly” hand customized aspect of maintaining a phone solidly a pet. Some version of my current phone has been around for ~10 years through various hardware iterations, all restarted from a backup image. I would be distraught if i had to recreate it without a backup, just finding my apps, logging in, finding wallpaper, rearranging icons, setting up shortcuts, etc. Maybe that’s the ideal state for a home server - a nearly no-op backup and restart process that you still manage as you need
As far as disaster is concerned, it's not that difficult to install software that really needs minimal maintenance. But it comes down to what you want out of the software and hardware that you run.
If you want to try out <insert tech here> to learn something, then just learn it, don’t try to fit it in your normal life and eat at your existing stuff. Don’t replace your mac with a chrome book just because you’re learning webdev, and don’t replace your home server with terraform just because you’re learning it. What if you learn it but stop needing it or never use it professionally? You’ll now need to maintain that skill to maintain something at home.
If you want something more than a blank Linux box for your home server, check out HASS.io, synology, QNAP, TrueNAS, or one of the many “hold your hands” distros/tools designed to make it less work. Even Portainer/Proxmox will give you a bit of a GUI without being too opinionated. I use a blank Linux box primarily, but only because I live with other SWEs who all want to mess with the shared server, and everyone wants their own thing and we couldn’t agree on anything else. We plan to switch to TrueNAS and give everyone a VM but haven’t coordinated the switch yet…
But still, for a single or less than 3 machines and/or in a single location I don't see the point.
For standard services, I use Apparmor with the default `apparmor-profiles`, as well as fail2ban with some additional firewall rules.
[1]: https://man.archlinux.org/man/systemd.exec.5
[2]: https://wiki.archlinux.org/title/User:NetSysFire/systemd_san...
Use userns-remap. Run the docker daemon rootless if you want but don’t stress about it. Set up auth to the docker socket. Don’t bother with running the processes in the container as not uid 0, with remap it’s effort for little gain.
Now breaking containment means having a local privesc on your Linux distro or breaking the auth on the docker socket. Like that’s plenty for drive by attackers.
I don’t think they ever will be. At least once a year there is a kernel bug where root in a non-root container/namespace can be elevated to root on the host
The WordPress Sandstorm app is slow enough at rebuilding the static side of our large site that I’ve been meaning to try forking it or building my own though. But Sandstorm itself has been great.
Generally your just serving a single user - you - so even potato grade gear is fine