Comment by cyphar | Hacker News Reader

Comment by cyphar | Hacker News Reader

I self-host the following at home. Everything is running under LXD (and I have all of the scripts to set it up here[1]):

  * nginx to reverse-proxy each of the services.
  * NextCloud.
  * Matrix Homeserver (synapse).
  * My website (dumb Flask webapp).
  * Tor (non-exit) relay.
  * Tor onion service for my website.
  * Wireguard VPN (not running in a container, obviously).

All running on an openSUSE Leap box, with ZFS as the filesystem for my drives (simple stripe over 2-way mirrors of 4TB drives).

It also acts as an NFS server for my media center (Kodi -- though I really am not a huge fan of LibreELEC) to pull videos, music, and audiobooks from. Backups are done using restic (and ZFS snapshots to ensure they're atomic) and are pushed to BackBlaze B2.

I used to run an IRC bouncer but Matrix fills that need these days. I might end up running my own Gitea (or gitweb) server one day though -- I don't really like that I host everything on GitHub. I have considered hosting my own email server, but since this is all done from a home ISP connection that probably isn't such a brilliant idea. I just use Mailbox.org.

[1]: https://github.com/cyphar/cyphar.com/tree/master/srv

> * Wireguard VPN (not running in a container, obviously).

I plan to use Wireguard too, so I shouldn't run on containers? Can you elaborate on that?

From the small research I did, I think you need a customized kernel on the host to do that.

I run it on the host.

> It also acts as an NFS server for my media center [...] to pull videos, music, and audiobooks from.

This is a bit tangential, but to clarify, do you mean that you listen to audiobooks on your TV using Kodi? Do you also have a way of syncing them to a more portable device, like your phone?

> This is a bit tangential, but to clarify, do you mean that you listen to audiobooks on your TV using Kodi?

Sometimes, though not very often -- I work from home and so sometimes I'll play an audiobook in my living room and work at the dinner table rather than working from my home office.

> Do you also have a way of syncing them to a more portable device, like your phone?

Unfortunately not in an automated way (luckily I don't buy audiobooks very regularly -- I like to finish one before I get another one). I really wish that VLC on Android supported NFS, but it doesn't AFAIK (I think it requires kernel support).

Why SUSE over another OS? I've used it and like it, though I see more ubuntu, debian, centos among servers. Any particular distinguishing factor/advantage, or just preference?

I've worked for SUSE for quite a few years now, and so I've gotten fairly used to running openSUSE on all my machines (and I do quite like things like the Open Build Service and other openSUSE projects). I'm am a package maintainer for a bunch of openSUSE packages (most of the container-related ones and a few others) -- so I might as well use them myself to make sure they work properly.

I'm curious about why you're using lxd. Is it just that you wanted to try something different from Docker and its rivals? Or is there a reason you think lxd is better for your setup? For a service per container, I figured minimal, immutable containers, rather than containers running full distros, would be better.

The primary reason is that LXD has an indisputably better overall security policy than Docker. They support isolated user namespaces (containers running with different userns mappings), user namespaces are the default, they make use of far more new kernel hardening features than Docker, and so on. If I'm going to self-host something at home and expose it to the internet, I'm simply not going to use Docker.

I used to run Docker containers several years ago, but I found them far more frustrating to manage. --restart policies were fairly hairy to make sure they actually worked properly, the whole "link" system in Docker is pretty frustrating to use, docker-compose has a laundry-list of problems, and so on. With LXD I have a fairly resilient setup that just requires a few proxy devices to link services together, and boot.autostart always works.

Personally, I also find it much simpler to manage a couple of services as full-distro containers. Having to maintain your own Dockerfiles to work around bugs (and missteps) in the "official library" Docker images also added a bunch of senseless headaches. I just have a few scripts that will auto-set up a new LXD container using my configuration -- so I can throw away and recreate any one of my LXD containers.

[Note: I do actually maintain runc -- which is the runtime underneath Docker -- and I've contributed to Docker a fair bit in the past. So all of the above is a bit more than just uneducated conjecture.]

Have you taken a look at podman / buildah? My understanding is that podman resolves all of the security concerns you highlight above while mostly maintaining compatability with the docker cli and existing docker images. It gets rid of the docker daemon so your containers (and restart policy) can just be managed by your existing service manager.

I only just recently discovered podman and I've been pretty excited. Having never used LXD and only understanding the high level differences between the two, I'm curious how it compares with regards to security and usability.

I'm a little bit too familiar with podman. LXD is more mature and actually implements all of the hardening features I mentioned. podman could implement them in theory, but doesn't. Its default security posture is very similar (though not the same as) Docker. Don't get me wrong, I do want to see podman succeed -- but I don't like the amount of unneeded hype around it. It's effectively a Docker rewrite by Red Hat (and other folks) that has some fairly important improvements, but it's not a revolutionary new concept. As for buildah, I am too biased to respond to that question.

Oh, and most of the Docker CVEs found in recent years -- including those I've found -- have also impacted podman. The most brazen example is that podman was vulnerable to a trivial symlink attack that I fixed in Docker 5 years ago[1,2]. It turns out that both Docker and podman were vulnerable to a more complicated attack, but the fact that podman didn't do any special handling of symlinks is just odd.

[Disclaimer: The above is my personal opinion.]

[1]: https://github.com/containers/libpod/pull/3214 [2]: https://github.com/moby/moby/pull/5720

I have to admit, the biggest selling point to me for podman is the removal of the central docker daemon. For my use case (personal workstation and home lab), it seems strange to me that I need essentially another service manager for these processes just because I want to slap them in a container. It definitely makes sense that there would still be some gaps though as it's a less mature product.

You've definitely convinced me to take a good look at LXC/LXD though. Thanks for the thorough response!

It should be noted that (unlike Docker), LXD can be safely killed and upgraded without your containers dying -- which is the main problem most people have with Docker's container liveliness model (even with Docker's --live-restore there are many issues). The main reason why LXD has a daemon is that it supports lots of management features (such as live migration and clustering) which cannot easily be done without a daemon.

You can use LXC directly if you want to avoid a long-running daemon.

Reply on news.ycombinator.com