The web server deployment model breaks at hobby scale
w.on-t.work
w.on-t.work
I'm hosting few hobbyist public websites. One is the library of hymns, songs, ... - mostly PDF, MP3, MIDI.
OpnSense router -> Traefik -> Docker container.
That Docker container runs on NAS, which is connected via 1Gbps port.
Suddenly, last week I started to get over 1000 RPS. Was it scanner, parser, crawler, DDOS? I can't tell, as user agent was default Mac OS.
It was using ~1Gpbs bandwidth. From many IPs. If I would have it hosted at AWS / Azure, it would cost me > $40 / hour for the bandwidth. Unless I use some of their CDN.
My router CPU went high, but it handled it ok. NAS CPU went high, but it handled it ok. Traefik loaded the server heavily: it had to both receive and send 1Gbps. And write access logs for all requests. My Alloy & Loki logging system got loaded heavily.
NAS already have cache. I tried to enable cache on Traefik. It went OOM in 2 minutes: https://github.com/darkweak/souin/issues/844
When I searched, I found that Traefik cache with proper memory handling is available only in Enterprise version.
Solved the issue by putting the site behind Cloudflare proxy and enabling it's magic to protect from the bot.
The best is almost not to do it at all. Just have a docker image serve http/1.1 and document that a reverse proxy is required to expose the service.
There are simply too many ways to do it and every sys admin/hobbyist has their own preferred way.
For the anonymous and authenticated caching issues the author goes into, I think once again it is useless for self hosting. Just embed a performant web server like nginx for example that is pre configured to serve static files. Use proper headers, and predictable/simple paths. Self-hosted versions are going to be low traffic and just need to work.
Have advanced settings for more control, but keep the default simple and working out of the box with minimal dependencies.
You say caddy. The next person will say nginx-proxy-manager. The next will say DNS challenge let's encrypt behind wireguard VPN. The next will say Traefik. The next will say CloudFlare tunnels.
Everyone has their preferred solution and its always the best and simplest.
Do you want a caddy service embedded with every web app you self host or are you running a single one serving all your apps?
From my understanding if you want acme http challenges to just work with let's encrypt you probably need to use a single, separated caddy to front all your apps.
In that case if I ship a caddy service pre configured to do TLS termination with my app it's just more trouble for you no?
My point was not to use http only as a finished solution, but to let the admin front the http-only service with their TLS termination solution of choice.
I self-host Immich with Caddy as TLS terminator, and it's far from obvious.
They all run on a small N150 PC in my closet, the same PC serves as my internet router. Both Immich and Caddy run in podman-compose, and there are firewall rules that allow incoming traffic to Caddy and outgoing traffic from Caddy to update certificates. There's also a tricky setup of Systemd dependencies that make sure podman networking and firewall rules play nicely together and with other system config, like the bridge for the 2.4Gz range internal Wi-Fi and for external 5Gz Wi-Fi 6 card.
If not for the LLM help, I would have spent many days figuring out all the rough edges of this setup.
Which is exactly the point OP is making. Hobbying webhosting is hard nowadays.
I self-host Immich with Caddy, and it's complex, really.
Caddy runs in podman compose next to Immich, there are firewall rules to allow traffic in and out and proper dependencies in custom systemd units in case podman networking goes bust after firewall rules are reloaded and a readme that covers all that.
Exactly the point OP is making.
If you want to put caddy in front of yours? Great it works.
For me, running a much larger setup? Great, it also works.
For users who never expose it beyond an IP address on lan? Great it also works.
No caddy. No cache. No proxy.
A lot of scrapers give up.
Log rate remains low. Actual visits even lower.
Peace of mind, for a while:
Somebody hasn't read the docs and doesn't understand how nginx's "if" works, because it's fantastically useful.
I just wish nginx / caddy / etc had better out-of-the-box support for letsencrypt. It would be way easier to set up if it was just built in to nginx. Let me specify my domain in my nginx config, and just make it work.
AFAU Let's Encrypt gets its money from Google. There are other minor providers of similar services over the ACME protocol, but this does not change the fact that they issue the the majority of the free TLS certificates.
We already had one centralized domain name system managed by the US Government that we have to hop through before publishing our content online for everyone.
Now, we silently and tacitly added another US-based one.
Here's how it could work: First, you generate a tls signing certificate. The signing cert gets stored in DNS. When anyone makes a DNS query, the response contains your root certificate's signature (HMAC). When you initiate a TLS connection, you check the root certificate in the cert chain against the signature provided over DNS.
The dns response would also need to be cryptographically signed by the dns provider. Instead of operating systems shipping trust roots of TLS cert providers, we ship the root certificates of DNS registrars. Eg, .com, .com.au, etc would each publish one or more root certificates that would get distributed with operating systems.
Trust already depends on dns (since that’s how you prove who you are to letsencrypt). So the whole system would rely on fewer trusted entities - which would hopefully make it more secure. It would be simpler to run - less organisations needing to do anything. And simpler to make use of, since you wouldn’t need programs like certbot to keep your tls certs up to date. This would also prevent the danger of malicious TLS providers being involved in MITM attacks on TLS.
That is, if they inject JS or modify the data in any way, they are liable for the content, otherwise not.
Computers are fast now. We should just encrypt everything that gets sent over the open internet.
The IETF put out a statement about this a few years ago. I agree with their stance.
> The IAB urges protocol designers to design for confidential operation by default. We strongly encourage developers to include encryption in their implementations, and to make them encrypted by default. We similarly encourage network and service operators to deploy encryption where it is not yet deployed, and we urge firewall policy administrators to permit encrypted traffic.
https://datatracker.ietf.org/doc/statement-iab-statement-on-...
An art professor I knew used to say about overwrought paintings "too much sugar for a dime". Technical people are often prone to this as well.
The details remind me more of the Bluesky PDS though. It's nodejs-based and the only officially supported way to run it is via a largely self-contained docker container.
I personally don't like apps that ship the entire sink (complex stack) when I already have an opinionated setup that already includes TLS/access control/etc.
Just tell me what the entry point is and what the dependencies are for it so I can decide if I already have enough to just sort it out myself.
By all means, create a kitchen sink container image as well, but don't have that be the ONLY artifact for running it.
I guess that breaks the plug-and-play nature, but you can't expect something internet-facing to be plug-and-play. (Maybe that's the real travesty here.)
This is the way (while also allowing people to use their own proxies etc., but being clear about those not being officially supported).
Ship your back end container, your front end container, tell people how to set up the off the shelf DB containers or whatever else your software needs and if you don’t need 20 different components like Sentry self-hosted versions do, you’ll be fine. The DB can be a container or not. The reverse proxy can be a container or not. Hell, if someone wants to build the back end or front end from source and serve them differently they can, just on their own time. That’s pretty close to the 12 Factor App principles.
Just avoid a lot of complexity when possible: do traditional SSR, or with some progressive enhancements or what I like to do in this day and age (unless maximum accessibility is paramount) only use your back end for the API and have a traditional SPA, none of that complex Next.js stuff. Your front end just becomes a bundle of files that you can put in a web server container to be proxied by whatever the ingress is. At most the Docker entry point would change some values in config.json that the app loads on startup in the browser.
If someone asks for Kubernetes support or whatever, just wish them the best of luck in setting it up themselves!
It’s fine to publish recommendations how to tune configurations for specific usages. But I only need the app, not the various things that you may think I need. And I really want the app to only solve its essential problem. It should not take care of the IT part other than providing configuration samples.
I just call /publish-local-docker subdomain.mydomain.local
And if there isn't a docker compose, it will create one, and then run it, and update caddy server on its own.
We'got all the world's knowledge compressed and queryable with LLMs: there are way less configuration issues that I cannot solve today than used to be the case.
Did those in charge of those standards royally fuck up HTTPS/TLS? Of freakin' course they did. The interests of the big players, not the small ones, have been taken care of. And the result is an ultra complex mess of fucktarded standards.
But it is what is: it's what we have. And we'll make it work.
Anyone serving hobbyists, commercially or on a volunteer basis, needs to accept that there will be some number of hobbyists “below the line” of what you can reasonably support. Those folks will just struggle with what you can provide to them, and you will never be able to address all their complaints, concerns, mistakes, etc.
You just have to mentally cast them free on their own. At most, encourage them to find fellow hobbyists who can help them level up where they need to in order to engage with your offering.
My homelab is also currently ~40 nix service VM's plus a handful of Ubuntu, etc long tail, and some external DO droplets for NAT, and a more robust vlan config than I'd ever been able to put time towards before... which I haven't had to do more than a dozen hours of manual setup on IN TOTAL thanks to Fable, Sol and local Qwen.
There's truth in this post, but I doubt it's longevity.
It's DEFINITIVELY overkill, and fills up fast, but nix making everything near fully declarative to roll fwd/back or shuffle between proxmox hosts at any time makes for a clean and easy mental model. Both for me and the AI, because it can infinitely retry a nix flake config until it gets it perfect.
I could definitely slim things down without disko, virtio, etc, and especially whole separate kernels.. but it's also not such a huge diff that I mind. I'm not adding new services all that fast anymore now that I've got my basics + wishlist set up, so I'll grab another box from the local pc recycler if I ever need space for another ~30 VMs some day rather than when I need space for another ~40-50 LXCs.
And in the meantime if OpenAI publishes a scary brag about how GPT-7 popped a shared kernel on accident I can be relieved that my private bookmarks repo isn't vulnerable while the internet burns down.
> you see that the reverse proxying rules you would need are more complicated than usual. you can’t Simply forward /api to your application and the rest to the frontend due to various reasons outside your control.
If the author is reading: Can you elaborate on what reason you are referring to that are outside my control?
I think at this point it's not a hobby, it's a burden.
An opinionated release of k8s (k3s, microk8s, etc) does wonders for making much of it just disappear.
You pay a slightly higher up front cost in complexity, in return for largely removing absolutely all the listed problems in this blog.
If you're going to run more than just one app... It's the route I'd recommend.
I think the market is already moving here, too. Synology is exposing very easy interfaces to run container workloads on their devices.
I think we're headed towards something like a combo NAS/router that handles storage and networking (by far the hardest parts of hosting).
No other.
Hobby scale you said ? It doesn't have to be that complicated.
I can fiddle with the page content in Developer Tools to make it work, but frankly, WTF, especially since Immich expects people to set up a reverse proxy for HTTPS access on their own and does not help them in any way (like providing a pre-configured caddy in their compose.yaml, I don't know)
You could theoretically upstream the fixed compose.yaml if you felt like it, but the project seemed unusually hostile towards the one fix I tried to upstream.
photos.example.com {
reverse_proxy 127.0.0.1:8080 {
header_up X-Real-IP {remote_host}
}
} photos.example.com {
reverse_proxy 127.0.0.1:8080
}
All because we forced TLS on everyone instead of solving the JS injection problem in some other alternative way. Like... legal, maybe?
Unless you also want to deal with security, backups, GDPR compliance, etc.