Self-Hosting Dozens of Web Applications and Services on a Single Server
cprimozic.net
cprimozic.net
- traefik (nginx proxy with auto letscencrypt)
- portainer (docker container management)
- fail2ban (basic security)
- logwatch (server / security stats by mail)
- munin (server stats)
- restic (cloud backup)
- unattended-upgrades (auto install security updates)
- apticron (weekly info)
- n8n (automatisation for e.g. quick info via telegram, if something not work)
Run every app that you want in your container.
I say this as someone who uses docker and docker-compose extensively and I'm very comfortable with a CLI.
docker system prune
For backups restic is a blessing. And so is Syncthing for getting things to the backup machine quickly (from mobile and non-unix machines).
On the bright side, it's a one-time cost and generally pretty simple, especially if they provide a sample `docker run` command. (Although yes, it would of course be better yet if projects would publish one so that it was easier)
On a side note this is actually a problem, in my experience, if I find something easy I might not cover every edge case - I can cover those if and when they come up - but given enough potential inputs and users the edge cases always come up. And then what happens if I am unavailable at that moment, the edge cases need to be handled by people who will find the easy solution I built up incredibly hard and daunting.
For long running projects it is certainly worth to invest into proper setup. It really is a lot cheaper in the long run.
It sounds more complicated than it is.
I can not remember a single time something bad or unexpected happened. Only the planned things - upgrading the distro every couple of years, and updating major versions of the things running in containers probably once a year or two. And maybe sometimes some unplanned updates if particularly bad vulnerability gets disclosed in a popular software/library. I am pretty sure I don't spend more than a few days per year to manage it.
If I had opted for a cloud vendor managed alternative, it would have been so much more expensive. I have definitely saved thousands or tens of thousands over the last 10 years.
But then again, I know how to manage it and I planned it out so it would not cause too much trouble for me. Prior to this setup I endured many painful moments and that "wasted time" allowed me to think of a better way to manage it and avoid certain problems along the way. Also available tooling has improved a lot.
Then again - this is for my personal projects and I would do it somewhat differently for large projects.
I run about half a dozen web apps on a single node on Hetzner with Docker swarm mode + traefik ingress + whatever the web apps need.
Any app I have is deployed in seconds as a docker stack. I treat my Docker swarm node as cattle, and I have an Ansible script to be used in case of emergencies that deploys everything from scratch. The Ansible script takes, from start to finish, only a couple of minutes to get everything up and running. I can do this with zero downtime as I have an elastic IP I can point at any node at will.
If I wanted, I could optimize everything even further, but it's already quite fast. In fact, I can get a new deployment on my Hetzner setup up and running faster than I can get an EC2 instance available in AWS.
Proponents of big cloud providers as the only viable option typically have absolutely no idea what they are talking about regarding availability, redundancy, and disaster recovery. It's mostly resume-driven development seasoned with a dash of "you don't get fired for picking IBM".
It’s a single server running a few containers with config files. The complexity comes when you outgrow a single machine or need stronger availability guarantees but none of that matters for a single-ish user setup.
After 20 years doing this, my postet example is stupid simple, that works for 80% auf all equirements.
For the rest 20% you need a good admin, or book a service with a good admin.
So easy :')
If you're hosting something commercially, you should get a commercial ISP plan. If you can get it at home, why would the provider not allow you to host your services that way?
That said, why would you do that? It would be very hard to scale this operation, so unless you're planning to be a tiny (and likely lossy) operation forever, get yourself a cheap VPS to start with, then migrate as needed.
This post is about self-hosting services for yourself, and perhaps to a few close relatives and friends. Many of us do that (have a look at r/selfhosted and check out the Self Hosted podcast), and OP's set up is one of the simplest around.
> ensuring that you have a static ip address
There are many ways to access your services without one. A mesh network, like Tailscale, ZeroTier, Nebula, is my favourite, but a regular VPN also works, and so does dynamic DNS.
• Backup is not to some cloud service, which I cannot control but done to a small server at home with a NAS attached in the basement.
• some smaller service during development I run simply from home from an small and old ALIX based on AMD Goede. Only probleme here, I need to upgrade at some point because some SSE2 instructions are not supported, which makes now problems with newer packeges including some self-compiled packages.
I think a web site/service which lets you choose "apps" and spawns a VPS instance would be very useful and profitable (Think "ninite for VPS"). I started to work on this but never had the time to continue. With an Ansible/Chef/Puppet recipe, this should be relatively easy to do.
I wouldn't be surprised if a freelancer would charge 100$ per hour to do this kind of work spanning multiple work days.
The nice thing is that it is mostly what you would do manually anyway and the commands are unlikely to change much/often since they will install latest versions of postfix etc. when you run the script.
I think this might be more doable since the scripts are easy to create and maintain so perhaps just a site with bash scripts for e.g. "Postfix + Sendmail" or "PHP7 + nginx"
FWIW I’d charge way more than $100/hr for this work.
Also see Cloudron. Not cheap but I’ve heard that people are very happy with their service. Basically you self host their “app as a service” platform so to speak.
Kind of like a super polished Sandstorm, but totally different sandboxing technologies (believe Cloudron uses Docker but not sure if they still do—- and I believe Sandstorm used Kenton Varda’s cap n proto technology which probably allowed for even greater sandboxing/protection than Docker, I would have to imagine..)..
(No affiliation)
The HC2 is discontinued now, but the HC4 looks like a nice, current option. https://www.hardkernel.com/shop/odroid-hc4-oled/ Maybe I should get one of these to take to my daughter's place.
The most important, and neglected, part of backup & restore, is the restore. If resources permit, this is where I like to use a battle tested solution instead of rolling my own with scripts. For my self hosted servers I use Veeam, but there are many good alternatives.
It's nice having the option to restore the entire VM to a specific state, and to also be able to mount backups and pull files if only a few need to be restored. It's also handy to be able to spin up a backup into its own virtual machine.
What if your server is already compromised? Doesn't that approach ensure your server remains compromised after you pressed the big undo button?
(And it’s also nice for being able to try things because your repo has a complete full snapshot of your setup that you can diff.)
1). On HOWTO articles about infra (1/2): I'd like to see more articles that lead with requirements, rather than setups that then justify the setup with requirements. Like, congrats, you managed to host a bunch of web applications via containers on a dedicated server. It's really nice for a super personal project and I'm sure it helped OP gain a lot of operational experience across several domains, but I just find this type of article to be "DYI porn" for a highly specific subset of DYI.
2). On HOWTO articles about infra (2/2): Is there any sort of comprehensive infra setup guide out there? Even something that just covers "core" AWS services (CloudFront, EC2, ELB, ECS, Lambda, CloudWatch, CloudTrail, and a few others...) feels like it would be incredibly useful for so many people.
3). "AWS on Rails" - I feel like we're getting close to a time when "AWS on Rails" emerges and we see the birth of a new opinionated tool that says "Most people just want to ship their product, not fiddle with cloud infra. Do it our way and you can focus on product instead of infra"
This would be the AWS certification at their different levels.
To take a personal example, we use multiple infra providers.
Some of our infra is gmail (currently working on automating it with terraform).
Some of it is other infrastructure providers that aren't US-based.
etc.
We're tired of that so we're looking into using a terraform provider [1], so that we can declare new users and instantiate their resources with a simple PR.
[1]: https://registry.terraform.io/providers/hashicorp/googlework...
If I tell them “I have Digital Ocean, I maintain the load balancer myself, the database myself, ELK myself, and you’ll be expert in Postgres, Ansible, Nginx and Debian”, I sound like an old fart.
The future for them in AWS is a click-on-the-button deployment and they’ll master nothing.
It does feel a bit like a Cinderella service however. The versions of the managed databases you can connect are sometimes several years old.
I know I’m being a beggar and beggars can’t be choosers, however, I believe that there are folks here who could solve this problem and make a nice living off of it. Hope someone reads this comment and builds something to address this.
For many scenarios, you can completely ignore IAM, but it's definitely not advisable.
On the VPC side, it's actually fairly straightforward, but you may need to come up to speed a bit on some networking concepts if (like me) that's not your background. Nothing too onerous though, especially if you have some technical background.
There are also some gotchas that allow you to too easily do things like create security groups or other resources outside the correct VPC. If you overlook that, you're in for some brick wall head-banging 'til you figure it out.
Or maybe as a startup, to-C website you don’t really care
EDIT: and this isn't necessarily difficult to grok. A lot of what you'll use from the network side is security groups, and they are straightforward. /EDIT
There are also actually some bad patterns in the AWS Console UI that don't help here. For instance, despite all the warnings they place on S3 buckets about making things public, they still allow you to appear to change subobjects to private. In a traditional hierarchical directory structure, the more granular subobject settings would override, but not so with S3. If you didn't know that, then you've just shot yourself in the foot.
But, I totally agree that the underlying nuance is a lot to take on when you start getting into X is required for Z which connects to A, yadda yadda.
That being said, if you choose a service and invest the time to understand it, you are availing yourself to a very wide world of technology "at your fingertips." You can most certainly say the same for DIY, just different shades of what you want to be responsible for I guess.
I write C++ business API servers. On the type of hardware I rent from Hetzner /OVH they're capable of processing thousands of requests per second. This would cover probably 90+ percent of real world businesses.
>"an API for everything"
Not sure what you mean here. API for what in particular? My servers talk to partner systems of real businesses. We consume their APIs and they consume ours. Integration is done by little plugins that translate requests should it be needed. I am not sure how Amazon will help in this department. The only generic API that everyone uses is email but you do not need Amazon for that. As for database - I always deploy one local to my business server.
>"good customer support"
Frankly in many years there was like 2-3 times when I needed something resembling support from Hetzner / OVH. In either case the response was prompt and satisfactory.
> Not sure what you mean here.
I assume referring to APIs for controlling and monitoring the infrastructure, not anything about APIs you may provide in your application or consume from external sources.
Old-school dedicated servers are so cheap that you can match a startups' peak autoscaled load and still pay less than their "idle" load.
There are many use cases where hosting fee is not the right thing to optimize for.
1. Have you done a security review and ideally pen-testing from a 3rd party? Are you fully TLS from end to end, are you aware of any exploitable vector from your API/UI and how you mitigate them
2. How do you handle software patching and vulnerabilities?
3. Do you consider your app Operational ready? Can you rollback, do you audit changes, do you need multi-AZ resilience, do you need cell based arch. Did someone other than yourself or outside your team look at your monitoring dashboard? Do you need to be paged at 3am if some of your dependencies or underlying node degrades. We have to answer a 50 questions template and review it multiple times..
4. Did you calculate the cost and forecast it with how the service/app may grow in the next 12-36 months?
While you still need to do all this when using a Cloud providers, you probably _should_ do much more if you manage the bare metal
If you have already done all those, Kudos to you, but I still find it hard to trust everyone who DIY
>" but I still find it hard to trust everyone who DIY"
It depends on what is the actual DIY part is. In any way it is your problem. Not mine. Sorry to say but to me your whole post feels like typical FUD scaremongering client to cloud.
>"While you still need to do all this when using a Cloud providers, you probably _should_ do much more if you manage the bare metal"
No I should not probably do much more. I do much less as I have way less moving parts.
I build products. That is my primary output. And the actual delivery is a binary executable that does the job. And it does it with the stellar performance that makes all that horizontal scaling infra totally unneeded. I serve the needs of my clients, not the needs of Amazon.
I share (some parts) of the sentiment.
Fiddling with a VPS and dockers (and Linux generally) landed me my current job but there's a curve and now I feel the need to up my game with a deeper knowledge and understanding of the different pieces and of the overall picture. Otherwise I am just another admin bash monkey.
This seems unnecessarily dismissive. (But maybe that means I am part of the 'specific subset'... :) ). IMO there is a big difference between 'tinkering' and 'self-hosting'. Projects you are tinkering with are typically transitory and can be treated as "pets". But in the 'self-hosted' world, stability and ease of maintenance are huge. In that regard I think the overhead of containers makes total sense, especially when running a bunch of stuff on the same server.
On the requirements side, one thing I would like to see though is an approach for determining how your applications will scale on cloud infra.
For instance on AWS, which RDS and EC2 instances will suffice out of the gate, and at what IOPS/bandwidth? And when will I need to scale them? The metric is simply how many users can I support with acceptable response times on a given configuration?
Sure, we know that's highly dependent on the applications, stack, etc. But I've often thought there should be some rubric for approaching this that doesn't require going heavy on performance analyzers, load balance testing, etc, which frequently seem like overkill out of the gate when you're making your initial config selections.
- They’ll leverage GDPR to make it only possible to comply to all privacy standards (SOC, credit card management…) if you use AWS,
- Worse, they’ll provide special Linux repos with vulnerability fixes that only they provide. Log4j can make all your distributions obsolete in a single day, and only Amazon Linux users will have that patch, this quickly.
- Then insurances will stop covering people who deploy their software manually, because vulnerabilities are too prone to happen.
The costs are bearable for early-stage startups that are mostly typical web applications with little resource requirements, or bigger startups that have so much VC money they are happy to burn it, but beyond these two scenarios clouds are very bad value for money.
Tradeoffs that will dominate the costs
1. Are you using a fast language or a slow language? (10-100x cost difference)
2. Are you using an efficient storage format such as parquet? (10-100x cost difference)
3. Are you using a reasonably efficient query planner job processor? (1-10x cost difference).
4. Are you using a good algorithm for the data processing task? meaning a good efficient SQL, An efficient imperitive algorithm etc. (unbounded cost difference)
The above tradeoffs will have a cost difference of up to 10^5 ( or greater depending on point 4). Once you account for risk, utilization, capex, and people costs. The cost difference between different compute platforms is usually negligible to the above points.
Cloud providers cut compute/storage costs every few years. You can get long term contracts with them that have heavy discounts for guaranteed spend. Switching compute instance types is pretty cheap, and switching compute platforms is relatively low cost when the time comes. Meaning that investors can generally expect your compute costs to fall YoY all else being equal.
Building out your own compute generally means adding new headcount to deal with colo things such as vender management, technical details such as networking, capes/opex management, and finally hardware selection. The odds that the hardware you select perfectly matches the compute you need in 2 years is low. The odds that a cloud provider has made a cheaper compute option in 2 years are moderate. The odds that your app needs something that’s available trivially in a cloud provider are high.
Infra is massively complicated. Not only do you have competing frameworks/modules, you also have these change in different releases of Linux. Some of these are simpler, some are more comprehensive. Some are really hard to use but powerful/well-maintained, others are simple but might not be so good. Some perform well at a cost, others might perform relatively less well but are easier. Sometimes it is worth the effort setting up ufw on a server, sometimes it isn't.
BY the time you got to something that was simple enough to create a guide from, and which didn't change each week when some vendor renames a configuration option, it would be very high-level and possibly not very useful.
Most of our applications are either:
- app developped in-house (django/flask): Procfile + deploy with git push
- standard app with a docker image available: deploy directly
Dokku comes with useful "service" plugins for databases, auto https (letsencrypt), virtual hosts... Overall, a good experience.
My only, tiny gripe would be excessive space consumption on the somewhat small 20gb SSDs you get with Hetzner VPSs.
My only, tiny gripe would be excessive space consumption on the somewhat small 20gb SSDs you get with Hetzner VPSs.
Multiple webapps (Symfony, Phoenix, Magento) on a 20€/m OneProvider server.
Databases backups are done by a cron bash script which uploads to a ftp.
It works fine, only real downside for my use case is the small downtime after a deploy. I probably would use something else for a frequently deployed pro webapp.
> there is some small downtime when I deploy new versions of things since I don't have any load balancing or rolling deployments
It's entirely possible to achieve, depending on your stack. `nginx -s reload` will reload the entire config without killing existing connections or inducing undue downtime. So if you can start a second instance of your "webapp" on a separate port/socket (or folder for PHP) and point nginx to it there shouldn't be any downtime involved.
> for users that are geographically far away latency can be high
That's true, but counter-intuitively, i found unless you're serving huge content (think video or multiple MB pages) it's not a problem. CDN can actually make it worse on a bad connection, because it takes additional roundtrips to resolve the CDN's domain and fetch stuff from there while i already have a connection established to your site. As someone who regularly uses really poor xDSL (from the other side of the atlantic ocean) i have a better experience with sites without a CDN that fit in under 1MB (or even better < 200KB) with as little requests as possible (for the clients that don't support HTTP2).
> CloudFlare (...) That may become necessary if I ever have trouble with DDOS attacks
I've personally found OVH to be more than capable and willing to deal with DDOS for their customers. OVH has been previously posted on HN for dealing with huge DDOS. That is of course if you have proper caching and you don't have an easy venue for a remote attacker to induce huge CPU/RAM load. For example, Plausible-like analytics can be such an attack vector because every request is logged in a database; something like GoAccess [0] is more resilient, and no logs is even lighter on resources.
Some time ago I was looking for an easier way to fix this. It seemed to me that a good way would be to have the reverse proxy (e.g. nginx or similar) hold the requests into the app restarts. Fit the user, this would mean a ~10s hiccup rather than 504 errors.
I didn't find an easy way to do it with nginx though and was sort of disappointed. Maybe other reverse proxies make this easier? Or maybe there is a stand-alone tool to do this?
[edit: one app I host can only be run single-instance as it keeps some state in-process. It can outsource it to Redis, but that seems overkill if it's only needed during upgrades, 10s/week or so]
Not exactly the same, but couldn't you serve a 425 Too Early with a meta refresh of 10s to achieve exactly the same goal? So change your nginx config to serve this "updating, your browser will refresh automatically" page, reload nginx, update the app, revert nginx config and reload nginx. Would that not address your needs, albeit in a more convoluted way?
I would rather a system that holds requests for 20s max if the backend refuses connections.
I think that's what nginx does by default, isn't it? Try again until "gateway timeout". The problem in this case is if you want graceful shutdown you need your backend to stop accepting new connections without stopping to process the existing ones, then update the app. If some client connections are long-lived, that's a hard problem and that's why in that case running a second instance makes sense if you can do it.
> Then every app consuming the API needs to be updated to support the 425 code gracefully.
That's indeed a problem, but your client should probably support some forms of server "failure" modes, including a "try again soon" type of reply (or interpreting 5XX as such).
> I have to manually do this switch in nginx before and after the update...
This can be fully automated. Just use different config files in sites-enabled. Then you can mv or ln to enable one or the other and run nginx -s reload. The whole process would be a few lines of shell script.
This might very well be a configuration error on my part though.
> your client should probably support some forms of server "failure" modes, including a "try again soon" type of reply
Currently the "try again soon" is passed on directly to the user. For a lot of endpoints, the client is a web browser for which this is not possible (short of turning the whole thing into a PWA... just for the ~10s a week the app is restarting...)
You don't want to import the mysql table itself on a fresh MySQL/MariaDB installation, it's a headache. So dump all your tables to SQL (raw data) and dump your user grants/rights with pt-show-grants (which in itself creates a small SQL file) that you easily import in MySQL/MariaDB.
ports:
- 127.0.0.1:3000:3000
To fix, just turn off iptables for the docker daemon and add the rules manually to UFW
The beauty of PHP is that more services consume no additional resources until they are used. This lets you run many many services without worrying about resource usage. In stark contrast to running everything in containers where you have at least a process per idle service.
When people go on about environmental issues and such, I cannot help to think all the useless cycles spent by 'modern software'. FaaS helps I guess. But that is not really unlike CGI; with my old setup it is actually practically the same but mine was less flexible (php only).
So many useful comparisons here, Bunny.net vs. CloudFlare, and the fact that you got this far without even using fail2ban!
Questions (not necessarily for OP, but for anyone)
- Give us an idea of the maximum load this server is doing in terms of requests per second?
- Anyone choosing AMD Epyc over Intel to avoid spectre/meltdown mitigation slowdowns?
- Any use of postfix or other SMTP forwarding via Postmark or another email provider?
- What is your Postgres configuration like for your heaviest loads? max_connections? Just wondering how Postgres scales across cores.
- Request rate is pretty low on average and peaks at around 15 requests/second. That's nothing really and it would probably take 100x or more request rate before I saw any kind of CPU bottlenecking or similar; my upload would probably bottleneck first. The biggest resource usage on the server comes from some expensive database queries made by some of my applications. - I'd definitely be down to use that kind of CPU but it wasn't available at the price point I wanted. Most of my stuff isn't CPU-bound anyway. - I used to self-host my own mailserver and webmail, but to be honest it was a disaster. Even after spending tons of time configuring reverse DNS, DMARC, SKIN, SPF, and the other list of arcane and specialized stuff you need, I still had tons of issues with mail I sent going to spam. I gave up and pay Google to host my mail for my own domains now. - I really haven't done much tuning. MySQL/MariaDB is the DB that gets the most load and its settings are most default. I have put a lot of effort into profiling my apps' query patterns and making sure proper indexes and schemas are in place, though.
Traefik can read labels applied to Docker containers (easily done with docker-compose) and setup the proxy for you as containers come and go. Even maintains the Lets Encrypt certificates seamlessly.
I also think Caddy 1 was just simpler to use in general, so it didn't matter as much. But I don't have much experience with that version, so could be wrong.
I was trying to configure the auto-HTTPS functionality to use DNS challenges, because my setup forbids any of the others, and this apparently requires plugins that are only supported by community efforts, so that's probably why I had such an issue.
The docs around configuring this were not immediately clear to me. Its like one part is over in this corner, the other over there. And the part about DNS provider plugins is off in a completely different direction. I definitely think things could be better organized. Also your one tutorial for getting HTTPS support only mentions the very basic cases. I think it'd be beneficial to see some more advanced tutorials using for instance some of the other LE challenge methods, something like this definitely would've saved me some time.
I wish there were an easier way to bind docker ports to Caddy automatically (without using dokku et al.), but for now I maintain a Caddyfile. Which, thinking of it, doesn't even require setting up a janky script for the rare times when I need to host a new service & modify the config.
I guess there's no reason to make things harder for myself 6 months in the future.
Related: https://xkcd.com/1205/
- GCP VM with “Google Container OS”.
- “Cloud init” config set on the VM metadata (can easily recreate the VM; no snowflake config via SSH mutations).
- My service runs in a docker container, reads/writes to a SQLite file on the host disk.
- GCP incrementally snapshots the disk every hour or so, and makes copies to different region. Any disk writes are copied instantly to another zone.
- Lets encrypt cert is read from the host disk and the docker container serves HTTPS directly (no proxy). Certificate is renewed with the LE CLI.
- The service logs to standard out, this is collected by the Google logs daemon which I can view with the web UI.
- Google have their own HTTP uptime monitoring and alerting which sends you an SMS.
I still use docker-compose and have a k3s cluster running, but I really like the simplicity of deploying via git.
Cloud-init for my core setups is here: https://github.com/piku/cloud-init
Otherwise I’ve been using it without issue for months now.
Well apart from that detail that's exactly what I do for my stack of personal sites. "docker-compose" is really hard to beat for all kinds for setups. And I get 64GB to play with for 50EUR/month from Hetzner, so I don't spend too much time worrying about RAM.
https://github.com/lavie/runlike/
But I've got those migrated to docker-compose files these days and I try to start with the docker-compose file instead of going directly into testing out docker run commands.
I rent three now total, one for production sites, one as a development machine strictly for vscode and another for development / staging / misc. waaaaaay overkill but it’s been a huge quality of life improvement.
For containers I just use a couple docker compose and a script to run them all.
Reverse proxy is through nginx proxy manager which takes care of LE renewals and has a nice web interface.
Sometimes though they'll send targeted email offers that are not on the site and are good for older hardware but at a really cheap price.
These projects are amazing but if you're gonna use them, don't forget to do backups, especially if your system is hosted on a SD card (for example on a raspi).
I'm just a little sad there's no standard packaging format across those distros, although i've spoken with yunohost and libreserver maintainers in the past and they seemed somewhat interested in the topic!
Later...
> I host my own analytics ... They provide an open-source, self-hostable version which I have it deployed with docker-compose.
Wouldn't docker-compose be the "already doing it" answer to the first question? It's pretty much your library of 'exact docker run commands', plus sugar to manage the running ones collectively.
I'm also confused why he's not using compose. Perhaps he is unaware that when you have multiple services in a compose file, re-running `docker compose up` only restarts containers whose parameters have changed in the compose file.
He mentions further down that he's using compose for one service, so obviously he is aware of it.
But I can't blame him. When getting up to speed on docker, I found that figuring out compose and how to use it effectively was frustratingly under-documented.
Here's what they did folks: they set cloud prices a while back, did not drop them as Moore's Law delivered more power but instead pocketed the profit, and meanwhile ran an entire "cloud native" development push to encourage development practices that maximize cloud lock-in (both through dependence on cloud services and encouraging complexity).
Oh, and bandwidth is ludicrously overpriced in the cloud. You can transfer hundreds of terabytes a month outbound for <$500 if you know where to look. No I don't mean from static data either. Look at bare metal hosting.
Hopefully through cron and not manual invokation! Certbot can safely be executed daily by cron, as it will only attempt renewal if it is required.
Automating certificate renewal is an very important step in ensuring availability. I feel like part of the on-call sysadmin initiation process (pre-ACME days) was getting a frantic midnight phone call because someone forgot to renew a certificate...
I suspect they are using cron, but this has been omitted unintentionally.
It provides a nice looking UI to manage reverse proxy with Let's Encrypt certs, auto renewal and a few other nice features.
https://github.com/keybittech/awayto
It's not a product, just a project. Check it out if AWS is getting you down! :)
I do the same, and have spent some time automating the backup of such a set of standalone containers [0], in case others also find it useful.
Heat pumps can achieve greater than 100% efficiency relative to electric resistive heating, because they move heat, rather than do work to create heat. These can achieve equivalent heating at 50% of the electricity utilization compared to resistive heating.[0]
Local fossil fuel heating (e.g. natural gas furnace) is much more efficient than electric resistive heating. The efficiency of local fossil fuel combustion is close to 100% of the energy content of the fuel, as it is turned directly into heat and that heat is dissipated into the home. More than 60% of the energy content used to generate electricity is lost in the generation process in the US.[1]
So, the heat from a computer does offset heating costs, but at less than 100% efficiency. For a small fanless machine, this is not likely to be a large expenditure, but it is valuable to keep in mind that heat from electronics is not "free".
[0]: https://www.energy.gov/energysaver/heat-pump-systems [1]: https://www.eia.gov/todayinenergy/detail.php?id=44436
We do something similar, multiple django sites and static sites on a single digital ocean droplet. We use docker for everything though, and have a single nginx listening on 80/443 that routes traffic to the sites by domain. Each site has a separate pg db and everything is glued together / configured / deployed via ansible.
But I can see the merits of containers, and might containerize these apps & services some day.
I see this sentiment quite a lot that k8s is too complex for small scale setups, but in my experience it scales down very well to small setups. Granted, you do have to know how to work with k8s, but once you learn that you can apply that knowledge to many different kinds of setups like k0s, bare metal, EKS/AKS etc.
With firewalld active containers cannot do networking, not even with hosts jn the same lan.
Everything else works beautifully though.
With iptables:
-A INPUT -i kube-bridge -j ACCEPT
-A OUTPUT -o kube-bridge -j ACCEPT
So yes, it definitely takes quite a bit of resources. I'm running this on 4 CPU cores and 6 GB memory, so 25% of 1 core and some 600mb of memory still leaves plenty of resources for the services. On a more philosophical note (as was mentioned below in this thread), it is a bit wasteful perhaps.
$85/month is a lot of money, but given the number of services he runs on his server, each service now costs roughly $3/month, lower than the price of the cheapest DigitalOcean droplet.
TLDR: You can self-host without your ISP knowing!
But using that then I think it would be more than feasible to do it from your home network if you have a fast enough upload link.
I have a server @home with Plex and few other services that are used mostly by myself and close relatives. Not something very public. It runs on a second-hand i7 NUC which is almost silent unless someone needs transcoding on content read from Plex. But it's "no warranty" services.
And I rent a $20 server at OVH on which I put more public stuff. Like blog for myself or others and for which I don't want to be linked _directly_. For instance a friend's blog was under attack because some people didn't shared her ideas. I was covered by OVH's anti-DDOS. If it had been on my home server, I probably would had some _troubles_. Same goes for the seedbox hosted on it. I don't have to set a VPN and put safety in place to make sure that not a single bit of data from the seedbox is shared without the VPN. I just run it exposing the public IP of the server and don't care. Worst case scenario is OVH taking down the server after (many) abuse reports.
The only thing I'd change in your workflow, perhaps, is switching from docker CLI commands to docker-compose. That'd make things a lot more reproducible and easy to read, and if you group relevant containers into a compose file, they're also automagically networked together, which is handy.
Your "trick" of grouping TCP ports by the hundreds is something I might steal, I've been using $lastport+1 for a while and now there's a bit of guesswork involved whenever I need to do things that don't go through the proxy. Then again, that's not often, so I might leave it.
https://costapiy.com/deploy_django_project_linux_server/ and here https://github.com/costapiy/server_setup
https://gitlab.com/stavros/harbormaster
It basically pulls Compose apps from the git repositories you specify, builds the containers and makes sure they're running. Pretty simple and works really well for me.
Don't get me wrong, my server also runs a ton of crap, like git/email/backups/irc/mysql/etc. plus a couple websites and some python scripts (3x doing continuous data logging, 2x providing some service), but it draws less than a traditional light bulb and also runs everything in linux containers. Unless you're doing large data processing tasks like machine learning on a weekly/daily basis, odds are you don't need more than two consumer-grade cores (I've got more since I repurposed an old laptop, but they're very rarely used). Large RAM is nice, though, I do have to say that. My 8GB is stretching it a little, considering that my favorite email server software is a Windows GUI thing and so I run virtualbox to have this guilty pleasure, which consumes about a quarter of the system's RAM just for one service.
Aside from Owncast, no idea what requirements that has, I think all of OP's services would be fine with a Raspberry Pi as well, just RAM would be tight with so many individual containers. Especially given the static site hosting, and doubly especially with that offloaded to a CDN, the only dynamic things (screenshot hoster, pastebin) are own use and won't see a dozen concurrent users continuously or anything.
Edit: I read over the -track API websites part. I don't know whether those might fit on a Pi as well, but that sounds like various people make use of it and this server is not just single user.
That's a great way to mutualize resources (reduce waste) and at the same time is arguably much greener than datacenter/cloud services which require dedicated infrastructure (a great source of energy/pollution), and often have layers and layers of user tracking/profiling and detailed logging which consume great amounts of resources.
If a €1200 built-from-parts desktop from March 2020 counts as modern desktop class hardware... this system uses 85W idle. It has an AMD CPU+GPU, is Intel and/or Nvidia any better in this regard or are you talking about more low-end hardware?
EDIT: e.g. for an optimized system with Ryzen 9 5950X and RTX 2070 Super apparently 40 W is doable for "sitting idly on Windows Desktop" according to reputable tests. Lower-class hardware can do less.
The desktop comparison to many laptops is more a mini-ITX based, embedded-graphics box with e.g. a quad-core CPU. There idle <=15W is realistic. (And the laptop idles at ~5, so still more efficient of course). And the range between for components inbetween that and high-end.
Desktop is going to be worse than laptops, but people tend to overestimate how much power a desktop system actually has to use if it's built well. Modern CPUs are efficient at power-saving, PSUs have gotten better, ...
Buying a raspberry pi for selfhosting is certainly not greener than repurposing your old desktop/laptop. Although i have to admit the economic incentives are skewed due to foundries and global supply chains paying energy orders of magnitude cheaper than we pay our electricity.
Edit: just realized the 70W figure is in a sibling thread, not a (grand)parent, but I'm still interested regardless!
I can use solar panels on my own roof to power an extravagant server, but it's still not really net zero (even if you would offset solar panel production and recycling) because they could have supplied to the still-very-fossil national grid if I hadn't been consuming it frivolously.
(Edit: I should maybe add that we don't all need to be saints all the time. My guilty pleasure is long showers, I enjoy those despite the power consumption being 10kW. I do better in other areas and vote with my wallet and ballot. I of course have no insight into what other energy choices you / OP / the reader makes so my comments are not meant as criticism towards any person.)
Arguably, none of these account for the pollution due to extraction/refinement of materials, which is another concern raised with building more stuff. Recycling is also a polluting process (usually less so than new extraction) but unfortunately most electronics is never recycled and R&D in the hardware world is mostly focused on raw performance and does not even try to optimize for recyclability.
If any government cared at all about climate change, they would mandate hardware manufacturers to publish hardware sheets (for long-term maintenance and interoperability [0]), outlaw throwaway single-purpose computers (as opposed to flashable general-purpose computers [1]) and planned obsolescence, and invest massively in two key areas:
- green IT R&D: there's very little universities around the world working on this, and they operate on a shoestring budget
- public-service repair shops for electronics: it's ridiculously hard (or expensive) to find electronics repair services even in big cities, but having skilled people do a 1$ part change on a device that would otherwise be thrown away (eg. soldering a new micro-USB connector or changing a burnt capacitor) goes to great lengths to extend the lifetime of existing devices
I'm interested if people have more links/resources to share on that topic!
[0] https://www.usenix.org/conference/osdi21/presentation/fri-ke...
[1] https://boingboing.net/2011/12/27/the-coming-war-on-general-...
"Repair is not an auxiliary, optional part of the economy. Repair is care, at every stage of the supply chain, and at every scale. Those in dire need of repair for their devices are, just like Terry Pratchett’s Theory of Economic Injustice predicts, in the crosshairs of widening economic inequality.4 Already-impoverished consumers are encouraged to buy ever-newer products (and other planned obsolescence mechanisms have co-evolved with the boom of overproduction for the Global North). These products are notoriously hard to repair by design and require repeated purchases, exposing the long-term scheme of the manufacturers. Mineral extraction necessary for the manufacture of new hi-tech devices inflicts death and immiseration on surrounding populations.5 A community that sees no value to repair is a community that cannot respond to the crisis of capitalism, and is merely its hostage. Repair, as an act of reclaiming technology, is ongoing in the Global North and South with complementary driving forces and problems."
[...]
"A classic strategy of anti-repair design is avoiding modularity. If the parts of the device are strategically bundled together so that failure of one part requires replacement of a whole unit, it is not “repair” anymore. While it happens with mechanical components as well, the electronic version of this strategy is worthy of closer examination. Unlike a gearbox whose miniaturization still results in an assemblage of separate gears that might be smaller and harder to replace, miniaturization in electronics was driven by manufacture of monolithic semiconductor structures. Control systems that have previously been implemented with, for example, a collection of discrete transistors and diodes (basic electronic components that are easily replaceable) have been revamped as embedded systems: for the same functionality they now use a microchip with software running on it. Access to the software is not provided. Access to the chip itself is not provided as it is commonly “globbed” (covered with a black blob of epoxy). On top of this, the manufacturer takes precautions to prevent you from replacing the entire controller with a different, customized controller on your own. Here I return to the robotic arm: what kind of a controller do you want such a mechanism to have? The odds are that the same arm might be deployed in a myriad of different settings and scenarios, and needs tweaking of the controller. The “body without organs” controller under the blob of epoxy offers no modularity, no chance to expand, reduce, or in any other way customize inputs, outputs, or processing capabilities. The individual components that might be at a higher risk of damage (e.g. transistors) don’t exist anymore, so every failure takes the entire block of silicon down with it. And finally, if product support is discontinued, the manufacturer goes out of business, or chooses to change the business model into selling a service rather than a product, the controller is expected to become an unusable brick. To make things worse, by making non-standard interfaces with mechanics and/or placing restrictive licenses on its software, the brick is supposed to be irreplaceable, hence rendering the entire robotic arm unusable (even if the mechanics are perfectly sound).
The loss of repairability is not a consequence of technological progress or increasing complexity—it should arguably be the opposite. Complex systems science pioneer W. Brian Arthur explains the two primary mechanisms of improving a technology: “internal replacement,” or changing the existing parts; and “structural deepening,” which means adding new components.10 Neither of these require that new parts and components cannot be modular, replaceable, and repairable. Complexity, in fact, is all about modularity and heterogeneity and can be an argument in favor of repair. The concepts of internal replacement and structural deepening, if anything, are the philosophy of repair as a creative process. New parts or mechanisms that come from repair contribute to an invention: potential new applications of the device, a new approach to manufacturing, and personalization of the item. A creatively repaired device is where the social network merges with the technological one. However, that is not in the interests of the manufacturing lobby: this network is one of capital accumulation.
The other aforementioned strategy of disabling repair is the legal one. To keep this grip of capital on the technology and knowledge of manufacturing and repair, the opponents of repair create the illusion of illegality: Looking under the hood should be taboo, understanding how things work should be perceived as illegal, and the concept of patents and protection of intellectual property should be regurgitated as respect for science and protecting the world from anarchy. Big manufacturers such as Apple also grasp at other legal straws such as privacy concerns.11
Bogus legal barriers run against the basic principles of science and engineering. Take, for example, the concept of reverse engineering. Finding out how a piece of hardware or software works by observing its inputs and outputs is an essential part of repair in modern technological systems. Often portrayed as illegal, this activity does not go against trade secrets laws. Instead, it becomes an issue concerning terms and conditions agreed to by user and manufacturer.12 Among legal contracts, “terms and conditions” represent a world of their own, with clauses that are often void, unenforceable, or plain illegal.13 The “opening box voids warranty” stickers mentioned earlier are a blatant example, but not the only one. Through lobbying, manufacturers erect new legal barriers where previously there had been none: when the Unitron Mac 512, a Brazilian clone of the Apple Macintosh was developed in the mid-eighties, it infringed no laws in Brazil. Only after heavy lobbying from Apple and threats of sanctions from the US government did the Brazilian government introduce a completely new “software law” to stop the sale of the reverse-engineered Macintosh.14"
source: https://magazine.scienceforthepeople.org/vol24-2-dont-be-evi...
If anyone has a good suggestion for an open source system where I can:
- Give people access to control their domains via a web interface (create accounts, set a catch-all, such things)
- Have aliases and server-side rules (e.g. blackhole email coming into spamaddr2021december@lucb1e.com; move From "bank" Subject "balance change*" Into inbox/notifications), preferably via the web interface
- Optionally: have IMAP and SMTP without a lot of moving parts (I've looked at typical Linux setups before and they're usually quite finicky)
Then please let me know!
> [list of features]
hMailServer it is?
And regarding SMTP replacement... for personal uses I moved to Fastmail years ago. It does everything listed except "control domains" - but does it in a reseller account (so you can have the control too). Of course it's all costs money, so...
My server isn't bound by CPU almost at all - the most intensive thing is the occasional trans-coding for jellyfin, however it does use quite a bit of ram for a couple of minecraft servers and ZFS. I'd really like some sort of preferably arm or maybe risc-v based server (in the future) that can take sata and a good bit of ram but most of the SBCs I see would require that my drives work over USB which is annoying and usually dont have more than 4/8gb of ram.
One advantage of having everything scattered around is that not everything is going to go down at once (eg: a DDOS or even a traffic spike on your machine will kill all your sites/services if you only have one machine).
I use: - Percona PMM[0] - Database/server monitoring - Backups to BackBlaze B2 using cron jobs - Simple deployments, usually using SFTP to upload files (usually for LAMP applications, also running Node.js apps using pm2 that can listen to file changes and restart the app)
I am actually working on creating some tools/scripts/guides to aid with self-hosting (for backing up stuff, uptime monitoring, alerting).
Why do you RAID-1 NVMe? They are likely to fail simultaneously so maybe make them independent and schedule regular backups from one volume to the other, or to a remote disk drive somewhere.
Even the cheapest OVH dedicated server, starting at about 50 EUR / month, has 16 GB of RAM and it's ECC.
Their middle-budget brand SoYouStart seens to only offer machines with ECC, I started using one recently, so I assume you are right and the main brand operates likewise.
If you are really worried about that, perhaps artificially stress one of the drives for some days before building the array so it is more likely to go first by enough time to replace and bring the array back from a degraded state?
"Replace one of SSD/NVMe drives in $server because I'm afraid of simultaneous failure (and probably still didn't configured backup for this server)"
Unfortunately that won't wash if you are renting a server of collocating. Replacing a drive without it having already failed would most likely result in a charge for unnecessary hands-on maintenance time.
Though if you are expanding the storage anyway you could do it with an identically sized pair of drives, sync the existing array over to one of the new ones, drop the extra, then you have to unmatched drives for a new array. If using LVM you can join that to the existing VG or (less safe but better performing once you are done) you can try reshape the two arrays into a stripeset for RAID1+0. And hope that the new drives are not from the same batch as your existing ones and have just been sat in a store cupboard for the last year.
Even just a nightly rsync would decorrelate what is right now nearly perfect correlation.
I wouldn't say much more likely than with traditional drives, unless you are getting towards EOL in terms of how much rewriting has been done but after that much time I'd expect randomness to separate things out at least a bit.
The main concern I have with either drive type is finding out blocks that haven't been touched in ages have quietly gone bad, and you don't notice until trying to read them to rebuild the array once a failed drive has been replaced - that applies equally unless you run a regular verify. Other failure modes like the controller dying are less likely to happen concurrently, unless there is a power problem or some such in the machine of course, but again these might affect all drive types and this is one of the reasons you need proper backups as well as RAID (the age-old mantra: RAID is not a backup solution, RAID increases availability and reduces the chance you'll need to restore from backup).
Having said that, my home server deliberately has different drives (different controllers, highly unlikely that even if the memory on each is from the same manufacturer it is from the same batch) in its R1 mirror of SSDs, just in case. The spinning metal drives it also has in another array were bought in a way to decrease the chance of getting multiple from one batch in case it is a bad batch.
> nightly rsync
The problem with that and other filesystem level options is that depending on the filesystem and what you are running, some things might be missed due to file-locking. As RAID is block-device level this is never going to be the case, though of course in either case you can't catch what is in RAM and not yet physically written.
Of course this problem will be present for most off-device backup solutions too, so you could use the same mitigation you have there for backing up between the drives too.
It doesn't look to be ECC.
> Why do you RAID-1 NVMe
That's what they set it up with when I first rented the server. What makes them likely to fail simultaneously, out of curiosity? I do backup everything irreplaceable as much as I can; I've not had to do it but theoretically I'd should be able to re-build everything from scratch with a day or two of effort.
- Nextcloud - Gitlab - A bunch of websites hosted under different domains - Matrix homeserver
And the load is still surprisingly low
Which means that you have some extremely unpopular services running...? :) /s
(Happy new year!)
It may be easy to maintain it as all your hosted services are in one place.
Main problem I faced is that the server has become single point of failure which caused lot of issues with customers. It’s really hard to explain why all your services are down at once.
I run everything in docker containers, except Wireguard and the backup.
I had an idea to "document everything" but the realized there would be no-one to maintain the services (neither my wife not my children are interested in IT). I then decided to document how to untangle themselves from the self-hosting (moving to well-established services, dismantle the home automation, ...). Thi sis going to make more sense in the long term.
Since I do not plan to die tomorrow but I do not know for my server, I tried to do a disaster recovery from scratch. It was surprisingly easy - after starting a VM on my laptop (simulating the bare-metal rebuild of my server), I was up and running in about an hour. Docker is simply fantastic for that.
I took some notes about the disaster recovery, which I promptly lost so I will have to run this again soon (maybe next week). Actually I will copy the encryption key for my remote backup right away as this is the only thing I won't be able to recover. EDIT: done :)
TL;DR:
- use docker containers for your services, it is a life-saver and makes everything easier
- use a docker-compose script to have everything on file
- document either your maintenance or exit strategy
- do a quick DRP test on a VM
This is a perfect solution for early stage projects if you're in a startup or for a personel project, but monoliths are....well....monoliths.
monoliths are KISS, and kubernetes can be YAGNI
Appservers can be trivially scaled horizontally regardless of whether it's one process that does all (monolith) or a mess of microservices.
Some hosts can reach incredible scale with a just few beefy dedicated machines, given the right software architecture.
If it was one virtual machine I'd say, sure, just use one box, you can bring up a new one automated in 10 minutes. But if it's physical, it could either last 10 years without going down, or 10 weeks, you never know. Physical machines is precisely the time to use a distributed system as a hedge against expensive failures.
Another consideration is cost; at $1200/year you could just keep buying servers and Colo them. It's more work but you get more (old) hardware to play with.
The same is true with VPS and "cloud" setups. As the OVH fire last year showed, offsite backups are not a luxury. The chances of that, though, are ridiculously low. Higher-end dedicated hosts have very low failure rates due to using higher-grade hardware and rotating the hardware every few years; they also usually have very competent and available technicians on site which are paid by your monthly bill.
> If it was one virtual machine I'd say, sure, just use one box, you can bring up a new one automated in 10 minutes.
The same is true with dedicated servers. If you house them yourself then sure bringing up hardware replacement is going to take longer. But if you use a managed dedicated hosting solution (as was mentioned in the article) the provisioning times are sensibly similar to that of a VPS (a few minutes).
At home I'm running an old desktop PC with Linux for my router. It's running on >10 year old hardware. I've got several other old PCs running, too, that's just the oldest, so not completely a fluke.
Assuming your one giant server will be as reliable as a 10 year old box sitting on your desktop is a gamble. All I'm saying is be prepared to lose.
I've known people who built their own boat and also built the engine for the boat. And it's a great hobby which gives you a great sense of accomplishment. But, if you just need to get some fishing done, buy the boat with the engine already installed.
you can't always rely on third party things to host your stuff if your intention is to become the person who runs the hosting services for other peoples' stuff. same way you can't rely on a boatyard's contractors to maintain your boat if your goal is to become an engineer for medium sized yachts.
Now, a containerization service like Fargate would fit his use case better since he is using containers.
However, cloud services come with their own complexities and the principle problem they solve (horizontal scaling) doesn't apply to his sites.