Due to the advent of low power small form factor PCs/SBCs like the Pi, and all the additional management options you get these days: conventional bare-metal, but also containers, VMs, and everything in between, coupled with IaC like Ansible or Salt. A single machine can now be far more useful than just classical Unix services like LAMP or SSH - but you can still have those if you wish in addition to the new stuff!
I recommend checking the subreddits /r/selfhosted and /r/homelab to get an idea what people do these days. These are by no means the only places where all the cool discussion takes place, but they are a very good jumping off point. If you’re interested in managing file storage that goes beyond a few TB, /r/datahoarder is a good place to look too.
(And yes, running an email server still sucks. This time, the complexity is no longer about setting up or maintaining the thing (Mail in a Box/Mailcow basically put this whole thing on auto-pilot for you, including initial setup), but about fighting the major email providers to force them to not silently drop incoming messages sent from your server).
What you said about email is also really true. I think it means your sending really has to go through reputable origins rather than through your own IPs. So the relaying/sending part needs to be decoupled from the actual storage.
All of this points to, where's the drop-in solution.
- For email, that's (IMO) Mail-in-a-Box/Mailcow on a VPS + an SMTP relay.
- For everything else, either a VPS or an x86 server/PC running Docker, paired with an easy reverse proxy like Caddy. I was able to quickly try out multitudes of applications with Docker, without commiting more than a few minutes to any application unless I liked what I was seeing.
My self-hosted environment is very much function over form. I don't have time to have fun with self-hosting. I run it due to two simple realities of SaaS platforms: the prying eyes, and the limited storage.
No prying eyes: pretty obvious. No matter how much each provider swears up and down that they do no, cannot, or will not read your stuff (if they do promise such a thing at all), I think it's wise to assume that they can, unless it's all E2E encrypted and you use an open-source client to access whatever service this is (not commonly available). I prefer to use self-hosted storage because I don't have to think about my data potentially being made public to someone who hacks into their systems, or sends a "give us all their data" letter on fancy letterheaded paper/email. Securing a system against external attacks is not such a huge amount of work, unless applying security updates is a problem. Once I feel I have true privacy on my own equipment, then I feel that I can freely express myself without self-censorship.
Limited storage: I prefer to archive as much as I can, because it goes away. As an example:
- several widely-acclaimed HBO shows have recently been wiped from the face of the Earth, because... reasons? As best as I can tell, it's some tax-reduction maneouvre that was mean to save them money, but at the cost of destroying a huge chunk of our shared cultural commons (because TV is a part of that, whether we like it or not).
- Youtube channels are dropping like flies thanks to drama, Youtube's copyright policies and other weird factors that mean the channel is there one day with 500+ videos, and the next day it's gone forever like the victims of the Ministry of Truth from "1984".
- There are benefits to having a full download of my entire GOG games collection available locally. Easier to navigate a few directories on my NAS and double-click on an installer than do the same with extra steps via a web browser or GOG Galaxy (which is still slow on old machines)
Self-reliance takes a lot of disk space, and that disk space gets pretty expensive in the cloud at TB+ scales (not even some crazy collections that people with 100TB+ of storage build - even just single-digit TB sizes are hard to afford). So the only solution is to keep most of it local, even if I make use of cloud storage for email backups and such.
The email part, I sort of wish I had programmatic email to messaging or something. Just some variant on it that isn't about the classic mail serving. Gmail does my email, but I don't really have programmatic access to that inbox or the ability to write filters easily or whatever else i want to do like trigger on stuff, do my own search. Current email provider APIs suck, so if actually there was one that was heavily focused on a great API that would be awesome. Also, some new domain separating that inbox from my personal one because I'm starting to feel like email masking needs to be come a thing.
> unless it's all E2E encrypted and you use an open-source client to access whatever service this is
This is something I want to know more about. VPN to the service itself through an open source client? And maybe also data encrypted at rest with a key to decrypt only you provide at time of access.
I guess what I'm starting to wonder, why is it we have all this infinite cloud and compute but when it comes to privacy we lose all that convenience because we can't trust services enough to host for us, instead we have to do it ourselves.
I feel that so much that I started a cloud provider from the ground up (m3o.com) thinking if I could see and understand every piece of the cloud infrastructure and all the services running on top were open source then maybe it would be the starting point for something. TBD on whether it will work.
Streaming is constantly asking a provider to give you a copy of that media, and the provider can always reply with a flat "no", or give you a "remastered" version that doesn't capture the charm that first made you like the media (see Star Wars - not in my personal opinion, but it is a controversy online IIUI). As such, I consider streaming a convenience that's great when it's available, but not something I'd like to be on the mercy of. The digital era gave us the ability to enjoy what we want, when we want it, and I refuse to let copyright dinosaurs ruin my fun.
Mail in a Box comes with support for Sieve filtering, but it's unpleasant at best for programmatic processing of email as you described it. I actually built an email server precisely with programmatic processing/sorting/transformation of messages in mind, but other projects took priority and I wasn't able to finish it.
> This is something I want to know more about. VPN to the service itself through an open source client?
Nope, I mean in the sense that the data is only ever decrypted and encrypted on the endpoints themselves. If the provider ever stores or forwards anything, it's encrypted blobs of data with no added metadata to even hint as to what's inside.
> we lose all that convenience because we can't trust services enough to host for us, instead we have to do it ourselves.
This mirrors my own reflections. The prevalence of centralised, gatekept systems is the trend of the day. I would greatly prefer a web of peered "communal" servers that each serve between 100 and 10,000 people each, to make sure that no one server dominates while also not putting the burden of server maintenance on each and every person who wants to participate in this. I guess the Fediverse is a shadow of this, but that's just for communication and not say, compute or serious storage.
> every piece of the cloud infrastructure and all the services running on top were open source
That wouldn't be a bad start. Some degree of transparency as to how you build and configure the control plane of such a cloud to not give yourselves undue power over machines would also be appreciated.
I know a NAS is not what you're asking about, but it goes give you a huge % of the features of a home server with only 5% of the effort.
Website server.
SSH server for remote logins.
I refuse to use a third-party service for these items.
All off-grid on a RPi (well two, because I'm too idle to consolidate them and save another watt at the moment)!
Mostly - for web servers, ssh and few vpn services.