Ditching PaaS: Why I Went Back to Self-Hosting
shubhamjain.co
shubhamjain.co
I notice a trend that the people who scoff at hardware specs are usually the same ones standing in line for 2 cores and 4gb of RAM for $50+/month. They'll laugh when you suggest utilizing an obsolete $50 computer with a decade old CPU (that's just sitting in a closet), but more than willing to spend $50/month on similar performing hardware from a Cloud vendor.
1. Those who love their Macs a little too much
2. Those who routinely build a PC.
i.e. Are you familiar with the hardware market.
I'm the kind of person who would rather take the 3y server and recomission it as a lower priority service than just do a 1:1 replacement. "Old" computers aren't as useless as they used to be. Computing power has advanced to the point where computation is arbitrary. For all intents, in most sectors, you can scale your compute capacity as high as your budget allows and you won't hit any performance barriers ever. There will always be more compute. It is a commodity now.
My stance is this; sure the new server is faster than the old one, but you know what's even faster? Dividing the existing workload between both of them.
What about the core count and memory capacity
How do you factor reliability into your analysis? I mean, the reason companies ditch their 3yo server is that their service life is nearing it's end, and odds are they will fail in the upcoming year or so.
Meanwhile, increased compute capacity means companies can use a single brand new server to replace multiple old ones, not to mention cheaper operational costs.
Unbelievable (new) specs, hosted and at prices points tough to beat doing it yourself.
E.g.
Hetzner
AMD RYZEN™ 9 7950X3D (current gen Zen 4)
16 physical cores (4.2 GHz)
128GB DDR5 ECC ram
4TB NVMe
completely hosted
They normally are trying to keep their IPs and Network clean of scammers abusing their resources, which inevitably hurt all customers.
Is there still more needed to complete their KYC process?
AWS is expensive and not without trouble but in my experience they’ve been best of the pack.
Otherwise, on-premises + collocation is never a mistake if the maintenance and upright expenditure is in your budget.
Isn't on-die ECC necessary just because DDR5 is less reliable than previous generations, especially at higher frequencies and density? Makes me wonder whether they use an actual ECC DDR5 memory.
Yet many other folks praise them…
What’s your experience?
I run a couple ryzen mini PCs on Verizon fios 1Gbps at home now for my hosting needs but I would jump at $60 colo in a heartbeat if I got even 100mb connection unmetered.
I also pay for 1Gbps symmetric fiber with dedicated IP going into my house ($140/month at the moment) so I also host some stuff right from my own place.
The duct-tape-maintenance vs your-time criticisms are 100% dead-on though.
Plus flexibility of scale up / scale down as needed for load, transient testing, etc.
Cloud is definitely not for everyone but it makes sense for some.
For larger projects I guess, but I have a few VPSes that have been running sites for a decade or more that require almost no maintenance (occasional apt-get upgrade). In fact it’s the frameworks / languages these projects were built in that cause most of the work (for which deployment target makes no difference).
Move the exact same workload to the industry's default but much more expensive choice, and the problems vanish entirely.
This is, unfortunately, one of those things that's really hard to judge about a hosting provider unless you have direct experience using them "at scale", as they say. Nobody puts that stuff on a sales page spec sheet or comparison grid.
Could I save money by hosting on real hardware at some popular, cheapish server-leasing place? Or colocating at one? Or hosting out of my own basement!? Maybe. Would it cause some users to consistently see dial-up speeds and dropped connections on gigabit Internet service because of either some quirk of routing, or bad peering agreements? Who knows!
If the thought of your service going down for a few hours because some random unplugged the power cable or your ADSL router crashed doesn’t make the CFO lose sleep then some raspberry pi is surely good enough. Just make sure you run it inside a safe if you store personal information.
I'm now running my primary project on Fly.io and I'm pretty happy with it overall.
"No matter how small is the service, no matter how rarely you need it, you’d need to pay the full price."
On Fly.io I'm running an app server, a db server, and another app server with a 1GB volume for a Discord bot. Everything fits in the free plan.
The thing about PaaS is you really have to do your research. It's not like VPS providers where all you really need to look at is how much compute and storage you get for a monthly price. PaaS have a lot more subtleties and yes, it happens that the startups behind them sometimes blow up or get bought out by huge public enterprise companies. VPS providers tend to be lower risk.
The tradeoff is worth it for me, but it really depends on your skillset, your priorities and so on. I can maintain a VPS, but I have very limited time, so I want to focus every spare hour I have on developing my product.
- Figured I'd need to screw with Systemd at some point. Nope, whatever Docker's doing restarts my services on a system restart, and auto-restarts if they break. I haven't had to lift a finger for any of that. My services are always just there, unless something really goes horribly wrong.
- Which directories I need to backup is documented in the shell scripts themselves. Very clear and easy.
- Moving those directories and my shell scripts to another server, potentially with a different distro, would be trivial. Rsync two directories (I've put all the directories I mount in the docker images, under a single directory for convenience), shell in, run the scripts. Writing a meta-script to run all of them would be easy. On a VPS I could have everything that mattered on a network drive, and that'd make it even simpler. Mount network drive, run script(s).
- Version updates are easy. I can switch between "use the latest" and "use this specific version until I say otherwise" at will. Rollbacks are trivial. If the services were public-facing I could automate a lot of this with maybe an hour of effort.
- Port mapping's covered by Docker. If these were public-facing it'd be pretty easy to add one extra container for SSL certs and termination (probably Caddy, because I'm lazy, though historically my answer for this at paying gigs has been haproxy). Like, truly, the degree to which I can interact with and configure this system entirely by using portable-everywhere docker commands & config is very high.
I've been running servers (sometimes private, sometimes public) at home since like 2000, and this is easily my favorite approach I've used so far.
I've used stuff like Dokku at work. I dunno—it's another thing that can and does break. If you're just self-hosting a few services and aren't trying to coordinate the work of several developers, IMO it's simpler and not-slower to just use Docker directly.
Would love to hear more about how Dokku broke as it will help me polish the project further :)
It was plenty solid, but I’ve definitely seen it fall victim to operator error :-) I’d certainly consider using it again to support a team, under the right circumstances.
Due to my love to shiny things, I keep wanting to find an excuse to move to a PaaS but I can never find a sufficient justification.
I find this true of pretty much all modern cloud development. So many people want to just pretend it’s another VPS alternative and are shocked that their misuse leads to high monthly bills. You need different patterns to take advantage of the strengths of PaaS cloud services and to avoid the weaknesses and gotchas. I PaaS all of the things that I can and my maintenance and support efforts have never been lower. My services scale to zero and I only pay for actual utilization.
If you’re standing up a lot of VMs in the cloud and pretending it’s just another data center of course you’re going to have a bad time and waste money.
I've always considered self-hosting to mean I'm managing hardware, but its clear the author here sees it more as a self-managed OS and infrastructure.
It actually feels very similar to the whole MPA vs SPA debate in web development. Maybe I'm just getting old, but self-hosting and SPA have specific meanings I learned years ago that seem to be getting redefined now rather than coming up with new names.
If you wanted to communicate that you're dealing with hardware I would imagine you would say co-locating or talk about your datacenter.
If that is the meaning now that is a more modern meaning, in my experience VPS was not part of a self-hosting concept before cloud providers were so common. At that time the options were your own hardware or a rented VPS, they couldn't both be self-hosting. Today that's less clear, hence my question of what is the common meaning today.
https://en.wikipedia.org/wiki/Self-hosting_(web_services)
> The practice of self-hosting web services became more feasible with the development of cloud computing and virtualization technologies, which enabled users to run their own servers on remote hardware or virtual machines.
If you buy their definition, it seems inline with the idea that self hosting is more about the software than the hardware these days.
I was a little surprised by the article but maybe it is just a redefinition over time as the industry changes.
[0] https://en.wikipedia.org/wiki/Self-hosting_(web_services)
The original difference was between using a service like WordPress vs. running an instance of the WordPress software yourself. Who owns the hardware it’s running on or where it is located is largely irrelevant for the definition.
Here is another reference: https://www.computerhope.com/jargon/s/self-hosting.htm
Self hosting is your hardware in your house/property with your software.
but times change i guess. I do understand why that definition would change. but I feel we should maybe name it slightly differently.
Lately I've been thinking of creating a bare-bones HTML website of my own and maybe I'll run it on a Raspberry Pi at home. I think that would qualify as 'self-hosting'.
> Self-hosting has become more reliable
Docker and Kubernetes really are the two best things happened to me in my selfhosting journey. They made billion dollar enterprise grade tech approachable in my homelab.
- I powered on a brand new mini PC, 10 minutes later it showed up as a node in my cluster and started running some containers
- Two servers died but I didn't notice until a month later, because everything just kept working.
- Some database file got corrupted but the cluster automatically fixed it from a replica on another node.
- I almost completely forget how to manage certs with letsencrypt, because the system will never miss the renewal window unless the very last server in my lab goes down.
I can’t count the number of times I’ve been frustrated at my current job because I have to wade through layers of ECS bullshit to do something.
All that said, if you don’t need containers, turns out you can get a lot done with two servers behind HAProxy + keepalived, running stuff with systemd.
I've been pretty sour on Kubernetes, but this looks like it could bring some of the niceties of PaaS to k8s. I haven't looked into the deployment process yet though, perhaps that's where all the pain lies
But I just learnt Kubernetes and suddenly all those implementations become interchangeable and my knowledge is transferable between homelab and work, and even across companies and platforms.
I’ve been running my personal cluster happily on k8s for years.
The other win is that there's a substantial cultural base to this way to go. Folks have been doing selfhosting for ages, but everyone has their own boutique setup some their way. A couple tools and techniques could be shared, but mostly everyone took blank slate configs & built their own system up, & added their own monitoring & operational scripts.
https://github.com/onedr0p/home-ops is a set of helm scripts and other tools that is widely widely used, and there's a lot more like it. It's a huge build out, using convention and a common platform to enable portable knowledge & sharing.
Self hosting did not have intellectual scale out at it's back, before Kubernetes came along. Docker and ansible and others have been around, but theres never been remotely the success there has been today in empowering users to setup & run complex services.
We really have clawed out of the server-hugging jungle &started building some villages. It's wonderful to see.
As a result, the software I run in my homelab for free, is the same software battle-tested in various enterprise environments, from 5-person startups to planet-scale mega corps. There are paid engineers and companies out there making serious long term commitment to it. This is truly amazing.
And I felt like we have had so many promises from various bits of software & professional teams about how they will heal and help us. But it hasn't planned out. The Mesos of the world, the OpenStacks of the world: they all have made similar claims to what you are saying, but something is so different this time. It's working. It's adding up.
Folks absolutely try to throw the "complexity" of it under the bus, but there's something incredibly simple about the pattern here of using apiserver to state your desires, and letting various controllers/operators work it all out, keep it going. It's a simple, repeatable pattern.
And the consistency of it all allows for systems like Helm to emerge. Not even great, but ok, adequate and demonstrating how portable the consistent platform of Kubernetes really is; it just works.
Something is different this time, even though so much sounds the same as previous dances around. It sounds complex, but many many people can get more onboard, the scope is more right sized to fit all the concerns, and the way that asking for things is just making a manifest is so much more accessible than before.
What is different this time? Why is this working this time? Why are there such more vibrant communities?
2 cores and 4GB ram is the recommended specs to run the 2008 game Crysis. It's hard to imagine that moving JSON around (which is probably 95% of services) is more demanding than Crysis. As I've begun really paying attention to the services being deployed you can actually get a lot of mileage out of the free tier especially now that they support running native binaries.
I host on a skylake about 7 years old I think, with 12 GB RAM. About 30 docker images running Home Assistant, Bitwarden, the *arr suite, jellyfin, minecraft ... Nothing fancy and I have heaps of free CPU and RAM.
I understand that one can easily load CPUs with compute processing, or RAM with video transformation but for a generic self-hoster of typical services I am surprised by the typical setup people have (which is great for them, I am just curious)
I get triggered by it because I get people in my company coming to me we should switch to cloud - but we are in cloud only that it is IaaS.
That said, I barely need to maintain most of my home infrastructure. I have CI/CD scripts do the bulk of maintainership for me these days.
After all, can you really write a good application without knowledge of how it will be deployed, and the challenges users face deploying things on specific platforms (eg. Windows, baremetal linux, kubernetes, docker, etc)? I would argue that you'd often write naive applications that gimp itself in unexpected ways depending on how the user intends to use it, without that knowledge. Depending on what types of applications you tend to write, this might be less of a valuable point to you. For example a static site web dev probably wouldn't be as interested in the infrastructure, they just need a server that can bind on ports 80/443. But I see a lot of incredibly naive applications written by potentially naive software developers out there.
Define “knowledge” you’re referring to. I have checklists for my ops. Define “really” and “good”. That sounds like perfectionist maxims without ground. Your listing of deployment targets can be extended by any BSD system, Android, iOS, IoT SoCs—what’s the point? If that’s your business requirement then you do it, if not then why bother?
If you think you can do both and create a “really good” app, or even product, well then you’re likely a genius. It depends on the depth, breadth and consistency of that “knowledge”—some people simply overestimate their capacity while others underestimate their potential. By poking around in Kubernetes is it deep enough for production or only broad enough to impress someone who’s deeper yet not broader? By reading the docs and textbooks and learning from people, would that be enough for production?
I said I hoped to forget that ops stuff because I have other duties to attend to and there’s too much variance and unreliability in those ops tasks, you need to make a context switch, hence IaC, hence managed, so you as a non-operator can focus on your tasks. Infrastructure and platforms are commodity goods now.
Many enterprises and companies use Kubernetes as at least _one_ of their deployment targets. So if you're a provider of software that has paying companies consuming it, does it matter if kubernetes is "deep enough for production"? It's being used in production, so I would argue, yes, from the perspective of a software provider, kubernetes is in fact deep enough for production. Though to one of your other points, define deep enough, because that statement seems like a moving target depending on who you talk to. But in general, if a piece of software doesn't work well in kubernetes, it's because the authors of that software hadn't considered any setup other than running as a monolith systemd service with an sqlite db for state. That will move me into the next quote:
> Define “knowledge” you’re referring to. I have checklists for my ops. Define “really” and “good”. That sounds like perfectionist maxims without ground. Your listing of deployment targets can be extended by any BSD system, Android, iOS, IoT SoCs—what’s the point? If that’s your business requirement then you do it, if not then why bother?
Knowledge is a fairly generic term, isn't it. It's a placeholder word for a series (or bucket) of facts about one or several domains. "Really" is a nuanced word that means different things depending on how it's used. For example, in my sentence fragment, "can you really write," I'm using the word 'really' to express disbelief in one's ability to perform the context. On the word "good" we can use context clues to extrapolate that I probably am using the word "good" to mean one that supports a wide variety of deployment targets with ease because you often do not know how your product consumer's infrastructure looks, but you can probably guess that if its intended deployment target is a linux server, these days, it should probably also be well supported in Docker and Kubernetes. So that is "good." There are a bucket of other concepts that make a piece of software "good" as well that I'll merely touch on for the sake of brevity, such as being well tested, generally lacking unchecked exceptions, runtime panics, is well performant when ran for long periods of time, etc.
Though, to be honest, I think you probably could have used context clues to come up with rough definitions of those words yourself, and I suppose your purpose was to glean what I think those words mean. In my opinion, the exercise seemed a bit more tedious than maintaining infrastructure though.
Well, yeah, I’d rather not waste my time. Time is money.
Anyway, this is a pick your poison scenario, probably. So, don't let me try to tell you one way is the best way.
It has probably contributed zero directly to my software engineering career, however, there are moments where a deep understanding of the quicksand under my app foundations can help, and shortcut strange debugging sessions and the like.
The time I spend is recreational for me. I can see where others, particularly Ops professionals, are horrified at the idea of doing lib and OS maintenance/updating for fun. It's a very yuckable yum :D
“Oh, I remember doing this a year ago in my homelab… it was a mistake.”
Otherwise, that time is his free time and he can do what he likes.