Hacker News Reader

Comment by PragmaticPulp | Hacker News Reader

Learning the difficulties of self-hosting is a rite of passage for tech minded people.

It seems nearly everyone goes through a “How hard can it be?” phase where self-hosting seems like it will be a trivial way to save a lot of money. Then you get to learn about all of the things that can, and will, go wrong over time. Great learning experience.

And then there are the rose-colored glasses people who seem to forget all of the troubles they go through with self-hosting. A few companies back we had a guy who insisted on self-hosting some things on premises to save on SaaS costs. The amount of time we spent waiting for him to figure out why the server was misbehaving each week easily cost several times more than any hypothetical savings. Yet when you ask him, he’d claim it was virtually hassle-free. Management liked it because it “saved a lot of money”… as long as you ignore the cost of the time we lost to dealing with it.

View on HN

>Management liked it because it “saved a lot of money”… as long as you ignore the cost of the time we lost to dealing with it.

bean counters tend to not evaluate your time as lost money. they are paying you anyways, so sunk cost fallacy or something. as long as they are not having to pay an invoice, they are "saving" money

I’ve had the opposite experience in recent years: Excess management and limited engineers leads to a desire to do deals, make contracts, and buy external solutions for everything. In many cases the integration costs are assumed to be $0 and we spent more time trying to make an external tool do something that could have been built in house in the same or less time.

In my thankfully short time at a company that operated this way, there was no upside either.

No decent engineer would stay there because there was nothing to build and management would always just contract out everything. Management, which was promoted internally from bad engineers, couldn't shop a good SaaS product or consulting firm and instead were consistently convinced into buying shitty Oracle products.

In-house products were badly done, which led to contracting out engineering, which led to no engineers, which led to getting screwed in the contracting.

Fixing that company would probably involve firing the whole tech department.

    we spent more time trying to make an external tool 
    do something that could have been built in house in 
    the same or less time.

Yeah. :-/

There's also a version of the "nobody ever got fired for choosing IBM" fallacy/truism at work, right? If management picks $WELL_KNOWN_PRODUCT_XYZ and things go sideways... well, they don't look foolish. But if they rely on... some random in-house engineer and things don't work out, they look like fools.

(Even though, in both cases, it actually just comes down to the same in-house engineers. Because the in-house engineers still need to do the integration work)

And we can expect it to continue going that way for smaller companies, since the tax changes in the US wrt R&D means it's better to buy than build, all else being equal, due to the tax deduction not having to be spread over time.

It's the converse of Goodhart's law -- if it's not measured/itemized, it's not part of the objective function for optimizing (costs).

Absolutely. The number of times I’ve pointed out a paid tool for $10pm is cheaper than a days salary per month to build in-house and maintain…

10 in total or 10 per seat?

SaaS mostly exists to extract profits perpetually.

This doesn't mean it's not sometimes worth it, but when you add all the licenses together, there tends to be considerable savings if you're willing to self-host.

In reality both sides of the debate are often wrong.

The one I think about was when I was building a 3D model of a new office in Unity and the options were either purchasing a material or trying to design one myself with basically 0 experience of doing so. Something like $40 for the material or however long it'd take me to work it out.

The $10 option was genuine though, it was for some SaaS tool to monitor endpoints and text if things broke (about 9 years ago when I had basically zero coding experience) - so just one seat, or me trying to hack something together with text/email notifications and a nice UI. Even now I'd probably still prefer using something like that over writing something in-house, although I'd be more happy to look at self-hosting an open source option.

> The amount of time we spent waiting for him to figure out why the server was misbehaving each week easily cost several times more than any hypothetical savings.

I’m not going to argue with your points, but we’ve spent maybe a months worth of man hours figuring out why some SaaS service wasn’t working this year. Only for it to be because some Azure or Office365 or Sharepoint online service wasn’t healthy, and, to be fair, because some of our systems aren’t build very maintainable so that it takes a ridiculous amount of time to figure out what is wrong. Still, it would’ve probably cost us maybe a work-weeks worth of time if we were more competent, which sort of goes against the SaaS idea when the licensing is also approaching the cost of self-hosting and hiring a Sharepoint person. Well, if a Sharepoint person existed of course, but I bet you get my point.

We are a probably one of the small subset of companies that have moved things back in house and off of 365. We have a lot of data and the previous IT guy just lift and shifted everything to the cloud. There was a lot of problems so we moved it back to our own data center... Funny no one complains about things being slow anymore.

I dunno, my experience has been it's way more straightforward than faffing about with cloud services. It's not quite zero maintenance, but it's really not a significant portion of my time (and the cost savings are substantial!). And that's work. My personal self-hosting I don't think I've actually logged into for more than a year.

It's liberating to know that my emails can't be wholesale data mined or someone can't take down my web project if I commit some ToS thought crime. But doing it for clients was a ball-and-chain that I have no desire to do again. So for personal stuff it's great. For business stuff it's like adopting a pet that you can't leave alone too long. I would keep business stuff outsourced.

> I would keep business stuff outsourced.

Maybe with the exception of Auth? Having some external SaaS startup be in control of your user database can be a bit nerve wracking. Especially if they get acquired by someone terrible.

I'm personally glad that Auth0 wasn't acquired by Oracle. What a nightmare scenario that would be... :D

Where I always go with this is to ask why our software is so damn horrible that it requires this much babysitting. Self hosting could be trivial if our software wasn’t slop.

I wonder how much we spend babysitting slop (either directly or indirectly through SaaS) vs how much it would cost to invest in engineering around automated and reliable deployment and resilient software?

Highly polished distributed file systems, databases, and orchestration tools that incorporated automatic replication and fail over would be a great start.

Don’t just say “it’s hard.” Of course it is. But is it cheaper to do this or to babysit rickety piles of junk?

Another possible way out is AI sysadmins. I wonder how far we are from AIs that can admin a cluster including upgrades and disaster recovery?

I can't imagine having to unfuck the complicated systems set up by an AI sysadmin when the system running the AI goes down.

Understanding how a system works is so 2022. Just let the AI dig you in deeper.

Our software isn't horrible. Our software is pretty good, but reality is horribly unpredictable, complicated, and replete with edge cases.

Our software is kinda horrible.

I mean, no other industry insists that users accept weekly automated updates because their products have so many critical security bugs.

No other industry is capable of automating weekly security updates, basically by definition.

If "handling edge cases" is not something our software is specifically designed to do, then it's horrible.

I think AI should be used to cut down crufts. It just makes no sense to me that Linux Kernel boot images are, how much? A gigabyte?

> It just makes no sense to me that Linux Kernel boot images are, how much? A gigabyte?

I guess the reason it doesn't make sense is that you're living in your own imagination where nothing is obligated to be sensical!

The current boot image (vmlinuz) on my debian system is 6MB and the initrd is 40MB.

no.

> 11M Apr 6 12:14 vmlinuz-6.1.19-gentoo

> 4.6M Apr 6 12:14 System.map-6.1.19-gentoo

And that's a full desktop QEMU "server" - and i don't use initrd, because i don't need every driver in the tree, just the ones that work for my system. No AI needed. You boot a "kitchen sink" kernel, lspci -k, note down what it says, then just enable those things in the kernel and build it.

Even ubuntu being "quite supportive" is 30-45MB.

The kernel source tree is over a gigabyte, but that has to support everything it supports - arm, risc-v, x86_64, etc.

> 2.2G linux-6.1.19-gentoo/

I have recently ventured into home-hosting some non-critical services and it's awesome. I can get 5Gbps residential internet and a dedicated IP through AT&T fiber, I have Solar+PowerWall for uninterrupted power... Once I got through the first few weeks of Network manager and router problems, I've had zero downtime the past 6 months. Since our workloads are very SSD and bandwidth-heavy, both of which are absurdly expensive on public clouds, we hit breakeven to our cloud services in less than two months.

Yeah, it's good when your internet connection is really reliable.

As a counterpoint to this, at my last house in Melbourne (Australia) I was thinking of doing the same. We have a "National Broadband Network" (NBN) deployed that most residential customers in the country are hooked into.

Shortly after moving into the house, the NBN needed to upgrade the network in the area. So, multi-hour outage. Then the same thing happened about 2 months later.

Thus, any idea of self hosting from my house was dropped and we've kept on using Hetzner. ;)

I use a vps for some things (my blog thst no one reads, or my teamspeak server for me a couple buds to use) but I still self host my plex, miniflux etc from home.

It works well enough.

Yeah, for those purposes it'd be fine. I was more looking to run stuff which is fairly actively used (~1/2 million inbound connections per month) though doesn't need high bandwidth. For that, multi-hour outages isn't going to work. :/

Within the runtime environment spectrum's opposing ends self-hosted and SaaS, there exist several more choices, of which I consider dedicated servers at some hoster the sweet spot for most projects/businesses, big and small.

I had an epiphany recently about the "servers are cattle, not pets" wisdom: every home server is necessarily a pet. With all that that implies.

And don't forget, the issue is always DNS.

Did Not Service

Reply on news.ycombinator.com