When resources are scarce, interesting stuff happens. Nowadays most outages are either due to expired payment, disk full, or provider outage. That's a good thing I guess? Just much less fun.
When resources are scarce, interesting stuff happens. Nowadays most outages are either due to expired payment, disk full, or provider outage. That's a good thing I guess? Just much less fun.
It seems nearly everyone goes through a “How hard can it be?” phase where self-hosting seems like it will be a trivial way to save a lot of money. Then you get to learn about all of the things that can, and will, go wrong over time. Great learning experience.
And then there are the rose-colored glasses people who seem to forget all of the troubles they go through with self-hosting. A few companies back we had a guy who insisted on self-hosting some things on premises to save on SaaS costs. The amount of time we spent waiting for him to figure out why the server was misbehaving each week easily cost several times more than any hypothetical savings. Yet when you ask him, he’d claim it was virtually hassle-free. Management liked it because it “saved a lot of money”… as long as you ignore the cost of the time we lost to dealing with it.
I wonder how much we spend babysitting slop (either directly or indirectly through SaaS) vs how much it would cost to invest in engineering around automated and reliable deployment and resilient software?
Highly polished distributed file systems, databases, and orchestration tools that incorporated automatic replication and fail over would be a great start.
Don’t just say “it’s hard.” Of course it is. But is it cheaper to do this or to babysit rickety piles of junk?
Another possible way out is AI sysadmins. I wonder how far we are from AIs that can admin a cluster including upgrades and disaster recovery?
I guess the reason it doesn't make sense is that you're living in your own imagination where nothing is obligated to be sensical!
The current boot image (vmlinuz) on my debian system is 6MB and the initrd is 40MB.
> 11M Apr 6 12:14 vmlinuz-6.1.19-gentoo
> 4.6M Apr 6 12:14 System.map-6.1.19-gentoo
And that's a full desktop QEMU "server" - and i don't use initrd, because i don't need every driver in the tree, just the ones that work for my system. No AI needed. You boot a "kitchen sink" kernel, lspci -k, note down what it says, then just enable those things in the kernel and build it.
Even ubuntu being "quite supportive" is 30-45MB.
The kernel source tree is over a gigabyte, but that has to support everything it supports - arm, risc-v, x86_64, etc.
> 2.2G linux-6.1.19-gentoo/
I mean, no other industry insists that users accept weekly automated updates because their products have so many critical security bugs.
bean counters tend to not evaluate your time as lost money. they are paying you anyways, so sunk cost fallacy or something. as long as they are not having to pay an invoice, they are "saving" money
SaaS mostly exists to extract profits perpetually.
This doesn't mean it's not sometimes worth it, but when you add all the licenses together, there tends to be considerable savings if you're willing to self-host.
In reality both sides of the debate are often wrong.
The $10 option was genuine though, it was for some SaaS tool to monitor endpoints and text if things broke (about 9 years ago when I had basically zero coding experience) - so just one seat, or me trying to hack something together with text/email notifications and a nice UI. Even now I'd probably still prefer using something like that over writing something in-house, although I'd be more happy to look at self-hosting an open source option.
we spent more time trying to make an external tool
do something that could have been built in house in
the same or less time.
Yeah. :-/There's also a version of the "nobody ever got fired for choosing IBM" fallacy/truism at work, right? If management picks $WELL_KNOWN_PRODUCT_XYZ and things go sideways... well, they don't look foolish. But if they rely on... some random in-house engineer and things don't work out, they look like fools.
(Even though, in both cases, it actually just comes down to the same in-house engineers. Because the in-house engineers still need to do the integration work)
No decent engineer would stay there because there was nothing to build and management would always just contract out everything. Management, which was promoted internally from bad engineers, couldn't shop a good SaaS product or consulting firm and instead were consistently convinced into buying shitty Oracle products.
In-house products were badly done, which led to contracting out engineering, which led to no engineers, which led to getting screwed in the contracting.
Fixing that company would probably involve firing the whole tech department.
Maybe with the exception of Auth? Having some external SaaS startup be in control of your user database can be a bit nerve wracking. Especially if they get acquired by someone terrible.
I'm personally glad that Auth0 wasn't acquired by Oracle. What a nightmare scenario that would be... :D
As a counterpoint to this, at my last house in Melbourne (Australia) I was thinking of doing the same. We have a "National Broadband Network" (NBN) deployed that most residential customers in the country are hooked into.
Shortly after moving into the house, the NBN needed to upgrade the network in the area. So, multi-hour outage. Then the same thing happened about 2 months later.
Thus, any idea of self hosting from my house was dropped and we've kept on using Hetzner. ;)
It works well enough.
I’m not going to argue with your points, but we’ve spent maybe a months worth of man hours figuring out why some SaaS service wasn’t working this year. Only for it to be because some Azure or Office365 or Sharepoint online service wasn’t healthy, and, to be fair, because some of our systems aren’t build very maintainable so that it takes a ridiculous amount of time to figure out what is wrong. Still, it would’ve probably cost us maybe a work-weeks worth of time if we were more competent, which sort of goes against the SaaS idea when the licensing is also approaching the cost of self-hosting and hiring a Sharepoint person. Well, if a Sharepoint person existed of course, but I bet you get my point.
The house's floor slab had a crack that wicked up groundwater when it rained too much, so my carpet would get mushy. Electronics off!
The anti-cheat software I ran (PunkBusters) went crazy after an update and banned anybody who connected. I noticed when my MRTG graphs flatlined.
Girlfriend once yanked out the Ethernet cable when we were having adult time. I didn't bother explaining that one to the players.
Heavy web browsing would cause lag, so I ended up learning how to do QoS via iptables (or its precursor, whose name I forgot).
At the time, the server used an astonishing 25 gigabytes of bandwidth a month. Always raised an eyebrow with the ISP tech when I called in, but they never complained.
Thanks for reminding me I'm getting too old. :V
1200 bps Unix shell access from TENet (Texas Educators’ Network) - thank you, Mr. Horner, my 9th grade US History teacher, for telling that nerdy girl who turned in all her papers in dot matrix print about what she could get her mother, a fellow teacher, to pay all of $5/year for.
Oh, and daring to have us look at Reconstruction’s sudden cut-off, Jim Crow and the fact that we were less than 30 years from its legal end in Texas, the Bonus Army, our rather late entry into WW2, the Japanese-American internment camps, and various shenanigans to do with the Vietnam War - you taught us that it’s possible to both love your country and to accept that it is by no means perfect.
I hope you’re enjoying a good retirement.
I have tiny recollection of this this but thanks to your comment I became acquainted with it again.
Thanks!
I still have ADSL. It's now 5 Mbps, but I don't recall what it was when dial up was common.
Higher speeds of DSL came along with ADSL2 in 2002/2003. There were higher than 1.5mbps before that, but you basically had to be across the street from the CO to get that.
Do you get cell service? We don't and that's actually the bigger hassle. Wi-Fi calling is kind of shit, and the coverage outside the house is appallingly unreliable. iMessage is frustratingly unreliable too. I'll be on Wi-Fi, but it won't have gotten synched up with Verizon and it'll refuse to send a message until it does.
We do have cell service. We're in an older 1950s development that was never rewired, but not far from newer developments.
The biggest problem for me is that downloads keep getting bigger and bigger. The latest AAA games are almost 100 GB and take days to download.
And either way, even that coverage is spotty and speed varies highly on the time of day that you're trying to use it.
All this is fine for a tertiary backup solution (behind Starlink) but not as a primary method of accessing the Internet.
That was the default speed for a long time.
They had a message up for a while that said they were unavailable because their servers were either underwater, or on the back of a looter's truck that was itself underwater.
Wow, I haven't seen or heard that name since the 2000s. Good times.
So did my work computer;)
I wonder if people will experiment more?