I got everything off the cloud and am paying less
twitter.com
twitter.com
I'm all for monoliths and hosted solutions, but let's not pretend that saving $1k / month is going to make or break most businesses.
What OP appears to have done is take on risk to save costs.
Another thing is, even when the cloud goes down, the story is "Global outages caused by Azure/AWS/GCloud/etc going down", and people are generally understanding.
If you have an outage of your machine, the story is "<Your Company> services down".
Are you meaning people using AWS/Azure/GCP/etc don't need to take care of these?
A lot of people here have fallen for the cloud marketing efforts. Nobody is saving any time.
If you were paying 15k a year to put a portfolio site online, then I can see where it can be too much.
I'm a sysadmin and have run schools on bare metal and on cloud. There are things like drive failures, hardware failures, connectivity issues, networking, user access, and all of those things that are much easier in the cloud. Much easier to spin up or down if you no longer need them.
If you're running networked storage, then that needs to have both a fail-over node and backup solution. You probably need to run extra hardware, cabling, etc. Cloud trivializes this.
If you're saving 15k in terms of hard cost, but spending 55k/yr for a sysadmin on-prem to maintain, then you're not saving money at all.
A disaster can be as simple as a failed disk, or overheated server because you're not a sysadmin and you put the server in a cabinet (I've run into this, dealing with faculty). Dead computer, no lessons for the week - lost time for the school and the professor in question.
Every place is different; you have to do a cost analysis and it's not as simple as "I saved 10x!"
All the time I think I'm reading apples and then realize people were talking bananas.
It takes 5 minutes to set up a Docker Swarm Mode cluster. It takes maybe 15 minutes for k3s or microk8s. After that, auto-scaling is dead simple, and no MORE complex than some shitty vendor locked-in cloud solution.
> that needs to have both a fail-over node and backup solution
ZFS pool replication, Ceph, GlusterFS, etc. Lots of options here. These are long-solved problems.
> A disaster can be as simple as a failed disk
Right, which is why you design your on-prem cluster with N+2 redundancy in the first place, and with a locked cabinet with spare parts. Cattle not pets, and all that. Do you think your EBS storage never fails? You'd need to do exactly the same thing in the cloud, anyway.
> but spending 55k/yr for a sysadmin on-prem to maintain
First, if you're paying only 55k for sysadmins, you should be planning to fail anyway. Competence is compensated quite a bit north of there.
Second, assuming the context is small business, you're going to have role crossover anyway, it's inevitable - chances are that your developer(s) is(are) administering this. Not every business is Facebook.
why not go cloud when the disaster happens while fixing your disaster, then close down cloud when disaster fixed?
True! It's also true that if you control your own servers, then you can fix the problem yourself instead of waiting until some provider somewhere gets around to it.
If you've got some spare time and the skills, alright. This can be a great way to save money. If you're especially confident in your ability to make this work and your margins are very thin, this might even be a wise move. You really can save a lot.
But what if you get slammed with traffic? Can you scale in any direction? Do you have a means of balancing loads with your $1000 server? How will you ensure it's secure? What if it lights on fire (figuratively/literally)?
The cloud does some useful stuff. You shouldn't always pay with your limbs for that, but in some cases, that stability, redundancy, scalability, and flexibility is worth every penny.
So no email, no important documents etc. live on my self-hosted services.
If the motherboard on my Unraid server shorted out right now, I'd be slightly inconvenienced, but I wouldn't lose sleep over it. Everything actually important is (also) on a service I pay monthly for.
Most of the truly skilled sysadmins I know recommend burst-to-the-cloud rather than pure cloud.
Yes, it's cheaper when everything is going well, it always is. That's not why we pay through the nose for AWS. We pay because Someone Else will be running around the colo facility swapping hardware and debugging networking issues if there's a problem.
Hiring a 24/7/365 redundant rotation of people doing that is a LOT more than $1000/month...
With many companies, the realistic cost of moving off cloud in labor would be higher than years of cumulative bills for cloud hosting. Even if you don't value your own time, you might want to consider doing something more valuable with it than devops.
That is silly for two reasons:
1. Cloud doesn't save you from outages
2. Outages happen, and are usually very rare, whatever your host
That said, I have fewer outages on my machines than AWS has on theirs. I run a service for a client, on one server, that has had 100% uptime during workdays for 7 years.
It gets even better (or worse, depending on your view point) if you factor in services you're dependent on. In my career I probably had more outages due external partners running their stuff on AWS than I ever did because our own servers where down.
That's not to say that AWS is bad, but it take a very skilled administrator to do AWS correct, and it costs a lot of money to get the required redundancy. The whole "I can't vacuum because US-EAST-1 is down" is because someone didn't want to pay what it costs to do redundancy in AWS.
Seems a little pointless to pay a premium on everything else then.
Add to this, that you can have outages due to operational complexity, which tends to happen in complex cloud native setups.
I've had far less downtime (read: zero) from my custom power redundancy solutions on my homelab than from power issues with my colo provider.
Small companies would be far better off with local power redundancy than some gargantuan complex industrial Liebert solution with a 4 hour SLA.
Yes. Have you attributed a cost to these stories?
Downtime is rare no matter what kind of infra you have. When you get downtime, fix it and the business will continue. In my experience people way over estimate on many services how bad an outage is. People aren't going to your competitor because of a single outage. Likely 98% of your customers might not even notice it. If you fix it and communicate honestly with your customers, the amount of money you lose is minimal.
Okay? Did you factor into that not needing the "cloud administrator", or whatever it is called now?
When your cloud goes down because payment got blocked, or your account got suspended for suspicious activity (pressing F5 in browser) or something similar, do you still get understanding?
If you are running something big enough and business critical enough and you roll your own custom _anything_, even if you save a gazillion, you are a hero for the next quarter. But if your custom solution goes fubar you are packing your boxes. The effort/time/cost of managing/configuring cloud anything is negligible if it allows people in companies where Software is a cost centre to deflect liability if something goes fubar.
It's pretty much the revamped 21st century version of the 80's meme "nobody got fired for buying IBM". Nobody will get fired at big corp for going all in on Azure or AWS. Look at CrowdStrike's cluster fuck, there were zero heads rolled for using the cloud services impacted by it. The very opposite would be true if it happened on bare metal servers. Google, Amazon, Microsoft, carry a 'too big to go wrong' halo with them (at least in public discourse) and that just sells.
If s3 goes down, and therefore your service is down, there's little you can do about it whilst your customers are screaming at you.
If it’s self hosted and it breaks you are in an IT firefight. If it’s a cloud SaaS product and it breaks you can twiddle your thumbs and yell at someone else and credibly shirk responsibility.
My point is that this is a bad culture for the company. There's no point playing internal politics and finger-pointing if your revenue is walking out the door.
I still believe running your own servers is cheaper but there is a tun of logistics you don’t want to really deal with.
Our "platform" team was 6 people, two network engineers and four "devops" (Sysadmins that could write code, but were still called SysAdmins then).
We had two datacenters, one was SAVVIS in Winnersh Triangle, the other was in Santa-Clara USA.
We used remote hands- rarely- in the event that a cable died or a harddisk needed to be replaced, though such events are exceptionally rare even at the scale we had.
Mostly we would collect up failures and visit one day every 3 months.
And there’s a huge range of sizes between “hilariously small” and “needs a whole data center”.
This conversation always seems to assume everything is and needs to be Google-scale.
In both the Twitter thread and the HN comments people seem to think VPS in no longer "cloud". IMO you are not "off the cloud" unless you own the hardware. A VPS is not that much different than an EC2 instance.
So in reality you just converted from cattle to pets[1] while just being on a different type of cloud.
And once you get into actual colocation you have to buy the hardware, have redundancy, and the colocation facility is probably charging you quite a bit (and perhaps even extra for bandwidth and power consumption).
[1] "cattle" refers to a disposable and easily replaceable resource, like virtual machines or containers, while "pets" are unique, carefully tended servers that are individually managed and not easily replaceable.
It seems to me that at this point, being an AWS customer is about as difficult as running a server manually. It's just different skills. The risk of running your own servers is certain kinds of downtime may be harder to fix (though if you have some "sysadmin savvy", ie, backups, it shouldn't be much), and the risk of being an AWS customer is accidentally paying 10x what you should.
The 'one stop shop' idea describes it pretty well https://www.investopedia.com/terms/o/onestopshop.asp
I always thought that all kind of servers in datacenters could be defined as cloud server.
The idea was, I think, related to network diagrams that showed everything outside of your own border as a cloud icon. (IE; very foggy and unknown).
It never really made much sense, but normally what seems to separate a MSP/service or colocation is the availability of managed object storage and an abstraction on the network.
However, now it seems to have been coopted and now means basically anything that isn't in your closet. However we ran many physical machines we paid for in colocation facilities, and when AWS was coming up there was an understanding that "that's not what the cloud is" among the masses. Despite managing them over the internet. So, a fuzzy term has become more fuzzy.
So much for us being "engineers".
For me control is the number one issue with computers today. Not cost, not features, not convenience or anything else. "Cloud" computing has been defined as "someone else's computer". Bloggers and other pundits have also cited a "war on general purpose computing". These complaints, and others, to me, are summarised by the issue of control.
When dealing with numbers like 15k/year... it's implied that we're not dealing with a full staff.
Maybe for a bunch of machines in a closet. But modern dedicated servers and similar are pretty easy to maintain. ;)
It’s been several years since I’ve run a business, but I had moved from AWS to a colo ESXi setup. I saved a bunch of money and got a better product. My costs were fixed, the hardware was better, the VM sizes more configurable, and I had high availability. I wasn’t spending anywhere near 10h / week to keep it running. Linux and most daemons are pretty robust.
I’d gladly acquire and exercise general sysadmin knowledge over some vendor-specific thing. AWS may run fine if you have a light load, but their API docs can lie, they have hidden constraints, and many apps end up using the wrong service or architecture and need to go through costly migrations. It’s hardly set it and forget it. There’s a ton of services and constraints to learn. There’s time involved in learning the tooling, the API, and the invariable bugs integrating with the devops tool of the day.
Then you need to factor in the time spent trying to speed up your app to compensate for the slow cloud machine you’re running on. You can choose beefier machine types, but you’ll pay through the nose. Performance can be wildly inconsistent if you’re on an oversubscribed host. So, you can get a cloud bare metal machine, but that’s even more money and good luck getting the spend approved. To get out of that hole teams will often layer in brittle caching that would be otherwise unnecessary.
These costs rarely get mentioned when we talk about cloud vs bare metal. That was a substantial time sink for the startups I was at. Large companies have whole teams dedicated to devops, which isn’t far off from having a traditional ops team.
This setup seems quite a maintenance hell. Let’s see what happens to his SLAs.
The hidden cost is the time he spends.
However, if running computers are your business, spending time on getting money is called having a job.
Everyone seems to think that managing AWS is free.
- database (postgres) - redis cache - webserver
How to exit the cloud (aka run everything in VPS) if you also need this?
- reliable and scalable block storage - queue / task management - trusted email-sending provider
Not questioning the trend, just seriously interested in finding solutions to this.
Like the of the AWS deployed sites.
i've been running my tiny business on the same provider as op for more than a year, and i've done very little maintenance to keep things up and running. granted, you need to set things up the first time, but after that, it just keeps chugging along.
https://logical.li/blog/emperors-new-clouds/
I wonder, though, why the topic is so contentious. This is supposed to be an engineering discipline, not a fight between religions.
if you're going to run your own servers with same level of support it will definitely be much higher than $120/month
but this is, alas, not the tech issue, but a more low level understanding the reasoning of why cloud is more interesting for certain use cases, and on prem better for other.
it is definitely not a one solution fits all
edit: https://rameerez.com/how-i-exited-the-cloud/
oh wait, he did not get anything "off the cloud" he is still running on a "cloud" based solution, just a different company.
LOL
Now a company with thousands of server should probably leverage big cloud offerings, but when we see a post that someone is able to save 10x by going bare metal, and they don't seem to be running a large operation, we should celebrate that, rather than parrot that it is a bad idea, oh god what have you done, no one will ever be able to run servers on their own, it's not worth it.
For comparison, I manage half a dozen servers for clients and I literally do 1 hour of maintenance a year.
It paid itself back in 3 months if I'd used AWS services.
Would you run your prod inference cluster the same way?
People seriously don't realize just how _big_ computers have gotten. The AWS mentality is still stuck in 2005 when a t2.micro was a decent computer.
The T2 EC2 instance family got announced in 2014: https://aws.amazon.com/blogs/startups/announcing-amazon-ec2-...
https://aws.amazon.com/blogs/aws/ec2-instance-history/
https://instances.vantage.sh/aws/ec2/t2.micro
https://instances.vantage.sh/aws/ec2/m1.small
Please nitpick better. Nothing is more embarrassing then being pedantic and wrong.
Also if you have everything ready to push to a cloud provider in the case of the server going down, someone like Vercel or Netlify (easier to maintain than AWS in my experience) then I guess you are pretty setup.
If anything, you could argue the tweet is propaganda, because the labels for the AWS bar chart is missing. (Not that I think that's what's happening).
Hetzner nodes aren't actually 10x cheaper than AWS instances, so there's stuff like bandwidth usage going on.
My point being there's no silver bullet, sometimes cloud makes sense and sometimes it doesn't.
It is not only the hardware cost that saves you money, but also the fact that if you rent dedicated servers you tend to develop your infra in a different way which is ultimately way more cost effective.
Dedicated servers definitely aren't a silver bullet, nobody is saying that. Managing your own servers requires work - how much, that is arguable and depends on many factors. To me the problem is more that AWS and cloud companies are selling a very expensive silver bullet, which really isn't even a silver bullet. In my experience most of the time it doesn't make sense to buy that.
> develop your infra in a different way
Totally agree, but for many people that ship has sailed.
They do if you enter an agreement with them for professional services; I'm not certain but I think there's an SLA also on their "managed" servers: https://www.hetzner.com/managed-server/
However, if you want a "no-human" SLA with bare-metal rented servers, I could recommend Gcore: https://gcore.com/hosting/dedicated
Their managed kubernetes even supports bare-metal nodes, which is actually something I'm using.
I don't want bare metal nodes, they put a high lower bound on the cost of a distributed cluster. I want a "herd" distributed across data centers, and a system that spins up new ones if the old ones die. I don't care if AWS shuts down an ec2 instance without warning, because a new one will automatically replace it.
The issue is not that people disagree, it's that:
1) People use surface level arguments that have been debunked a few times (SLA, backups, reduction in headcount)
2) There is an aggressively high frequency of these arguments.
If you couple that with the marketing spend of major cloud providers; it's easy to paint a picture of at least a small contingent of people who's livelihoods depend on keeping cloud growing; combined with a contingent of people who have skilled into a cloud (sunk-cost fallacy) and don't want to reskill and you have a decent self-reinforcing propaganda machine.
It likely helps things that there's nobody except a tiny plurality of nerds who really care to refute the marketing claims. There's not many colocation or hosting companies with even 1% of the marketing spend of even the third largest hyperscaler (google).
I wrote about the three major reasons you'd want to use a cloud provider here: https://blog.dijit.sh/gcp-the-only-good-cloud/
If I take the Hetzner AX102 (16 cores, 128GB ECC RAM, 2x1.92 TB NVMe SSD) that costs about $0.2 per hour. An EC2 instance (on demand) with 32 vCPUs starts at around $1.2 per hour, so 6 times as expensive. And this doesn't include storage which would be something like an additional $0.3 per hour for AWS.
Of course this isn't a fair comparison. Using on demand instances is somewhat unfair to AWS, but reserved instances also wouldn't be entirely fair as you're only locked in for a month with Hetzner (with a setup fee for most server) and not 1-3 years. I'm guessing with the CPUs here and assume that 2 vCPUs are roughly one real core. And the storage is not comparable at all.
You do get a lot of other things with AWS, but you usually also pay for those. Which can certainly be worth it. But I am slightly tired of people arguing that you save money with the cloud because you can scale down automatically. And this argument doesn't really work in many cases if you get that much more hardware for your money.
Remember the age old saying: Time is money.
If you run your own servers, you are spending your own time and thus money to manage them.
If you buy services from hosting ("cloud") providers, you spend money to have someone else manage them.
The question then becomes: Do you value your time or your money more? And then you pick the one that spends the one you value less.
Some people value their time, some people value their money. YMMV.
Running a server by yourself isn't more time consuming than having AWS do it for you. The issues that cause very long downtimes are flooding of the server room, subpar connectivity, low quality hardware, CPU is on fire.
Just do not host your production services in your basement. Any VPS saves you from these, at 1/10th the AWS cost. This is obvious to anyone that has the basic skills to run a Linux host.
The fact that it needs to be spelled out means that these days few have the skills and/or the others have been convinced that it is "a waste of time" and "makes only sense if your time is worthless" which is total FUD.
Sure, buying a VPS or whatever from a hosting provider might be cheaper than buying a full blown cloud solution, but at what personal time cost?
Time is a finite resource, much more than most of us ever realize. Worse, time is a resource that cannot be replenished. Money, on the other hand, is a replenishable and theoretically infinite resource. Sometimes, paying big bucks for premium services is the cheaper solution to moving on in life.
No, these are people who don't care about setting up their own servers.
Similarly to how I just bought a NAS unit from Synology and just use Tailscale because I don't care about assembling a homebrew NAS or setting up my own VPN infrastructure, most people don't care about setting up their own servers.
Why do we not care? Because time is finite and we have more important shit we need or want to attend to.
I mean, I admin my own servers. But I learned Linux as a kid when time was cheap. If you didn't have that experience then yeah it may make sense to just use as much cloud as possible. Sysadmin is just unpleasant even when you do know how to do it.
Time is finite and there are more things in this world than we can ever afford to care in a single lifetime, not caring is a matter of budgeting that finite time so we can live our lives in a fruitful manner.
Wrong, wrong, wrong. You spend a similar amount of time, but on different knowledge and tasks. And I'm saying I find my skillset more enjoyable and more useful compared to the vendor lock-in and Tower of cloud services of Babylon that is AWS/Azure/Google.
It's a truism that you can do stuff a lot cheaper yourself if you have these skills than paying a cloud to do it for you, but cloud services have grown like crazy for over a decade now and show no sign of slowing down. UNIX is fundamentally a user-hostile operating system, it will never change, and it's nice for those of us who learned it that we can sometimes convert our rapidly-obsoleting skills into savings (sometimes), but it doesn't seem likely to stay that way. The Linux vendors just haven't improved the usability of their platform at anywhere near a fast enough pace to keep up with the cloud vendors.
What I have witnessed is actually the opposite; as mentioned elsewhere in this thread I was on a team of 6 sysadmins (who knew perl/python and C as a requirement for employment, so more like "devops" but with worse tools): and we ran 1% of web traffic on a few hundred physical machines.
However, Ubisoft has 5% of it's total headcount as IT staff, my last job has roughly a similar number of staff for a multiplayer game as I had for a huge SaaS web-store which even ran it's own payment systems (IE; before Stripe existed) and had regulatory compliance issues to take care of. -- and Ubisoft is using AWS/Azure/GCP and has moved away from bare-metal hosting, and my last job only ever ran on cloud providers, as it was du-jour when they started the company.
I have yet to see any evidence that it actually saves time. I think it's weaponised short term thinking, because going from the console to a running system indeed takes less time, but the tinkering with CDK/Terraform/IAM/VPC and trying to architect your solution to be cost effective and reliable is taking at least an equivelant amount of time it seems.
Statistics don't lie here, if "devops" and "SRE" is the new sysadmin then headcounts have not fallen. Operations staff are in as much (or even more) demand as they were in 2010- We just made operations more complex and vendor specific.
Related to this observation you really should watch DHHs keynote talk at Rails World 2024 from about minute 20 to 30: https://youtu.be/-cEn_83zRFw?t=1296
He addresses exactly this phenomenon.
EDIT: changed URL to include timestamp to the start of the relevant content.
How to scare a computer engineer in 2024? Invite him to configure a server.
Of course HN is a site frequented by people who are wont to start those same service companies so it is not surprising for there to be a tendency to play up the 'risks' of doing things yourself instead of 'leaving it to the experts'.