Hacker News Reader

Comment by mabbo | Hacker News Reader

My goodness is it Monday already and time for yet another of these articles?

At a given scale and software maturity, it is cheaper to buy your own hardware and pay for your own team to operate it. Generally, this is when you start thinking of your hardware needs in terms of fractions of an entire data center, but in simple cases may be smaller.

This almost is not ever true for software startups, who get more value from speed, flexibility, and not needing to pay a team to manage hardware. Just let [cloud services company] handle it for you until you reach that point.

And then when you do reach that point, you get to write the article about how everyone using the cloud are fools because it's cheaper to self host.

View on HN

The cost differential in my experience is generally higher on the low end. Once you get into the hundreds of k per month in cloud fees you can get steep discounts, and some of the gap starts to close.

You also need devops resources to manage a cloud setup - most of my jobs in recent years have involved exactly that, and it's not appreciably cheaper than managing "raw" hardware resources these days (ever since IPMI etc. became the norm); for some time I did consulting and provided devops services on retainer. The amount of time spent managing actual hardware was a rounding error of the overall devops cost for those of my clients relying on colo's or managed servers.

Since many colo providers also provide managed servers and cloud services, the flexibility is also not really an issue - you can start with cloud services, rent managed servers as you know what your base load is like, and can tie them into the same setup, and do the same with colo services if/when it's cheaper (the tradeoff between managed and colo is trickier than with cloud - the cost differential there is often dominated by realestate costs where the relevant data centres are; if you're somewhere expensive, managed servers elsewhere will often be cheaper than colo near you)

I've heard many times that while retails AWS prices are high all big AWS customers are getting discounts. Just curious at which scale this starts in terms of $$.

Somewhere around $100k/mo or $1mm/year.

This usually requires a term spending commitment and gets you an effective discount of around 5-10%. Higher spends can yield significantly deeper discounts, of course. I think you'll find it difficult to get specifics at higher spends due to non-disclosures.

Google 'AWS EDP negotiated' and you'll find some more info on the process.

Numbers I heard involved a much steeper discount than that, but may well have been an account they saw as particularly important. If you can only get a 10% discount you're still paying massively above the odds.

If you’re getting a 5% discount on $1 million per month you’d need to host using a team of 1.5 people at SF salaries and benefits to break even.

If you were getting a 50% discount, you'd still not be competitive with hosting your own or using managed servers most of the time.

> You also need devops resources to manage a cloud setup

Don't you also need them in a colo?

Yes. But people use "you have to hire people to manage it" as an argument against on-perm. When in practice, most companies hire "devOps engineers" or "cloud engineers" to manage their cloud infrastructure too. And those salaries seem higher than the infrastructure/ops guys of yesteryear. So resource cost isn't really an advantage.

don't forget the "cloud cost engineer", just to figure out how much it will cost to run the 20 services that should have been part of your app with random ways of billing the usage of it

What nerdix said, basically. People assume most devops work when you own hardware is hardware related, but for a typical setup, if your hardware related devops work make up more than a few percent of your total devops cost something is wrong.

As soon as the machine boots over PXE and is on the network, it becomes software. Hardware actually becomes easier to manage as you get more identical machines as you can do differential analysis to figure out which components could possibly be out of spec.

I have seen multiple thousand host clusters managed by 1-2 people, and day to day ops looked like absolutely nothing was happening. This included messing with hardware racking and networking. The most resource intensive was just uncrating, moving and routing cables.

Managing hardware is bogey man, false toil.

> This almost is not ever true for software startups, who get more value from speed, flexibility, and not needing to pay a team to manage hardware.

Exactly. I cringe every time I look at the AWS/Azure running total, but at the same time I realize that paying a separate team would be even more ludicrous at our current scale.

Right now, I can single-handedly administer 100% of our IT infrastructure as barely a side-concern because nearly everything is inside the Azure/AWS bubble. Delegating IT management tasks to non-wizards is actually feasible too.

The moment you start racking servers on-prem, you need a very specialized workforce to keep them alive. I suppose you could vendor parts of this out too, but then you are really splitting hairs regarding why you brought crap on-site in the first place.

There's always the in-between. It's not a manage your own physical server or nothing scenario.

Software startup can use consumer grade workstations and self host at their office (or at home). Cloud at scale is VERY expensive. And if you do not need the scale, but reliability, it is easier to rent two servers at coloc.

> Software startup can use consumer grade workstations and self host at their office (or at home) (...) And if you do not need the scale, but reliability, it is easier to rent two servers at coloc.

Yeah, just don't forget the backups and monitoring, the networking setup (load balancing between your two servers or workstations at the very least), patching, moving to new servers/workstations when you outgrow the current ones (while also taking the risk of overprovisioning) etc.

Or just throw it all in a PaaS, ideally a container-based one (like Google Cloud Run or AWS Lambda or Scaleway Serverless Containers) where you won't have any lock-in and migrating away to something that makes more sense at your new scale would be easy. The only concern that would remain in that case is data backups to a different provider/place - scaling up/down is automatic, monitoring (at least the basics) are already covered, and there's no patching outside of your application's libraries.

It is a startup, not a space program. Backup can be done with simple ssh+rsync script. Patching is a weekly weekend cron script+reboot. Some heartbeat monitoring... If it goes down for a few hours, not a biggie.

For "throw it all in a PaaS" you need dedicated person, monitoring, billing etc... It is more expensive, not simpler. Cloud provider may also terminate your accounts, change settings etc...

If I had a nickel for every time that restores from "simple SSH+rysnc script" backups failed, I would not need my job as an infra engineer. You can do that, but being sure they work is hard

My clients spend a lot more on AWS bills than they would on self-hosted infra; $1000ish compared to $200/mo from self-hosted or a pair of $1000 servers colo'd for about $100/mo. They get a lot more - Reliability and support channels. That's what they're paying for. Absolutely, I recommend that they when they scale, they move their most expensive parts in terms of AWS costs (bandwidth-intensive things like gameservers and media bouncers) from AWS to self-hosted, because those are the things that are too expensive on AWS and have reliability properties where "Throw a bunch at the wall, fail them fast" is a good guarantee. For your production backups, don't rely on simple scripts. Don't rely on complex scripts. Rely on commonly tested solutions of other people's amortized experience.

SSH+Rsync is the wrong way to go about it, you will fail to restore backups. You need something that allows you to make snapshots. Fortunately that's easy and cheap if you use something like Proxmox.

It would be interesting to map out what kind of industries and startups require a load balancing and multiple production servers for the first few years of setting up. A single 32/64/128/256 core server is pretty decent for most work loads, and anycast services are decent in terms of static content delivery.

Was there a specific kind of startup that you had in mind for this?

Backups and monitoring are you responsibility anyway! You are making a distinction where there is none.

Cloud is cheap when starting. Saves time

Try to run LLaMA in cloud...

VPNs and hybrid architectures exist for a reason. If 99% of your IT infra is boring crap but you have this one special unicorn machine, maybe throw it on an employee's fiber connection and set up a point-to-site VPN for that machine.

Does it make sense to abandon the entire cloud because of 1 use case?

> abandon the entire cloud

I was never on cloud. I have a big home with fiber connection and some backup connectivity. Setting up new server is like $50/year in electricity. If project works I will sell it, and let new owner deal with cloud and scaling it up!

Just reading price list from cloud providers gives me a headache. We will charge you between X and Z, and hopefully you will not bankrupt on next bill. Also we may terminate services anytime for whatever reason.

> unicorn machine

If you have a specialized startup, that is 90% of your cost! Not some sort of unicorn single machine. If you can use consumer grade hardware without extra cost, that is major competitive advantage!

> VPNs and hybrid architectures

What is that? Do I have to study that? Seems like a major obstacle!

That unicorn machine had better not use unicorn data transfer. Transferring an average of 100Mbps from S3 to your unicorn? That will cost you $1-3k / mo depending on how you configure it and what tier you’re in. Never mind that buying the hardware to sustain these data rates at home or in the colo is so cheap as to not even be worth mentioning and can easily be done on 20-year-old gear.

Of course, those prices to rapidly up if you use serious data. Want to train your fancy ML model to draw cats based on your giant data set of customer cat photos stored on S3? Want to do it on your nice nVidia box at home? That, by itself, might cost as much as an expensive Silicon Valley FTE who could manage an entire installation in a colo facility nearby.

> VPNs and hybrid architectures exist for a reason.

And that's why the rent-seekers got that covered too by outrageous egress bandwidth fees, to ensure they still get their cut no matter what.

> This almost is not ever true for software startups, who get more value from speed, flexibility, and not needing to pay a team to manage hardware.

It's not that simple, some costs on AWS can be high and some tasks don't need the availability/reliability guarantees AWS provides. So even for small startups sometimes having own hardware can be a good cost saver, through with some risk attached to it.

For some mid-sized companies with relative fixed resource utilization sometimes renting rack space or renting bar metal (in a server center, with service for parts replacement) can be a good cost saving measure.

In the end while AWS is flexible and convenient with a lot of services easily accessible it is also expensive (at least when it comes to compute power) so the question is most times how much money do the flexible, convenience and services save. For many companies this is a lot. For some companies (of any size) this is not enough.

I mean this is why there are such funny things as being able to buy a "GPU compute min server center in a container" (through that's outside of the price range of an small company, but smaller versions similar to this exist, too).

> It's not that simple, some costs on AWS can be high and some tasks don't need the availability/reliability guarantees AWS provides. So even for small startups sometimes having own hardware can be a good cost saver, through with some risk attached to it

I agree. I'd like to add that AWS's value add can be the lack of need to understand the product configuration, just the SDK. Your start up already has developers that can read SDKs, you might not have someone on your staff that can set up an elasicsearch, redis, or other service.

When you grow enough that the fine tuning of such services might matter, you likely can hire the right people to run those in house.

I've been on a team with ~6 DevOps members for 5 years, and it's been very helpful to be able to pick up and drop services as we evolve.

I think it depends. My best friend works at a company that took four years to architect and build V1 of its product on prem. They had to build a lot of things by hand that are standard components in the cloud. They architected and built V2 on AWS is a single year. The higher cost is considered small by them compared to three additional years of waiting for a system to offer key features that upsell customers.

If you are software startup, then you don't need all those cloud services. You need a server, install webserver, database, redis or whatever you need and you're good to go. Going cloud as startup is almost always over-engineering. Just go simple. Your total costs will be couple of hundreds $$ and you'll serve a million people with this simple infra. Probably much more.

bUt WhAt AbOuT sCaLiNg

>This almost is not ever true for software startups, who get more value from speed, flexibility, and not needing to pay a team to manage hardware. Just let [cloud services company] handle it for you until you reach that point.

>And then when you do reach that point, you get to write the article about how everyone using the cloud are fools because it's cheaper to self host.

at least read the article, this is about a software company that started in a colo and literally wouldn't be a viable company if they tried to use the cloud. It's more about the fact that if you have the skills you can avoid having to burn a huge amount of VC money by running your own hardware

I always tell people to do the spreadsheet. There is a point at which bare metal or colo becomes cheaper and that point will depend on your work load.

If it’s outbound bandwidth intensive that point will come sooner than say a storage intensive work load.

You always hear stories like this where people didn’t do that and were lazy and massively overpaid on cloud.

I am often surprised how people in our field don't run the numbers, so they are not talking nonsense when these hard numbers are so easy to generate.

Also, people seem to forget that not all startups are consumer lo-CPU requiring applications. If one has a product that requires non-trivial compute, going immediately colo can be essential. In my own previous startup, my calculations showed an AWS monthly expense of $96K, while I could spend $60K to acquire higher end hardware than AWS to build a server cluster and colocate that for a measly $600 a month.And forget about the "it's hard to run a server, let along a server cluster" propaganda - it is not hard at all.

> "it's hard to run a server, let along a server cluster" propaganda - it is not hard at all.

It's not hard at all if you have skills to manage hardware switches, deal with rack power budget, configure the power on sequence, establish procedures for hardware failures, establish procedures for disk management, redundancy, early earnings and replacements, deal with rolling out firmware updates for all components, provision resource limits between apps, provision and secure iLO or similar access, configure hardware monitoring and alerting, maybe run a SAN, ... (and lots of other things)

Sure, it's not hard when you have those skills. And you can offload some of that to another company if you're doing single servers rather than full rack(a). But it's not exactly a common set of skills either.

Edit: Also you can easily forget things at small scale. Just remembered LTT's "Sure we can host our video archive ourselves. Oops, forgot to turn on scrubbing. Aaand the old archive is gone."

If you have that sweet investor money just go all cloud!

> I am often surprised how people in our field don't run the numbers, so they are not talking nonsense when these hard numbers are so easy to generate.

Agreed. Unfortunately the tech industry is driven largely by fashion instead of objective engineering as we'd like to think. Trying to diverge from the fashion du jour (here, cloud.all.the.things) is difficult no matter how solid the numbers are.

At my last couple startups I've run a recurring review of all the cloud costs so at least there is good awareness of how fast the expenses are rising (faster than revenue in every case).

Sadly with "microservices" and "microfrontends" and what not, it does become a CPU/memory expenditure.

Every split service has overhead and requires some minimal allocation to start its own server, own database etc.

>and not needing to pay a team to manage hardware But you pay for a team to manage the cloud instances and services. Usually more than the hardware guys.

Most of the hardware stuff is done by the data center anyway. You just create a ticket for remote hands and the data center staff is going to rack the server you sent them or replace a hard disk etc.

Not so true. Not all remote hands are made equal. Some are there just for simple repairs / maintenance tasks. Don't be surprised that it may need a lot more hand holding.

Experienced it all first hand.

Yup, I'm a bit fascinated that an entire industry is built on the drug dealer / puppy / loss-leader model -- "here; try this for a while for free; you'll like it!"

No question, it is easier to spin up an instance and ler them handle the scaling. And unless you are really careful, it's then very hard to extricate yourself and provision your own servers in a/multiple colo sites...

I always like the control of vertical integration, but evidently enough like convenience to support a huge industry (and there are other advantages, but I don't think that cost is on the list of advantages)

> This almost is not ever true for software startups, who get more value from speed, flexibility, and not needing to pay a team to manage hardware. Just let [cloud services company] handle it for you until you reach that point.

This is an old myth. Today you can get dedicated servers provisioned in minutes with an API call. A competent DevOps person can manage 10s if not 100s with proper automation. When you have a certain baseline usage of compute moving to dedicated can give you massive savings. And if you're a heavy user of say, RDS, the cost difference can be even more dramatic while also gaining on performance. For ephemeral workloads, testing, etc. the cloud makes sense of course.

I'd say testing/CI is one of the best cases for renting dedicated metal because of the huge speed increase available, everyone hates slow CI, and CI workers are easy to setup so if hw fails (it probably won't) it's not a big deal, you don't need to scale up again immediately.

Agreed, but I include CI in baseline usage. With testing I mean compute for feature branches, experiments.. that kind of stuff.

If you are only running in one cloud availability zone, with the backup plan being, we'll spin up new servers during downtime, maybe.. but if you are distributed over 2-3 hosting areas geographically for redundancy, I highly doubt that any cloud application is cost effective at any scale.

Right. I pay under 2k for server costs. And combined with all the different platforms I use and the convenience features and not having to hire my own devops person, I assure you I am saving A LOT.

There are few companies that operate at the scale at where it is cheaper to run your own servers. Once you hit that, it is a great problem to have.

Cloud doesn’t eliminate the need for dev ops. Sometimes it can help, but overall I’ve seen an increase in cost, workload, and stress.

The cloud is very much a your mileage can very situation. Just remember cloud providers are very interested in selling you a product due to their extremely high profit margins.

Eh. Sort of. I disagree with that statement. So far the devops of managing my cloud stuff on heroku has been a few days a year. And we have a better pull-request environment setup than most places I worked with that did or didn't use cloud.

Point is, devops as a service is great for small scale. I've done the math of what it would cost to switch away from heroku, and at this point it is quadruple our current costs, because of the time investment, and that's over a few years. And then moving to our own servers? even worse.

Reply on news.ycombinator.com