Hacker News Reader

Comment by mmmBacon | Hacker News Reader

A few years ago I was trying to start a company and get it off the ground. We had to make decisions on our tech stack and whether we were going to use AWS and build around their infra. Our business was very data heavy and required transferring large datasets from outside to our databases. Even in our early prototypes, we realized that we couldn’t scale cost-effectively on AWS. I figured out that we could colocate and rent racks, install HW, hire people to maintain, etc... for way less than we could use the cloud for. I was shocked at the difference. I remember saying to my cofounder why does anyone use AWS, you can do this on your own way cheaper.

Later I worked at a FAANG and remember when Snap filed their S1 when they were going public they disclosed that they were paying Google $5B and we were totally shocked at the cost compared to our own spend on significantly larger infra.

I think people don’t realize this is doable and it’s great to hear stories like yours showing the possibilities.

View on HN

Disclosure: I work on Google Cloud.

> paying Google $5B

You were off by 10x :). The annual commitment was $400M/yr on average. Snap’s S1 [1] said:

> We have committed to spend $2 billion with Google Cloud over the next five years

[1] https://www.sec.gov/Archives/edgar/data/1564408/000119312517...

> I remember saying to my cofounder why does anyone use AWS, you can do this on your own way cheaper.

I agree with everything you say - I'm convinced that a huge part of the cloud's financial success is due to how it allows CTOs/CIOs to indulge their fantasies about having a mega-scalable app - even if their workloads are very regular and predictable. Along the lines of buying an expensive sports car but never driving it fast, you're just paying for the kudos it brings you in the eyes of other people.

Having said that, we are happily using the cloud for our small app because it makes no sense to build out our own infrastructure for a single VPS and database.

> I agree with everything you say - I'm convinced that a huge part of the cloud's financial success is due to how it allows CTOs/CIOs to indulge their fantasies about having a mega-scalable app - even if their workloads are very regular and predictable.

It's a sane choice given incentives, too. Cloud bullshit features prominently on an awful lot of technical job postings these days.

But yeah I've definitely seen some heavy, expensive cloud setups that could have run on a toaster. Smaller-scale B2B stuff seems especially prone to this—like, what's your max reasonable traffic? If you took off like crazy? What's the size of your market? Come on. Throw in really half-assed and inconsistent use of automation tools and lots of relying on shitty cloud web dashboards and hope, and often the toaster (or a couple low-end co-located servers, more seriously) would be easier and safer to manage, too.

From a developer perspective, I prefer to work with cloud providers because I can do more with fewer developers because I'm not dealing with sysadmins too. I can throw up a EMR cluster with minimal effort and get a working product out the door quickly.

These things don't matter for large, established companies because they already have DevOps, SysAdmin, and Development teams. But for smaller dev shops, it absolutely makes a difference when you can generate a good bit more efficiency from your development staff.

That may be the case for a startup. But in the average corporate IT environment the on-prem server infrastructure and procedural/political jungle is so terrible that you just have to get away from it. Before "cloud" became fashionable it was politically hard or impossible to host your project somewhere outside the bad corporate infrastructure, but now there's a way to present the choice in a palatable way.

I will chip in here. Although I agree with all that you guys have said. Your comments seem to be skewed to a perspective that all companies are being run in U.S.A, Canada, Germany, etc very developed countries where power is constant and can be relied on. The skilled labor to run these servers is available for hire. The parts and infrastructure are available to rent or buy. Majority of the world is not like this but there are formidable companies all over the world running, and for them, this is one less unreliable aspect of their company.

To give you an idea, if you want to run a data center or your own servers in some countries you need a standby generator (because electricity is not a given) and the Diesel used to run these generators are imported and the economy of these countries is shaky so the exchange rate fluctuates, so suddenly the cost to keep your site up becomes a variable and is now subject to government announcements (not even in an evil authoritarian way), policies and import taxes. In the face of all of this, having a steady AWS bill with reliable infrastructure becomes priceless to these companies

You nailed it. The infrastructure in developed countries is taken for granted by its consumers. Websites are often not tuned for low speed internet, or for people across the ocean, because everything works so smooth in developed countries. Also, having a server that serves to the US and Europe if you live in a remote area obviously cannot be on-premise, it has to be in the cloud served in those countries for latency reasons.

And you cannot have a website that doesn't cater to US and EU users, unless the website only solves local problems.

Even so, having say a bare metal server (à la Vultr) can be way cheaper than having a cloud setup in one of the most flexible kinds of providers.

The detail to look at is that flexibility is a feature of the cloud offering, and an expensive one at that. If you don't need it, you need to find a way to not pay for it.

If web site caters to EU customers, just rent it from the EU datacenter, no need for a cloud. A Hetzner server costs $100/month, an equivalent amount of cloud resources will run you into thousands.

And how much does Hetzners nosql/relational db, emr solution, faas, etc cost? Solutions aren’t built the traditional way anymore. You don’t go get some cots server and write everything yourself. So while you’re futzing around getting a database installed, tuned and setup the engineer using a more complete cloud platform has already moved onto business logic.

The person above wrote that she can't use local server to cater to US and EU customers. I pointed out that you don't need cloud to resolve that problem.

Hetzner is a cloud, as per their own literature but my point being if you buy Hetzner you’re not getting a complete platform. You are buying into a platform just less of one. Hetzner also advertises themselves as thrifty, not exactly instilling confidence for critical infrastructures. My point remains though, buying a VPS is equivalent to an EC2 and this is not how things are built these days. Also note the posters name is WilliamEdwards, I know if you called me a she (I’m a he) it would be very offensive to me. It’s best to use gender neutral or pay attention to sex/identity if possible (not really here) and use the correct pronoun.

VPS is in no way equivalent to an EC2

Sure it is, for almost every use case. For edge cases, it’s not exactly equivalent sure. If you just need a place to run code then it most definitely is.

You wouldn't run your own datacenter, you would colocate. Even if that's not possible, you can still rent servers for significantly cheaper than cloud offerings.

Wow... really a first world perspective here.

I believe you are saying that some countries don't have data centers where you can co-locate. Most of the places where AWS has a datacenter have other datacenter companies that offer co-location.

If you can rent an AWS server, you can also rent a Hetzner server, or put some servers in a Hetzner DC.

The you co-locate to the first world. Or somewhere that can get a data center built with resources that you trust.

I ran data centers for a living in Northern VA and had all sorts of international clients. Egyptian schools who rented servers, Brazilian Protestant ministries who shipped servers to us, etc. There were some decent data centers in Mumbai we had to get VPNs built for, and we had a least one legit client in Lagos, Nigeria.

So its true [0]

[0] https://pics.me.me/me-i-prefer-mangoes-to-oranges-random-per...

3rd world perspective: They're not far off from the mark.

You'd want to co-locate with ISP who already has infrastructure for continuous services through blackouts et al, or you could have a datacenter if it's small-ish, because you already have some infrastructure to keep operating during blackouts.

I think you'll struggle to find any place without a few datacenters nearby, at least in a neighboring country. Of course it's going to be a bit further away if you're in central Africa.

Even then, you can colocate in a datacenter anywhere you want, have equipment delivered there and pay remote hands to install it for you for a very reasonable fee.

Of course this doesn't make sense if you just want a small webserver, but that's not who we are talking about here.

I support your comment.

Dropbox did the same thing a few years back - moved everything from Amazon S3 to their own storage.

My guess is they did it for cost reasons.

That S3 is eventually consistent with object updates (HTTP PUT) might also screw up things for a company whose core value is synchronized storage.

I suspect that is behind many problems with Google Drive.

Sometimes old versions seems to take precedence over newer ones, for some reason.

I don't mean to sound daft, just clarifying my own understanding, but isn't Dropbox eventually consistent (as a system)?

Oh, sure, but when they think they have written something to S3 and got a successful HTTP response back from the API, perhaps they want to be able to tell clients to go fetch the new data from the bucket. But those clients may not get the new data then, due to eventual consistency.

S3 is immediately consistent for new objects unless the service received a GET on the object before it was created. It's easy to use this to make an immediately consistent system.

Yes. HTTP POST (new objects) is immediate, HTTP PUT (updates to objects) is eventually consistent.

If you want to use this to create new objects all the time, rather than update ones you already have, you now have to keep track of which objects in your bucket are "old" and should no longer be there. But yes, totally doable.

S3 ListObjects calls are eventually consistent (i.e. list-after-put). EMRFS [1] and S3Guard [2] mitigate this for data processing use cases.

[1] - https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-f... [2] - https://blog.cloudera.com/introducing-s3guard-s3-consistency...

Since the move to S3 is only 'a few years ago', my guess is this was not the reason for them to move away from S3 since service at Dropbox is pretty much the same.

They might have implemented a metadata store 1 layer before S3 to guarantee read after write consistency since write to a new object is consistent in S3. Only updates are eventually consistent.

They did, but at Dropbox's size it should be pretty obvious. I mean, their whole business model is essentially acting as a cloud storage provider. Once they got big enough, of course it made sense for them to optimize their infrastructure instead of letting another company take x% off the top.

> Once they got big enough, of course it made sense for them to optimize their infrastructure

Isn't that true in all cases?

There is no doubt that rolling and maintaining your own infrastructure can be and is better than dumping cash on the AWSs of this world. The only question is what size marks the breakeven point.

I think it is actually only very true in a small number of cases. First, you need to be big enough where the cost of the people needed to support your infrastructure can be amortized over a (very) large number of machines. Second, Amazon, Microsoft and Google pay some of the best salaries around, so they have some of the best infrastructure people working for them.

Looking at the comments here, I think it's clear that there are a relatively small number of use cases where roll your own is a better idea, primarily where you have a huge number of servers basically all doing the same thing with lots of data transfer (which is comparatively expensive in the cloud). This may be cheaper to manage when a small team of people if you're essentially cloning similar setups.

How many other companies do you know that have same problem as Dropbox?

An average company that uses does not provide IT infrastructure (like storage in this case) to a massive amount of clients.

I expect many companies do that. They prioritize growth over costs (stock price over profitability).

Then they get to a certain inflection point on the growth curve and splurging for unpredictable capacity slowly is replaced with reasonable costs for the known capacity.

You’re not paying Google $5B for raw infra, you’re paying for cloud services like top-tier horizontally scalable databases, global availability, CDNs, datastores of different flavors, and transparently managed monitoring and hardware fault resolution.

Yes, you’re paying for a ton of stuff there that you probably don’t use and then are susceptible to bugs that have nothing to do with your use case. At $5B it would not cost anywhere near that much to replicate. Your infra would be better tailored to your workloads and your sw teams which would further drive costs down. The upsides are that you only drive features you need and keep things simple. The downside is that your need a big org to do this. It’s a long term play.

Snap was spending 2B over 5 years - so 400M/yr. By comparison, Uber, who operates under managed colo, spends nearly 200M/yr alone on real estate for their datacenters. Who knows how much they are paying in engineering salaries so manage those datacenters.

Personally I don't think there's a one size fits all solution, you will have to do the math (like I'm sure Snap, Netflix and others have done) to see if cloud is worth it. However, I agree, for most teams the default should be cloud.

400M/yr was just Snap’s Google Cloud spend, they also signed a 1B deal with AWS for redundancy.

Unless that's capex for building, or includes server capex that they depreciate, that seems way high given Uber's scale.

A pretty dense cabinet should only cost ~$1400/mo at wholesale (1MW+ rooms) rates, and $200M is 143,000 cabinets.

And it's public that Uber uses multiple clouds as well.

Disclaimer: I haven't reviewed public Uber filings, would be very interested if there's any data that indicates they're really spending $200M on opex for real estate (which would be equivalent to $400M/year on cloud, which is either opex or potentially mix if there is some reserved instance-type cloud spend).

The 42U rack might be only $1400 per month but the servers to put inside are $5000 upfront per U.

42U times $5000 divided by 3 years is still only $5833 per month per rack. Add the $1400 and we have a total amortized rack cost of $7233 per month. (naively assuming you could fill every spot in the rack with servers).

But servers can't be included in a "Uber spends nearly 200M/yr alone on real estate for their datacenters" figure anyway.

The rack is $2800 per month, not $1400. The lowest advertised pricing does not include enough power to supply half of what's in the rack.

Add $200 per server per month to have a gigabit uplink.

Add $4000 per server per lifetime for VmWare licenses.

These are reasonable estimates of course. Could be multiple of that depending on what hardware is used and what colo. Physical hardware easily gets as expensive as any cloud.

Not easily; you’re making the argument that there can be high expenses for collocating but you’re talking about a -lot- of computational power.

I know that hardware varies a lot for sure, but for context I put together 3 racks 50% density with an 800gbit backplane for around 18k eur/mo.

I spared no expense, official juniper QSFPs (which are egregiously overpriced) and top of the line Dell servers with full out of band licenses.

And once there: interconnected bandwidth and IOPs became “free” (or, no extra charge).

We put the same application in cloud And it costs us 40k eur/mo with a heavy amount of optimisation, with half-sized instances and an aggressive optimisation in bandwidth/iops.

Clouds “sticker price” to us is 4x that of physical. You can buy a lot of human time for that price.

Physical infrastructure is always overprovisioned because it has to be planned long in advance and the smallest unit is huge (8 cores? 128 GB of RAM? for a small server).

If anything that's an argument against physical infra, not in favor of. Although it's fine if one wants a lot of the same big servers (an Hadoop cluster, or video computing cluster, or a CDN), which are the few use cases where physical can make sense (and hybrid cloud probably makes even more sense).

With AWS, you'd never spin half of that infra upfront. You'd spin a few VMs and start running stuff. If the project goes well, spin more or bigger VMs, otherwise spin down. Cost is very dynamic and the company doesn't have to spend half a million upfront, which is a big financial problem for many companies.

Anyway. We can both agree on AWS being overpriced. The reference on costs should be Google Cloud if one is cost conscious, not AWS. Google Cloud is often half the costs of AWS for the same thing.

AWS vs Google comparison, bit old and instance types have changed but relative pricing has not moved https://thehftguy.com/2018/11/13/2018-cloud-pricing-aws-now-...

And this one more recent since they've released high memory instances to compete with SoftLayer: https://thehftguy.com/2018/11/13/2018-cloud-pricing-aws-now-...

My costings are based on GCP; with sublime discounts even.

And, I would really agree with you if not for 2 things:

1) "wasted" cycles are not wasted, the CPU will clock down.

2) Kubernetes was designed specifically for this, the idea is you slice the CPU up so much that you dont waste much resources.

It's astonishingly capable of consuming all resources.

By the by; RAM is never "wasted", it's used for various caches.

As the others say in the comments, understand completely re: servers. And network. And some management overhead. I will write a blog post laying out our COGS at Kentik including all these factors, hope it'll wind up helpful.

> A pretty dense cabinet should only cost ~$1400/mo at wholesale (1MW+ rooms) rates, and $200M is 143,000 cabinets.

The 200M is for a year - at $1400/mo, that's 11905 cabinets.

Well those numbers are... weird

Also Uber seems to be much less computing/BW demanding than Snap

> Uber, who operates under managed colo, spends nearly 200M/yr alone on real estate for their datacenters

200 million a year, huh.

The cost of commercial office space in the U.S. can range from $6 per square foot in low cost regions to over $12 per square foot in New York City. On average, a 50-cabinet data center will occupy about 1,700 square feet. At a median cost of $8 per square foot, the space alone would cost about $13,600 per month. [1]

Are Uber renting on the order of two million square feet of data centre? Do they have sixty thousand cabinets of hardware?

If they do, i would absolutely love to see a quote for how much it would cost them to run in the cloud.

I think it's far more likely that number is bullshit.

[1] https://npifinancial.com/blog/data-center-pricing/

>Office and data center rent expense was $194 million and $221 million for the years ended December 31, 2017 and 2018, respectively.

https://s23.q4cdn.com/407969754/files/doc_financials/2019/ar...

Page 124

So that number includes (and is almost certainly dominated by) the cost of their office space.

Exactly. Calculating TCO for on-prem vs cloud is tricky. Any time we have done it cloud came out the winner. I also found exceptions: predictable static workloads requiring a huge amount of bandwidth, mostly outgoing. An average company that needs some CPU and storage for various unpredictable workloads benefits from cloud services greatly.

disclaimer: former uber engineer

we ran the numbers continually and like others mentioned, it was a no-brainer to build on-prem. that said, there are a few use cases where aws/gcp are a great fit. people selling cloud without putting in real engineering work to enterprises generally make my life harder than it already is.

@nemothekid Can you provide a source for your claim on Uber's real estate costs for their data centers? I couldn't find anything on Google that corroborates your assertion.

Uber 10-K has operating lease payments in 2019 of $196M and in 2020 of $216M. That's not all data centers but a lot of it is data centers.

Why would you assume that? SF office space goes for way more than data center space.

You can look up the prices of Uber's current office space - they currently pay 16M/yr for their HQ.

> Yes, you’re paying for a ton of stuff there that you probably don’t use

This is the antithesis of any cloud. You only pay for what you use.

> At $5B it would not cost anywhere near that much to replicate.

If you can recreate GCP for $5B in capex, there are likely some VCs lurking here who would like a word with you.

Sorry but that’s not correct. Have you ever looked at the economics of cloud compute at that scale? You absolutely are paying for a lot of tooling that’s irrelevant to your use case. This is because GCP has to be a general provider. They have to offer the broadest possible solution to capture as much share as possible. This means there are money losing services (also low margin) that are offered to provide a complete end-to-end solution. However, the fact that those services exist means that the cost has to be made up for elsewhere. These economics are not some complicated rocket science and exist in many other types of businesses.

Like cable TV.

That's not a perfect analogy but it's easy enough for laymen to understand so I'm stealing it.

If I use S3 + EMR what exactly falls into the category of "a ton of stuff there that you probably don’t use"?

>> Your infra would be better tailored to your workloads

How do you scale elastically with an on-prem infra? What part do you tune more to your own workload when an average company cannot even tune the GC for Java for their own workload?

Your claims do not reflect on reality, it is rather your imagination. I have migrated countless companies to the cloud, and almost every single migration was driven by a single factor: cost. Everything else was an added bonus: inscrased security, availability and elasticity.

IMO, the cloud providers' main advantage is that they will professionally manage all of the underlying hardware. Optionally, they can also manage the low level pieces of software also (e.g. databases, file store, etc.). It's a safe bet that GCP, AWS, Azure et al. can manage/architect their data center much better than the vast majority of companies.

The last point is something that “we”, as an industry, can often be blind to.

If your business is something that is fundamentally not “pushing bytes”, you really, really don’t want to know about routers, firewalls, the OSI layer, SHA256, RAID arrays, all the way to this week’s JS framework. All of that is a big annoyance, and paying AWS to “take care of it” makes sense, even if it comes at a higher price: more often than not, the difference wouldn’t be offset by the time, effort, and risk exposure that you would have to allocate when building your own.

This calculation is different if your business is primarily digital. The gentleman upthread making a game, for example, is perfectly right: his company naturally developed a culture that can evaluate and manage every aspect of its digital operations, because it’s part of its core business, so it makes sense to put that knowledge to good use and save money.

But if you’re incompetent to that degree, AWS isn’t for you because you can’t even write software to run on it. You should be using a fully managed SaaS at that point.

In many cases yes, but there are shades of grey. Some systems cannot be shared, for whatever reason.

There are three things that make sense to me.

Firstly, if you are big enough, you can manage/architect your data centre better than a cloud provider. You can afford to hire staff to do it, and they can build something specific to your needs. Companies like this should be on-prem.

(It doesn't matter if the company is "digital" or not. It could be a huge retail chain, or a government agency, whatever. The only requirement is to be big enough that you have big needs and can afford a big spend on staff.)

Secondly, if you are small enough, you really, really don’t want to know about routers, firewalls, the OSI layer, SHA256, RAID arrays, etc. Dealing with that would mean another couple of full-time employees, and you can't afford that. Companies like this should be on a SaaS (ideally one less full of gotchas than Heroku).

Thirdly, if you are in between, you have the capacity to deal with system administration, there are some advantages to being able to shape your infrastructure to your needs, but you really don't want to get into real estate and millions of dollars of CAPEX. Companies like this should be on rented physical hardware.

What doesn't make sense, at any scale, is renting VMs.

Do you mean PaaS, not SaaS? What other good PaaS exist besides heroku?

I did mean PaaS, not Saas! Apologies, and thank you!

I wish i had a good answer about PaaS. There are hosted Cloud Foundry services. I think CF is more sensible than Heroku, myself. The various serverless platforms are PaaS of a specific kind. I think SalesForce counts. Is OpenShift still a thing?

> What doesn't make sense, at any scale, is renting VMs.

I disagree. The big cloud providers will let you rent VMs across multiple availability zones in a single geographic region. That gives you better redundancy than you could get by renting one or more dedicated servers in a single data center. Yes, those same cloud providers offer bare-metal instances, but those are absurdly expensive for a company that only needs small-scale computing power but still cares about uptime.

If you're small, you can get that redundancy on a PaaS. If you're big enough to move off a PaaS, you can rent physical machines in multiple datacentres.

Size is not the only thing that dictates whether one should use a PaaS. It's entirely possible for a tiny generalist team, or even a single developer, to develop an application whose requirements prevent it from running on a PaaS. But that doesn't mean the team or solo developer should have to handle all the hassles of operating physical servers. (Source: I was that solo developer for many years.)

I'm not super familiar with Heroku and similar platforms, but if I don't rent EC2s (which I take to mean what you meant by "VMs"), how do I host and serve bespoke backend components?

Even if all it does is suck data in, perform some computation, and send it off somewhere else - are you saying there are SAAS providers that are better equipped for this?

> Firstly, if you are big enough, you can manage/architect your data centre better than a cloud provider. You can afford to hire staff to do it, and they can build something specific to your needs. Companies like this should be on-prem.

It's interesting that competent engineers is only a question about cost to you. Where I am at (north of Europe) it's really hard to find good engineers. Even though it doesn't mean much about competence it's not mandatory for people in the IT industry to have a CS degree even if they're managing infrastructure in the cloud for millions every year.

I've been working in the industry at different companies for a bit over 10 years and I would say that a huge majority is just an average person who knows basic concepts about infrastructure but wouldn't be able to design and implement any of it by their own. This includes network infrastructure for the biggest ISPs in the country.

I work as a system engineer and the teams I've been working with the last 5 years, independent of company, have been looking for engineers constantly. The pay well and have great benefits but it's just not enough people out there to take these jobs and manage a full on-prem infrastructure.

So a part of the money you're talking about that should be used to hire all these really competent people are instead used to pay for cloud services where we know that professional people with far more hardware (and software) knowledge is managing the infrastructure. And this is really convenient for a lot of companies.

> The pay well and have great benefits but it's just not enough people out there to take these jobs and manage a full on-prem infrastructure.

I’m from a small province in Canada and we have similar problems here sometimes. One of the questions I sometimes have to ask clients when they make a statement like that is: “do you pay well for the area? Or do you pay well enough to attract talent from out-of-province?”

This often leads to them arguing that “but the cost of living is low here!” And inevitably I have to mention my friends who left the province to go to the Bay Area for 10 years and came back with $500k USD in stocks they collected, on top of what they had left over after paying their high cost-of-living rent.

I feel your pain. Companies running in less desirable areas have to somehow pay a premium for top tier talent. One way, as you mention, is to outsource, whether to cloud providers or part time contractors.

This isn’t meant as a brag at all, but I do the independent contractor thing here and make “pretty darned good for the area” money. Inevitably clients will ask me to come work full-time for them, and we have a painful conversation where I tell them what my taxable income was the year before, as well as the investments I made into equipment and licenses I use to provide the services I provide. The result so far has always been to continue being happy with me as a part-time contractor!

Edit: sorry for the typos, written on mobile with the swipe text thing

if it's ok to poke, can I ask what type of equipment and licenses? Is it like a home tech stack to test new infra setups on?

p.s hope you're staying safe from COVID

How much do you think a competent System engineer should be paid?

I think that's way too hard to answer since it depends on so many factors. But again, my experience isn't that the pay is too low. It's not like there are 10 people coming to the interview with no one taking the offer because of the salary. It's more like no one have applied for the job in three months and the external recruiters sniping LinkedIn didn't find anyone.

But to give some kind of estimate just so you know what page I'm at when I say they pay well I would say somewhere between $60-75k.

too low.

So I checked on glassdoor, which is known to lowball tech wages, for Washington D.C. So It isn't NY or Silicon Valley/Forest. The Median wage for a System engineer is 182K USD. At the current conversion rate that is 167.7K Euro. Unfortunately, that's your problem.

The cloud providers' main advantage when you're unicorn scale is their engineering team acts as an extension of yours. We regularly hop on calls with AWS to scope out features for them to build for us. Our parent org can pretty much snap their fingers and Google will dispatch their engineers to fix/build whatever they wanted.

That and very preferential pricing.

And a lot of people don't need that. There are a lot of applications that are highly cost sensitive but where losing some data or having some unavailability isn't a deal breaker.

Avoiding cloud works well for small requirements too - worked for a company that ran a global store; we've opted for dedicated servers on OVH, all together had about 40 of them. It was 1/10th the price of running the infrastructure on AWS. We still used S3 though.

Yeah, I'm seeing this too.

I recently talked with someone who said their costs dropped in half from switching off a major cloud provider to OVH's dedicated servers. Performance went way up too.

For $200 / month you can get a machine with a high end Xeon 8 core (16 threads) CPU, 128 GB of memory, 4.5 TB of SSD space across multiple disks with unlimited incoming / outgoing traffic and anti-DDoS protection.

No reputable cloud provider is going to come close to that price with their listed prices.

Of course there's trade offs like not being able to click a button and get it within 2 minutes for most server types but if you have consistent and predictable traffic, going with an over provisioned dedicated server seems like a very viable option.

I've worked with several start ups and my experience is the exact opposite of what you said. Due to costs of scale, it is virtually impossible up beat cloud architecture. Ignore the fact that you can get started for free it next to free with cloud. Even as you grow, it is difficult to do it for cheaper.

Did your business get off the ground?

> why does anyone use AWS, you can do this on your own way cheaper.

Because they can calculate TCO correctly.

Is it just me or is Total Cost of Ownership being talked about way less often these days?

It's easy to game. Source: was a sales engineer for ISPs and Data Centers.

So assume you're doing a 3-year or 5-year TCO -- you build in credits, discounts, and bundled options. Execs are looking for a 3-year apples-to-apples spend, and you make your TCO look fucking amazing. They see the low price and decent technical options and they bite.

3 years later those credits vanish and they're paying full OpEx costs. And after 3 years they're now invested -- stuck -- in their space/circuit/whatever. You can start raising the price or negotiating new contracts.

Same thing with the cloud, for that matter. Cut a glorious bulk deal with Microsoft for Azure space, and then after you've moved everything to MS they can start nickle and diming you -- cuz the cost of moving that load to AWS or GCP isn't cheap, and you're not going out and buying more hardware and going back to CoLo are you?

Thanks for this perspective, I guess the gameability of the metrics explains a lot.

Exactly that. A lot of people do not understand that and get into trap of those credits "thousands of dollars worth" just to be scalped by their vendor-locked costs after those credits vanish.

After all people who make decision on customer side are in same trap - in 3-5 years if not earlier they won't be at the company anymore and it won't be their problem.

Yes and also there is a ton of hand-wavy, without any merit bullcrap floating on HN. Without any in-depth analysis, numbers, apple-to-apple comparisons. AWS and Azure would not exist if moving to the cloud was a bad move. There are so many companies whose primary business is not to build data centers, yet they require some computational resources. These are the companies that are the primary users of cloud computing. Based on my experience, most of the financial companies fall into this category, including banks. Other industries that I have experience with include, travel, gaming, pharmaceutical, logistics, and a few more. Microsoft is doing a great job to get the most enterprise customers, while AWS has stronger offerings. The biggest winners with cloud migrations are the companies who can start to auto-scale while previously was impossible because the on-prem datacenter had no such features, even if they had, they could not sell the extra capacity to other companies like AWS. Other great cost optimization opportunities include the option to try to run the workload on different node types and find the best fit. Also not an option with most on-prem DCs. S3 itself can solve problems that are very hard to solve. For example, you have different security zones and you need to copy data around in your DC. This becomes not necessary anymore with S3, just give different users different access level and you do not need to copy the data around anymore. This was one of the big selling points for one of your customers. I could go on and on. In the last 5 years, we have saved several millions of EURs for our clients and made businesses possible that was impossible using on-prem resources. But some HN knows better and they argue that all of these companies which are rolling on AWS are morons and they should have built they own DCs because it is cheaper while AWS became a ~10 billion income business unit. There is some irony in this, I guess.

Reply on news.ycombinator.com