Hacker News Reader

Comment by revolvingocelot | Hacker News Reader

>Ms Wang and Mr Casado have suggested that firms should think about building their own private clouds to keep costs down

Yeah I distantly remember "on prem", too. Does everyone, everywhere really need to be equipped to hyperscale at a moment's notice? Wouldn't it be better not to be beholden to a megacorp that serves you with 30 pages of opaque billing for business-critcal functionality? Doesn't hiring external consultants to manage your fucking cloud infrastructure seem like sort of a risk as compared to keeping the services-wrangler in-house?

...I don't have an MBA, so these are half-serious questions. If anyone can chime in with (at least) a half-serious answer, I'd halfway appreciate it. I understand that The Economist is the voice of global capital, and global capital has decided that there's money in them thar clouds, but still!

View on HN

I've mentioned it before and will again: AWS spent a decade openly and privately advertising how much "cheaper" the cloud was. How economies of sclae meant you'd save boatloads of money. Except, it was never true. You will potentially save money if you're just starting out and have no idea if your idea will be successful. A massive capital outlay is difficult and stupid at that point. You will also potentially save money if the tool you use to make money is something you can easily scale up and down, and actually DOES scale up and down. IE: I host a fantasy football stats app and every Monday night, Thursday night, and all day Sunday I've got millions of users, and thousands the rest of the week.

If you don't meet those criteria, unless you are absolutely horrible at negotiating with your vendors, you aren't saving money going into the cloud. I don't care that "datacenters are expensive to build". If you are large enough to need to build your own datacenter instead of renting floorspace for pennies on the dollar from Switch or Equinix (although Equinix is kind of pricey) or Windstream or insert provider - then the datacenter is a rounding error in your 10-year IT budget.

The problem is "I moved company XYZ to the cloud" is now a requirement for an IT exec to show up to the country club on Saturday. So whether it makes any fiscal sense at all, the board wants to know "what our cloud plans are" and most execs are more than happy to oblige in order to pad that resume. In my experience, most of them leave long before the bill comes due, and people start asking why the IT budget quadrupled in 3 years. The best part is, most of these companies can't pass along that increased cost to end-customers like a Netflix can because they aren't actually making money on the apps they're hosting in the cloud, those are all internal business facing apps.

You have some really good points, but your premise is simply not based in reality.

You're painting the picture as if AWS is the 2nd version of the IBM saying: "No one ever got fired for using AWS". This is simply not true...at least not the part where "IBM" (i.e. AWS now) is somehow inferior to the competition.

> If you are large enough to need to build your own datacenter instead of renting floorspace for pennies on the dollar from Switch or Equinix (although Equinix is kind of pricey) or Windstream or insert provider - then the datacenter is a rounding error in your 10-year IT budget.

Then you clearly don't understand the TCO of using a data center vs cloud. The datacenter cost itself isn't the cost - the people who operate it are.

On a computing resource basis can you operate SQL Server on a rack in Equinix for cheaper than that in AWS? Sure, but who is going to pay for the person to patch it every week, or upgrade it to the latest version, or upgrade the SAN disk, or upgrade the hardware every 3 years, etc. etc. Let's take this a step further - who is going to hire these people? And fire them? And train them? These are all transactions costs that get hidden in the "$0.20/hr cost" of spinning up an EC2 instance.

If you think cloud computing it's just about a lift and shift from a traditional data center, then you don't understand how to calculate TCO.

P.S. - For the record, many large internal IT groups do in fact do simple lift and shifts from traditional data centers and don't fully realize the value of cloud computing resources. However, these companies will usually mature eventually and evolve services to ones that are more advantageous (i.e. setup SQL Server on EC2 and the migrate to RDS). AWS (et al) have the ability to do this with ease.

You still need the IT team to babysit the AWS infrastructure.

I've seen this "even a lift-and-shift reduces TCO" argument before, but it only ever works if you pretend that AWS lives up to its zero-management hype. Maybe, for someone, it does. Not for any deployment I've seen. This never prevents management from pretending, of course.

EDIT: ah, I've seen that the argument has retreated to "it reduces IT headcount." Again, maybe for someone, but I have yet to see it actually work, and every time management has just hustled the TCO calculation.

> You still need the IT team to babysit the AWS infrastructure. … it only ever works if you pretend that AWS lives up to its zero-management hype.

Who is saying zero-management? If you have a consultant lying to you, that's not a cloud problem and I've heard the same people lie about all kinds of on-premise services as well.

What I've heard the major cloud providers say is true: you need _less_ staff and you can spend time on different things, which has been generally true. For example, instead of needing to configure a bunch of network hardware (which is a distinct skill you need to hire) you can have a general devops person use Terraform to manage routes, firewall rules, etc. without also needing to be a Cisco expert and spending time operating the underlying hardware.

The key question to ask is where your infrastructure was to start with. If you have an A-grade ops team, a lift-and-shift will probably not be much of a cost savings because you presumably already have economies of scale working in your favor, unless you have enough bursty workloads to pay for it that way. The catch is that most places do not have that high-end ops capacity so what you're really doing is taking your limited IT capacity and reclaiming the time which is currently going into things like managing Dell firmware updates.

Surely for any company of size you understand there will still be people managing the AWS infra. Having tens to hundreds of small dev groups go at it willy nilly is a recipe for disaster and overspending. Someone still has to patch the server instances, manage the people who access it, train them on how to use AWS, etc. Those costs don't go away because you've moved to the cloud.

> Those costs don't go away because you've moved to the cloud.

Sure, but you can go from "I have 5 people managing my 150 instances of SQL Server" to "I have 1 AWS sys admin". The point wasn't that the costs are eliminated, it's that they are baked in.

Generally at big companies you needed people to manage the infrastructure, and people to manage the layer on top of that infrastructure though. With clouds you can consolidate those groups into one group that is smaller.

I think Cloud still costs more overall, never been convinced it actually saves money. I do think it tends to increase developer velocity though, and that can be worth it.

> If you think cloud computing it's just about a lift and shift from a traditional data center, then you don't understand how to calculate TCO.

This is especially true for more than basic services. If all you need is a bunch of Linux servers, sure, you can find people who will keep them patched for you without too much trouble (although it'll almost certainly turn that “rounding error” into a serious number) but once you start building things which are higher level with legitimately hard problems to solve and need to have good solutions for all of the monitoring, security, ops, etc. needs that staffing becomes harder and you're going to have to sell your executives on the idea that the best use of your expensive A-team is cloning S3, Lambda, etc. when you can get that benefit immediately without a massive up-front payment and have the same people working on a strategic benefit.

> cloning

No need to clone; all of this already exists. Take a look at CNCF.

I'm trying to be charitable here, can you expand on that and explain why that's not an invalid comparison? For example, the word immediately after your quote was “S3”. That's not something the CNCF can solve for you since one of those is a service and the other is an application you can run — at the very least you need an _extensive_ ops and security if you're relying on it as much as S3 – and, assuming you're talking about Rook, you're comparing it to one of the largest, highest-scaling services in existence but here's what you're on the hook to do if you do it yourself:

* Kubernetes

* High-level storage services: Ceph, Cassandra, NFS

* Lower-level storage for whatever you pick for the previous point

Each of those, and the combined service you build, needs staffing for operations, monitoring, security, etc. support. If you follow any security standards or are in a regulated industry, you're on the hook for documenting compliance on all of those, too, especially if you need to be able to demonstrate things like immutability.

You certainly can do this but that's a number of full-time jobs requiring expertise. If you have enough storage usage you can potentially justify the savings over the rates you'd be able to negotiate but that's a big up-front cost which you hope but are not guaranteed to recoup.

Don't forget Minio for S3.

I took your previous comment to be about on-prem cloud technology. It exists and does not need to be "cloned". If you were taking about staffing resources, we've pretty well established you're going to need staffing to manage both cloud and on-prem tech.

You need to document whether it's on-prem or in the cloud.

I excluded Minio because that's not branded as a CNCF project but that doesn't make the comparison less invalid: if you're building an S3 replacement, you will need to spend a lot of money up front hiring people, buying hardware, and building an equivalent service. That's a significant 7+ figure commitment just to get to the point of saying that you have an equivalent service.

If you buy storage by the petabyte or use a large enough amount of network egress, you can definitely make that money back but it's a major commitment.

Again, you don't need to build an S3 replacement; you can deploy it onto k8s. This is the same skill set whether in a public or private cloud.

Agree, I don't think anyone is buying SANs that don't need them. If you're spending 7 figures on your AWS bill, you should be looking into other options.

> Again, you don't need to build an S3 replacement; you can deploy it onto k8s. This is the same skill set whether in a public or private cloud.

That is building an S3 replacement. You aren't writing every line of code personally but you are selecting components, configuring them, operating the service, and testing it, which is a major commitment. Simply running k8s at an equivalent level is a non-trivial commitment.

> Agree, I don't think anyone is buying SANs that don't need them. If you're spending 7 figures on your AWS bill, you should be looking into other options.

Note that the 7 figure commitment I mentioned was what you'd need to build an S3 equivalent. Simply having 24x7 staffing and hardware easily puts you in that range.

Nobody is arguing that a startup needs a datacenter; that is a ridiculous straw man. I'm well aware of the costs, having managed multiple colos as well as multiple cloud footprints. I also know you can lease most hardware, if you prefer not to depreciate assets.

You're casting this as a much more complicated concept than it is. It is essentially the same old "build vs buy" argument. No, it does not make sense to manage a datacenter to run a single wordpress blog, nor does it make sense for a telco to run their infra on top of a public cloud. These are all truisms.

My point stands that if you have a seven-figure AWS bill, you already have plenty of associated staffing costs. Once you hit the break-even point between the two, you can decide whether you want to continue to pay linearly, or go down the path of increasing RoI.

> On a computing resource basis can you operate SQL Server on a rack in Equinix for cheaper than that in AWS? Sure, but who is going to pay for the person to patch it every week, or upgrade it to the latest version, or upgrade the SAN disk, or upgrade the hardware every 3 years, etc. etc. Let's take this a step further - who is going to hire these people? And fire them? And train them? These are all transactions costs that get hidden in the "$0.20/hr cost" of spinning up an EC2 instance.

Who cares? This is IT and is not relevant. And they have Brent and he will solve everything.

> who is going to pay for the person to patch it every week, or upgrade it to the latest version, or upgrade the SAN disk, or upgrade the hardware every 3 years, etc. etc. Let's take this a step further - who is going to hire these people? And fire them? And train them? These are all transactions costs that get hidden in the "$0.20/hr cost" of spinning up an EC2 instance.

You're implicitly arguing that NoOps is not only a functional model, but also scalable and much less costly than having a {Dev|Sec}Ops staff. The last time this was publicly debated between Adrian Cockcroft and John Allspaw, it was immediately obvious to everyone that there is no such thing as NoOps. You can have developers support their own apps in production on the cloud, but that takes away time from developers and comes with its own set of downsides (most prominently, a disinterest in what they consider mundane or boring). Once your developers are spending, in aggregate, more than 40 hours a week do ops work, you'll gain efficiency by hiring a DevOps Engineer whose entire career focus is all the things developers aren't typically great at. I've seen this in-person as an employee at medium, large, and a Fortune 50 business. I haven't yet seen a single app, not even serverless, that requires zero maintenance. I have also seen highly-distributed, performant apps run on cheap hardware with no SAN or VMware to drive up the cost and complexity. That same app, made up of mostly microservices, was much more expensive to run on on the cloud.

You might want to stop throwing around terms like TCO when you also clearly don't understand how to calculate it.

I'm starting to be of the mindset that us-east-1 is already "Too Big To Fail". I guarantee you that conversations are being had in boardrooms right now about how to reduce dependency on that region, if not AWS entirely.

> I'm starting to be of the mindset that us-east-1 is already "Too Big To Fail". I guarantee you that conversations are being had in boardrooms right now about how to reduce dependency on that region, if not AWS entirely.

If you're not blasting your AWS account manager and asking what their plans are to avoid this happening in the future, then your CTO is not doing their job.

If AWS itself is not internally working out how to "get off us-east-1" for its global services, then it is failing in its role as infra provider.

AWS has a lot of stuff about how to make your infra redundant and resilient to failure, in their environment, mostly about replicating across AZs and then across regions.

The fact that their "global" services are not actually "global" but are "everything goes to us-east-1" is broken and if they're not fixing it, then that is when you start looking at other alternatives.

> You're implicitly arguing that NoOps is not only a functional model, but also scalable and much less costly than having a

I'm not. You implied too much.

> You might want to stop throwing around terms like TCO when you also clearly don't understand how to calculate it.

I do. I never said that people were completely eliminated (see one of my child comments). Everything you implied was that "well devs are doing the effort and that doesn't get calculated". That wasn't my argument - you just added that nuance as a retort.

> I guarantee you that conversations are being had in boardrooms right now about how to reduce dependency on that region, if not AWS entirely.

I guarantee you they are not. Also, how many data center providers let you simply spin up your whole environment into another region, point n click, and complete it in hours?

Source - work with 100+ companies with PE folks that sit on boards.

> I guarantee you they are not.

I'm speaking from firsthand knowledge. How embarrassing for you to think that because you know some Private Equity folks (generally the worst people at managing technology), that 100% of the rest of the world is doing (or not doing) exactly what they're doing.

You wrote paragraphs about how expensive it is to hire people to do stuff in a datacenter. What is your argument, if it is not that this is more costly than outsourcing it?

This entire debate should be recast the simple "build vs buy" concept that it already is.

> I'm speaking from firsthand knowledge.

So am I. My reference to PE firms was that my anecdote of the few boards I'm on is supplemented by their experience.

> This entire debate should be recast the simple "build vs buy" concept that it already is.

Because that's entirely the premise of my argument. Did you read the parent comment and what I was responding to?

PS - Say what you want about PE folk, but they are the most ruthless capitalists. So IMHO they are far more incentivized to drive cost efficiencies than anyone else because profit = EBITDA = valuations.

>Then you clearly don't understand the TCO of using a data center vs cloud. The datacenter cost itself isn't the cost - the people who operate it are.

I guess I find that a bit humorous. I've been architecting solutions that span on-premises and cloud for nearly a decade so I'm intimately familiar with both. I don't actually have a horse in the race. You want cloud, buy cloud, you want on-prem, buy on-prem. But the marketing about one-size-fits-all is EXTREMELY tiresome. And the idea that every single business can or should be moving their entire infrastructure to the cloud is as well.

The fact you imply that somehow you don't need people to manage the cloud makes me seriously question what at-scale deployments you've been a part of. I have yet to see a single enterprise customer save money on staff by moving to the cloud. More often than not what they end up doing is laying off the "tape jockey" and replacing him with an SRE who costs 3x the salary. Congratulations? Now you're paying more for an SRE, and you're backing up to S3 at about 5x the cost of what you were spending on tape? But I guess the 7 years of backups you're legally obligated to keep are more agile?

>If you think cloud computing it's just about a lift and shift from a traditional data center, then you don't understand how to calculate TCO.

I don't think the concept of cloud computing is lift and shift and never said it was. But that's not really relevant to OP's question, what is relevant is what's happening on the ground.

What's actually happening is that most established businesses "moving to the cloud" are lifting and shifting and finding out the hard way. I'm not saying that's what they should do, I'm not saying that's at all a good idea, I'm telling you that is what is happening. And if you think "well they'll mature into a more cloud friendly model" - you're as delusional as their management. What ACTUALLY happens is after they lift and shift and get a year's worth of those bills, they lift and shift the stuff that never should've been in the cloud in the first place back on-premises. Hopefully by that point they've learned their lesson and have a thoughtful evaluation of their applications: re-home, re-write, leave in place.

> But the marketing about one-size-fits-all is EXTREMELY tiresome. And the idea that every single business can or should be moving their entire infrastructure to the cloud is as well.

100% agree. Didn't mean to imply this. As with anything....it depends.

Wholehartedly disagree. I've been in large scale IT for 20+ years, working for deep pocketed companies. The costs of running old school IT with on prem data centers were orders of magnitude higher vs. what we pay for well architected serverless solutions these days. I remember having a $30MM/year line item for Oracle alone, running on expensive hardware that was sitting idle 90% of the time. Time sharing expensive hardware is the way to go.

Another thing is costs of building something new. In the old world it would mean doing the math on resource needs now and in the future, since adding Hw meant capital expenditure. No such problem now and its easier to get started, see if it works, and scale or kill early.

And of course as others have mentioned, keeping it all running requires expensive expertise. Esp. now with the internet that looks like automated wild west. You cant release a service without it being auto-probed and auto-attacked within 2-3hrs.

Nope. With the cloud, the costs have went down for any large coMpany, even those that just moved the old crap to VMs running on someone elses hw.

> I remember having a $30MM/year line item for Oracle alone

I think I found your problem...

>Nope. With the cloud, the costs have went down for any large coMpany, even those that just moved the old crap to VMs running on someone elses hw.

Are you implying that there isn't idle cloud hardware?

There is, but there's less if you're using managed services where possible and it's visible in a way which wasn't true for many large organizations. I can go into any cloud provider's console and see a detailed breakdown of what I'm paying and why, and how much of it is over/under-utilized, but getting that same report from an enterprise IT department is usually a project.

If a cloud machine is idle is you can shut it down and stop paying for it.

Cloud providers (especially Amazon) can definitely be a bit greedy / shady about pricing and lock-in, but the key thing is that they have room to lower prices whereas a bespoke data center does not.

Most data center machines run at a tiny fraction of their capabilities; seeing a CPU running at 25% is definitely something that stands out. Because of this, cloud operators can oversubscribe these machines in various ways, packing workloads of multiple tenants onto a single box while a single-tenant bespoke DC can't. They can also do things that schedule DC-wide resources (namely power) in ways that renting space in a colo cannot; they can shed load in insightful ways when a power overtopping event happens that can minimize the impact (e.g. throttle / nuke VMs that can operate in a degraded mode before impacting other critical services). No hetergeneous, multiple-private tenant DC is going to oversubscribe their rack- or row-level power delivery to tenants and require them to hook in to a smart system to manage overtopping events; they just won't trust their tenants to do the right thing, as they have no power over the tenants' machines and power draws; the cloud providers do.

At the end of the day, the economies of scale are just hard to beat, in terms of COGS.

The real question is whether they'll pass the savings from this on to the customers or the shareholders' dividends, and so far it seems like it is the latter.

> Most data center machines run at a tiny fraction of their capabilities; seeing a CPU running at 25% is definitely something that stands out.

Kubernetes fixes that, as does VMware. They can both distribute workloads efficiently to ensure good utilization. VMware has done that for 20 years.

AWS doesn’t oversubscribe on EC2 instances outside the T family.

> Most data center machines run at a tiny fraction of their capabilities; seeing a CPU running at 25% is definitely something that stands out.

Are you familiar with VMware?

Is VMware free?

The problem is not just the datacenter itself but 1. all the personnel needed to operate the datacenter/private cloud 2. the lack of agility when everything needs to go through internal IT provisioning (at some companies this is a huge bottleneck).

A few things are relatively cheap:

Compute on preemptable VMs/spot instances is cheaper in cloud.

Object storage, if you would have required a vendor support contract on prem, up to a PB or so.

managed kubernetes, if you would have required a vendor support contract on prem.

It's all good until you remember you have to now higher an entire team to manage your bare metal servers. Companies don't like adding new employees, even when it's comparable in cost to hiring an outside vendor or paying for an outside service. Firing 10 people is very messy, reducing your AWS budget by 20%, however is not.

Are you also factoring in that you often need 2+ DCs for redundancy? That seems like a lot of operational difficulty unless your company already had a physical presence in multiple locations.

I know there are also a bunch of requirements for where you can store user data these days due to privacy regulations.

Fantasy football is actually interesting in that it happens “off hours”. So if you are a company that has an “on hours” business, running Fantasy Football is free if you utilize multi tenant hardware.

> Fantasy football is actually interesting in that it happens “off hours”.

I entirely associate fantasy football with working hours. Ditto a variety of other kinds of sports betting. Like, I know people do these at home, too, but there's a lot of it happening during business hours, at offices.

The peaks though (by a significant margins) are when games are on and people are watching the games unfold.

The money is not the issue. Cost is not the same as value.

>The money is not the issue.

Money is ALWAYS the issue. Whether you can pass along increased cost to your customers or not is entirely dependent on what your business is and the competitive landscape of it. I don't care how "agile" the cloud makes you, if your cost of IT is more expensive than the revenue you bring in, you're bankrupt. It's the running joke around here about people who build autoscaling k8s based infrastructure for a site that could be hosted on a $5/month VPS with even a little bit of tuning and caching. Complexity for the sake of complexity is called stupidity unless you're doing it as part of an educational experience. Going to production with needless complexity is a great way to ruin a good idea.

>Cost is not the same as value.

I guess it's a good thing I didn't state that cost is the same thing as value?

If we’re discussing major corporations, they are willing to pay a premium for predictability and reliability of service. Cloud vendors provide just that: if you want more resources, you can scale up until you hit capacity, then you can call the account manager to increase it. Very little downtime (except for the occasional outages). All of these are known processes.

With operating your own DC, you have to hire really great people and hope they don’t fuck up. You have to plan much in advance for capacity changes. It’s very much possible that something goes wrong in your compute infrastructure and simply nobody in your team has the expertise to fix it. Put it another way: can you pay/hire the same kinds of engineers that work at AWS/GCP? If not, cloud is the safer route. Cost does matter, and this is why companies sign multi year contracts to get discounts. But business continuity is critical.

It's all fun & games until you can't make payroll.

What does that have to do with the cloud? A company where the "cost of IT is more expensive than the revenue" or has "needless complexity" is making bad decisions and will execute poorly whether it's cloud infrastructure or internal systems.

But if you need a server then (in most companies big and small) it's far easier to spin up a VPS in the cloud than deal with internal IT.

> "I didn't state that cost is the same thing as value?"

You're arguing over cost, but the cloud is often much greater value compared to traditional IT. Spending 2x for 5x more value is a valid decision. If it doesn't work in your situation then don't do it, but it's not the same answer for everyone.

> build autoscaling k8s based infrastructure for a site that could be hosted on a $5/month VPS

I think the goal is that it should be the same amount of engineering time to set up both options, and both should cost $5 per month.

Today, that is broadly untrue. Although a few niches like appengine offer this.

There's more to it than just the tech and what is cheapest though. Like risk management. Cost might be higher for a less-risky thing.

> The money is not the issue.

Money is COMPLICATED and THE issue.

Money is complicated because you have income and expenses. Time to market and velocity can help you get more income. This needs to be balanced by expenses.

When you're growing or your needs are uncertain than a hyperscaler can help you get moving quickly. This aids in time to market. But, you do this at a cost. Hyperscalers cost more than running it all yourself.

If your workloads are steady state or changing at a predictable rate than it can be cost effective to run your own infrastructure.

There are other elements to all of this such as global location of hardware compared compared to end users.

Money is the issue because companies are in business to make money. Sometimes companies take a loss now in order to make money later but it's always about the money.

No, but it can make value go negative. Money is always the issue. When isn’t the issue of a private business not ultimately money?

Cost is a key component of value though.

Like anything it's just pros and cons. Sure you save money by doing on-prem or just buying servers and colocating, but you also get all the headaches that come with that.

I used to work at one of the most popular fast food chains. I had one project that was internal to them and one that was customer facing. The internal product, a search engine for documentation ran on-prem while the e-commerce store ran on AWS. Running the former on-prem was just fine.

I'm sure glad we were on AWS for the customer-facing site, though. We'd run ads during the Macy's day parade, college national championship, etc... dear lord I could not imagine wrangling the necessary people to scale that out from an infrastructure perspective. It was very nice just doubling the number on our ASG, failing over to the secondary database, launching another DB instance with double the capacity, and repeating for the second one.

Between those popular ads and DDoS attacks I'd say AWS was a win there, but internal tools don't benefit from the cloud as much. Also not an MBA, but I feel like that's a rule a thumb you can add to the repertoire.

I don’t see any cost analysis, you’re just glad you were on AWS so you didn’t have to do it. Could be a convenience fee in there big time.

or, more likely, it would have been impossible to scale at a short notice and the site would slow down (with luck) or go offline for a bit (most probable).

It's not about hyperscale, it's about time to market.

Traditional corporate datacenters were dogshit when it came to provisioning new resources for business capabilities.

In today's world, we need the ability to spin up and down deployments very quickly. We don't need huge numbers of resources, just fast response.

Also experience. My team has been doing cloud deployments for the last 18mths, now we are on a gig at a company that has onprem. In theory the onprem infrastructure is comparable to cloud (K8's, GPU's, HDFS, Kafka) but the reality is that to get a set of K8's pods for a release has taken 2 weeks, 10+ meetings (calls) countless messages and 40+ emails.

And the ones we got on Friday were wrong (we asked for 3 * 16GB ones + 1 * 2GB, we got a 16GB project container with 4GB pod limits).

For our hadoop request - yes it got provisioned in >8hrs (mostly I think because no one is using it) but none of the access instructions work and everyone who has used it is mysteriously unavailable - so we can't access it.

Compare and contrast to self service from any cloud.

Why : well it's demand management. If you do self service and say "pay for what you want" then IT's costs bloom and the CIO is shot. If you do that in a cloud scenario then Cloud costs bloom and the CIO says "you guys have spent your money badly". This means that you need police and you need people who you ask to give you what you need if they are happy. These people obstruct you and then get things wrong.

You bring up some great points. There are two that I see.

First, there are internal processes. All too often they end up needlessly slowing velocity. This isn't a technology problem but a people problem. Using a public cloud enables people to circumvent these issues, in some cases.

Second, there are costs. What does it cost to give someone the features they ask for in the time they want it? This is a requirements, design, time, and cost problem.

Yes - thinking about it, it's a market failure.

Basically small markets (internal IT) can't sustain the investment to support the requirements design time cost issue - while the public cloud folk can say "any colour you like so long as it's black" and there are enough takers to pay for it all.

I have worked with hundreds of companies moving to AWS, and believe that this is the answer.

People are investing in cloud because its faster, more agile and better, not because it's cheaper.

The top line looks expensive, but if you then do the correct analysis on the cost side of the equation (taking out staff costs and DC costs) then I think it is either close or in the black from a cost perspective.

If the business case was really bad, I don't think we would see the rapid uptake we have seen.

If you think from an economic standpoint, Amazon should price AWS only a bit cheaper than the on-prem options. It's a way to maximize their profits.

Then you have competition (GCP, Azure, Alibaba) which will tend to lower the price, but for now the cloud industry grows. Their goal is to attract new customers who didn't use the cloud before.

> In today's world, we need the ability to spin up and down deployments very quickly.

Do you really? In my experience, admittedly outside of SV, pretty much nobody needs that.

I think this is a common need, even in smaller organizations. Spinning up/down deployments doesn't imply you're doing it in response to scale needs (most orgs don't have those); often it means you're simply experimenting with new tools or new product offerings. Having the ability to do that without waiting for infrastructure provisioning, and having the ability to only spin up the resources you need to see (for example) if a given version of a web framework can run your site well, or of a new database meets your needs, can be a significant value add.

> often it means you're simply experimenting with new tools or new product offerings.

That only makes sense to me in the context of a business being an IT shop to begin with. Most businesses aren't, and their new product offerings require factories and dies and labor and materials. Besides which, the problem of provisioning is entirely overblown for this scale of business. I can spin up new servers all day on our on-prem vSphere cluster.

sense to me in the context of a business being an IT shop to begin with.

Lots of businesses outside of 'IT shops' use computers. I currently work in Civil Engineering and more and more of our product and service offerings have an IT/Online/Web based component which involves setting up some sort server.

I can spin up new servers all day on our on-prem vSphere cluster.

I what price? At my old job an on-perm vSphere 'machine' cost my department a lot more than what AWS was charging.

We've had the same vSphere cluster for 5 years now. I forget what it initially cost, but over the timespan in question I think it would be very difficult for Cloud-anything to cost less. Hell, the licensing for the software we use to do business costs an order of magnitude more.

I work in a very boring industry very far away from SV and spinning up and down machines for doing calculations and analysis is a godsend. I have a 'cluster' of 6 machines with 128 GB of RAM each set up and ready that I can start up with a script and only pay for the less than 200 hours a year I actually need them. Sometimes however I need 256 GB of RAM so I just change a parameter in my script and, magic, I have 256 GB of RAM. For other workloads the optimal setup might be 100 1 core machines with 2 GB each. So I type some commands, and now I have that instead.

And since it's my 'private' cluster I don't have to worry about queuing my jobs and waiting for the machines to be free. There is no way my department would have bought me those machines as physical hardware.

Fortune 500 companies have hundreds of internal apps that each have their own stack. Multiply 2 or 3 for lower environments.

At that scale in a single company the ability to provision verticals weighs heavily in architecture and design choices.

"Should we spin this new feature out as a microservice or fold it into an existing app so we don't have to wait weeks and do crazy cost accounting for new environments?" And the answer has follow-on effects because now it has to be accounted for in a specific teams' backlog and we encounter yet more waiting.

It's a need born of our tooling. The only way to stay sane is to operate extremely simple services, or to put every service in its own container (or VM) that can be torn down and re-created automatically. The result is a need for one or both of: excellent container deployment tools; fast & easy ways to spin up "a new machine" (a VM) on demand.

have developed on both AWS and non AWS server and db. There is no deference now days. Depending on if you want to hire the right people with the right skills is another thing.

People also seem to be overlooking the benefits of downward optionality - you effectively transform a fixed cost (servers and staff to run them) into a variable one, that you can flex down cheaply (no long-term leases or redundancy payments). It lets you test things out (e.g. entering a new market) and cheaply wind them down if they don't work out.

I have worked on reliability for some private OpenStack deployments, on-prep K8S deployments. Views are my own and don’t reflect on any open source community or company.

To operate a production cloud you need expertise at all parts of the stack. This includes network, kernel, maybe firmware, orchestration, etc. You also need adequate planning where your on-prem data center teams continuously upgrade your hardware/firmware and add capacity. From an organizational perspective this is an entire VP or SVP level organization depending on where you are.

Operating large infrastructure is hard. If it’s not your core competency there is really no point in dealing with it. Either way, it takes enormous effort to hire and keep a team skilled to operate infrastructure. You will lose your team to big-tech eventually.

You don’t need external consultants to manage your cloud infrastructure. Your existing engineers can skill up to use it. Public cloud will commoditize what typically takes a lot of time to plan for capacity, DNS, networking, scaling, etc.

In short - its prioritizing speed of execution over everything else. The opaque billing is just cost of doing business. But unless tech is your core competency, opaque billing is much easier to deal with than all the other situations that come up with managing and maintaining your own infrastructure.

On a previous job we had a huge GPU server on prem because we wanted to use it for deep learning.

It would have been much, much cheaper to rent gpu in the cloud and when we knew _something_ about our workload we could buy an on prem and maybe rent extra if needed.

But we weren’t cleared for cloud.

Regarding in-house expertise: I think most companies would like to have in house expertise but they can’t hire the expertise they need so they use consultants.

that's really surprising to me, because I used to run a GPU cloud that was cheaper than amazon, and the margins were insane.

I've looked for some GPU on the web and it's super expensive to rent. I was expecting to rent that for say 50$ per month (considering I can rent some webspace for less then 10$ a month) but everything was more expensive like 600$/month (I look for entry level GPU, used just a bit, I'm a student) ... Did I look correctly ?

if you want an always-on gpu for $50/mo? That's unreasonable. Consider how much "building from scratch" even a cheap "minimal" GPU unit costs (let's say 3k GPU + 1k supporting PC), whereas a cheap "building from scratch minimal webspace" is a fraction of a single CPU unit, so let's say $100-ish/unit dollars, to be really generous. A rented non-fractional GPU should be easily minimally 40x the cost of a website.

Don't expect these ratios to be authoritative; there are obviously different economies of scale and many many details I papered over, but this should just give you an idea of the back of the envelope "relative costs"; just that 5x is way out of the range you should have guessed.

It all comes down to utilization as a percentage of time. If the workloads are infrequent enough, GPU cloud providers on going to crush the onsite equipment.

However... I was also sort of surprised, because last time I priced this out, you can have pretty low utilization rates of the equipment and still come out ahead.

the other thing to consider though is that if you are under pressure by management to not utilize GPU cloud, you might do fewer experiments than if you had at least a GPU workstation under the desk. (that, presumably, you can share with other workmates)

I appreciate your point about "they can't hire the expertise they need" but I feel it's missing the "because": because those hires are so expensive? Because they only exist as consultants? Why can't they hire the expertise?

For company I am working at, it is acceptable to pay more for even long term consultanst, than to pay market price for experienced developers. The reason explained was, that it is different budget line - HR does not want to "overpay" for emplyees, so consultancy is the only option.

I would so that about 3/4 places I've consulted, I wouldn't want to work full time.

Mostly because they simply didn't know how to run an IT office, and so they couldn't build a culture where good IT people want to work. When they finally get a strong candidate, they end up overloading the poor hire, because finally, it looks like they can get their infrastructure fixed or dream project off the ground.

because nobody wants to be hired for the position. it isn't even about the money, there's simply no candidates - and any candidates competent enough are already consultants and/or freelancers.

I.e. those hires are expensive. I am sure there would be candidates with an offer of $1M/year.

of course, everyone has a number - but also, for that money you'll just hire a consultant and will probably come out ahead.

You have to run the numbers for your specific company, expected scaling behavior, compute work load, and expected system administration overhead and compare all scenarios.

I will say that if your work load is relatively fixed in magnitude and especially if it's bandwidth intensive, you can probably beat the "big cloud" providers by at least an order of magnitude. If it's extremely bandwidth intensive you should never consider mainstream cloud as outbound bandwidth is absurdly overpriced... like 1000% markup or even more. I can't exaggerate how insane bandwidth pricing is.

If you are just prototyping something, your work load is highly elastic, or your product is rapidly evolving, the flexibility and outsourced sysadmin of cloud will probably make sense.

Edit:

There are edge cases too. If your work load is, say, inbound bandwidth and storage intensive, AWS might be great since S3 is cheap and inbound bandwidth is free. Just keep in mind that you might pay a lot for egress if you ever need that data out. Will you?

Run your own numbers and be detailed!

Almost all “on prem” business IT is managed by external contractors. I was one of them. We served about a third of the commercial premises in a small Midwestern city, driving out to visit the server closet, change backup tapes, click through Windows Updates, and perhaps make the rounds of desktop users with any questions or concerns. We sold a monthly package of so many labor hours, and we’d visit each customer site about once a month. We’d get the keys from an office administrator who perhaps knew how to provision user accounts. The biggest, richest customers might have had 1-2 help desk on staff, probably not domain admins and certainly not empowered to do any decision making or design. That was for us.

It was quaint, adorable, personal, and hilariously inefficient.

You're talking "Mom and Pop" scale businesses that have minimal computing needs other than printing and Internet access. Most "on-prem" IT is much larger. My company (also in the Midwest) has mainframes, SANs, vSphere deployments, Nutanix, as well as supporting ~3K users.

These businesses ranged from "Mom and Pop" to about 50 employees. There'd be a domain controller, Exchange server, file server, PBX, SQL server, and application servers for 2-3 line of business apps.

It seems very clear to me that these kinds of businesses are much better served by cloud. No one of them is that valuable of a customer, but there are so many.

Companies that are large enough to have an IT department with multiple highly skilled professionals onsite, or who could even contemplate doing a real data center (most "on prem" is in a closet), I agree is a different calculation.

Big corp financing is weird, and IT is often its own cost center responsible for its own budgets. This means that every time my department asks IT to do something for us they send us a bill. And that bill is always several times more than going to an external provider. At a place I worked, hosting in-house would for example cost a minimum $150/server/month for something with the same spec/performance that would cost $20/month on AWS. It could also take up to 10 days to get that (virtual) server set up. As a project owner and someone responsible for my project/department budget using an external cloud over in-house hosting makes rational, economic sense

Building up an IT infrastructure that can provide services at roughly the same quality and cost as AWS takes a large investment in talent and infrastructure. And many companies don't see the value in making that huge upfront investment for something they might not need. Better to start with AWS, and once you get to the point where you can lower your costs by 50-70%, then start looking at moving in house.

> Building up an IT infrastructure that can provide services at roughly the same quality and cost as AWS takes a large investment in talent and infrastructure

no, you missed an important point here.. no one, at any scale, has been able to compete directly with AWS on the combination of service, scale and cost per unit.. its intensely competitive

Bank of America built their own cloud. They say they save 2 billion a year.

https://www.businessinsider.com/bank-of-americas-350-million...

It seems no company wants to be in the business of in house IT Systems. They think moving all their infra into the Cloud absolves them of all the responsibility of owning it themselves, they don't give a shit how much it costs but then tell you everyone has got to save money and tighten their belts.

BoA’s scale of operations are probably orders of magnitude larger than most other businesses.

Often it's an OpEx vs CapEx play. Also large orgs crave treating staff as interchangeable parts and IT hiring is thorny. Throwing too much money at the problem can be preferable.

Enterprises would rather not maintain inhouse infrastructure for many of the same reasons they engage MSPs despite persistently shoddy services.

It depends on who's running your company. The bog-standard MBA's in middle management love to minimize capital outlay, preferring to expense everything possible. Sure, there are tax advantages, but it becomes a rule-of-thumb in most Fortune 500's, and winds up infecting IT as well. What a lot of blue chip companies STILL haven't grasped is that IT is as strategic, differentiating, and important as any custom machine they can design, build, and install on an assembly line, and the software engineers, as much or more, important as the product engineers.

Scale (of your application) is usually not the reason to use clouds.

Agility, flexibility, security, global deployments, and consolidated billing and operations are far more important.

Exactly. The reasons to adopt (or not adopt) cloud, like any technology, should revolve around business questions, not technical ones. Of course, technical ones are critical to business success, so you can't ignore them, but they are subservient to the business concerns.

Security is certainly and quite clearly not a reason to use clouds. If they are any reason at all it is to avoid them.

But the others are spot on.

Being able to blame someone else for a security problem is a reason to use the cloud.

Security is complex but many companies find it easier to offload it to managed services and best-practices/reference architectures in the cloud compared to janky internal systems.

Offload what exactly? You mean making the same decisions, but through an opaque badly documented panel? Or you mean paying somebody to keep your OS up to date?

Going into the cloud does not alleviate any security decision. You can't buy security.

> "You can't buy security."

Of course you can. In this case, many of those decisions are already made for you in the cloud.

A simple example would be using RDS for a database service. It's secure by default and only the credentials are an issue, compared to running a database of your own somewhere internally that may be exposed in dozens of ways. Less decisions, less surface area, more documented reference setups to follow, more scanning and analysis tools means better overall security posture with less work.

I bet in the 1920s, people were questioning whether or not businesses should generate their own electricity on site or whether to go with the new-fangled "cloud" solution.

The question for businesses comes down to economics, of capital expenditure vs operating expenditure. If I buy my own hardware, I have to spend capital and depreciate it as part of the operating expenses. If I lease hardware (whether on prem or cloud), then that's pure operating expense.

There are accounting and tax implications. There's also the "stick to your knitting" of a business. Do I need my own payroll service, or do I outsource it. Do I need my own accounting, or do I outsource it. Do I need my own cleaning service...

It's not always easy to bypass the cloud, especially when you have to deal with things like GDPR and host client data in their respective region.

MBA here. The cloud is not a good value, the cloud handicaps your organization by placing your most valuable resource - how you compute - in 3rd party hands. Doing the financial math, the cloud is absurd, it is a very bad value; if it were a better value, the cloud providers would not have such high profits. Doing the flexibility math, having one's own datacenter - even a rented colo cabinet - provides far more flexibility than working through a cloud provider's ticketing system. Doing the organization knowledge math, having in-house skills to create, modify, and completely customize one's backend infrastructure with the knowledge that you're numbers are pure - nothing but your jobs are running on your hardware - grants you the visibility to optimize effectively.

I've done this: built a 17 server cluster, housed in a colo, for a high compute startup producing 3D avatars from photos automatically. The hardware for the cluster, including a Federal Reserve class hardware firewall, was just over $50K, with a month to month colo bill of $600. This was in contrast to $96K per month if the same configuration were at AWS. This was 7 years ago, so adjust the prices, but you get the exponential higher expense point.

On the other hand, what if your tiny company spends only four figures per month on AWS and has only a single developer who also acts as a sysadmin as needed? Is it even practical for a company operating at that scale to colocate servers in an environment that has anything like the private network and multiple per-region availability zones of AWS? Not to mention that, presumably, the developer would have to implement and test their own equivalent of AWS's auto scaling and multi-AZ database services. To me, AWS is worth the money.

We colo’d our own servers as a two person team. We didn’t have multiple availability zones, but we’d have the ability to fail over to leased machines if we needed to via automated builds and frequent offsite backups, and that was good enough for our needs. But we never needed to over the few years we ran this setup. This was after running on leased dedicated machines (which is a good first step so you can figure out what you actually need without putting out the money for hardware).

A private network is ridiculously easy to set up. In our case, we had two switches, one public, one private. Every machine had 4 NICs. Just bind some to public addresses, some to private, plug in to appropriate switch.

Unless your traffic is extremely bursty, auto scaling is probably more trouble than it’s worth. We were serving millions of users with three dual-socket machines. And if you really need it, you can try to handle base load with dedicated servers, and bursts with AWS infra. But at <$10k/mo on AWS, you’re not getting anywhere near the horsepower I’m talking about.

> I've done this: built a 17 server cluster, housed in a colo, for a high compute startup producing 3D avatars from photos automatically. The hardware for the cluster, including a Federal Reserve class hardware firewall, was just over $50K, with a month to month colo bill of $600. This was in contrast to $96K per month if the same configuration were at AWS.

You also have to include the human time spent setting up and maintaining those servers. And also the fact that you can probably use a managed service on request instead of 17 servers running constantly. Not even mentioning redundancy.

You can afford far more humans than you need to maintain 17 servers for $95.5k/month in savings…

Yes, the human time is an investment that pays back multiple times over. Why the core of a technology business would be in the hands of a 3rd party dismays me. Of course, those 17 servers composed dev, staging, and live, plus fallbacks, ultimately composing quite a few APIs and related services. They were all used for high compute, where a task would floor 32 cores for 3.5 seconds, and that was essentially what the cluster was for, beyond the accounting and tracking of its use.

> core of a technology business

Most businesses are not technology businesses, they deliver goods or services to their customers where IT is part of their costs to deliver.

Investing in IT people to run that bare metal is a huge expense that doesn't improve your goods or services directly.

As for "where a task would floor 32 cores for 3.5 seconds"... means that your internal IT infra had to support that maximum requirement headroom 100% of the time, even though it may not be used for 99% of the time (depending on load patterns etc).

That $600 figure doesn't include the minimum in-house 5 person staff + 1 manager necessary to run a 24x7 high-9s operation that could run operations such as live-migrating workloads to standby systems in order to fix faulty hardware. Additionally that same $96k could be used to run your compute in multiple datacenters. Back of the napkin math suggests that it'd be a wash once you expand to only three DCs.

I understand it was a startup and may not have needed high-uptime ops from day 1, but a lot of organizations do, and the cost is not that dissimilar.

Yes, it was a startup, with a staff of two. Once we had the cluster up, self monitoring, and rotating on command it is literally a half day a week to maintain the entire setup. The hardware firewall was our best investment, as it load balanced while fending off significant DDoS attacks from hackers. It also is entirely required to use an experienced network engineer for such a task. Far too many people try to wing this, and that's where all the publicized horror stories arise.

Not sure how an MBA helps here, but if all you are doing is straight compute, then yes, cloud is a waste of money. The benefit of cloud comes when you have a bunch of different teams with different optimal solutions such as nosql, sql, queue services, data processing pipelines, data warehouses, object storage, CDNs...

Reply on news.ycombinator.com