Hacker News Reader

Comment by throwaway2037 | Hacker News Reader

throwaway2037

Most large corporations treat these categories of employment as different budget line items with different rules and limitations: (1) full-time employees, (2) individual contractors, and (3) large contractor "body-shops" or outsourcing providers. Many times in my career, I have seen layoff a few from (1) then way over spend on (2) or (3). The mid-level manager who makes the decision gets to "claim" that expenses were reduced in (1) and "win" at year-end reviews. Yes, I know: This is total non-sense, but I have seen it many, many times at mega-corps.

View on HN

Related to this, the massive advantage of AWS is that it allows staff to buy infrastructure without having to raise purchase orders. If you ask permission for spending, it's a very onerous and frustrating process. If you just deploy stuff and get billed for it, it's much easier! Even if that's more expensive than having your own cloud. Worse, if you have on-prem, you have to have staff. They might even be permanent. Businesses hate having to have important staff.

Not coincidentally, this results in massive overspend until someone notices and has to painstakingly go round checking what all your instances are for. And AWS is very profitable (well, margin rather than accounting profit).

Now, look at the billing model of AI, especially once flat rate goes away. People can spend millions on tokens without ever having to ask a manager! Obviously this is going to rake in money hand over fist, because it will be years before anyone catches up to ask "are we actually getting value for money here?" rather than "quick spend more tokens".

> If you ask permission for spending, it's a very onerous and frustrating process. If you just deploy stuff and get billed for it, it's much easier!

This point is underappreciated as it appears in many forms and can really help reconcile things that seems obviously wasteful (they may actually be wasteful, but sometimes financial structure makes this hard to determine in an honest capacity).

Capital costs and operational costs are a similar dichotomy. When I was in graduate school, the university was breaking ground on new buildings at the same time that staff layoffs were underway. On its face this seems grossly unreasonable, but staff salaries were paid from one funding bucket and capital improvements (new buildings) were funded by a completely independent state-level allocation process and those buildings that were breaking ground had essentially been locked in 5 to 10 years prior.

> massive overspend until someone notices and has to painstakingly go round checking what all your instances are for.

and when you reduce your companies AWS spend by 30% (which is usually easy) you get a good review/bonus/promotion. So not really much incentive to pay too much attention to cost until you can get recognition for doing so.

There are only a handful of companies in the world that AWS would be more expensive than having your own cloud. Just the baseline costs of having 24/7 availability requires minimum 5 full time engineers (3 shifts + cover for sick / vacation). So that's $1 million a year right there, and we're already well over the cloud spend of the vast majority of companies. If you want multi region that's now $2 million. And we're still just covering the cost of the people who have to be on call to deal with hardware issues. Now how much is it going to cost to develop the software tooling required to manage the whole thing from your office?

I agree with the basic premise and math here but getting to true 24/7 uptime in the cloud will require a minimum of 2 full time cloud engineers or infra people. If you're spending another 500K in usage costs you're almost at a million. For most, the cloud is the right option because maintaining servers is a pain and modern clouds offer a lot of additional features that may be useful, but it doesn't mean it's a massive win for the bottom line. There are lots of small and large companies that could run servers on prem or in their own datacenter and it would make financial sense, but they'd have to attract the talent to build and operate, which is harder than just paying enough.

This is apples and oranges, though. AWS does not manage your whole product. Are you going to have 24/7 on call for the parts that you manage? If not, why do you need it for the parts AWS manages? The reality is that most smaller companies do not need this level (and frankly I think most people significantly overestimate the overhead of running your own box. A heck of a lot of cloud deployments could be two servers in a co-location with one of them being a warm spare, and have equivalent service levels for their product in practice).

Eh this is true but in practice "my RAM stick went bad" is 0% of our downtime. It's almost always that we screwed up something in the app layer. And my company of ~100 does have 5 engineers on call 24/7. It's baked into our salary to do that.

(1) is considered a cost while (2) and (3) might be called an investment. Company can lie to investors that (2) and (3) costs could be removed at any time without interrupting any important business processes.

>The mid-level manager who makes the decision gets to "claim" that expenses were reduced in (1) and "win" at year-end reviews.

I worked at a company where every 6-sigma project ended up that way. They eventually had to stop giving bonuses for 'saving money' because the projects would only save money for a month or two before ending up costing extra in the long run.

Reply on news.ycombinator.com