Hacker News Reader

Comment by trabant00 | Hacker News Reader

> This isn't simply about having fewer staff (although that can reduce internal politics and inefficiencies), but making fewer decisions and more importantly reducing the potential for making the wrong decisions

This is a myth directly from cloud providers marketing campaigns: that using the cloud simplifies infrastructure to the point that you don't need as much and/or as qualified staff.

In reality it is quite the trap: without experienced system administrators AWS and the like are free to dig their hands deeper and deeper into the pockets of their unsuspecting customers.

View on HN

> This is a myth directly from cloud providers marketing campaigns

I agree the cloud providers like this idea, but I'm not sure I agree it's a myth. Businesses need to choose what competencies to focus on. For nearly every other area, businesses rely on suppliers. Auto manufacturers use machines built by other businesses and parts built by other businesses. They also obviously do some amount of in-house part and tool design. It's a balance and I think that it's a lot more likely that most businesses will want a good partner to provide infrastructure (ultimately at margins less than AWS) instead of building it themselves.

> Auto manufacturers use machines built by other businesses and parts built by other businesses.

Yeah but they generally aren't renting those machines. Renting critical pieces of business infrastructure that aren't easily replaceable is opening yourself up to rent seeking on the part of your supplier. Remember the Oracle business model--lock people in and then keep raising the price just below what it would cost to rebuild from scratch.

I don't think the renting example is that true. AFAIK auto manufacturers certainly do "rent" many of the factory machinery they use, and they certainly "rent" the maintenance contracts from the companies who service them (the big exception I'm aware of being Toyota, which makes their own factory robotics).

In other examples, many airlines do lease airplanes, and even among the airlines that do own the planes, they still usually lease the engines.

As the GP said, there's a balance to be struck. Delta doesn't have the expertise or the economies of scale necessary to build their own engines from scratch, but they do have the ability to lease those engines and then have their own teams maintain them. Similarly, most companies don't have the know-how or the EoS to run their own data center, so instead they buy cloud services and then have a smaller team to maintain it. It's a win-win. Some companies like BofA might be big enough where they can run their own cloud and it be cheaper than using a cloud provider, but that's definitely not a common thing.

Could you clarify "AFAIK auto manufacturers certainly do "rent" many of the factory machinery they use, and they certainly "rent" the maintenance contracts from the companies who service them (the big exception I'm aware of being Toyota, which makes their own factory robotics)."?

Rental only makes sense if there is a marketplace for used machines. Eg, planes and plane engines are components that aren't really tied to one place or company. (The "that aren't easily replaceable" from the parent post.)

But industrial robots for car manufacturing seems like rather specific hardware. If, say, the Ford plant in Claycomo, Missouri no longer needs a machine for making the F-150, who is going to rent it? How is rental a better cost-savings for Ford than buying the machine and later selling it?

I tried looking this up, but failed. The closest I found was that "over half of industrial robot purchases in North America have been made by automakers" at https://www.robotics.org/blog-article.cfm/The-History-of-Rob... .

Also, '"rent" the maintenance contracts' doesn't make sense to me. I sell maintenance contracts to my customers, I don't rent them, and I don't understand what "rent" could even mean in that context.

The difference between renting and purchasing gets murky (sure, not literally) when you are dealing with sensitive hardware that you rely on but cannot maintain yourself. Such systems usually require frequent maintenance by the supplier. Sure, you could end your maintenance contracts, but then the systems you have purchased will very rapidly become useless. The up-front cost of hardware is often very small compared to the ongoing cost of maintenance contracts, and so the relationship ends up looking a lot like rental.

The parent comment mentions that Delta rents their airplane engines. I imagine that they have the capability to do general maintenance on the engines, but that the engines need to be periodically sent back to the supplier (probably GE or Rolls-Royce) for a teardown, safety inspection, and rebuild. Without regular inspections an engine can't be flown, and is essentially worthless. Not much point in actually owning the engine if you can't use it.

I guess I have a naive view. I think of "buy" as something which is a capital expenditure, can be depreciated, etc. while "rent" are things which fall under operational expenditure.

With that in mind, I can see support contracts as an operational expenditure.

But if "over half of industrial robot purchases in North America have been made by automakers" then that's a lot of capital expenditure. I never got the feel that the companies which make modern industrial robots primarily rented them out (akin to IBM's renting out of tabulation machines 100 years ago). I really thought they mostly sold the robots.

Many of these industrial systems are built from reusable components. In my experience at a Tier 1 OEM for the US domestic auto market, a line could be torn down and re-purposed to another line by adding or removing physical components, modifying the physical layout where appropriate, and then adjusting the PLC programming.

An assembly line isn't really a "product," per se. It's more of amalgamation of several products and technologies, many legacy, that forms a cohesive but loosely-coupled whole. You may have several vendors' products / systems in a line. So what you end up with are ongoing contracts that provide for installation, programming, troubleshooting, maintenance, etc.

There's actually a whole specialized industry that does nothing but design assembly lines. It's pretty neat.

Thank you for your response and insight!

They probably do sell them. I believe most industrial robotics systems are built custom, or at least heavily customized. But I have no idea (and I'm sure they closely guard) what percentage of their revenue comes from direct sales and what comes from support contracts. Somewhat related, I recall reading that one of the things that caused financial issues for GE Power was broken incentives around support contracts. To meet quarterly targets the sales team started offering to trade larger up-front prices for reduced costs on support contracts so as to frontload revenue. Too many customers saw the golden opportunity and GE ended up losing a ton of money.

It's more common than you think. In industries that are locked into legacy apps, cloud costs are massive. Same for industries where storing massive amounts of data for decades is normal. Healthcare and banking spring to mind. I wouldnt be surprised if big pharma are private cloud too.

There are quite many different types of airlines. If you show up for some flight, it might actually be operated by personnel that are on the books of some other airline.

There's Boeing which also used to have an airline. It was made illegal IIRC.

And there are things in between these two extremes.

Some airlines have a heavy maintenance organization. Some have outsourced it. Different companies have different market segments and safety records etc.

The point is, there's no one true way for all.

Why is it better to be locked into your own company’s IT department rather than AWS? I would argue AWS gives a better ROI than 95% of corporate IT departments, and generally less hostile to boot.

One hopes you're not arguing that companies shouldn't bother trying to figure out if they're in the 95% or the 5%.

No, it wouldn’t make sense for me to make the 95/5 distinction only to prescribe a 100/0 response. My post was a call for sober judgment, which is admittedly not very popular here.

Making up opinions and then arguing against them is not popular here.

No straw men here Greg. Looks like you’ve confused this thread with another.

Many people did not realize that IT infrastructure is a utility, it should not be owned all the time by most of the companies but purchased when needed. AWS couldn’t exist if it wasn’t better than on-prem IT. How better is defined is different case by case. Usually it includes TCO, ROI, turn around time, efficiency in terms of capacity management and fee more. These are the things many small companies do not have to deal with or easily solved. Yet there are many companies choosing AWS to solve these problems for them. In my experience AWS really start to make sense if you are spending more than 100.000 / month on infrastructure.

I think a company that saves $2B (meaning they were able and willing to spend even more before they had this idea) is more than able to have multiple competencies. Acquisition and vertical integration is how many companies became massive.

Of course it's a question of scale, but a lot of arguments I've seen usually come down to CapEx vs OpEx accounting which is silly because a dollar is a dollar.

CapEx vs OpEx isn't about raw numbers, it's about who can make the spending decisions and how quickly they can be made. At most companies getting a couple additional servers is a lot easier when you're running on public cloud vs running on-prem.

The general trend with cloud investment is that it makes utmost sense early in a startup or business expansion when time to market is the most important and uncertainty is high, but once the business stabalizes the assets move in house because of costs.

The cloud is powerful, but for long running businesses who can afford to invest in the competancy it's stupidly expensive. Hybrid cloud installations are ever more the norm, private clouds increasingly capable (PaaS, IaaS, SaaS), and toolsets like Kubernetes reshape how a lot of those legacy apps get looked at.

the difference is CapEx is dropped all at once and OpEx is over time. so the accounting is not the same

You can use financing arrangements to avoid dropping the money all at once -- for example: A service provider offers you a monthly subscription or "lease-to-own deal" where you sign a contract to buy the software or hardware in exchange for monthly payments to be made over a period of time; that could be indefinite, or you become the owner of the hardware after the end of its useful life. Or just go to a bank, and they will be happy to create a loan and write a check out to your vendor for the lump sum, and so you are only responsible for making required monthly payments of the interest plus any principal, etc, required by the contract.

So no... not necessarily... CapEx is not necessarily paid out all at once. So the accounting ought to be the same whether its a CapEx purchase under a Subscription Agreement/Loan, or if its an "OpEx" payment for 1 month of services.

in the case of computer equipment that needs replacement very 4-5 yrs (or less depending on compute and other reqs) you’re perpetually in a lease and this starts to become opex

Yeah but you don’t need data center engineers, real estate specialists. Power specialists. Etc.

Sure. You just need more people to keep an eye on what your consumption rates are. Or to handle access control systems or manage service accounts and client certificates, ensure firewalls, additional tooling around firewalls to ensure things aren’t getting exposed outside. Countless cost projections on top of a complicated billing system.

I mean. You save the headcount in one area and you consume it in another. And you pay overhead in outsourcing costs.

It’s likely worth it. But you need to determine that on a case by case basis.

My (large) company is too incompetent to do hosting properly so cloud was a blessing for us.

> Sure. You just need more people to keep an eye on what your consumption rates are. Or to handle access control systems or manage service accounts and client certificates, ensure firewalls, additional tooling around firewalls to ensure things aren’t getting exposed outside. Countless cost projections on top of a complicated billing system.

At a certain scale, you need all these things with internal clouds too. Resource provisioning, basic security, and managing "insider risk" don't just go away when you own the infrastructure.

Not nearly in the same way.

This is like saying that "You also need to have a cashier!" when you're comparing a shopping mall to a corner store.

How so? Presumably someone with an internal 'cloud' needs someone to be responsible for a large kube cluster, to handle a uniform way to do scheduling/resource management, etc. Having an engineer who knows how to do this in AWS doesn't seem more expensive than an engineer who can roll their own internal 'cloud' system...

Just curious, do you put 'cloud' in quotes to indicate that you're using a different definition of the word?

> Countless cost projections on top of a complicated billing system

I think it's a loooot easier to design and build a cost-efficient multi-tiered high availability application than it is to make reasonable cost projections in whatever micro-currency every seconary and tertiary service uses.

And I'm ever surprised by how time-intensive it is.

You can outsource the physical data center and connectivity without paying massive premium for a software stack which in many ways requires more expertise to operate at scale and redundancy than the in-house alternative, while costing an order of magnitude more money and locking you into a potential competitor’s ecosystem.

You can rent dedicated server or onprem ? people always go from cloud to building your own cpu in these threads, never in the middle

Yeah we literally turned our $7k a month AWS bill into $1k a month by changing to 3 dedicated servers (Fully managed!) and cloudflare. It's better in every conceivable way.

For anyone doing this... look around.

Tooling has only improved.

We were able to cram a large set of applications into a group of instances using CapRover.

Instead of 2-3 instances, and a whole bunch of RDS, we ended just recreating the databases there along with minio for storage (we write a small amount of data to S3 storage).

what instance types were you using? do the dedis have the same specs as the ec2 instances? what about contracts (do you have one on the dedis?) are you only using servers or other aws services?

Colocation is another in-between. Build your own server, but the racks / datacenter / power delivery / real estate / network is all rented from another group.

Colocation is probably ideal for anyone using custom hardware: like GPUs or FPGAs. If you're using "normal" CPUs, just buy dedicated instances instead.

Ya, I feel like this must be coming from the younger crowd who is just used to seeing all the cloud marketing comparing cloud computer to building your own data center.

Anyone who has been in the industry since pre-cloud explosion knows that most people just rented bare metal and at the more complicated end they might purchase their own hardware and colocate.

And for those that own their servers, many lease them and HP or Dell or whoever maintains them on site if anything goes wrong.

Yes, this also seems weird to me. And I think it might be a cultural thing, I noticed that in Europe renting dedicated servers is far more popular than in the US.

Maybe it’s because wages are a lot lower in Europe? Better to outsource to the cloud than hire more people if people are expensive.

IDK where you get the 'wages are a lot lower in Europe' thing. There are lower cost and higher cost places, but people are expensive and talent is mobile.

I don’t know, in the Bay Area 150k is entry level and big companies pay 250k or much higher to senior engineers. Whenever I hear about pay in Europe it’s a fraction of that. It may be worth it for a different lifestyle, I’m not making that argument. Regardless, from the company POV it must lead to different choices about buy vs. build.

This is why you see a lot of insurance companies, for instance, with their tech folks quartered in Ohio. Easily half the cost for 90% of the talent.

I'm surprised that more companies aren't playing moneyball with nice, but cheap, locations.

Bay area or NYC sure, that's like being in London, but what about the Midwest, or any of the places where you can easily set up an IT shop without paying a premium for space?

My impression is that programmer and IT salaries are SF>>NYC>>London, even if living costs aren’t similarly different.

how will you attract the talent?

- Get your own bedroom. - 30 minute commute. - Not everyone you meet will be in tech. - You'll be rich compared to everyone else in town. - Partner with local academic institutions to offer recognized research projects and training. 20% time or 3 month project stints etc. Adjunct professor ships for people with an existing research record. - Throw in X free flights to east/west coast, home town etc.

there’s 30 minute commutes in san diego, la and sf if you live in the right areas. what about weather? cali has really nice weather. schools / doctoral candidates / etc are basically useless in the real world. if i’m making enough money the free flights are useless. nothing you said here makes me want to move to the middle of the US. exactly what i’m looking at is the lack of opportunity. say i decide to move for some job and hate it. now what? pick up the family and move again?

The Bay Area is a tiny fraction of the U.S.

Believe it or not most U.S. programmers don't live in the Bay Area.

They 100% are. They're not even close.

More likely about trust/security/privacy issues.

The popularity of MBA is nearly zero in the EU

Renting bare metal is exactly how the business worked prior to the rise of EC2. e.g Rackspace, Linode.

You can rent vs buy at any level : rent only machines, rent a 1/2 a rack, rent a full rack, rent a cage, rent 1/2 a dc, and so on.

Same with connectivity: you can plug your machines into a lan managed by the hosting provider, or run your own routers and peer with their in-house ISP, or you can buy peering directly in the building, or you can rent lightpath from a telco with pop in the building, or you can go rent a backhoe and start trenching across the parking lot...

If you rent the backhoe... make sure you call before you dig and don't cut one of my fibers, please.

Well spotted.

Sure you do, you’re just blending them into the rate, at a significant margin.

Cloud is awesome for companies where the capital investment needed to deliver the SLA is too expensive.

If you’re big, and you run the numbers, lots of services are better in a datacenter that you manage. Many of the early advantages of cloud tools aren’t a differentiator today. If anything, the granular billing is a nightmare to manage in many large enterprises. There’s a reason AWS heavily targets .gov customers — the Feds probably waste billions on idle services.

I’ve found that the shitty to support or known workload services are best somebody else’s problem. Email is an example of both. Most other things are trading IT complexity with accounting complexity.

Counterpoint: Granular billing of real money can actually drive behavioral changes in ways that fake money and/or letting one department slide because you’re hoping it’s head will rubber stamp your promotion doesn’t.

If you reduce companies to "developers hiring developers, led by a couple of business people," how resilient will that company be? I'd say it keeps them at a disadvantage for either elimination or acquisition, but dependent nonetheless.

Maybe cloud usage can be an indicator whether a company has plans to maintain independence into the future, despite the loyalty-engendering words of leadership.

If you build your own DC you would outsource that to a consultancy or contractor.

Nor do you have to pay the rent on the buildings where those things happen.

Why does this argument apply to AWS but not to desktop IT providers, paper office supplies, janitors, cafeteria services, and everything else that a business hires vendors for?

There's nothing simple about debugging opaque error messages & client quirks on AWS.

I don't necessarily think you're wrong, but I think you're missing the point.

Situation: some manager is tasked with updating some antiquated COBOL monstrosity. They're given twelve peanuts, some shoe string, bubble gum and a little bit of duct tape. The experienced staff have all been aged out, quit out of frustration or laid off and replaced with fresh college grads for half the price which was a great deal and significantly reduced costs for several consecutive quarters, earning a well deserved healthy bonus. Unfortunately reliability of the system has been suffering and the young engineering team unanimously agrees the old system should be replaced.

Option 1: do it in house with the young team of rockstars the manager has been bragging to management about. Unfortunately, the manager secretly has reservations and has doubts about the team's experience, let alone skills. It's too risky. Not too risky to the company who can easily absorb the loss and move on, but too risky for the middle manager who will look bad and possibly get fired.

Option 2: outsource the entire thing to IBM or Oracle. Tempting, but not in the budget. Also, significant parts of the old system are on IBM anyway and the cost of hiring IBM people to fly out has been increasingly expensive after the latest round off layoffs saw Scott put out to pasture. No, it's too risky. Not risky to the company who can have uptime guarantees in the contract, but risky to the manager who will look bad if he comes in over budget.

Option 3: The Cloud. It's the 'in' thing. FAANG are all all in on it, and if it works for them it could work for us. Much of the new team used AWS for their web dev class in college, so the skills are fresh. There's very little risk to it. The staffing requirements will be dramatically reduced so costs will drop. The manager will be able to deliver an awesome demo next quarter and the S3 instance they'll host it on will only cost one or two of their peanuts. The only risk is if the manager doesn't get promoted before the problems start, which shouldn't be a problem because AWS has such great onboarding programs. It is slightly more risky for the company though, who may find themselves no longer in control of critical infrastructure.

Risk for the company isn't the same as risk for a mid level decision maker. Small fuckups carry the same penalty as huge fuckups. (you get fired) From my fake story, doing it in house has a relatively high chance of being a minor fuckup, and AWS has a relatively low chance of being a major fuckup. The manager is going to choose the low probability risk, even though the low impact risk is the company's best interest.

And let's be honest... a remarkably large fraction of small IT shops are really bad at certain basic competencies, notably backup reliability and timely application of updates. AWS is pretty good at sysadmin and accounting 101, even if it stumbles at 201.

> Risk for the company isn't the same as risk for a mid level decision maker. Small fuckups carry the same penalty as huge fuckups. (you get fired) From my fake story, doing it in house has a relatively high chance of being a minor fuckup, and AWS has a relatively low chance of being a major fuckup. The manager is going to choose the low probability risk, even though the low impact risk is the company's best interest.

Insightful. Can you clarify what you mean by "low probability risk" and "low impact risk" ?

probability - Odds of a project being a success at achieving it's goals.

impact risk - Odds of a project failure causing large impact to company.

Most medium size companies still would be using a data center provider today to handle the server/hardware maintenance and security. I think it comes down more to how likely is a workload going to change or interact with cloud services or api’s. If you have a CRM system used by a 15 person sales team growing one person a year, it probably is cheaper to self host in some ways. At the same time, if it goes down, someone internally needs to fix it. If all the sudden you want to run some data analytics project on the prospect and customer data, potentially the project is road blocked if more server space needs to be provisioned, etc.

If you already have IT staff in-house it ends up being a duplication of roles and an added layer of complexity. I get some servers and/or resources can benefit from autoscale. USGS gets slammed after a major earthquake so having autoscale capability is important. Or you want to co-locate some servers (payroll) for disaster recovery purposes. There are some legitimate uses cases, but often cloud just feels like renting servers that you could own and manage in-house.

wholeheartedly agree. Also very few companies, need scaling like netflix. But hype of AWS makes inexperienced architect to choose stack without looking into pricing and complexity. Later projects are continued by throwing more resources on cloud.

Reply on news.ycombinator.com