Hacker News Reader

Comment by washywashy | Hacker News Reader

I pretty much never see engineering salaries factored into these types of savings projects. I assume because engineers are already viewed as a sunk cost or maybe it’s just because it’s way less tangible. Have seen many designs describe how X saves Y dollars but ignores the engineering effort to maintain and build it. Half the time I suspect it’s just so people have something to work on, rather than it being some critical fix.

View on HN

A better strategy as a company this size would be to write the PRD for moving and then call AWS and negotiate.

I think it's likely that they tried this. But DynamoDB is expensive to consume probably because it's expensive to run and maintain. If you develop for a particular use case, a lot of optimizations can reduce these costs. For a large enough business, the fixed costs of in-house are easily amortized. It'd be hard for AWS to compete.

That was where my mind went to when I saw the headline. Granted that while I’m not on the Finance side of things and am in fact a developer, “six million” didn’t seem like much at all considering engineer salaries. It’s certainly an achievement, but at what short and long term salaried cost?

$6m across 2.5 years is like $15.5m, how many engineers man-months to breakeven, I'm pretty sure it was worth the work.

$6m in perpetuity is like infinite and engineers can be fired.

It really isn't. The first years dominate the value and later years are worth nothing due to inflation. Google a calculator and use a reasonable discount rate and I suspect you will find that todays value for an infinite perpetuity is a lot less than your intuition might guess. It always surprises me.

But they will save more than $6M the second year because AWS will up their prices.

What?

It costs me $10M to run something every year, it now costs me $4M to run something every year. I have $6M in my pocket every year now in perpetuity. Compound that annually with the assumption that I maintain or increase top line revenues and thats pure extra profit.

Note - I admit, all of this ignores two key things (a) we dont know the engineers salaries who built this and (b) we dont know the ongoing maintenance costs.

A primer on valuation: in many financial contexts, $1 of operating savings may be worth much more than $1 of investment.

That is because an investment is a one-off, so it's actually worth $1, but the savings are recurring, so they are worth the same number of years that a company's profits are valued. Depending on sector and investors' beliefs in the future of companies, this factor is typically in the 5-20x range. That means that $1 of savings is well worth at least $5 of investments.

Factor in anything you want!

In an ideal org, perhaps. In many places, forming that team starts a process where it continually finds reasons to still exist, so your $1m is yearly until a reorg.

Sigh.

If that $1 of investment doesn't yield any returns that's not an investment it's just an expense.

So yes $1 of savings is worth more than $1 of spending.

You could potentially take it as an investment loss for tax purposes. Whether that’s proper depends greatly on the circumstances surrounding how the money was accounted for and spent.

That's not how taxes work. Investing $1 doesn't count as a loss for tax purposes.

I did not claim that investing $1 is a tax loss.

If anything, it reduces Uber's exposure to AWS' proprietary technology. I don't know how to measure how much that's worth but they probably do.

Companies this size almost certainly have different terms of use. I worked for a smaller, but still ASX200 company that had a custom contract, and assigned staff that would drop by 2-3 days a month. Of specific note was that if AWS wanted to stop doing business with us they had to give at least X notice (from memory that was 12 months for us).

For our risk profile this was more than enough time to migrate off any AWS' proprietary technology.

That makes it worth less to avoid exposure.

This usually comes from people who have never done a mass migration at scale.

You’re always dependent on your infrastructure. Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate.

No, just use Terraform and Kubernetes is not the answer.

The typical enterprise is dependent on depending on the source between 80 - 120 SaaS products - ie outside vendors.

*> Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate.

I'd assume it takes fewer millions to migrate your own tech stack from AWS to somewhere else than it takes to migrate from AWS proprietary solutions. Is that reasonable?

No because you still have to deal with permissions, integrations with AWS services like networking, training, security audits, regression testing, often physical network connections (Direct Connect), DNS…

And you’re dealing with your PMO department, project managers, finance, security, contract negotiations, retraining your ops department…

And you know that Aurora MySQL instance that was suppose to prevent “lock in”? I bet you someone somewhere in your org thought about creating an ETL job and then said forget it and used “select into S3” to move data from MySQL into S3.

As a project manager trying to ship code so you can show “impact” to put on your promo doc, are you going to choose for your team to spend weeks to write an ETL job to prevent “lock in” or are you going to tell the developer to write that one line of SQL?

There are all sorts of choices you can make that will save time and money and ship features that actually deliver value instead of worrying about the boogie man of “lock in”.

And I really hope that there was some better technical reason than just saving $6 million dollars a year for a multibillion dollar company to go through the migration.

Thanks for the insights. So in the case that it's actually more expensive to migrate your own tech stack somewhere else than, say, migrate from AWS proprietary to GCP proprietary, it seems there might be other reasons.

The difficulty would be worse of course if you depend on anything proprietary from the cloud vendor.

But the main question is, once you do all of this work and spend time to be “cloud agnostic”, does it add business value?

In the case of Dropbox, it made sense to move from the cloud. In the case of Netflix, they decided to move to the cloud.

But you can’t stay completely “cloud agnostic”.

Let’s take a simple case of using Kubernetes and building the underlying infrastructure using Terraform.

The entire idea behind Kubernetes is to abstract your infrastructure - storage, load balancers, etc.

But eventually, you still have to deal with what’s underneath. I used AWS’s own Docker orchestration service for years - ECS. But I just learned Kubernetes last month.

I still had to know how to troubleshoot problems with IAM permissions, load balancers, view CloudTrail logs for permission issues, know how the underlying storage providers worked, make sure I had the right plug installed for K8s to work with AWS’s infrastructure etc.

Once I got all of that figured out, then I could go through the tutorials and mind map the difference between ECS and AWS’s Kubernetes implementation - EKS.

But I had years of experience with AWS. I could have never easily troubleshoot the same types of issues with Azure’s or GCP’s version of K8s. Now multiply that by an entire department.

Once everything is configured correctly, a developers experience would be the same across environments

Migrations at scale are always a pain from one system to another.

Source: I worked at AWS in the Professional Services department for three years. I’m mostly a developer and I dealt with the “modernization” side of “lift and shift and then modernize”.

Indeed. Everyone enjoying the Broadcom / VMWare situation is running around saying how glad they are that it’s “just a bunch of VMs” and this migration will be a walk in the park.

It could be worse. It could be better.

$6M/y is something like 20 heads (depending on where they are, could be more). So probably it's a win. Hard to see that this could take more than about 5. Add cost of hay and water of course.

It's less that 20 heads. The gross spend for each engineer is probably closer to $0.5 million annually. You can layoff 5% without any impact on the company and save so much more. A company like Uber ($130B market cap) isn't going to bother with building something internally to save $6M/year. The only reason to do it is improve efficiency that actually improves the user experience, which then we're talking about a big deal. Sometimes those things happen only because engineers don't have anything else to do and someone needs a promotion...

is it? if you consider the value 20 engineers could drive instead in that time

if you assume they wouldn't have had anything else meaningful to work on during that time to save money, then you have a different problem in the company. $6M seems like the value 1 engineer can drive in a company at the scale of Uber

You don’t need to consider the cost they could drive during that time. You have a direct and tangible savings for engineering time invested. That possible value they could otherwise derive is moot and hypothetical, this is the real deal!

But if we’re being honest, there isn’t actually any meaningful quantification of engineering time to understand return on investments at this level (not to say there’s none, but it sure does get wish washy). Corporate and engineering strategy isn’t so carefully weighed, and to believe otherwise is to fall victim to the pseudoscience that is software estimation. You just have to estimate directionally if a given proposal has you heading in a better direction in the long term, pursue that, and course correct along the way.

Put another way, the end state justifies the means and resourcing. It’s rarely possible to fully understand either the costs or benefits with much accuracy up front. You slowly put more resources into projects that show promise, and revoke them if the projects do not appear to be heading in a value add direction.

>You don’t need to consider the cost they could drive during that time.

You don't need to, but you 100% should. "Opportunity cost" (cost of not doing something) is real.

This is the problem with all refactoring/migration projects. It's very easy to get a lot of people to agree a company should migrate from Node to Go or Monolith to Microservices (or to clean up a mountain of tech debt), but it's much harder to justify the time it takes away from building things your users care about.

True, but often the project that was supposed to build something users care about turns to dust. On one side, you have rosy projections. On the other, a cap on gains, so sure everyone picks the first, but nobody measures if it worked.

One can build a great career working only on key, promising initiatives that never amount to any value in the end. By the time it's clear the project lost money outright, you are on to something else.

A quick look at their careers shows they hiring eng seemingly anywhere but the US so maybe they've already saved the dollars there.

At Google we had a conversion chart between things like memory / cpu and SWE hours, so if you had an idea that would save 8TB of RAM and take 4 hours to do it, you could decide if it would be worth your time or not.

Because they often are a sunk cost.

E.g., you can’t lay off an entire SRE team and have nobody on the on-call rotation. If some of their project work is cost control that is basically free cost savings.

Spot on. $6M/annually is not much of a saving for a company like Uber ($130B market cap). It only makes sense if it's also more efficient and actually improves the app.

Yeah in one quarter from a quick google it looks like Uber profits $1.6 Billion. Basically why I never thought about cost savings projects after I learned to put things into perspective.

People really struggle with large numbers in business I've noticed.

Reply on news.ycombinator.com