Hacker News Reader

Comment by zevv | Hacker News Reader

One recurring pattern I have seen at multiple customer sites is that scaling makes the engineers lazy to optimize. One production performance calamity requires the team to add CPU as a quick fix, and from that time on the baseline for the product's requirement has been set to the new number of CPUs.

"Back in the olden days", if your product was slow but the number of CPUs was fixed (or could not be increased instantly), the solution was to go and fix your code.

Basic system level skills are now no longer taught or practiced at the appropriate levels, so teams end up without engineers who actually know how to profile and optimize.

The cloud providers are the big winners here.

View on HN

"scaling makes the engineers lazy to optimize"

I've lost track of the times I've heard "compute is cheap! engineers are expensive!" Except... that compute cost will live forever. The time it takes someone to debug a bad loop or poor query is at worst a one time cost. Longer term, it may even make other stuff faster in the future.

Looking at some numbers on cloud, yeah I don't believe in that statement anymore. End device compute might be cheap, but cloud services certainly are not.

We've also seen a rise (or maybe I just notice it more) of stories of "I changed these 3 lines in 5 minutes and saved my company $40k/month!"

And then you'll get responses like "pfft.... that's hardly the cost of one part time FAANG person who makes $680k/year - what's the point?"

And around we go...

Because it's cheaper now to throw hardware at a problem than actually try to fix the code or root of the issue. This wasn't the case a couple of decades ago.

It also means less engineers are needed for most companies.

> Because it's cheaper now to throw hardware at a problem than actually try to fix the code or root of the issue.

It's not cheaper, it's just more opaque.

Back when your service was deployed on that 2 CPU box and it was too slow for obvious reasons, you optimized it and then it was good.

Today you just shrug and increase that kubernetes cluster from 16 to 48 nodes and forget about it. Costs a lot but the bill shows up somewhere else, in most groups the engineer doesn't even know what it is.

It isn't engineers that are lazy, it is management that tells the engineers to work on features that make money not performance that will save money.

Sometimes management is correct in that decision, sometimes it is "penny wise, pound foolish".

"scaling makes the engineers lazy to optimize"

More often I think that's more of an overall engineering department time budgeting / culture issue.

Reply on news.ycombinator.com