Hacker News Reader

Comment by paxys | Hacker News Reader

I was thinking about this recently. I work at a large company with untold millions in AWS spend. I'm 100% confident that I could shave off a few thousands (maybe even tens or hundreds of thousands) from the bill with a little bit of effort on my side. If I go up the management chain and ask if (1) I can make this an official project and put it on the roadmap or (2) I can do this on my own time and keep some % of the savings for myself as a reward, the answer would be a very clear "no" to both. So overall, as an end developer I really have no incentive to work harder and ensure lower operating costs for my company, and I'm sure most developers in the industry are in the same position as me.

View on HN

If your company has a spending commitment with AWS in order to get a few percent savings, and it's just barely hitting the contracted amount, it may not be worth the effort to pursue any cost savings. Suppose your company has committed to hit 5M in spend, and they're just barely inching over the line at 5.01M. You might spend a bunch of time and labor expense knocking it down to 4M of usage and not really move the bottom line at all.

> (2) I can do this on my own time and keep some % of the savings for myself as a reward

This is a textbook case of perverse incentives.

Yep, that's how AWS sales make money.

Theoretically, your company could reduce their commitment to 4M next year, but the AWS sales would start negotiating hard against it, like "you will not get the same discount with less commitment".

Not only will you not get the discount you won't get the commit at all, AWS only reups commits with incremental growth. You can spend less; you just don't get commit pricing.

I agree with this comment.

Ironically, I was asked by a manager some time ago if we can imagine using some (more) resources from AWS to reach the next spending commitment. If you're just below the threshold, it's probably inconvenient.

EDIT: To give some context: you can only do this, of course, when you know that you're not really wasting resources, otherwise you end up with just burning money to save little to no money :)

Spending commitment on AWS? Are you referring to ec2 reserved instances?

Even if you have one year reservations on your instances, starting service migrations/deprecations now would pay off quickly enough. Your commitment expires in 6 months, on an average basis.

When you're a large enterprise customer you get private pricing in exchange for committing to spending a certain amount over a few years. This is unrelated to reserved instances and such.

Can you explain how this is perverse incentives? Wouldn’t this actually be a good way to align incentives of the employee and the company (it makes both of them want to save money)? Or do you mean it makes the employee increase cost unnecessarily from time to time so he can reduce it later?

I am not the parent poster, but I think your last sentence covers it well.

Part of it is that it creates an incentive to create wasteful systems, only to "optimize" them later to rack up a bonus. Even if it gets changed to only pay out for reducing spend incurred by other engineers, it's possible to collude in such a way to extract bonuses from the company.

A better way to have aligned incentives for the company and the employees would be to allocate a bonus pool for the entire company, from which AWS expenses are taken out of, but that might be a bit unorthodox.

> allocate a bonus pool for the entire company, from which AWS expenses are taken out of

Also a perverse incentive.

If we use ec2.small, the customer's query will take 3x longer but be half the price. Let's turn off the nightly security audits. We can live with quarterly backups, right? What do we need all these logs for, anyway? We could hack something that works together in 2 weeks, but if we spend 3 months, it could be really efficient, let's do that...

This is one of many insights that hint at why biz-facing cloud architecture is so popular, wasteful, and profitable.

The incentives are designed to form an enormous cash siphon. From aggressively marketing toward fearful & liable (or maybe just tech-cost-illiterate) upper-management to the silencing effect that the low-rung experts experience when sounding the alarm.

Companies make and spend so much money that this doesn't matter. Thousands, 10s of thousands, 100s is pointless in comparison the potential of building features. A developer costs about half million a year, (salary, bonus, rsu, taxes, benefits)

If they are paying you to saves 10s to maybe hundreds the company is losing money on you so they won't do this.

If your at a public company, look at your company's quarterly reports and see what it would take to many any kind of impact on net income.

How are they going to be sure you're not just farming cobras[1]?

[1]https://en.wikipedia.org/wiki/Perverse_incentive

Maybe there should be a sort of "anti-saas-sales" role: you get commission on whatever costs you're able to justify as superfluous. After all, the person at AWS makes commission selling you the stuff.

Funny downvotes. Although not structured like this value engineering is common in other areas.

Unfortunately it seems like businesses only wake up and asks for cost reduction on the infrastructure spend side once the problem is out of control. At that point, the level of operations effort to get it under control feels more like a "big rewrite" than a collection of small tweaks.

It's opportunity cost. If it's not a problem then you can just make more money with your time.

That's why in our company we have 2 type of engineering effort, core projects impact and improvement. Not all time is spent on the impact. A balancing act is needed

Nah, that'd imply they think about this objectively. Most companies simply don't.

Most time this "opportunity cost" is then spent on useless hacky features that are never used and forgotten right after release (redesign anyone?).

Of course there are exceptions to all of these, but IME the majority of companies are either focusing on pennies or ignoring it completely. Not sure why you don't see balanced approaches more often. Maybe this will change with less VC money flying around.

Not a commission but don't you have performance reviews? Promotion boards? Surely it can count for a bit, if the effort is as little as you say.

some of us care enough about our craft to do it without the backpatting or the extra money.

Reply on news.ycombinator.com