Comment by ar_lan | Hacker News Reader

Comment by ar_lan | Hacker News Reader

Parent Full thread

Humans have proven time and time again that we are very good at optimizing for specific metrics, and apparently generally very poor at choosing the right metrics to target.

School is a prime example, in my opinion - there is so much I "learned" in school that I'll never remember, because I didn't take a proper approach to learning - I just optimized for grades. I had a 4.7 GPA and could hardly tell you a thing about US history or recite a lick of Spanish, because I simply did not care about anything but getting the A so I could get into the college I cared about.

In the work environment, I can see a lot of parallels to this. If I'm competing with my coworkers, my incentive is to outwork and outshine them. A common thing I see is when a coworker does a bit of innovative work - it's almost guaranteed some other coworker will intercede before a chance at applause is given to discuss its obvious flaws, the plethora of alternatives out there, etc., leading the developer to feel like their implementation was not good. (For the record, I'm not talking about general criticism - I'm talking about the general pattern of not celebrating someone's achievements and then discussing how to continue improving, but rather a "why did you even do this" mentality).

Planet Money had an episode on such metrics[0]

The host shares a personal story from time when he was a cashier. The grocery chain measured performance as items scanned per minute and shared internal leaderboard.

Some items are hard to scan so the cashiers ended up skipping scanning them – so the store gave them away for free. But the managers were happy about their KPIs. (Look how many items are we suddenly selling per minute!)

[0] - https://www.npr.org/transcripts/669396192 (both podcast and transcript)

I love this podcast.

Costco has such leader boards and keeps a separate board for part timers vs full time cashiers. I asked my friend who works there why the part time cashiers have a much higher (upwards of 2x) items per minute rate than the full time employees. Costco only schedules part time cashiers during busy hours where the lines are long and there are assistants getting the carts ready.

I heard about a call center that rewarded customer service agents for short calls, so workers started immediately hanging up on some calls to bring their average call duration down.

I can verify this does happen in some call centers.

I swear this happened to me with Grubhub. I had a problem with my order and used their chat. After I wrote my problem the agent replied with a standard answer and then immediately:

"Is there anything else I can help you with?"

And then immediately after that:

"No, ok I will close this chat."

> I heard about a call center that rewarded customer service agents for short calls

Almost all call centres reward agents for short calls.

There's a target time. Something like 6 minutes. You can't just hang on customers

Smart companies optimize something more like "percent of calls resolved on first contact" (or "1 - percent of calls requiring the customer to call back").

The supermarket I worked at in high school did this, but you usually didn’t get incentive pay if you worked less than 15 hours a week.

The supervisors were pretty swift and always looking for shrink. If you were letting dog food or whatever walk out the door, they would figure it out quickly and you would be gone.

The GMs were well paid and profitability and shrink were their metrics.

> Humans have proven time and time again that we are very good at optimizing for specific metrics, and apparently generally very poor at choosing the right metrics to target.

Maybe it's Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."

As another example: Agile dev team measuring velocity (story points per sprint) to predict how much work they can do in the future.

If velocity gets turned into a target, corners will be cut (less testing, poor code quality) in order to get those story points for that sprint. Code quality suffers and tech debt accumulates, reducing output down the road.

On the flip side, as time goes on teams could just progressively estimate a greater number of points to stories of similar difficulty during the planning process. So velocity goes up but actual deliverable features goes down or stays the same.

I agree that it is easy for a metric to loose it's original meaning. I've had luck with OKRs; Having a non quantitative goal, and possibly changing quantitative based targets aimed at making the goal happen.

One cannot overstate this and human capability to optimize for metrics ( and what is apparently rewarded ). Anecdote time. My former manager had an idea to measure the amount of average alerts per hour and tore into people, whose values fell below a certain treshold. What did people do? They stopped doing harder alerts whenever they could and did 'easy ones' to pump up their stats. I have seen it since and whenever I do, I have a quick talk about people not being idiots and importance of metrics not being used for evaluation ( or if you do, to be ready for its consequences ). In case you are wondering, former's managers response was to double down and hide the stats so that people don't know where they stand.. eh.

"When a measure becomes a target, it ceases to be a good measure".. I think this phrase all too often... haha

If we don't have metrics, how are we to fairly and objectively compare one worker's performance to another?

It's not that metrics in general are bad, it's that bad metrics are bad.

What's the best metric for metric quality?

It's metrics all the way down...

How is obsessive focus on metrics making anything more fair? Even if you have metrics, choosing which ones are important still ends up being subjective, and the metrics are usually incomparable between projects. So the metrics alone don't tell anything useful and can still be easily gamed to make the manager's preferred employees look good and disliked employees look bad.

Metrics are CYA insurance for managers to promote or PIP employees without getting into trouble with HR

Let me ask you another question. How do you identify the additive employees from the multiplicative employees? You probably don’t want all multiplicative employees on a team, but you surely want some of them. How do you create metrics that don’t encourage too much of one or the other? How much is too much of one or the other?

The performance evaluations I've been through claim to be evaluating how an employee does vs expectations. That doesn't demand comparing workers to each other.

In theory, you have a list of things to check for each category of performance. If you tick enough boxes, you get a certain category. In reality, it seems managers stack rank, and then apply the required curve to the stack rank. If everybody on your team does their job as expected (or better), too bad, X% need to be flagged as poor performers as declared by the performance curve.

I mean, what if you just don't?

at my company, raises have more to do with the financial situation of the company than subjective reviews (although we do have these, they just don't factor into compensation that much). if the company is doing well, everyone gets a nice raise. everyone gets the same percentage bonus every year which is calculated based on the entire company's performance. maybe this makes us unattractive for very competitive people, but it leads to a very cooperative environment. senior devs don't really have anything to lose by spending a few hours helping newer people, so they just do it.

the theory is that high productivity people will not want to work there, as everyone else benefits from their excellence. If companies that retain these high productivity people perform better than companies that don't, evolutionary pressure will force these egalitarian companies out.

I think this theory may be true in some fields like hedge funds or heavy sales performance based businesses.

In software, I am not sure high productivity people care about this issue, as long as their compensation is high enough.

> high productivity people care about this issue, as long as their compensation is high enough.

That circles back to the original problem of how do you identify the high productivity people to give them high compensation. It works when the person spending the money is close enough to evaluate employee performance. But when it's some middle manager spending Zuckerburg's money you run into the agent/principle problem. The managers will do what's best for them (raises for everyone!) and not the enterprise.

I think high productivity people by definition cause everyone* else to benefit from their excellence.

So high productivity people who don't want everyone else to benefit are a bit of an oxymoron.

(* or at least most others)

LeBron James wants his whole team to succeed, but he still expects to be paid the most.

If you can afford LeBron, great!

If you can find a dev who can do for your company what LeBron can do for the Cavs/Heat/Lakers, great!

The reality is that you should be shooting for Shane Battier.

Being productive and not blocked is all some people want. Some might want to do less. And some might want to feel like they are the mvp. Michael Jordan didn't mail it in because Scotty was sitting at home.

Well, one reason is that without metrics, prejudice and bias is much more likely to win the day. "I just don't feel like they're doing a good job" is affected by all kinds of things unrelated to actual performance. At least having some metrics require you to think about whether that feeling is backed up by reality.

I've been thinking about this recently, what if we're doing it all wrong? what if, instead of your manager giving their reports a review, and then also giving raises based on that we go with a more market-based approach. Hear me out:

Every year, and once a year, all the managers in the company submit bids to the employees that they want to work under them. Employees then select their favorite bid, and that becomes their new manager. Employees are free to accept/reject bids based on any criteria: Salary/PTO/On-call requirements, etc.

There's a company that does this and more:

https://news.ycombinator.com/item?id=20252759

Hah, I think I heard of that company on a podcast, it may have inspired the idea

You are conflating justice with “objectivity”. They are not the same thing.

How so?

Metrics don't mean it's fair it just means you're judging someone on a handful of numeric values rather than them as an individual. If you judged me on the metric of turning up on time I'd be doing appalling [0], yet I'm still one of the first people to be brought in on every critical project in the company.

Obviously metrics vs just manager instinct/observation will benefit and harm different people in different ways.

A combination would be better. But life isn't really fair or objective, end of the day I'd rather work with someone I like and can socialise with and know working with them is stress free (which means my work is better) and can be trusted to just do their thing even if their performance is lower than someone who's a pain to work with but maybe puts in more hours.

Making software is a combination of engineering + artistry, it's not flipping X burgers an hour, objective performance isn't important it's the contribution of the whole individual.

[0] : I did have a manager judge me on this before, all it meant was I'd stay up all night, go into work, sit there like a zombie doing close to nothing, go home then sleep. Doesn't matter if I was barely doing anything, by their bum on seat metric I was doing great.

What is better, an objective measurement of the wrong thing, or a subjective measurement of the right thing?

Which is more fair? Note that objective measurement of the wrong thing will reward people for stuff other than quality, and probably stuff beyond their control.

Can you do that even with metrics? I really doubt that.

no better alternatives, if you do not do grading etc at school then what we have is a small group of kids learned a lot on their own(1% of them I guess), the rest 99% will just waste their youth totally and learned nothing at all.

Reply on news.ycombinator.com