Hacker News Reader

Comment by zentiggr | Hacker News Reader

As I read through the 50-odd comments, I wound up with a kanban / 5 whys level thought. All these points of discussion revolve around pluses and minuses of metrics, but the next level is "Why establish metrics at all?"

At a deeper (very generalized) level, we've been infected by production line, Deming style, efficiency focus.

If you are handling previously defined routines, great. Optimizing can help you.

As soon as you are working with adding value, be it in code, education, product development, metrics are all premature optimization.

Promote the idea that you can't apply optimization techniques to creative steps and encourage managers to get off the numerical crutches and make value judgments. Takes a better manager but then that's the point anyway, right?

View on HN

"we've been infected by production line, Deming style, efficiency focus"

Could you say more about this? I'm trying to form an opinion on this Deming-ism. My impression is that the actual Mr. Deming gave different advice: "Eliminate slogans, exhortations, and targets . . . Eliminate quotas, numbers, numerical goals."

I presume OP intended to write Taylorian?

Possibly but Taylorism is a term I have heard used

https://en.wikipedia.org/wiki/Taylorism

Yes, to me it seems that the real enemy is more Taylorism, especially in how it relates to advance planning.

Generally I think the problem isn't in goals and measures, but in poor goals and measures. Deming said "The aim defines the system". So if you focus on short sighted metrics you will define a poor system. Metrics and reward systems need to be carefully considered and well thought out with regard to the larger scale system outcomes.

That, in turn, tends to have a lot to do with the incentive structures of the person coming up with the metrics.

If you've got someone who's been tasked with improving large-scale system outcomes, and they're developing metrics that are meant to be a means to that end, then you might expect them to come up with a well-crafted set of metrics. Of course, you can only expect them to implement those metrics to the extent that they're able to alter the system in a way that allows for collecting them. The best plan in the world isn't worth much if nobody has the power to execute on it.

If, on the other hand, they're being designed by someone whose task was, "We need to be more data-driven; come up with some metrics and give me a dashboard," they're going to do exactly that. No more, no less.

> the problem isn't in goals and measures, but in poor goals and measures.

Part of the problem is that, in some[0] cases, all measures are poor.

0: probably many, but it depends on how you count[1]

1: which, for that matter, goes right back to the original problem

If it were only premature optimization, it wouldn't be that bad. But it tends to lead to actively counterproductive optimization. Teams get incentivized to do things that increase their KPIs even when what they're doing delivers negative value.

For example, I've definitely seen this happen on Agile teams where management makes KPIs based on velocity or ticket flow: Suddenly, lots and lots and lots of waste and technical debt are generated due to implementing unnecessary things. Because unquestioningly implementing tickets as fast as you can increases velocity, while spending time to stop and talk about whether or not something is a good idea, or can be solved in a better way, decreases it.

I can think of two deeper whys.

1: Dilution of responsibility.

Metrics are defined by committee, which in turn answers to another strategic leadership committee. No single person is responsible for failure. Mediocre managers recite the entrenched KPI mantra because it cannot fail.

2: Differentiability.

If you can't measure how well you're performing, you can't tell if you're improving or regressing with any given change. Total revenue or other coarse metrics are so far divorced from any given decision that they can't provide feedback about it. Just like gradient descent needs the underlying neural network to be differentiable, a business strategy needs the performance to be differentiable. KPIs are an attempt at that.

Ultimately, KPIs suck but give a fairly predictable average outcome. Not doing KPIs is a risk that most leadership is unwilling to take. KPIs sit in the same category of concepts as airport theatre security. They're a product of (find something you can measure and try to manipulate that number, no matter how meaningless | something must be done. This solution is something. Something is better then nothing. Therefore let's do this).

> If you can't measure how well you're performing,

There are a lot of assumptions baked into that statement, not least among them: (1) that being unable to put a number on it means you can't tell if you're improving or not; (2) that this all boils down to a number anyway (or a reasonable set of numbers); (3) That such numbers are actually predictive.

There are two primary reasons people complain about metrics in my experience: (1) They're difficult/tedious to get and/or (2) they lead companies to focus on the wrong actions. These would belie the claims you're making. (I can't tell if you're making them personally, or just echoing what you've heard, so I'll argue against the generic theory)

Turn your scenario around: We have a system. Experts in the field agree the system is more likely to cause poor decisions than not. Should we continue the system? (And trying to gather metrics on whether the system works or not is not the solution, because the question at hand is if metrics accurately guide decision-making.)

You are preaching to the choir. I am just explaining why KPIs are prevalent and probably not going away any time soon. The business world is full of mediocre people who will cling to mediocre but predictable methodologies rather than take any kind of risk, especially if that risk is based in their own competence.

The devil you know, etc.

In the expected environment such as large corporations, the desire for an objective (even if bad) metric is driven by the fact because yes, being unable to put a number on it does mean you can't tell if you're improving or not - in every case you'll have some people making points why they believe it's improving and other people making points why it's becoming worse, with the arguments mainly driven by politics; in the absence of such numbers, any claims about a suborganization will be even less reliable as bad metrics - bad metrics can be manipulated, but no metrics means that claims can have no relation with reality at all.

And regarding your latter paragraph, the answer to "Should we continue the system?" depends on what alternative systems can plausibly be implemented for making the same decisions. If the current system causes poor decisions, but the alternatives are even worse (or there are better alternatives which we are unable to implement), then we should keep the current system.

Yes, the whole system would work better if people could just agree on what best needs to be done, and which activities are useful for achieving these goals, but often they can't. Metrics are a solution to a particular problem - an environment with limited trust and plenty of subjective politics, where any non-metric system for evaluating success and assigning benefits will be gamed as well. There possibly can be other solutions, but "simply" not having that problem isn't a solution.

I'm not sure how to train machine learning models before defining metrics. I wonder how many other approaches/tools make metrics unavoidable. I think the key here is not marrying the metrics, and being happy to swap them out.

Which businesses succeed without a business plan? Without metrics, how do you keep your runway open?

People manage to survive without planning out the number of breaths they will take every day. A competent engineering team is going to do well even without target lines of code per second goals.

I would argue that's counter to your intended point though. If you go into the E.R. or the hospital, one of the vitals they measure is in fact the number of breaths taken per unit of time. In a normally functioning human we don't have to worry about that metric because it is self regulating. If on the other hand you are in critical condition, measuring your breaths is a very important metric.

So as I've said elsewhere, the problem isn't measurement. The problem is as Deming said "The aim defines the system". Well thought out measures in accordance with clear goals and aims can be valuable. Like anything though, it requires careful consideration and may need to be revised based on outcomes.

I see a lot of straw man arguments in this thread. There are bad metrics - I think that’s uncontroversial. But you (and others) appear to be making the argument that there are no good metrics - and pointing to bad metrics is irrelevant there.

When it comes to creative endeavors metrics rarely assist (ignoring code performance metrics like resident memory size or movie sales)

You focus on your goal and try and take the best path there. LoC written or test coverages are distractions imho. Trust the skilled talented people you’ve hired as they are motivated to make the correct choices.

Reply on news.ycombinator.com