Hacker News Reader

Comment by networkchad | Hacker News Reader

I’ve often wondered what is the solution to Goodhart’s law. Obviously a business can’t abandon metrics. Perhaps this is where qualitative management skills come into play - humans in the loop making good decisions instead of blindly marching to the output of an automated reporting system.

View on HN

I'd argue that the point Goodhart's law isn't that metrics are bad, but that all metrics aren't created equal. In the case above, it seems like the real goal was improved code quality. Number of unit tests isn't the best proxy for that goal, so it wasn't the best choice of metric. You don't want developers creating tests for the sake of creating tests. (There's some irony here in that it's not the type of behavior I'd ascribe to professionals). The "solution" is a metric that's a better measure of what you actually want.

Even professionals have limits. I've worked in companies with the kind of management which kept adding this bad proxy metrics and pushing initiatives which had a totally expectable bad effects on the product quality. Most devs used to fight the management on this, but grew progressively tired of this continuous fight. At some point the experienced devs either left or just gave up and started giving the management what they asked. Us juniors followed suit. The management was happy, the actual workload diminished because we let go of "low priority" tasks and we even go a juicy bonus at the end of the year because of how good we were doing.

The company tanked six months after that, now it doesn't exist anymore.

There's only so much you can do when the management is hellbent on doing stupid things.

You might be misconstruing the point. I certainly wasn’t insinuating more and more metrics. If anything, it’s the opposite: a core understanding of what’s really important helps you focus on the few metrics that matter.

In that context, I’m not really sure what point you’re making, unless it’s just to share a personal anecdote. Are you implying that management shouldn’t have any quantitative measures and should only be qualitative?

You need good quantitative measures, not just random numbers.

If you sell, say, water bottles, you probably want to know how many of them you can sell at any given moment, in order to not overbook and have to reimburse people. In this case, keeping track of how many water bottles you do have in stock probably helps, keeping track of how many labels with funny jokes you can stick on a shipping box in an hour doesn't. But if you start tracking the latter and handing down bonuses and layoffs based on it, people will max that metrics out - at the expense of your actual stock capacity.

Quantitative measures are dangerous, especially in the hands of people who believe they are better than qualitative ones because they're "objective" or whatever. Because not only they aren't, but they are also better than qualitative ones at hiding their biases and soothing your own.

> Are you implying that management shouldn’t have any quantitative measures and should only be qualitative?

Many managers would do a lot better this way. They'd still make stuff up, but would at least be forced to admit it.

You realize that you just reiterated the same point I made in the OP, right?

Goodhart's law is always in effect. It can't be solved because it's not a problem, it's a fact of nature with annoying implications. It's the echo of the observation that efficiency is fitness as its consequences ripple from the lowest levels of reality through systems made of people.

You can make an engine as effective as the laws of physics let you, but you can't solve the limits of thermodynamics. You can only do your best within them. Same with this, for maybe the same reason.

In my experience, there is no good solution long term except changing the metrics themselves every so often. When new metrics come in, as long as they are not totally boneheaded, they improve things. Then people start learn to start gaming them and some of the more sociopathic folks start doing so, then others copy by example and soon enough the metric is useless at best or detrimental at worst and it is time to move to a new metric.

It helps to have someone with a hacker mindset think about the metric being designed so the obvious ways in which it could be games are taken care of and their own metrics/incentives are aligned with the company goal.

Reply on news.ycombinator.com