The danger of surrogate metrics (2007) | Hacker News Reader

The danger of surrogate metrics (2007)

blogs.msdn.microsoft.com

29 pointsJtsummers8 comments

I observed a (non-engineering) team successfully use this argument to avoid being measured at all except through the customer feedback that they were a primary channel for.

As such, they were oft quoted as being amazing. I can't say that they weren't - my interactions with them were tangential to their main purpose - but the work that my team had that depended on them or their input was often not done or missed the point completely (and criticism dismissed as they are the posterchild team).

I think you should use surrogate metrics. You should just know them for what they are, and be able to argue as to why they aren't representative when they are bad and when they are good.

This means you can't make somebodies job, bonus, or promotion depend on the metric.

But it means you do get a heads up when something might be going wrong. And it means you get a heads up when something might be going right.

It also means you have to deeply understand what a metric really is, and what each particular metric is to the mission of your company and your job. This is hard.

Yes! I've seen these called guardrail metrics. Having a single metric be the one to optimize works pretty well if that optimization is what you want. But optimizing a teams work is messy and generates organizational instabilities. Tuning the guardrails feels like the best way to make sure that you get enough instability for interesting things to happen but not enough that other teams start deciding to take over whole functions just to get work done.

See also Goodhart’s Law: When a measure becomes a target, it ceases to be a good measure.

https://en.m.wikipedia.org/wiki/Goodhart%27s_law

I know this is from 2007, but it's funny this is coming from Microsoft... with all the W10 metrics, telemetry and spyware bloat that replaced the old and effective "Send error report" button. Kind of seems like they didn't follow their own advice - provided user satisfaction is actually what's on the line here.

In the bio/pharma scene, this is most cruelly illustrated (in recent times) with this example failure: https://blogs.sciencemag.org/pipeline/archives/2017/05/18/no...

tldr:

- statins have a measurable effect on HDL and LDL levels that correlates really well with how much they reduce your risk of dying, and that risk reduction is real and big.

- therefore HDL/LDL levels shall be a great surrogate metric for new drugs; let's go find some.

- we found this great drug that does even better than statins do at HDL/LDL level changes

- two years and 12k patients in trials later: this new drug doesn't affect mortality literally at all.

- ...crap.

Every consumer internet company needs to heed this warning. Instead of digging deeper and deeper into metrics like "time spent in feed" or "upvotes per session" or "minutely active users" (because daily and monthly active users is so 2015)... It's pretty simple. Find a way to ask your users what they actually think of your product (then, yeah, by all means use data to validate/explain that)

But in the end, isn't "what a customer thinks about your product" also a surrogate metric? There are a lot of products that were loved by the users that still failed to be a sustainable business.

If you give a valuable product away for free, for example, your users will love you.... but it you will soon run out of money and go out of business if you don't also have a way to generate revenue.

"What a customer thinks about your product" is a input, not a metric. You can still fail, just like a chair factory can fail to produce chairs if provided with unlimited amounts of wood but no nails or glue to hold the pieces together.

Reply on news.ycombinator.com