Metrics-driven product development is hard | Hacker News Reader

Metrics-driven product development is hard

blog.doubleloop.app

74 pointskiyanwang35 comments

The last company I worked at was very metrics-driven, and used a really methodical, numbers-driven approach to make big, unpopular, counter-intuitive bets that failed catastrophically, while ruining morale, and trust in leadership, and gutting the company.

After that experience, my view of metrics-driven product development is that it's a way to offload the burden of thinking too hard or understanding where you're going. When people talk about metrics-driven product development, I think of drivers who blindly follow their GPS' directions into a lake. Obviously, metrics can be a useful supplement to a deep understanding of the product, the domain, the users, and the company, but keep it on a leash.

> After that experience, my view of metrics-driven product development is that it's a way to offload the burden of thinking too hard or understanding where you're going. When people talk about metrics-driven product development, I think of drivers who blindly follow their GPS' directions into a lake.

There's also the flip side - metrics-driven development used to keep other people from thinking too hard and scrutinizing work. At my last company, many product decisions were justified using a litany of metrics including AB tests, which people just swallowed because "the data speaks for itself!"

I once took a look at the app we were using to run the AB tests and not a single one had reached statistical significance. The company was basing its decisions on data considered garbage even by the AB app's naive p-value algorithm. I thought that after dozens of A/B tests at least one would be statistically significant just by dumb luck but no, not a single test had even the veneer of credibility.

But what you're describing is noise-driven development. It's not an actual metric value until you've somewhat ruled out noise as the explanation.

I believe that's a distinction without a difference. Most scientists have a hard enough time understanding statistical significance in well controlled experiments and they are (ostensibly) trained for it.

There's so much noise in business data that it's like reading tea leaves, except no one can prove it because by the time they gather enough data they realize that the management and/or market dynamics have introduced even more noise so everyone keeps chugging along, blissfully unaware that they're indulging in statistical fantasies and reinforcing whatever beliefs they already hold.

Accidental or otherwise, we lost something. We had a situation where we had a three legged stool of product/marketing, QA, and Development. If any of the three were too far out of line, the other two could drag them back.

But we killed QA, and Ken Schwaber gave us a treadmill to run on forever where structurally, nobody has time to think past the end of next month. So Product has been running progressively more roughshod over everything for the last 10 years. And since 50% of devs have 5 years or less of experience, that means most people don't know that things used to be better.

I recently had a revelation that merely using a product can often reveal the organizational structure and the cultural incentives that went into making it.

A company that is famous for being metrics driven and having teams that optimise for their own metrics is Facebook. You can tell that the product has "seams". Each page and widget feels like it was individually optimized and then stitched together. Overall it somehow feels off.

Apple puts design on top and has a kind of dictatorial veto over all aspects of the product. Hence products made by Apple feel seamless and cohesive. They are also more polished because when you solely rely on metrics you make it as good as it needs to be and then stop.

Sounds like you've independently discovered Conway's Law, one of the most terrible, true things you can say about software organizations:

https://en.wikipedia.org/wiki/Conway%27s_law

Apple relies on intuition and experience and uses data to support decisions, or not (and data be regularly overruled). Rarely does pure data drive decisions. People coming from other companies usually fail if they can’t adapt to a “data second” mentality.

I feel like Apple’s method is much better than blindly trying to interpret data and let that drive you over the cliff. Data isn’t a replacement for understanding of your product and users.

That's interesting. I was actually just guessing that Apple is not data driven.

I would be grateful if you can share Apples general approach to product development.

The closest publicly available info is an HBR article about Apple’s organizational structure:

https://hbr.org/2020/11/how-apple-is-organized-for-innovatio...

Are you aware of the work by Meehl on actuarial vs. clinical judgment? One of the conclusions of that line of work is that fairly dumb, mechanical data-driven rules ("actuarial judgment") consistently outperform expert opinion ("clinical judgment") even on what we think of as fairly complex topics.

The more surprising conclusion is that the dumb, data-driven rules outperform experts even when the experts get access to the same data and the decision made by the data-driven rule. Why? Humans have a tendency to think they're better than dumb rules and override these dumb rules in ways that make their decision ultimately worse.

I haven’t, but I don’t think it’s a stretch to assume that conclusion doesn’t apply in creative fields.

The funny thing is that people say that about every field, yet one by one, Meehl and the people doing work along the same lines consistently disprove them.

The result has remained remarkably consistent across many fields where people were sure it wouldn't apply. At this point it's stable enough that I think the conclusion is a good base assumption, and it would take serious evidence to show that it does not apply in any given field.

What is a creative field ? Can you write a novel this way ?

Metrics are great as an objective measure of feature releases/goals/success, but they don't provide the whole picture in terms of product success or direction.

Years ago I interned on a messaging team at a FAANG company where everything was driven by ~3 metrics. We had a screen with the trailing 30d average of our metrics, future work was prioritized based on the estimated impact to the metric, and the success of our team was based on these 3 metrics.

My intern project involved some machine learning modeling using the (anonymized/scrubbed) content of the messages sent with our product, and despite the team having existed for >12 months, this was the first time anyone on the team had read a message sent with the product.

In this case, the sharp focus on metrics was super useful for prioritizing features and evaluating our success - but was done so to a fault when the team didn't have a concrete idea of what people were trying to do with our product.

The usual solution to this is to spend more time talking to users - it'd be interesting if there was a process that can help teams know how much time to spend on metrics vs qualitative feedback

Metrics-driven product development sounds like a good idea but in practice ends up being Cargo Culting.

That's usually because people rarely reflect on the metrics they're deriving meaning from, rarely abandoning ambiguous metrics as misleading, and rarely imagining new metrics to find utility. Those are very hard tasks that take time and deep thought.

At my last (very large but not FAANG) employer we collected huge amounts of metrics in our apps, ignored them entirely, and based everything on executive whims. Is this Whim-Driven Development?

This is sometimes called "reality."

Known as HiPPO -- highest-paid person's opinion.

That’s because metrics are great for optimization and fine tuning features but terrible for innovation

One reason might be that great innovators don't naturally gravitate towards middle management (or management consulting), so the metrics are flawed out of the gate.

We don't have the tools to measure the complex dynamics and factors that result in innovation.

Do you have evidence for this claim? I think metrics are absolutely invaluable for innovation.

What is innovation but throwing 100 things at the wall and seeing which two of them stick? And to quickly get a sense of which things have stuck to the wall, metrics are the only reliable way.

It probably depends on the type of innovation. If you are trying to do things differently, metrics are invaluable — most modern internal innovation teams use metrics to build and manage an idea funnel.

However, if your innovation strategy is to do different things, you will likely be focused on market differentiation, customer development etc. Common practice is to refer to AARRR metrics for this, but it’s not like metrics themselves are the main focus.

Anecdata: on a couple of projects our metrics have told us that do things differently will not get us to target, forcing hard choices to go do different things to get there. But it is usually an uncomfortable fight to get the boat turned around, especially if people have already invested a lot of money and ego in the earlier decision tree.

Also purely anecdotal, but I think it takes a different skillset to pivot correctly, which execs who work in a corporate environment focused on doing things differently often lack. That’s where external consultants usually get involved.

Metrics are really good, and really can help you reason about what will happen to usage when you add, remove or change a feature. There are two better / essential tools: First, dog food / use your product if you can. I build recruiting software. When I hire developers, I am a user, just like my customers. You'll really feel the impact of defects and bad assumptions have, and it will really help you understand how to make users happy. The second is to work closely with a few customers. This way you have someone who can heat check the metrics and confirm they are having the same experience you do when you dog food, to make sure you are not being led to an alternate reality.

We have been using metrics driven product development lately. What we have been doing is hypothesize what will grow our product feature based on org identity, talking to customers, looking at the market. We define some metrics which we believe represents the success metrics in the best way possible. Some are based on domain expertise, some are just best guess. We use these metrics as a way to ascertain the hypothesis. We are not using it as indicator to build the next thing. The thing is that the success hypothesis or strategy might be wrong so we are willing to throw that away and start all over again. Not sure if it is the best way to do it.

Metrics is insanely useful to help inform all kinds of product decisions. The most basic example I can think of, detecting when people aren't using a particular feature you've built.

I think where it gets tricky and where even FAANG companies fail at, is the intersection of intuition and validation. Intuition can guide you along initially, ie, we need to build this thing because this it could become our differentiator and would be useful. Well then you need some way to validate that your intuition is correct. So how do you do that?

At FAANG companies metrics can come from all kinds of sources, not just a few various MixPanel event streams. UX researchers can go out and do real world case studies and bring back statistics. User feedback itself can come through Marketing outreach or surveys. PMs collect research on similar products and add valuable data to goal against. All this to say, when you hear the word Metrics, do not assume it's just a single stream of usage data.

Yet still, even when FAANG companies get all these data points from all over, they still make invalid decisions too. It's really hard for everyone!

The point I'm making though, use Metrics to help inform, but also understand that Metrics should never become dogma. They will help give clarity in many many things, yes, even early indicators of things. However, they are never going to invent the next big thing. That will be up to your creative teams who work outside the metric bubble to push in different ways to see where things could go.

metrics driven development has killed facebook.com. On its own, it is too short sighted.

How do you measure long term success? hint: u can't.

Blind application of ML is therefore, also dangerous for similar reasons.

metrics are useful for tightening the tactical decisions like, fuck, our landing page conversion is low so let's invest there, but strategically metrics are unable to be creative and innovative and all the other things you need for a great product.

Metrics are great for testing a quantifiable hypothesis, but not all worthwhile hypotheses are quantifiable, and in ignoring the latter we discount an essentially human side to creativity and that’s a big turn-off for me.

Metrics driven software development gets more meaningless, the bigger is the customer base, and frequent are releases.

A good way to make a big organisation chase meaningless things without your bosses realising, and even get a promotion for it.

Can you give an example of this? I am having a difficult time understanding your meaning.

Google Chrome is a great example "developers in the hamster wheel" of metrics.

Keeps getting constantly redesigned for seemingly no reason, and certainly not all for the better.

Google Chrome's (or any other established software's) useless changes are absolutely metric-driven. The metric is some employees' salaries.

Reply on news.ycombinator.com