Don’t Let Metrics Undermine Your Business
hbr.org
hbr.org
But this advice is so hard to implement. It's like the whole null hypothesis p-value debacle going on in academic publishing right now. Sure we can agree it's no good, but there's no standard alternative.
Standards are powerful. They break lots of ties. All else equal, if you have a hunch and I have a measurement tied to objective reality, people will side with me. Nobody ever got fired for choosing Java. Nobody ever got fired for doing the thing that made The Metrics go up.
It's so reassuring to just say "We still disagree, so let's just test it." We're submitting our dispute to adjudication via Science by resolving via The Metrics! It's much harder to be objective without these kinds of numbers, and I like being objective! I want the scientific method to tell me I did the right thing and I'm a good person.
Because without that, when we're trying to figure out what to do, it's just us, you know? It feels like guessing, and that's hard.
Jobs didn’t have a data scientist telling him to make the iPhone. A data scientist would have told Notch to make a Call of Duty clone instead of Minecraft. There was no data scientist behind Gates, Warren, Zuckerberg, Musk, or anyone of their ilk.
Looking back in history, some great discoveries and inventions were held back because the “data” said it wasn’t possible. Everybody thought ulcers were caused by stress and bad food, then Barry Marshall came along and literally trusted his gut by eating a bunch of bacteria to prove that’s how ulcers were formed.
At best, data science helps you squeeze a couple extra bucks from someone else’s ingenuity. Once Jobs died, the data scientists got their hands on the iPhone and chopped it up into different models. And it was the data scientists who made your Facebook feed ad-ridden junk. And at worst, data science is destroying our society. It’s the data scientists who are driving this insane data vacuum that is plaguing the internet and our daily lives. You thwarted the 2016 election by weaponizing data against America. You did it again with brexit. No doubt you will be out in full force for every major political event for the next few decades, sullying the discourse of the electorate.
Bit of a rant... sorry. I hate this over-reliance on data and the smugness it’s captors carry.
When you have a killer idea, scaling is obvious; just do more. More people, more machines, more materials, etc. You don’t need a number to tell you how much more to scale, because you always want the max anyways. Tim Cook was told there weren’t enough CNC machines in the world to manufacture the unibody MacBook. So he bought the company that makes CNC machines. What’s the point of metrics when you can shift the entire world to meet your needs? Great things are like that.
Metrics are for mediocre products and services. We use them in banking because people fucking hate the banking system and we have to pull teeth to get customers to spend money. McDonald’s uses metrics to gauge how shitty they can make their beef without people noticing. Hollywood uses metrics to decide which movie franchise to reboot or make a sequel of. Metrics are a sign that you are done innovating and you are squeezing the sponge before someone else’s innovation dethrones yours.
And you bankrupted the company building factories you didn't need, because iPhone demand has its limits.
what metrics give you is a way to give the average joe (which is most of us) a way to take calculated risks when there are multiple stakeholders each with their own opinion of what should be done.
metrics - like statistics - can be used effectively as a guide or can be used obsessively like dogma.
2. Apple has a very large Product Marketing team who do extensive quantitive and qualitative research on what products they should make, features should be delivered etc. This is the same for Microsoft, Facebook etc. I worked at Apple and each team will have a Product Owner/Program Manager assigned who decides what is done where as Engineers decide how it is done.
3. Data Scientists aren't solely responsible for releasing more iPhone models or how many ads to show. Those sort of decisions are cross-functional involving Finance, Product, Marketing, Strategy etc.
4. Data Scientists are basically engineers. They largely do what they are told to do by the "business". So not sure why you seem to think they are root of all evil.
Your last point kind of weighs in on a point I didn’t quite get to; that data can be found to reinforce any decision. If business wants to do X, they will have the data scientists prove to them that X is good. Data can be a huge force of tyranny in this way, this is where my disdain for the field comes from.
Both vision and drive and the ability to execute are essential.
What happened at Apple was engineers came to Jobs with a prototype and he saw the potential and drove it relentlessly forward. Same for the Mac - he saw what Xerox had done and recognized the potential.
Yes he trusted his own judgement - but I'd say if you saw either Xerox prototype in the age of terminals or a touch device in the age of fiddly keyboards it doesn't take a genius to see the potential - it just takes a hero ( courage ) to bet on it.
Finally, on the dark arts of advertising, moving from the 'gut' to being data driven - sadly I'd say that's what has made it more effective as a tool.
Part of the reason metrics work so well in marketing is because a lot of accepted industry practices are inherently malicious acts towards other human beings. So unlike other optimization processes, you don't care all that much about side effects of optimizing to metrics. For instance - do customers hate your guts for your advertising spam? Doesn't matter - the product becomes associated with a strong emotion, making it more likely some of them will buy it few months from now. Mission accomplished.
Interesting, does that really work? Got data you can point to, to back that one up?
I take your broader point of them not really caring - we are seeing that in data driven politics in the UK right now.
Obviously this host/parasite interaction will evolve - but at the moment it seems the population doesn't have many effective defenses. It's a bit like the opioid epidemic or obesity epidemic - currently people are being given the politics they apparently want.
It's the basis of the concept of "brand awareness", on which many billions of dollars are spent every year.
I'll give you a single data point as an example, and I'm sure you'll be able to relate. There's a certain on-line currency exchange operating in Poland that has a tendency to run really cringy, annoying ads in cinemas, and those ads somehow get played 3 or 4 times in a row in the span of 5 minutes. After sitting through them a couple times I got really, really sick of them, and had pretty negative feelings associated with the brand. Fast-forward three years, I have my own business now and foreign customers, so I find myself needing an on-line currency exchange. Guess which service I checked out first?
I think you place too much emphasis on the "genius" that combines these novel ideas, and I'm not saying it isn't required, but I think there's a lot of these ideas that die and a lot more data behind them than is talked about.
For the iPhone to work it had to to be at a time where the tech was right, the price of components could enable it to be sold at an acceptable price-point, the UX had to be tested on countless people.
To be more succinct, the right amount of optimisation, data driven improvement and innovation combine with the right timing.
Ceaseless optimisation can lead a good product to become horrible, but you need numbers.
Surely you can see the symbiosis between things like data driven battery improvement and optimisation and the possiblity of something like the iPhone. The iPhone would not exist without data driven decisions.
It wouldn't exist without vision and risk taking either.
Except I wouldn't say data science is destroying our society. Maybe data science done the blind lazy way is, but shitty science isn't representative of good science.
A concrete example, if at a restaurant we expect meals to be delivered within 7 minutes and we see the nightly average is currently 8:08 then some team members might start pushing to go faster and cut corners, compromising discipline and training effort--even if the company's values include accuracy and precision above speed. The values are laid down by the visionary founders, but the only metric that the workers have is an exact number and target. Somewhere along the chain between founding and the last hands on the food the values can be lost, and I feel like chasing the metric has a lot to do with it. Everyone pushes to make the number "because" of some factor, in my opinion probably pressure from above or a colleague.
Pardon me if I've abused any terms, thanks. The factor that I am sure of is entering scarcity mode rather than abundance mode, and I'd like to know more about how reading metrics affects us and whether there are known methods to systemically remove metrics from view such that we can remain true to all of the values outside of the metric. If we're short of a target then there is a gap in training or experience; nobody should be so driven by metrics that it overrides their acting in abundance.
I know this can be said to be coming from fantasy land but I've come from working in a number of domains in a short period of time, I feel like there's something in here even if it's hard to find.
On the strategic level your performance metrics are giving only a baseline for your capabilities, strength and weaknesses. That should only be affecting the implementation of a given strategy and never the definition of it.
And as always, if a metrics system is ill defined from the start it will steer you in the wrong direction. Like a broken speedometer in a car. And we didn't even start to talk about the corresponding incentive and bonus program yet...
>* the steps from A to B and from B to C might have been tiny and unnoticeable, but the steps from A to M added up to a big leap.* [1]
The metrics looked great, getting more beer out of less ingredients, in less time. Until the 70's when they hit a tipping point and drinkers stopped buying, tanking the company.
[0] https://en.wikipedia.org/wiki/Salami_slicing [1] https://beerconnoisseur.com/articles/how-milwaukees-famous-b...
It's especially bad on a lot of data science teams because there can be a lack of serious management experience and know-how if the team has a background mostly in academia. Academia is full of extremely dysfunctional behavior, even moreso than corporations, which is shockingly hard to pull off!
Everything described in this article is old hat for experienced, knowledgeable executives, people in the know have been talking about these ideas for at least half a century (dating back to Drucker, Deming and others).
While I agree with the authors' recommendations, I disagree about their diagnosis. I think Wells Fargo's main problem was a lack of internal controls, which is a boring, but necessary part of any large business.
It's a rehash of tried and true principles that have been described numerous times before almost always without attribution to original sources.
What's wrong with that?
Well, for one the authors are passing off work that is clearly not their own. Fine, whatever, it's a b-school publication, nobody takes their intellectual output seriously anyways.
But the more important problem is that you lose a sense of where this knowledge has come from, where it's been applied, how it's been applied, and other important context.
In other words, reading this article is basically a waste of time. It's better to stick to a handful of classic business texts (Drucker's a good place to start and yes, he goes over all the recommendations these authors make in Effective Executive). Reading Kaplan on the Balanced Scorecard is another good place to start.
Thank you for saying this!
A thought I have had in the past few years is "there's been nothing new in management since Drucker and Deming."
I go back and forth because some people have done a nice job of updating their ideas/applying them to modern disciplines, which is valuable.
However, what you said about where that knowledge has come from/been applied/how/context is sorely missing.
That these ideas are still not common knowledge leads to so many unnecessary mistakes.
Both very real problems that have existed forever. The solution, as the article suggests, is not relying on just a single metric, and not losing sight of your wider strategy. There’s a really good article on Reforge about the problems with the recent trend of a single “North Star Metric”: https://www.reforge.com/blog/north-star-metric-growth. Stacey Barr’s another thought leader on the subject: https://www.staceybarr.com/measure-up/a-single-kpi-that-meas....
Metrics absolutely cannot replace strategy and vision. It needs to be a core value of the company that you’d rather do the right thing by the wider mission than drive a metric at any cost. Managers at every level need to live up to that. This is harder the bigger the company. It’s one of the things OKRs, love or hate them, try and help with. Like with everything it’s whether you comply with the spirit or the letter of the law. And whether people get rewarded for latter!
An example I think everyone can relate to is in education where grading is a rich source of perverse incentives. In developing minds, no less. Yet the metric persists.
(Just to be clear, the alternative to grading is not no feedback, but personalized, situation-specific feedback. Freeing yourself from a bad metric can open doors.)
It seems like in the case of employees at a company, just knowing that a parameter is being measured can result in it implicitly becoming a target even when there is no explicit focus on it (especially if it is the only or one of a small number of metrics that is discussed).
I have observed that in general most humans are very bad at considering the incentives created by rules and rewards they impose or seek to impose.
This is very common, though, and a real problem associated with it is people pick proxies without clearly documenting (or even thinking through) what they want to measure, why they pick a particular proxy, and how the proxy might fall. As a result, people remote from the decision (including the original decision maker) tend to incautiously apply the proxy because they aren't even aware that it is not the actual figure of merit.
Largely, because what you are wanting to optimize is something like current value of the future income stream of the firm. So a proxy is taken, which inevitably has bias, and you need to understand why hat that big is likely to be a continuously sanity check your results for reason to think that you may being caught in that bias. Which you probably can't measure well (though maybe something has improved since you adopted the proxy), because if you could you'd have a better proxy to start with.
The problem is when companies choose qualitative and thus interpretational metrics.
Just stay to the simple easily trackable and quantitative ones and the make it your job to interpret them qualitatively.
I read a lot about how data-driven companies measure this and that, often through questionable, privacy-violating measures. What I don't read about is how do these companies ensure the metrics are actually valid - that they're correctly sampling the population[0], or that they're measuring what the authors think they're measuring, or that they're not being misreported or otherwise gamed (very common if the value of a metric impacts someone's career or even workload).
--
[0] - e.g. voluntary surveys usually don't, telemetry increasingly doesn't either as more and more people are aware of it and disable it.
Don't choose things that can't be tracked quantitatively. Track things quantitatively and then make qualitative decisions.
At Square, we had "Make Commerce Easy". This was qualitative and thus up for interpretation. It became even vaguer when it became "Economic empowerment".
These things can't and shouldn't be tracked as if they are quantitative metrics but instead used as guiding stars.
So the trick is just to separate the two and as I said. Use data metrics for optimizing and not for creating or measuring of qualitative goals.
Decisions are often inherently quantitative, e.g., you allocate an actual dollar amount to a budget. Or a project go/no-go is a crisp binary decision. I think you mean don't make decisions directly based on quantitative metrics but on qualitative assessments informed by quantitative metrics.
I don't distinguish that hard the second I get into the qualitative so many factors (like experience) inform the quantitative work.
Sometimes the qualitative goals is created because of the quantitative data.
What you are saying is certainly also what I mean but I don't think I only mean that but I see what you are saying and yeah it might just be my english.