The more metrics you track, the less you know

At <FAANG mega-corp>, I used to work on an internal team which basically did nothing important, certainly nothing users ever think about.

The problem that we were trying to solve, well, we had already solved it about 3 years ago, yet the team was still expanding, and our product managers kept spitting out new (mostly useless) project ideas.

One of the main areas of focus was metrics. Leadership was obsessed with metrics, we measured everything. Not just plain empirical metrics (like # clicks), but also very complex metrics, theoretical models that we had data scientists work on, etc. At some point maybe 25% of the team worked on metrics related stuff, it was crazy.

Why were they doing this? Was it only to keep us busy, so that the team can continue to grow? Or maybe they were desperately trying to find metrics by which they could prove the team was actually still doing meaningful work? I don't know.

At some point I left, and moved on to a much smaller team. This team was working on a brand new product, one that customers actually pay money for!

One of the first things I said to my new product manager was "so, what metrics do you care about? how do you measure success?".

He was taken aback by my question. He seemed genuinely perplexed. He paused for a few seconds, then said "metrics? What do you mean by 'metrics'? I look at revenue, when the line goes up I'm happy".

I thought it was... insightful.

There's a difference between evaluative metrics and diagnostic metrics. You have a handful of top-level metrics that tell you whether things are good or bad on a dashboard. When they change, you have to dig in to see why, so you need more metrics. You segment, you cohort, etc.

Otherwise, you see GMV is down, and you don't and won't understand why.

One thing I see far too rarely in choosing metrics is ensuring that every metric has one or more countervailing metrics to balance the choices made. In other words, if you are optimizing for a metric, you also need to measure what is being sacrificed for it. The classic trio of cheap/fast/good is a great example of this, as are most engineering-style tradeoffs, like capacity vs range.

You will often see managers create metrics for the things they want to improve, without realizing that making a metric without measuring the countervailing metric is creating an incentive to throw the countervailing metric in the shitter. A good example of this is call centers which measure the time of call resolution without measuring customer satisfaction with the call. This is how you end up with a low quality, high volume call center. You can do the opposite by measuring only satisfaction and not speed. Now, maybe what your business needs is high volume, low quality, but that should be a choice you make. The problem with not measuring the countervailing metric is that nobody (officially) cares about how bad that metric gets. So, if you have a tradeoff situation, you automatically get an extreme even though that may not be the most profitable place on the spectrum. Perhaps your call center would be better for the company if it was volume 9 quality 2 instead of volume 10 quality 1, but if you only optimize for volume you can't make that choice.

This even applies to metrics you really want to go up, things like sales and profit. A manufacturing company that optimizes only for sales while ignoring manufacturing metrics will hire salespeople in preference to manufacturing workers until the inability to deliver product starts affecting sales. That might sound like a nice problem to have, but it can be a real problem, and perhaps sales 9 manufacturing 2 instead of sales 10 manufacturing 1 would result in better sales next year. It's better to have metrics which allow you that choice.

To re-paraphrase a common saying: Half the metrics I track are useless, the trouble is, I don't know which half.

Obviously for core business metrics there's a direct cost associated with measuring, validating, and maintaining access, but even for more technical metrics that's true.

The issue is, without a comprehensive metric collection and analysis system tracking multiple metrics, troubleshooting reduces down to a red light / green light level, which is not super useful. Is your site "down" because your uplink has gone offline, your servers, or is it your metric collection system throwing a false positive?

Same can be said for business metrics - is revenue down from customer acquisition, deal size, or orders per customer? If you don't track more metrics than the minimum, it's hard to tell.

The author's framework-"you just start small and only add one at a time as needed"-and given example illustrate exactly the process that leads to too many metrics. Pick a few metrics, add another, add another, add another. If you have 5 teams working on a product you quickly end up in the 50 metrics scenario.

Too many metrics can be a problem but it's not the real problem. The real problem is choosing metrics without any regard for the decisions they're supposed to inform.

When you understand that the purpose of all measurement in business is to reduce uncertainty for a decision to be made, everything comes into focus, and you'll have a natural constraint on your scope of measurement.

I think the framing should be around what makes sense on a dashboard, not how many metrics total that the entire company (or whatever cohort) wants to collect.

A dashboard should not distract from what you're trying to do. It should show only the information you really need [1] so that you can focus on actually driving.

I worked at one company where we had so many signals on ours that it looked like the dash of a 747 [2] to me. Pilots go through a lot of training and tests to show that they understand those and how to respond to different scenarios! Software startups are a bit more seat-of-pants than that.

We can have dashboards for different purposes and roles, for different activities we want to drive. But the dashboards should focus on use. And if there are metrics that don't fit on any of the dashboards, they're not about driving anything - maybe consider scrapping them.

[1] https://www.pinterest.com/pin/58054282668632186/

[2] https://www.reddit.com/r/cockpits/comments/ayr4z7/boeing_747...

> Most companies have far too many metrics. Companies might have a dozen dashboards, each with 4-5 metrics, leading to 50-100 metrics being tracked at any given time.

I think it’s important to divide “internal” / operational metrics, which a team monitors but doesn’t expect anyone else to care about (say, error rate of the DB or some internal ops task latency) vs. “external” metrics which are rolling into OKRs / KPIs or otherwise being reported out as health metrics of the team.

I struggle to believe that most companies with 50-100 metrics are actually using most of them as external metrics.

I think this article is just wrong, or at best describes the author’s experience in a dysfunctional org.

There are a lot of pathological metrics patterns. I used to work at a FAANG/MANGA that prided itself on being metrics driven; the challenge was that most people suck at picking metrics. The most common anti-pattern is choosing metrics based on what is easy to measure.

The most valuable metrics I have encountered are metrics that measure directly your teams success in their stated mission. The problem is a lot of teams have bad missions. I tried to tell every team that I worked with, that their mission should read like a problem statement not like a technology statement. For instance, instead of saying that “we are the team that owns the foo service“, the team needs to think about the business problem that inspired the foo service, and make solving that business problem their mission statement.

Once you have clarity on the business, problem that your team exists to solve, then you can start thinking about metrics that measure how well you are solving that problem. These are the most valuable kinds of metrics.

Now, the thesis of the article was that teams had too many metrics, and that this was bad. Once a team has clarity of mission, they have to implement technology to accomplish the business problem that is their mission. After you have clear metrics that tell you how well you were accomplishing your mission, then you need metrics to tell you how well your technology is functioning. Do not mix the two kinds of metrics. It is a happy coincidence if the technology functioning well directly, corresponds to how well you are accomplishing your business mission. More likely, the business metrics and the technology metrics need to be kept separate. You need metrics around the technology so that you can predict whether you are nearing a problem, detect whether you have a technology problem, And identify the nature of the technology problem that you have. Good metrics around technology will allow you to do all these things. You should not add metrics arbitrarily, but you should analyze your technology for where it is likely to break and prioritize adding metrics in that fashion.

My last point is about team just functions. Other anecdotes in this comments thread have described organizations that lacked clarity of mission or that had completely solve. Their mission yet did not pivot to a new mission. In those cases, you end up with a lot of make work, and that make work may consist in part of implementing new metrics. This is just plain org dysfunction and not really a metrics problem.

There was a paper from Hitachi (I think) decades ago.

Their thesis was one should devise a metric to solve a problem, and discard it once the problem was solved. Then on to the next problem.

Metrics can be interpreted in so many ways and is not the solution to the problem itself. A metrics provide some data points, but data points by itself does not solve any problem. That is why metrics is only part of the truth here. Consider everything else NOT taken into consideration by metrics

My last company lost its mind and started getting into meta-metrics like:

X% of systems have metrics. Y% of uncovered systems get added to metrics in Q2. Z% of metrics captured meet their SLO.

Meanwhile the SLOs themselves were pulled out of thin air because management only talked to upper management and not actual users.

Treat metrics like automated tests - if they are not green you have action to take.

You can thus have thousands but hardly ever will you have thousands all red.

And when you create a new metric, TDD style, the company will not be able / designed / changed to make it green.

... but still rely on metrics. Yes, metrics are something easy to compare how things changed, but the system as a whole is more complex than that. There are things that are binary (you get there, you lose), there are things that are multifaceted and is not easy to put an linear number to it. And the Goodhart's Law (https://en.wikipedia.org/wiki/Goodhart%27s_law) should be taken into account, specially when you lower the number of metrics. They may be a guide, but be careful if you put them as target.

I think of metrics like I do accounting. Accounting is all about measuring parts of the business. You track every dollar sent, lost, owed, etc. Every economic transaction is measured. But nobody wants to just stare at a list of raw transactions. You have to summarize those figures: you group them into revenues and expenses, assets and liabitilies, and you break it down by department, by function. You create a picture of what your business is doing. In the end, all some people really care about is profits, some people want a measure of solvency, others want to know how big your company is. Accountants have actually has thought deeply about metrics and written papers about why the measure what they do and I think something close to the golden rule is: does this information have the capacity to influence someone's decision?

Information is meant to help you make a decision: should we add more servers, should we end this service, can we sell this feature, are our customers satisfied? Every metric should help you answer a question. Ask yourself, what decisions are you trying to make and what information would help you make that decision?

It is very easy to see the benefits that metrics provide (typically you start at no metrics at all), but incredibly hard to measure the costs.

This bias is ubiquitous: metrics, unit testing, abstractions, startup funding, documentation, security, etc. All of those typically start at zero. All of those are perceived as "good", therefore more is always better. You rarely find people with the balls to say "we need less documentation" or "we invested too much effort in security". Kudos to author for recognizing the same issue with metrics.

And he's right: the solution is to recognize our bias and for things where cost is hard to measure just assign some cost to it. Though instead of million dollar value suggested by author you should just treat cost as exponential.

Having 3 metrics is better than 0.

But having 50 metrics is way worse than 3.

In fact, having 50 metrics is likely way worse than 0.

Very relevant to me right now.

But doesn’t the “ What About the Details?” section acknowledge that you will many of the other metrics later anyway for different people, projects and problems? Isn’t the message then “You should know which metrics are relevant to you and only look at them?”

The focus on "data driven decision making" tends to get things stuck in a weird local maximum and result in https://en.wikipedia.org/wiki/Goodhart%27s_law .

"Not everything that counts can be counted, and not everything that can be counted, counts."

I agree with the sentiment but the example is not very compelling. Tracking averages is its own pit of problems.

To be able to understand your funnel in a marketplace you’ll need to at least track profile and product views & click-throughs in addition to the metrics mentioned. How can you possibly tell what’s wrong when all you have is a average GMV metric going down?

Priority metrics vs primary and secondary metrics if this is a tool that help us?

Counterpoint: you don't have enough good metrics.

The beatings will continue until metrics improve

Indeed, knowing what metrics need to be monitored is vital.

But to do that, you need to understand the system producing the metrics.

Not interested, based purely on the title. The article may have insight, but these anti intellectual clickbaitesque headlines are too tired to heed

You can never have enough metrics.