Andreessen and Mixpanel Call for an End to “Bullshit Metrics”
allthingsd.com
allthingsd.com
For real businesses (read: those with actual business plans) the closest single metric of importance is Customer Lifetime Value. And the equation is very simple. Make your cost of customer acquisition less than your LTV and you will be making money.
The bigger issue I see in your comment is that sites at different stages in their lifecycle will want to measure different things. Customer lifetime value is the most important number for a mature business but early stage startups are nowhere near getting a realistic number for customer lifetime value. If I am the founder of a startup I want detailed knowledge of how users are interacting with the product I created to address a problem domain. I can get that info immediately after I start acquiring users, well before I have an idea of whether I have a viable business and what my customer lifetime value is going to be.
Really Instagram probably has a few OKMs (visits, uploads, hearts, comments, follows). All of those are important and should be tracked. In their documentation and educational videos, I've not seen Mixpanel focus on visits as a key metric. I'm not sure why.
I like Fred Wilson's analysis [1] of different revenue models it puts this stuff in perspective. The old "drive traffic too then harvest it with AdSense" model is losing a lot of steam.
For our most recent press releases, we reported many of these "bullshit" metrics because thats what gets acclaim and attracts attention to help the company gain the necessary notoriety to thrive. Internally, we still use real, actionable metrics to drive our development. We just don't report them to outsiders because its either sensitive information and likely uninteresting outside of the context modeling or users experience.
-- Pretty much how the world works.
Death can be alluded in many different ways, depending on the nature of your business, but it usually means making money, hitting a milestone that allows you to raise money, acquiring users, and so forth.
Once your company evolves, and has beaten death for the foreseeable future, a new set of metrics come into play and are now the most important. These may be similar to the death-evading set above, or they may be something new like increasing revenue, retaining users, increasing engagement, optimizing CLV, and so forth.
The point? If a metric means your company is not dying, or your company is growing, it matters. If it doesn't fit into those buckets? Probably doesn't matter.
It actually does a better job correlating with our growth than purely the number of customers we have. We've also been public about that.
Admittedly, we do use a different metric to determine how "valuable" our product is in a more honest way. We're talking about non-bullshit metrics that highly correlate to growth. Not all valuable products necessarily grow.
This article for more me is pretty straight forward. It's a well crafted PR play (whether made by commercial means or not is irrelevant), that draws readers to putting more importance on what metrics are used to determine valuation. Something that they can draw from using a service such as Mixpanel. The reality is that you, nor I, can definitely provide one metric that clearly correlates to financial valuation. Every case is different and every case warrants negotiation.
Mixpanel appears to be a great service btw. I hope my tone doesn't dictate otherwise.
Should companies selling, say, food products be mandated to show pictures of the actual delivered product on advertisements, menus and packaging? That's not going to happen without additional legislation.
Should resumes of individuals highlight workplace weaknesses, medical conditions, personality faults, past failures, major conflicts, etc.?
The web is no different. Unless individuals and corporations are persuaded to do otherwise, they will continue to use metrics as they please.
About the article, the problem stems from the fact that proper analytics is hard and is (arguably) getting harder with more advanced packages.Shouldn't it be going in the opposite direction?
It is a lot easier to track discrete downloads or pageviews than some other, more insightful metric, so people will naturally gravitate to the cheaper metrics. Until this is reversed, bullshit metrics will reign.
Anyways, my beefs:
First: how do you decide what data to send into the package?
The more data you send, the better (sure), but at a certain point you are just duplicating your internal datastore, so that is too much, right? But not enough and you'll miss a chance to understand a phenomena that you didn't predict seeing (isn't that the point?). After you decide, then you write a crapton of code to send it all (what about backfilling data when you want to track something new?).
Second: once you are collecting the data, how do you know what metrics to actively track?
This is definitely existential, but it's back to the core problem: doing analytics properly is hard. Why couldn't the software let me define some properties about the type of app I am running and suggest some strategies (you have a subscription SaaS app? Try tracking paid plan retention, signup funnels, etc...). Maybe it could go even further with reverse funnels, as in: what events are the most important and work backwards. I could see some automation and discovery possibilities there.
Third: do I really have to dig around trying to find something useful?
All the data is there, the software should tell me what is useful or interesting. It's definitely a hard problem, but I would throw money at software that could send me this email: "Looks like users who experienced event "ABC" also performed your highest priority event "Signup" at a 13% higher rate. This observation is 99% confident." Of course, you'd need to investigate a littler deeper to see if that isn't just a fluke or something stupidly obvious (like: people who view a page signup at a higher rate than those who don't), but at least I might learn something.
I know this is certainly a pipe dream as of today, but I vow to shower someone with money if they can do this.
In my opinion, the next generation of analytics software won't just have more bells and whistles, it will fundamentally shorten the time to some sort of real "AHA!" insight.
"First: how do you decide what data to send into the package?"
My recommendation is you think about this gradually. Don't instrument everything. Start by picking 5 metrics you really, really care about. One of them should be your One Key Metric (OKM) - a metric that you would be the company on if you could only pick one thing to measure. I think the pressure of picking 5 helps you decide what to measure.
"Second: once you are collecting the data, how do you know what metrics to actively track?"
When you start gradually, there isn't much to pay attention to. Stay focused only on a set of metrics. Add more metrics when you want to understand those specific metrics more deeply. Add more data to split those metrics into groups to dig even deeper. Keep driving your OKM up as much as possible--sometimes you have to do it with something that correlates to it.
"Third: do I really have to dig around trying to find something useful?"
Today, the answer is yes. Businesses are fundamentally very different. Analytics is a lot like science experiments in high school: You start with a hypothesis, you test to see if you were right, and if you were then you came up with a conclusion or answer to your experiment. It's hard work to build a good business. I think the first step is for companies like us to help you understand your data. The next step that I think is more important than simply pulling "insights" out automatically is to help you take action on it so you grow.
I just signed up fresh to Mixpanel to play around and you guys have solid technical on-boarding around just installing and collecting any data, but not a lot of details on what to collect. I'm sure that getting someone to collect just anything is a better first step than some sort of high level tutorial, which is why you do it that way.
I like the OKM idea a lot. Perhaps that could be a pretty solid center stone for on-boarding a new user. I just remember being overwhelmed by the blank slate every time I signed up for any analytics package.
If you want to learn more in the abstract we recently came out with an education series: http://mixpanel.com/education
A simple library of use cases with how to implement them in Mixpanel would be an awesome start. No coding required, just need a someone to write these.
For instance, I had a question about how to track the effectiveness of different blog posts. When a user signed up, I wanted to know which blog posts (if any) that user had read. This is useful because content is the biggest driver of signups for us.
I couldn't find any info anywhere on how to do this with Mixpanel or any other analytics tool. I emailed Mixpanel support and got an immediate & thoughtful response from Woody (which was awesome), but I would've much preferred if there were a self-serve library.
1. Come up with the questions first, then decide what data you need to answer them. It's so now easy to track every mouse movement of your users that a lot of people just track everything assuming they'll find something useful later. It doesn't usually work that way. More is not better, it's distracting. Even worse, people tend to pick the easiest things to track, which aren't the most useful. If you start from a question and backtrack, it might be more work, but you'll definitely get something useful out of it.
Coming up with question isn't always easy. We're trying an experiment to help people with questions via an analytics/engagement "Cookbook" (http://www.klaviyo.com/cookbook). You pick a question, fill in the variables and then we tell you what to track to answer it. We're still fleshing it out, but that's one idea we've got.
2. Don't look at analytics only in retrospect, use them actively. People don't make most decision at a single point in time, it's about building enough momentum to catalyze action. Because of that, you can do a lot more if you have a way to communicate with them and can effectively leverage what you know to build that momentum. Someone doesn't sign up? When they come back, can you show them content based on what they didn't do last time (e.g. viewed pricing page, but not feature tour...highlight the feature tour). Someone signs up and doesn't get completely set up? Are you sending them an email with instructions tailored to where they stopped and why they might have stopped there.
Related side note: I've gotten plenty of emails after signing up for something asking if I want "help." While it's a nice gesture, as more people send those emails, it gets old fast. Why can't you use what I've done so far to anticipate the questions I might have or give me reasons to get back on the horse?
When it comes to ad-driven startups, you could actually say that the "real" metrics suggested, like active users and engagement, are the actual bullshit metrics. For many ad companies like Google(search) or YouTube, high engagement by the users adds nothing to the bottom line.
Not once in this article there were the words "Profit" or "Revenue" and yet they talk about business and companies.
Here is a surprise for you. Real companies are interested in real profits from real revenue and you count revenue and profit only after the money is actually in your bank account.
Otherwise one day you get the kind of terms of service just like Instragram these days.
Anything recent come to mind?
Great metrics for e-commerce are churn, lifetime value, repetition of purchases, average revenue per user. On the marketing side, I think it's useful to understand what kinds of marketing produce certain kinds of repeat revenue.
Publicly, I think understanding how many customers purchased something in the past 30 days is useful. Potentially, the number of customers that purchased something above $X.
Then cash flow suddenly becomes essential.