I was in a company that had a number of products. We had one product which I'll call Workhorse; it was boring, but provided almost all of our revenue. Despite that, only a few people were on the Workhorse team, because the product was un-sexy and management didn't like it. Management wanted to have a hip startup vibe, and Workhorse just wasn't it.
So management put tons of money into acquiring and creating new products, in the hope that those would become the next big thing. These products varied in quality, but none of them were profitable.
I worked on one of the smaller products, which was of middling quality. We didn't even generate enough revenue to pay the salary of a single developer, let alone the dozen people on the team. It didn't matter though, because we projected the attitude that the executives wanted.
In order for the rest of the story to make sense, I have to spend a minute on a seemingly unimportant detail. The company as a whole referred to subscribers as "people who pay for our product." It seems like a reasonable definition, but led to some problems.
Most of our products were straightforward. With Workhorse, you paid a flat fee of $45 each month to use it. Some of the other products charged a fee for different tiers of plan. The product I worked on was unique; we only charged money to businesses that made money when they used our product. Our product was rarely successful at making money for our users, so only about 50% of them ever ended up paying for our product (thus, becoming subscribers).
So Devin calls us into a meeting. He tells us that we've done a great job and built something really special. I knew we hadn't, but who doesn't appreciate having their ego stroked every now and again?
Devin cuts to the punchline and tells us that he's done a study, and found that our subscribers have the highest subscriber satisfaction of any product in the company. We beat Workhorse by miles, and even all the other products couldn't hold a candle to our subscriber satisfaction.
Most of our company's users only used Workhorse, but Devin believed that was because we weren't trying tell sell our other products hard enough. Since our subscribers were happier than Workhorse's subscribers, we should make a big push to convert all the users of Workhorse. In fact, since Workhorse had so much market share, and it's probably the best way to make customers aware of our product, we should cut the price of Workhorse from $45 per month, down to $15. Then we'll charge more for our cooler products, and aggressively market them to Workhorse users. We would make piles of cash.
This was the point where I raised my hand. I asked which users he surveyed, and he said he surveyed subscribers. I asked him if he included non-paying users, and he said of course not; you're only a subscriber if you pay for our service.
I explained that this was a problem with the study. Our product only charges you money if you make money. Since our product isn't very good, we only make money for half our users. So the 50% of people who got zero results from our product, Devin just ignored. He didn't survey them. Then the 50% who it did work for, they were pretty enthusiastic about our product.
Meanwhile Workhorse defined a subscriber very differently. You got a 1 month free trial, and after that you became a subscriber.
That means Workhorse surveyed any user who stuck around for more than a month, while we only surveyed the 50% of users who had success with our product. We threw out most of the people who were probably dissatisfied.
I told him that you can't compare our subscribers to Workhorse's subscribers because they're apples and oranges. Furthermore, Workhorse had tens of thousands of users, and we had a few hundred. You can't survey 30 users and compare that with a survey of 5k Workhorse users.
Another engineer piped in and said that he was also concerned. By excluding our unhappiest 50% of users, we were biasing our study in a way that Workhorse wasn't. I looked around and saw a few other people whose eyes said they were also worried, but they stayed quiet.
Devin got mad, he did some shouting, and I shut up, let it go, and started applying for jobs.
Devin took his plan to the executives and they loved it. It was a vindication of everything they'd been doing for years; all the time and money they spent on transforming the company had been successful. They didn't have to be a big, boring corporation with a big, boring product. They could be an exciting company with trendy products.
The executives told their investors that big changes were coming, and the company would be more profitable than ever. We had a series of exciting products, and we had internal studies showing that with a new marketing push, we would dominate the market.
They cut the price on Workhorse from $45 per month to $15 per month, as Devin has outlined. Our customers thought this was great. They liked Workhorse at $45, and they loved it at $15. Sure, some aggressive salesperson would call them every few weeks and try to get them to pay for some product they didn't care about, but customers got used to ignoring the calls.
Revenue plummeted. Investors were furious, and even accused the executives of fraud. They believed that the hype from executives about big profits, followed by the complete collapse of revenue, was the result of deliberate malfeasance. I can tell you from being in the room that it wasn't malfeasance. Someone told them what they wanted to hear, and claimed it was all backed up by numbers. They bought it, and I've already told the end of the story in my previous post.
For me, the moral of the story is that you should be wary when someone tells you want you want to hear. It's tempting to go along with that message because it makes you feel good. Unfortunately, it can also destroy your business if they're wrong.