How we reduced our cancellation rate by 87.5%
blog.reemer.com
blog.reemer.com
I'll probably write something about this eventually. There are a lot of generalizable tactics which repeatably work well. (Email engagement is probably the highest bang for the buck, considering that you can implement it in about an afternoon and, coming from the starting point "We send no email", it will virtually immediately produce visible results.)
Your writing and approach (viz. "engineer who thinks like a marketer") has changed how I look at what I do from "write code" to "scalably reach profitable customers so they'll pay me for a technological solution to a painful problem."
And agree re: email engagement!
Thanks.
I understand a premium, but a mass market gym could be 40 to 60 dollars, and a crossfit gym was 140 to 200 dollars.
Worst they could do was say "no".
Some time later, I cut Tarsnap's account attrition rate by another ~50% by adding a line to the "account will be deleted soon" email: "If you've decided to stop using Tarsnap, I'd love to know why." This wasn't deliberate -- I added it for the simple reason that I really do want to get that information -- but it seems to be causing people to stop and say "hmm, I can't think of any good reason to not use Tarsnap, so maybe I should keep using it after all".
I wonder if they feel a bit of social pressure to stay. Basically when you say "I'd love to know why" all of a sudden there is an actual person who would be clearly harmed by their decision to leave.
Interested in learning how a cohort analysis can help your business grow? Get in touch – I work with select clients to help identify growth and retention opportunities, and build features to realize those opportunities.
Kind of killed it for me. Now I am not sure whether I should trust the results.
I have no doubt kareemm is a fantastic at his job. I have simply seen to many cases where a case study is used to create sales. Again nothing wrong with that. It just kind of makes me sceptic when I see fantastic results combined with a a sales message.
Thats just me.
I routinely distrust the gushing claims by paid TV advertising actors about how the Magic Bullet changed their entire life. That hardly implies I think the product is trash, but the alleged results are obviously designed sucker less discerning consumers.
And the general answer your question is yes, as far as I'm aware, employers routinely seek references to back up the claims made by potential hires.
I'd say the best way to see if a cohort analysis can help you is to try it for yourself.
Happy to answer any questions I can for ya - gratis - if you do try it out. Email in profile.
I am sorry cause I can't answer this without sounding like a naysayer and I normally hate that.
It's just when you have been around on the net as many of us have for so long you just realize that normally if something is too good to be true it probably is.
For the record. I think there is nothing wrong with you trying to get work that way I don't find it wrong or anything.
I just went from wow to oh he wants to sell me something.
Occupational hazard I guess :)
Plus the guy blogs pretty regularly. My personal sense is that someone who writes an article sharing his process and results for free deserves the benefit of the doubt — and at least a few clicks around their website to check it out — before posting in a way that suggests that his integrity may be suspect.
I never questioned his integrity. In fact I think I went to great length to say that it was not him but that post.
When you say something like, "Now I am not sure whether I should trust the results"... I think you're implicitly questioning the author's honesty.
Same goes for this: "It's just when you have been around on the net as many of us have for so long you just realize that normally if something is too good to be true it probably is."
Anyway, don't mean to debate you on any of this. You had asked where you were wrong, so I just thought I'd share how your comments came across to this outside observer.
That has nothing to do with the integrity or honesty of the person, which I have repeatedly praised.
It's the difference between saying "you are X" and saying "what you said is X"
Yeah, technology isn't really revolutionising things that much. I mean, someone told me you could send a electronic letter around the world, instantly and FOR FREE! There's gotta be a catch. That's too good to be true.
And someone else told me that there's a phone company that does FREE online voice and video phone service all over the world. No way is that possible.
And then someone else told me about an online encyclopaedia that has waaaay more than my Encyclopaedia Britannica, but I don't believe them because they said it was free aswell. Can't happen.
(To quote Bill Gates) Who can afford to do professional work for nothing? What hobbyist can put 3-man years into programming, finding all bugs, documenting his product and distribute for free?
The internet is changing things.
I reacted to a post that made great claims and then ended up with trying to sell me something.
The internet is changing things, but humans are often the same.
That "free" email you talk about? It runs ads. That free online phone call? You're paying more than you should be for another product, so that they can give you this one for "free". That free encyclopedia? The "please donate" ads are ads!
So yeah, there's almost always a catch, but we either don't realize it's there (because it's indirect) or it's subtle enough to not bother us.
>> That free encyclopedia? The "please donate" ads are ads! Donating is not mandatory. So it is free. It doesn't waste my time nor my privacy.
In his defence though, to be fair, this article says:
1) What the problem was.
2) What he did.
3) What he found out.
4) What actions he took as a result that lead to the improvement.
That's a lot more than you get from other conversion experts I've seen (usually the write ups I see stop at (2) or (3) and you're left with the pay-for-more-info).
Seeing exactly what actions were taken for each of the things they found makes me err on the side of, yeah, this was a cool post, not just a sales plug.
It's really good to see this sort of stuff turning up on HN.
I was in fact trying to go after the ball not the man.
Assuming a normal distribution (probably not that accurate, but it's just a guess), the final cancellation rate would be about double that measured until now.
Am I missing something here?
So while your analysis is correct and the cancellation rate will likely rise over time, the key thing to check when comparing a pre-change cohort to a post-change cohort to test for improvement is that the cancellation rates at two weeks post-signup are significantly different. Obviously its the most ideal if you can run the tests in parallel to reduce the risk of selection bias. However, sometimes that is not always feasible, and with a result like 85%+ improvement is not really necessary to do so in order to assume that the control was beaten.
For instance, they could know the average number of cancellations in a week, and see how that number has dropped.
I think this would work out, provided it doesn't matter too much whether the two actions they've taken (the two emails) occur right when the gym signs up or later on.
You're right that an average time to cancel of 61 days implies some may cancel at, say, 80 days. But some gym signed up 20 days before this cohort analysis began, and could have been the one to cancel had it not been for these changes.
Think of it like kissmetrics having an "edit the code that generates this report" option. You can leave your email or drop me a line if you want to try out the beta.
=)
What were you trying to measure in KISSmetrics that you were unable to? Did you try using the Data Export feature? http://support.kissmetrics.com/apis/data/data-export-setup
Btw- You can import data into KISSmetrics as well: http://support.kissmetrics.com/advanced/importing_data
I'm the Product Manager at KISSmetrics, so I'm genuinely interested in your issues to understand what you're struggling with. We try to cover key use cases like cohort reporting, but perhaps missed something for you.
Thanks, Jason
- since all the relevant data's in our db, i don't need to import data into a 3rd party service if i haven't been tracking it. i just modify my sql query. no need to futz with data importing, syncing, yada yada. - using Excel is a matter of writing sql, running the query, then importing a CSV. simple. - data manipulation is easier - Excel is designed for it.
I wrote the SQL and ran the analysis in my post over a couple of hours one afternoon. I can't imagine it being easier using any 3rd party service.
If you want to email more, drop me a line - email's in my profile.
One thought on ways to analyze the follow-on problem of customers canceling after 61 days (a problem similar to what I've seen at every web company I've ever worked at).
First, perform the same cohort analysis you’ve already done, but look at the cancelling customers vs retained customers at day 1, day 15, day 30 and day 45, then use this analysis to figure out your triggers (things like # of Facebook posts needed by day 15, % of profiles claimed by day 30, etc).
Once you have your triggers, you can make proactively calling / emailing problematic customers a key part of your daily routine. While discounts might still be the way to go, this trigger based approach is one I've seen work well. Additionally, because you are in touch with problematic customers it often gives you insight into what do next.
I wish you could work out more concretely why the situation improved but with three substantial improvements (that likely impact different customers in different ways) that's difficult. I could imagine "drop[ping] prices by 15-60%" would help those not using your product fully for example as even if they don't use all the features they don't feel like they're overpaying.
First, you changed two variables simultaneously, so its tricky to tell how much either contributed. The cynic in me, suggests that the article could actually be summarised as 'we cut our prices'.
Second, you right:
> So we improved our onboarding to help a gym owner export a CSV of their members’ email addresses to send to us.
Certainly in Europe, that could fall foul of data protection legislation, you'll need to make sure that the customer has given permission for their data to be past to 3rd parties.
On price specifically: In a subscription business like this one, you have to meet a minimum utility requirement in each month that a customer is able to cancel if you want to retain customers. Each customer's minimum utility is different and could even be comprised of different factors/features depending on the breadth of the product offering. But there is one factor that cuts across all of them: price. A significant element of churn is price because the initial purchase thrill may decrease over time and result in customer cancellation requests at a certain point in their lifecycle. So, cutting price is kind of an easy way to reduce churn in a subscription model...particularly because people bought in at X and are now paying fractional X. Boom - happy customers. Also, price-cutting is habit-forming, and the customers who received a reduced price will come back asking for more reductions in time.
On features: Multiple times, I've seen the "get more people using our product" as a good way to reduce churn. I won't comment on permission customer marketing and whether or not what OP did was legal, but the results of a feature like this are great, and seems like he added more than just this one. He added improved functionality and invested in his product at reduced prices - great deal!
On onboarding & cohort analysis: OP was right to focus on onboarding features and adoption to improve stickiness among new cohorts. He would have also been smart to raise the price for new customers if he materially improved the product (which it sounds like he did). Over the same time period, he could have had newer cohorts of higher paying customers, making the older ones less important to the financials of the business. By "hiring" higher priced customers to increasingly recent cohorts and continually "firing" older-lower-priced customers, the balance of his revenue would have shifted to these newer, more valuable customers over time, making the older-less-happy customers less important to his business. That's how you really turn the crank on a subscription business, and if your onboarding is good enough to continually improve retention in new cohorts, you've really nailed it.
Overall, I don't mean to be overly critical of OP's choices. I aim to highlight where optimizing for customer churn alone can harm the financials of the business, particularly around price-cutting for existing customers. He's doing lots and lots right with his cohort analyses, onboarding improvements, and assumptions about churn impact of new features. However, we're in business to make money, so these have to be balanced with the health of the business itself.