How our freemium plan failed
baremetrics.com
baremetrics.com
The 60 customers that were lost during that time period - were they pre-existing customers from before the freemium switch, or new customers? Did they leave because the technical issues or because they saw the freemium version and decided they didn't want to pay any more?
The author was certainly a victim of their own success, as their conversion rates are pretty good. They just need to find limitations for the free account that doesn't consumer more resources than they can support.
And I agree our lack of preparedness to scale was part of this, but it's shortsighted to only look at a simple conversation rate for determining success.
"Success" is relative and we define it on a higher level than just conversion. The fact is, software isn't infinitely scalable...or rather it takes orders of magnitude more resources for it to approach that level.
There's a certain inflection point where the time spent "getting prepared" simply outweighs the potential benefits. We could have spent years "getting prepared" but there was a point where we just had to say "ship it"...and that's what we did.
But ultimately we exchange at least a few emails with every cancellation and try to understand exactly why we were no longer addressing their need. 99% of people are extremely willing to help out here.
Also, Exit Interviews: http://www.extendslogic.com/business/jobs-to-be-done-cancel-...
Any service which lets me sign up online but not cancel online is incredibly distasteful.
Personally, I am usually happy to give feedback to fellow business folks, and I wouldn't be offended if someone politely asked for it from a service I was cancelling. However, if a single reasonable action to cancel and no further contact resulted in my card being charged because I didn't play along, that service should not expect a happy outcome.
We respond with: "Bummer to hear, but happy to take care of that for you. Anything we could have done better?"
Then that usually sparks a longer conversation, but if we don't hear back within 24-48 hours, we just go ahead with the cancellation.
We're also very generous with refunds and certainly wouldn't keep someone's money just because we delayed the cancellation a day or two. That's just silly.
I suspect that both observations can be explained by assuming that the amount of value generated by a product follows a power law. Charge a lot, and you will restrict your market to only the head of the distribution. A nice consequence of this is that the consumer surplus (the amount of value you generate in excess of your price) is highest in this part of the distribution, which means all your customers are very happy even if you're charging them an arm and a leg. When you drop your price and service the long tail, you get a lot of customers who derive only a tiny bit more value from your product than it costs them, which means they get mad at you really easily.
The beauty of this argument is that it doesn't put customers (who are less demanding while paying more) on pedestal.
I run SaaS where there is one plan only. So all customers pay the same price. One observation I've made is that customers who are the heaviest users (thus extracting the most value) are the most forgiving ones and the easiest to deal with.
It's not about the price they pay. It's about net value they extract. The more value extracted, the higher tolerance threshold.
This is probably the single best way to explain the phenomenon that I've read. You're absolutely right. My boss and I talk about this a lot, and I'll absolutely be stealing this in our next conversation about it.
The consequence of this is more of your features don’t work the way the customer wants and hence even if the overall value of your product is high for the customer, they experience many more annoyances using your product. Annoyances seem to generate far more complaints than lack of value. If I use a product that offers me little value I almost never complain about the annoyances I just stop using it, it is the high value products that don’t work the way I want where I bother to write a complaint.
As the article hints, the real limit isn't computing resources (CPU/storage/etc), but people resources. Committing these resources to non-paying customers is generally not a winning proposition.
Upfront-card-capture free trial periods and liberal money-back guarantees give your customers the zero-risk ability to guarantee that a product is a win-win, but filter out people who don't take your product seriously enough to consider paying for it.
I'd advise any founders working on SaaS products to provide some sort of free trial mechanism, but probably less than they imagine!
I'm not trying to be cheap. I just have no reason to believe you aren't going to try to grab a few dollars from me hoping I'll forget to cancel. And while you might have a liberal refund policy many companies don't so the risk I feel includes previous scams. On top of which, from experience, many companies make you jump through incredible hoops to cancel and there's no way for me to know if your company is one of those.
And, if I was an employee of some company. The odds of me feeling comfortable using a company credit card to trial or even having access to one seems low.
(1) A limited free demo - just one button click from homepage and it loads in your browser!
(2) A more extended free demo - but requires e-mail signup / free account creation.
(3) Fully-featured free trial period, includes saving to server, more export options, multi-user collaboration, other operations - but requires entering CC number and choosing plan.
Even after all that, our default customer service policy is to cancel subscriptions (available on website without talking to a customer service rep) and to give refunds immediately when asked, extending into 60-days of past charges.
At the end of the day, I definitely agree with your sentiment: nobody wants to get stuck paying for something that doesn't work for you. I think we've achieved that for our customers, but it took some iteration to get all of these tiers together, and other companies will have to come up with their own free trial plans. Just like Baremetrics is doing per their blog post.
(The other side of this is chargebacks: businesses do not want chargebacks! Stripe, for instance, charges merchants like us a $15 fee for chargebacks [disputed charges reported by a customer to their bank], plus the time it takes our customer service staff to respond to the dispute, plus losing the payment from the customer! We are incentivized to avoid chargebacks.)
As a b2b it requires a bunch of extra planning, trials, evaluation, and then mgmt approval. I guess I'm just saying that it's much harder in b2b to see who is taking it seriously and who isn't.
(nice job w/ cl btw)
It sounds like any usage spike would have resulted in the same outcome. If you had been featured in Oprah's "Favorite Things", and ended up attracting the same number of users to a paid plan, which of the events would not have occurred?
1. Server resources would have been strained.
2. Engineers would have been reassigned to scale instead of features.
3. Existing customers would have churned as a result of inability to scale.
The only notable difference is that you did it to yourself. On the flip side, it also enabled you to turn it "off" by limiting the free account, which you would not have been able to do if the users were from a referral source expecting to pay for account anyway.
1. Have you determined the capacity, and cost of freemium vs paid users?
2. Your pricing should, at the least, cover your costs.
3. Before any large marketing push, make sure you can handle the estimated load, and have a plan to 'scale out' (add application servers, defer or async possible DB calls, etc)....
I'm definitely a big fan of freemium, and specifically I really like limiting accounts by features (less by usage), but I think it's 100% imperative to control the rate at which new users are given access to your system. We're currently in beta which makes it easy (by only allowing users at our pace) but I think in your case (given the amount of processing required) it would have been very advantageous to build a line for the free account and let people wait for a bit. That builds demand, lets you control user-flow, and people who are very interested can pay to skip the line. Anyhow, just my thoughts, obviously every scenario is different and you guys know this area best.
Also, there's a misnomer that you're supposed to link pricing to features or usage. No, you link pricing to the customer's ability/desire to pay. Sometimes this correlates with features and pricing but often times not. Multi-user access is a good example of a feature that naturally appeals to a company with resources (i.e., ability to pay). Automatic collections is not.
The alternative is shared password lists that never really go away.
They're still going to use customer support and infrastructure as much as other customers, but there's going to be that little bit of extra friction for users as they find out what the password is, and communicate any changes to it.
Personally in the context of Cronitor, I'd drop the user levels, and just segment on monitor counts and features. Maybe bump Slack integration up to the team level, since that's the sort of feature which is genuinely useful to a team, and less so to an individual.
Assuming "over 1,000" means "over 1,060" then 53 paying accounts actually means "less than 5%" which fits with their assertion that "the average B2B conversion rate is around 3-5%".
While I applaud converting 461 "potential paying customers" into 53 paying customers, the 11.5% rate seems more like an internal metric. It's important to focus on what you can achieve and so segmenting out those with the requisite "subscription revenue" to be able to convert is useful for refining internal goals, but there is a reason why the average conversion is so low: you always get customers who can't pay.
If other businesses were excluding those customers the quoted average would be greater than 3-5%.
I often download a 15 or 30 day trial of some program and that sounds like a lot of time, but it can take a day or more to really evaluate something and I need to multiply that against some guess of the probability we will really adopt it. Well we have lots of tasks that have a 100% probability of "must be done" and those are more important and then the trial ends and I forget about it.
If you make somebody put some chips in the pot upfront it makes them more serious, like they not only have something to gain but also to lose.
It's a timely subject for me. We're preparing to test this at Cronitor and we'll write a blog post with our findings.
I like a lot of the features of the app, but it would be a lie to say this trial thing didn't push me over the edge into a "buy."
if the "work" is just spinning up some new AWS nodes, the math is easy. If lifetime value of a free guy > cost of AWS node, do it. If < cost of AWS node, don't do it.
If the work is lots of staff related hand-holding - then it depends. If your staff is at 100% capacity and you need to hire more staff, it is closer to the AWS example above. If you have staff sitting around doing nothing anyway, then maybe it is "free" as in an already sunk cost.
In their case it seems like they had some engineering flaws in their system (who doesn't), that made the scaling up part harder. Good to think about how to scale up early so you don't hit a wall.
But free users can be put to work in other ways, and that value may be very hard to peg down accurately. A good 'free' user that promotes your service to paying users, for instance, an employee that uses the product at home that advocates its use in the business where he/she works. Or maybe free users give you content that you can monetize in some other way.
And that's the hard part, assigning a value to the contribution. The cost part is the easy bit, servers are cheap. But support is costly and even then it may still make sense to have a free tier, it's never going to be an easy decision.
They also probably figured that's true given they already raised $500k so they actually do have additional capital at their disposal. They're not bootstrapped anymore. Smart move to pull the plug and rethink things.
In general, I think most engineers these days overuse databases. At Google, common wisdom was "never, ever hit the disk during routine serving", and when I was in the financial industry we'd regularly process the NYSE TAQ data (50GB, about 2B trades) on a single machine nightly.
It'd be really, really nice to have growth gated on technical problems. :-)
Edit: I can't speak for the team and I've never worked on Baremetrics' code base, but I have developed against the Stripe API quite extensively.
Otherwise you're wasting your time and your users time.
* Unless you want to eat the costs for a few years. In that case you need to really develop your brand to become top of the line in your market.
Nobody is saying scaling out to an additional 1k customers is easy. The great thing about scaling slowly is that you can grow into the problems and address them as they come up. Suddenly doubling your userbase, most of which will ultimately churn out without paying you a dime, and having to deal not only with the technical but also the support ramifications, is a recipe for failure. As the article says.
If it's not, someone else would happily figure out a way to take care of your customers.
I'm not sure what we're disagreeing on, really. The blog post literally has "failure" in the subject. It was an experiment that didn't turn out well, thus they ended it and are now moving on.
Sorry, but it's not quite that easy. Stripe has rate-limiting and paginates response data. Providing metrics like "churn growth rate" between 2 dates requires you have all the data every day. You can somewhat rely on webhooks but not for historical data prior to a customer's sign-up date. Oh, and Baremetrics has plenty of queues.
(Source: former Baremetrics engineer)
In theory, it should be "easy" to scale since each account is fully discrete. More accounts, add a few more "workers" to pull jobs off the queue and everything is happy. Even on the front-end each company is hitting their own private data set, their own indices, etc. So unless you are discovering individual data sets / companies that themselves have a couple orders of magnitude more events than you are prepared to process... Even still, the total number of discrete metrics, and the cardinality of the metrics doesn't change all that much just because event counts themselves (total billings) increase.
So I can definitely see from both perspectives here... why is something like this so hard to scale up? Best answer is because the real world presents challenges well beyond the theoretical complexity of the problem at hand.
Many companies operate the freemium model with much less than 11% conversion rate -- but the additional customers they have to carry with the others are not so a heavy burden.
The advantage of the freemium model is faster scaling and adoption of the service -- but with the cost of an additional burden ... and you have to know, if your business can carry the weight or has to do differently.
Our free plans were limited to an extent that they didn't blow-up our service. Our primary concern was that as we've built out our product, introduced more advanced features, and signed bigger customers, the risk that a free user could degrade service for paying customers no longer seemed worth it. We considered isolating free users in a separate silo but maintaining that infrastructure also didn't seem worth it.
Ultimately these are hard problems to test because rolling out a change like this requires a lot of work. Even if you release it via b-test you've already made a real investment to get that far.
For those curious, we are keeping the free trial model but we do hope to continue improving trial conversion.
Edit: For clarity
This doesn't just go to being prepared to handle growth, but realizing that most customer acquisition strategies either decay over time or reach a point of diminishing returns, and putting yourself in a position where you need growth and volume to survive and betting on your existing customer acquisition is a formula for failure.
Like all thing, Growth is neither good nor bad on it's own, and should never be more than a means to an end.
For what its worth, we'll be signing up and trying out BareMetrics shortly. I love what we've seen of your product so far and look forward to seeing what your team has delivered.
One other comment -- I love how you pulled in the output of your product to so clearly support the point of your blog post. Not a bad job selling your product while discussing technical issues :)
Combined with the fact that your About page shows 5 people, this is easily the most interesting part of the post :)