Usage-based pricing isn't always best
chrisreuter.me
chrisreuter.me
An example from my experience: Datadog. It makes sense that we pay for usage, after all things like logging and metrics scale in wildly different ways from product to product, and charging flat rates or per-user rates doesn't factor this in well enough. The problem is that for a feature-complete Datadog deployment, there may be 10-20 axes of pricing – how much log data, how many unique metrics, how much of your logs do you want to index, and so on.
The problem comes with cost estimation. Because there are so many axes, we can't just take our size and figure out a cost estimate, we have to model out different scenarios. Does a traffic spike for us equal a logging spike? or Metrics? How does it impact second order pricing like log indexing or archival? Even when you know exactly how much, e.g. logging you generate, it's still all approximated modelling, and many teams won't have the necessary input numbers at the beginning anyway.
Stripe's usage-based pricing works well because it's so direct – it's just X% of revenue. Datadog's is a pain because there are many factors, all of which are quite far removed from revenue.
I understand that at some level this is required. They're pricing for compute and storage, essentially, and a cloud provider offering those wouldn't price them at a flat rate or per-user rate, but it always felt much harder than it needed to be. I'd almost have rather paid more, but a known amount.
However,
In the case of Datadog it’s even worse that they don’t allow you to cap costs. If you have an accidental spike (e.g. some deployment gone haywire and relaunching containers and suddenly you have 100x the amount of containers as usual in a 1 hour time span), they will not refund it. I had to plea a lot for me as a solo developer, and they ended up locking me into a 1 year contract in order to be reimbursed.
I’m convinced it’s entirely deliberate and they prey upon these types of things. It was a terrible, hostile experience, because I had to pay a huge sum I did not get any value out of. I was so scared of using their product for the rest of the year that I simply wrote off the yearly subscription as a loss.
I'm sure it is too.
My colleagues are investigating Datadog as a vendor and I think they're off their heads. We're meant to be tightening our belts and I think we'll waste a metric fuck ton of money on Datadog for a gram of value. Hopefully we don't go with them.
If you've got 20+ people, it's a no-brainer, the value is immense. Fewer than that and it's a tricky call.
It's a great product though, and it's in my list of "things I'd sign up for immediately" if starting a new company.
To be fair to them, I believe they bill at something like a 98% high water mark for the month, so you can spike briefly like this.
I completely agree with the point though.
With these usage metrics and a little bit of automation (for example the Webhooks integration: https://docs.datadoghq.com/integrations/webhooks/), you could stop shipping telemetry at a certain threshold.
Disclosure: I work at Datadog.
The problem is that because of the billing complexity, it's hard to predict how billing will scale with usage. There are just too many axes. Even steady-state billing is hard to price out before signing up, so much so that the answer we received from sales was to just try it and see what the bill is.
This becomes a road block. "I'd like to move us to Datadog! – Oh yeah, how much does it cost? – I won't know until after we've moved." – these conversations are hard to have internally.
The product is fantastic, I'm a big fan. And the total price isn't necessarily bad, it's just opaque, and required a lot of work on my side to model out the pricing, present the business case, get sign-off, and manage the contract over the years. Not something that I, as one of ~8 engineers, wanted to spend my time doing.
For what it's worth, that's how Stripe Payments works. But if you want to also use billing, invoicing, other additional functionality, then it's X% + Y% + $Z/month.
Like in Electricity markets - you can either get market pricing (cheaper, but very volatile) or fixed price.
The problem here is the statistical distribution of the usage will bankrupt you as a company if you try that. People mentally want to model everything as nice, polite distributions through a combination of all the statistics classes they've ever taken using them (where the "uniform" distribution is the default and "gaussian" is if you want to get fancy) and internal cognitive biases around the nice distributions generally encountered in most real life and a general lack of experience with pathological ones, but the real distribution of consumption is grotesquely pathological and has huge spikes in the tail. The probability of one of your 10 biggest customers, which collectively account for ~90% of your business, getting an unexpected 10x or greater spike, isn't negligible like a naive high school statistical analysis might suggest. It is instead virtually certain.
I've simplified this problem down just to concentrate on that bare statistical problem, which is something engineers should try to develop an intuition for. However, as we de-simplify the problem by re-introducing back all the real world complications, they all tend to make this problem even worse. For instance, give a company an all-they-can-eat deal, and if they accidentally spike your usage because the intern accidentally committed a debug logging message that spiked their usage by 50x, they have no particular incentive to fix that.
Naturally, in the real world you'd put limits in the contract... but if you think about it, flat pricing with limits that you can realistically hit because the limits really ought to be thought of in logarithmic terms, rather than linear ones, is just usage based billing with extra steps.
I've also taken advantage of the local topic of conversation being a logging service. Other businesses are not necessarily so pathologically distributed. Email is more reasonable to charge per seat, partially because people are used to the pathological part of the resource consumption distribution being cut off... or to put it in the terms everyone is used to, people are used to being told they can't attached a 3.5 terabyte attachment to their email and that emails to a few thousand people at a time generally cause trouble. Since the number of emails an account can use is bound to some degree by human attention and the size and therefore resource consumption of the email is bound by size limits, it tends to be less pathological. Email also permits throttling, and indeed, if someone is suddenly receiving millions of emails per hour they may be happy that you are throttling their inbox. Although in the end it's probably still more pathological than you may intuit.
Even here though, when you return the real world back into the picture it doesn't mean a flat pricing model perfectly works. Give them an $X/month all-you-can-eat, and one of your customers will find a way to provide email addresses to every resident in a country or something. Put limits on that and it's just usage with extra steps. There is a sense in which usage-based-billing is the only real option, it's just a matter of the details of how you price and market it.
In a sense, yes. But the difference is that with flat pricing, you can at least easily predict what your charges will be. With usage-based pricing, you can't unless your use case is very simple.
For example, let's say you have a product that manages AWS cloud spend in a way where you can always reduce AWS costs by 10%, and you charge 2% of cloud spend for this product. This is a great example of usage based pricing: whenever the customer is paying you 2%, they are saving 8%. This is a no-brainer, and the customer will use your product as much as possible.
On the other hand, lets say your product can sometimes save 30% on AWS spend, and sometimes it can't save anything at all. For example, maybe it's great at optimizing spot instances but not reserved instances, or it's good for some AWS locations but not others. Now pricing based on cloud spend sucks because you're forcing your users to think hard about whether to apply your product to each individual piece of cloud infra. This is a crappy user experience, and customers will leave if something friendlier comes along.
On a side note, a more subtle example here is Heroku. When you're a small Heroku customer, paying a 200% premium on cloud spend is fine because you save so much by not hiring a team to manage your cloud infra. However as you grow, the price you pay increases much faster than the value you receive. As a result, lots of customers graduate once they hit a certain size.
This is a general pricing principle I've been thinking about: you want your price to be proportional to your value. Doesn't matter if you're charging by usage, or per seat, or a flat price -- you want the decision to be a clear win for your target customer all of the time.
Take Hacker News. If they charged $50/mo, I'd pay that. But if HN charged $1 per homepage visit, I'd start visiting a lot less. Maybe once or twice a day, but not 20x/day. Both $50/mo and $1/visit are simple pricing methods, but my value is not the same with each visit given the homepage doesn't change much hour by hour. So maybe it's really worth $5 for the first visit of the day and 25 cents/visit after that, and thus the $1/visit system wouldn't work well (for me).
For many tools, this isn't the case. I've worked on a lot of enterprise software that charges on a per-user basis. This makes sense for something like enterprise content management and CRMs - the number of people using them is roughly going to correlate for their value. If you hire a new salesperson, you expect to make more sales and thus have a willingness to pay a little more for a tool that helps with that.
I hate them the most, after seeing them in companies with 10^2 - 10^3 of employees.
whenever you get an intern, or when an employee could help on a job that isn't his, you have to get them a license. And by the time the licensing process has happened, someone would have found a solution without using the software.
Then, instead of getting new licenses, some employees will just waste time to find someone who can access the right functionality on the software.
Eventually, if your company doesn't have a strict whitelisting policy for the software that is being used, employees will just end up using their own open source software or free smartphone apps, and will ditch the software that will charge on a per user basis. If your company has a strict policy, they will just hate it.
I would prefer to have a bundle, and a salesperson that properly estimates the value for each company, so that each employee can have the proper tools.
We're definitely thinking of different kinds of software here, as this would never be the case with the kind of stuff I'm talking about. Whatever CRM your company uses, every salesperson is using. It would be incredibly obvious if they were using something else (their manager is regularly looking at reporting in the CRM), and it wouldn't ever fly. To your point, many people do hate it, but you can't find a way around it.
Similar with services like Box, Zoom, ServiceNow, JIRA, etc. If it's important that each person have their own account, then a per-user model is generally pretty reasonable. You'd typically need this if you need to easily track which user is doing what and/or if you need user-level access controls.
A flat fee based on some tiered size of the company might still make sense and reduce some friction. But the amount has to work for both the buyer and seller and per-user pricing is just a natural fit in a lot of circumstances.
As the guy who knew the app's logs best, it fell to me to code up and then maintain the monthly report to generate those customers' usage data and send it to billing. (The things we do for love.)
Our app's log data wasn't complete and predictable enough to generate completely accurate usage data, so some assumptions were needed. (The problem was sometimes-missing timestamps for "usage session complete". Yeah I know, WTF? I inherited that code.)
Things got a tiny bit ugly. Customers -- whales of course -- started demanding weekly, then daily, usage reports. They demanded backing data for their bills. And, it became really hard to actually fix the defects in the app's logging so it was possible to generate accurate usage -- it would change the billing too much, and we'd have to explain it to the customer. It was a pain in the you-know-what. Especially for a guy like me who prefers to write really accurate software where other peoples' money is involved. This didn't threaten the business, it wasn't nearly that serious. But, wowee, it was unpleasant.
Lesson: If you're going to add a new SaaS pricing model to an existing service, think of it as a BIG change. Make sure you can capture accurate and commercially defensible metrics for your pricing model. Get devs to think it through. Get testfolk to think it through and work out some system tests. Sales people, don't just tell customers they can have their own pricing model, but only if they sign up by the end of the quarter.
They had a really wild architecture with FPGAs that, if I remember correctly, sat in between the CPU/disk and was used to offload some of query execution.
It was wildly expensive + expensive to maintain, so the company ended up (unsuccessfully, I think) jumping to Hadoop. Crazy times :)
Usage based pricing is fine when it’s on predictable metrics, number of logs sent, number of rows sent, number of emails/sms/etc sent, number of seats, users, licences, etc. When it’s on abstract terms like credits or abstract resources like warehouses (with no idea what resource a medium warehouse utilizes) or error prone metrics like scanned gigabytes per query, it might become problematic at some point.
Charging e.g. by # of processed rows is more predictable from end-user/admin perspective. That’s all. But obviously they make more money the way they price, what do i know? :-)
It’s akin to a poorly developed benchmark that doesn’t consider all the real world factors but tries to win on one vector.
Here are some other intangibles:
(1) Subscription pricing is difficult to scale upwards. The concepts of upsells and new products are difficult to execute and increasingly result in lower and lower adoption. Each new capability needs to be “sold in” with a new sales motion creating endless amounts of feature creep and unused features.
(2) If your costs are variable and your revenue is fixed, you better hope your customers don’t like you or your product is really simple. Otherwise, you risk more margin compression as customers use you more and more beyond what you modeled. The difference between your ARR and the cost to serve gets smaller. Like a perverse Innovators Dilemma.
(3) Innovation tends to be simpler with usage based pricing, when done right. In a subscription model, product teams tend to be much more conservative. Each new sellable feature needs to be fully supported with training and enablement for sales teams. When the product fails to grow like the first product, then the sales team becomes increasingly more conservative. It’s a viscous cycle.
Usage- based pricing makes a lot of the above moot. If any existing or new benefits of the product are offered to all customers, then it actually aligns the incentive of your company and your customer. Use more to drive revenue (you) or use more to get more value.
Of course, marginal value isn’t always the same, so that’s when usage tiers develop.
Now how you quantify all this I’m not sure. But there are far more simplification benefits to usage based pricing.
It would be much easier for us if folks liberally sprinkled IAAS "buyer" / "seller" or "provider" / "consumer" in their opinions.
Anyone know what this is referring to?
Mike Stonebraker talked a little bit about this in the context of MapReduce on the dbt podcast recently: https://roundup.getdbt.com/p/ep38-a-romp-through-database-hi...