Hacker News Reader

After seeing billing done right, I cringe every time I see billing done wrong.

For example, people building foreign keys to prices and receipt items. In what cases do you expect to retroactively change what you charged all customers? Copy that data to the receipt item, don’t give your accountant a headache and heart attack when you change pricing. I’ve also seen people using those same kinds of rows to determine what features a user gets access to. Don’t do that, use capabilities/roles. When a payment succeeds, update the user’s capabilities. When a payment fails, remove capabilities. This also allows support to just give a customer capabilities without a line item or changing their plan, because some customers are just annoying and it is less expensive to give them a feature. It also allows more easily selling custom-tailored solutions to bigger clients since the features aren’t linked to line items.

This article is more about usage-based billing, which is probably harder to get right, but feature-based billing is something I’m more familiar with and it’s so annoying to see all the ways people will fuck it up.

Your first point about foreign keys was the first contract I ever did. Someone built a billing system in perl+mysql for an ecommerce site. When prices changed so did historical invoices. I was hired to work this out because the existing developers couldn't. They only realised realised that something was wrong after a customer sued them after being paid the wrong amount in a refund.

I rewrote the entire system to snapshot the committed values at the time of purchase and fixing all the old invoices from database backups.

The way you'd assume they'd do it if they wanted relational values would be to never be able to edit prices, only create new ones.

> never be able to edit prices

I can't think of any database that makes a row strictly immutable, as in carved onto the platter immutable. People will find ways to undo any soft-guards you put into place, all in the name of being helpful.

OP here, totally agree!

Feature-based billing is super hard too, and people underestimate the complexity even more.

We were genuinely surprised that some companies wanted to use Lago even if they ""only"" had a "feature-based/subscription-based" pricing, that someone new would consider as "trivial".

(I'm a billing engineering team leader with more than 25 years experience.)

This is a wonderful example of how engineers underestimate the complexity of billing.

> Suddenly, months, years, calendars, etc... cease to matter

No they don't. You don't send a bill every hour, or even every day. Typically you invoice monthly. And if you invoice monthly, what date do you aggregate the invoice? Setting aside the obvious problem that months are not homogenous, do you start on the anniversary date of signup, or the start of the month? Well, anniversary date is generally better, because it spreads out payments, collections and cash flow over the whole month, which reduces a lot of effort in collections and finance, and reduces the impact of any issues - but it's also significantly more difficult to engineer; how do you bill February if they started billing on the 31st of January? Monthly billing is far easier from an engineering perspective, but is not generally the ideal method from a business operations perspective. And get this, if you start with monthly billing and have significant income, and then you want to change to anniversary billing, you get all kinds of income recognition (P&L) problems that you have to deal with; it's really complex.

And what do we mean by "signup", anyway? Do we bill services individually based on when they are created, or do we bill based on the account creation date? What if there is an account hierarchy?

Hourly or unit based billing does literally nothing to solve these problems, because they are endemic to the billing domain.

> Similarly, consumption-based plans are relatively straightforward. Simply multiply unit costs with the per-unit costs.

Right - and what happens when you raise or lower your prices? That has to happen at a particular point in time, which is typically in the future. What if we give tiered discounts (first 1000 minutes at 2c, next 1000 minutes at 1c)? In both cases, we no longer can keep a single per-unit cost, but rather we end up with a cost matrix, which can become really complex when you add other factors (such as discounting).

Also, what happens when you are charging, say, 5c per megabyte... and your incoming CDR feed gives you bytes? Do you configure this as 0.00000005 cents per byte? When do you do the rounding? On an hourly basis? Daily? Monthly? What do you display as the unit price to your users? What if another plan is charging 5c per Gigabyte?

> An example of a hard-to-solve essential complexity is currency conversions and taxation when dealing with international customers. That's an externally imposed set of problems that can't be hand-waved away.

Currency stuff should be very easy in practice, but I agree that tax is hard. That's why you need a multi-stage pipeline for invoice generation. But that's perhaps a story for another post...

Those are all problems which have been "solved" by many companies. I'm sorry but none of this screams complex to me, all your problems have 2-3 "correct" choices, or you could just copy what almost all big companies do.

The whole point of TFA is that billing systems are way more complex than they appear to be. If you're trying to build a billing system, how do you "copy" what these big companies do, when most of the effort is neither obvious nor visible?

OP here, that's exactly the point.

It's easy in theory: 'just do it', 'just copy', BUT the challenges are in - the tiny details (that can have a huge impact: we're talking 'billing' and 'money' here) - the maintenance: it's a living organism: you will inevitably change your pricing, your payment processor, the way you bill, your back-end/front-end etc

> The whole point of TFA is that billing systems are way more complex than they appear to be.

Technically the point of TFA is to market billing software. Billing systems are complex in that they can consist of many different moving parts, but billing system are also simple in that they are easily understood. Billing has to be easily understood, else you'll quickly scare your customers away. Complexity and simplicity are not necessarily at odds with each other. The meat of TFA is mostly about how billing systems can quickly become a time consuming endeavour which again isn't at odds with simplicity.

> Billing has to be easily understood

I disagree. Billing has to be correct - that’s billing rule #1. You can have an easily understood pricing plan, but implementing that plan correctly and consistently in a way that you can defend and audit can be very complicated.

Just take the simple example of implementing anniversary billing for a customer who signs up on the 30th of the month. What date do you bill them when the month has less than 30 days? What about when it has more than 30 days? What about when your customers are spread through 4 time zones? What about when daylight savings starts and ends? Do most engineers understand time and time zones well enough to get this right?

These questions are fundamental to correct billing at any kind of scale, and the answers are not simple, or at least I don’t think they are.

> but implementing that plan correctly and consistently in a way that you can defend and audit can be very complicated.

We established earlier that billing is likely to be complicated, but that doesn't mean it isn't simple. Simple things can be complex. Simple refers to ability to understand, while complexity refers to scale of interconnected parts. They are not at odds with each other.

> the answers are not simple, or at least I don’t think they are.

I think you can make a strong case that business isn't simple, but billing systems only need to serve the needs of business after business has already figured things out. You cannot bill at all, even without a formal system, if you haven't answered these questions. A billing system merely needs to adopt the answers that are already understood.

> Simple things can be complex.

I’m sorry but I consider complexity and simplicity to be antonyms - and so does the thesaurus! https://www.thesaurus.com/browse/complex

> billing systems only need to serve the needs of business after business has already figured things out

Billing is a fundamental part of your product and your GTM. Getting it right gives you an advantage, especially in terms of trying different approaches. Getting it wrong can be a burden on the business, especially if you find out that you need to re-price.

> These questions are fundamental to correct billing at any kind of scale, and the answers are not simple

As someone who also works in billing and subscription management, I 100% agree with everything you said and want to add an extra thing:

Even when the answers ARE simple, there's so many of them that you end up realising you want to pay someone else to make it their problem instead.

...or you could just copy what almost all big companies do.

This is a dangerous strategy for a startup. A big company can afford to make a mistake and fix it even if it costs tens of millions. If you copy their mistake before anyone has realised it's a mistake you probably won't be able to afford to fix it.

It's also very likely you'll only be able to copy the externally parts of what the big company does, without really understanding why or what's going on behind the scenes. Really, you're advocating cargo culting a solution. That's a bad idea.

Right! The decisions and complexity are all hidden. Billing has to look simple otherwise people wouldn’t pay for stuff.

Take anniversary billing, it seems so simple, just store the billing day-of-month in the database and run the billing using the sql:

   select do_billing(customer_id) from customer where billing_day=extract(date from current_date)

And this works great in December and January, but if you don’t have good financial controls you might miss the fact that you’ve dropped an entire week’s worth of billing by the end of the following year.

> you could just copy what almost all big companies do.

What almost all big companies do is pay someone to make the problem go away.

It's interesting that you consider what are solved problems in the world of RDBMS to be difficult, but currency conversions simple.

Out of curiosity, does your system use a NoSQL database?

> Right - and what happens when you raise or lower your prices?

The billing table has "CustomerID, ResourceID, HourID, Units, SKU_ID" columns. You raise or lower your prices on the hour boundaries. The new rows inserted after the price change will have the new SKU ID with the new costs.

Irrespective of when you send an invoice to the customer, the report simply does a "GROUP BY" over the (CustomerID, ResorceID, SKU_ID) columns and some range of the HourID column. You sum up the Units * SKU_ID.UnitCost value, or whatever.

This isn't exactly rocket science for any normal database engine.

I suspect the error people are making is not recording the change-of-billing-model as a "slowly changing dimension", and instead trying to do complicated rules-engines in code...

> Out of curiosity, does your system use a NoSQL database?

No, the last system I built was using PL/PGSQL. It's brilliant for this application.

> The billing table has "CustomerID, ResourceID, HourID, Units, SKU_ID" columns. You raise or lower your prices on the hour boundaries. The new rows inserted after the price change will have the new SKU ID with the new costs.

Like I said, this is a very simplistic way of viewing how to do billing. Sure, today you only have the hour dimension when choosing a price. But then sales asks you to give a discount to a specific, very large, customer. For only a single resource. For only the first three months. Suddenly, and very predictably, your billing is not so simple any more.

The problem is not that simple billing is difficult. The problem is that billing is not simple.

> Like I said, this is a very simplistic way of viewing how to do billing. Sure, today you only have the hour dimension when choosing a price. But then sales asks you to give a discount to a specific, very large, customer. For only a single resource. For only the first three months. Suddenly, and very predictably, your billing is not so simple any more.

I'm not sure if I'm missng some part of the complexity here, but with the solution jiggawatts proposed, wouldn't the above scenario be fairly trivial to solve?

Just implement a new SKU for this special one-time (more like first time...) discount that the account manager applies to the account.

If you want to be fancy, you could code in (or leverage one of the dozen cloud tools, depending on where you're deployed) to enable the SKU for the customer at period X and remove it at period Y and replace with the previous SKU.

To ease reporting, in your main SKU table you could have a parent_sku column and a sku_type column explaining that this row is a special discount child SKU of the parent_sku. Add basic databse restrictions and validations.

Billing is complex and requires lots of domain knowledge, but IMO it's not really THAT hard from my experience.

EDIT:

IMO a harder scenario for any billing system to deal with is account manager A made a sweetheart deal to make customer B less angry, removing one SKU completely from last months bill, giving them a 25% discount for the next invoice, and giving them some freebie.

These types of requests usually end up with a lot of issues unless the business logic promised to the customer already exists.

Well, the problem with the first scenario is that if the discount is percentage based (say, 10% off), then it's going to require code to calculate. It's no longer just an SKU - it's an SKU which does a calculation, which is presumably not part of the platform already. And so we have a new "discount" SKU - but which line items should the SKU apply to? How do we know when to stop using that SKU when the discount runs out? And anyway, why should an SKU have a discount % field on it? Why do we get different behaviours based on some magic SKU field? It's a breakdown in cohesion... and so the spaghetti begins...

You can also argue that we can just "get someone" to apply discounts manually, but isn't that the job of the billing system? Manual labour doesn't scale too well, and a lot of mistakes get made - asking a customer to return an accidentally applied discount is a very, very difficult conversation, and I've worked with companies who simply had to absorb years of incorrect discounts because doing so was the only way to keep the customer.

Dont forget co-selling discounts. First 100 messages free with a monthly resource. And is that per monthly resource, or each monthly resource?

Also, here is an example of 'looks simple' but 'raises many question'

1/ Should it be a discount on future invoices, or a refund (an angry customer will ask for a real refund: aka $$ on their bank account)?

2/ Do you want to keep track of how much has been discounted/refunded to which customer (so that you can identify deeper trends: if a segment is being discounted many times: maybe you can adjust things, or maybe it's fraud?)? In that case, just erasing a line doesn't help

3/ If you refund, how do you deal with taxes that have been paid already?

Btw, we've written a quick primer on Refunds, Coupons, and Credit Notes: to lay out the pros/cons

https://github.com/getlago/lago/wiki/A-Primer-on-Refunds,-Co...

#3 is an absolutely classic example of unanticipated complexity. Tax might differ depending on individual line items on an invoice, so if you’re refunding $50 on a $100 invoice, how do you compute the tax payable for the invoice?

If memory serves, we ended up solving this by pro-rating the tax against the refund amount, on a per-item basis. And, I think, some smart accounting.

Oh and then you have the problem of when tax is becomes payable. Is it when the invoice is raised (accrual based)? or when it is paid (cash based)? And if memory serves, it depends on the size of your company. :head asplode:

Where is the SKU (equivalent) applied? IME the metering data is sourced from or near to the service. We cant move these per-customer-per-period variants there. So instead you have a fixed usage type/metering and use arbitrary “pricing plans” that have differential pricing to be applied post hoc.

The problem, apparently, is, that even in the largest software installations there are a lot of snowflakes to be taken into account (your "very large customer").

Yeah, as engineers we want to say "well tell them we can't do it". But in reality, if the customer represents 25% of your income and they are dead set on getting their way, what you are really saying is that we should lose the customer - which translates to, "we have to sack a quarter of our team". (I had a CEO who used to use exactly this analogy with us).

> "well tell them we can't do it".

hah yeah i've tried that a couple times over the years, it never worked.

Another aggravating thing i ran into is in the industry i was working in (pharmacy) prices are calculated using a contracted formula. The pharmacy (my system) calculates a price and then sends the rx to the insurance company. They calculate using the same formula (hopefully) and if their result matches or comes out greater than what we calculated then they pay the invoice. However, if their calculation came up less than what we sent they'd pay their amount. It was a cat and mouse game constantly.

Well one very large insurance carrier had a bug in their system that under the right conditions would come up a dollar short for a very common rx. I understood what was happening and called and called and bugged them to fix the issue. Finally, they just ended up saying "we're not going to change the system. end of story.". It blew my mind, I was right out of college and too naïve to consider they'd just flat out refuse to fix the bug.

I actually saw something similar in telco, where a carrier was rounding charges to the cent, but charging in fractions of a cent. So for example a CDR might be worth $0.001 but my customer was being charged $0.01.

My customer complained, but in this case they prevailed because the contract said nothing about rounding.

This can be really significant. They ended up having to compute the charges and send the carrier a bill for the difference. It was worth tens of thousands of dollars per month.

> The problem is not that simple billing is difficult. The problem is that billing is not simple.

Love this quote!

I suspect doctor_eval and yourself are talking past each other a bit. You seem to be conflating metering and billing in your model:

> You raise or lower your prices on the hour boundaries. The new rows inserted after the price change will have the new SKU ID with the new costs.

In the big systems Im familiar with metering, pricing, and billing are discrete. Yes, usage will be metered to a SKU. But there are arbitrary pricing plans that denote price/tier/discounts per SKU. So the price is only joined to the SKU usage, per customer, at bill generation time.

> (I'm a billing engineering team leader with more than 25 years experience.)

As a software engineer with >25 years of experience, now running my own SaaS business and dealing with subscription billing, invoicing and tax issues:

1. I agree with everything you wrote. And more. Taxation introduces an entirely new level of complexity on top, and lawmakers design laws for humans, rarely thinking about automation. If you don't believe me, look into EU VAT rules, and enjoy the mess with GB/XI prefixes, the case of AX, and parts of ES that aren't really in the EU VAT zone, but look like they are.

2. The responses to your comment are typically dismissive and are to be expected from this forum. Usually the replies are from people who do not do the thing you do, but think that the thing you do is simple and have all kinds of opinions about it. I used to care more about replies, these days I just smile and move on.

1. Thanks! We were lucky that we dealt only with the (non-US) Asia/Pacific region, where all the tax regimes we encountered in the countries we operated were relatively simple, mostly straight percentages. But we did have to deal with tax system transitions in a couple of countries, and it was obvious to us that tax is one of those things that was quite arbitrary, could be changed overnight (which happened in Malaysia after an election), and needed to be factored out as its own domain. As I’ve been trying to argue, there are no “perfectly spherical cows” when it comes to billing… simplifying assumptions end up killing you slowly.

2. I think it’s often young people who haven’t encountered externally-imposed, non-negotiable complexity yet. I try to take it easy on them because I suspect I might have been like that once :)

EU VAT aren’t too bad compared to the North American ones. But I grew up in the EU so it clicks more with me and dealt with foreign customers

I agree. Billing is fearsomely complex and not ever going to get any simpler. I feel the old rule about not rolling your own crypto should equally well be applied to not rolling your own billing, tax or finance systems. That said, the alternatives all suck. If you are a F500, sure go ahead and spend $$$ on a SAP implementation. But whats a small 10 person company turning over a mill or two to do?

I think you've possibly simplified by solving a different problem to what's described in the article. The problem space described is subscription plan & pay per use (with what sounds like might be percentage pricing as well). This is quite different from per unit billing.

> Some of this complexity is self-imposed

Having spent years working on invoicing & payments systems, the business will impose complexity (which you can only say no to a certain amount of) and then legal will pay you a visit and explain all the many rules that apply to billing & invoicing that vary across each country. Internationalization isn't just a tax complexity.

Good grief this was informative!

Thank you for taking the time to share that.

Of course! Thank you for the kind words!