Billing systems are a nightmare for engineers
getlago.com
getlago.com
Dates are no problem, everything is billed at the end of month, quarter or year.
Upgrades are downgrades are simplified, customer can legally change tariff only on predefined dates/events.
Usage - yes, but exampled in the article do not even scratch the complexity of this
Idempotency is not really needed, the billing process is not continuous, it is a batch job.
Cash collection is out of scope of billing software.
Taxing was clear. Yes, there were multiple billable items with different tax rates, but not too complex. All customers were local.
The nightmare part was:
- customers can have multiple consumption locations, location can have multiple meters and customer can request to split the bill on multiple invoices or combine multiple bills to single invoice as they please
- meters can be replaced at any time
- meter readings are available on dates totally independent from billing cycle. Most of consumption data are mere forecasts.
- when the actual consumption data is available, everything must be recalculated AND compensated (on a correction invoice showing the difference)
- actual consumption data can be wrong and may be corrected at a later date, even multiple times
- consumption points can be added or removed or moved to different customer at any time, but this information is only available after the fact
- the prices can change mid-billing cycle for some customers but the consumption data is not available with that granularity
- customer legal information (company name, address) can change mid-cycle
Bonus points if the meters can send data out of order.
When connected they would emit events as they happened, but when there was poor service or disconnection they would buffer them and send them as a burst once they reconnected... in reverse order.
I forget the finer details, but the service I was writing was supposed to notify of trucks entering and leaving geofences, inactivity etc. anything you can think of where the order of events is important.
https://www.ibm.com/docs/en/db2/10.5?topic=tables-bitemporal
In a time-series (or event) table you have one date - when the data point is valid (event occurred).
In a temporal table you have two dates - interval when the data is valid.
In a bitemporal table you have two intervals. First describes when the data is valid in real world, the second when does the system know about the real-world state.
I.e. if company changed address on 2022-05-01 but the system will not get that information until three day later (2022-05-03) you have two validity intervals.
Then you may ask - what was the company address on 2022-05-02? And what did the system think the address was on 2022-05-02? You will get two different answers. This is important if you need e.g. to make a correction to previously issued invoice. You need to know what data was used for original invoice and what data should have been used.
Apparently the billing system provided for 7 'counters' and hence the smart-meter had to have exactly 7 counters.
Indeed, we only scratched the surface of the complexity of metered billing, we'll do a deep dive soon on this topic (it does deserve its own post).
I think for the'nightmares' you mention: - some might be specific to utility (not applicable to an online business), such as '- meter readings are available on dates totally independent from billing cycle'. - Some topics might be simpler for a utility co: tax for instance (you know where your users are, and they are limited to a geographic region)
- But some nightmares such as '- the prices can change mid-billing cycle for some customers but the consumption data is not available with that granularity' do really resonate for online businesses too, indeed
Thanks for sharing, great insights for our future post!
Sounds like a nightmare to deal with, of course this is only assuming it is enforced. If it is not well who cares, companies like Lyft/Uber view laws as a starting point in negotiations when expanding into new areas not as a brick wall of dogma.
I think we call ourselves CPAs :). In all seriousness the 'solution' depends on what industry we're talking about. Short term rentals? Avalara. Software as a service? Various accounting firms and startups. Etc.
The meter being replaced shouldnt be a major issue, this is relational (at least we treat this all as relational) data that should be captured (by the customer information 'CIS' system), and should be available with a 1-2 day delay. Similar argument applies to other relational aspects of the premise under consideration. Not saying that those are easy (more ways for this process to have gaps).
Additionally, it didn't mention financial regulation changes. India had changes sometime in the last few years which required whole new systems built that were specific to India customers. One example of complexity is this: does the system apply to customers who are in India or does it apply to businesses who are in India (the owners might not be)?
He also did not mention fraud, which is basically uncollected spend that results in charge backs typically. If you get too many chargebacks, you'll get fined. If you get even more chargebacks, you'll get kicked off the card network and will no longer be able to accept cards within that network.
Post pay (customer gets the product and pays afterwards) models are subject to more fraud, uncollected charges, etc.
There's also the fun of dealing with downstream payment processors. I haven't worked with a processor or bank yet that was very reliable. Sometimes they'd return 500 http codes and then process a payment, requiring manual intervention. Sometimes that happens with files of payments because most banks deal with files for transactions.
IIRC, I've read this is one of the reasons that systems like SAP are so popular. They might be an ugly monster from a technical perspective and seem overly complex, but they provide thoroughly tested implementations that handle all that stuff.
Products like SAP do seem complex because of (i) the complexity of the billing problem and all the edge cases, (ii) they were not built with an API-first or engineer-first mindset
We're only at the beginning of the journey, but that's one of the reasons why we thought of open-sourcing our billing API, so that the work could be forked and tweaked to address edge cases if needed (no need to wait for SAP to update the product: and they will never prioritize small edge cases). Also, it seemed to be no middle ground between 'Stripe Billing' (it's great for B2C Subscriptions) and billing systems that could address the millions of nuances of pricing between 'subscription' and 'usage-based' (most companies are a mix of both).
This is a component classified as "basis technology" relied upon by many modules for billing (e.g. should a daily salaray be calculated in a country for a particular person? Should an invoice be considered "overdue" yet, based on the number of business days passed since issue? etc.).
So more agile players can get involved more quickly and entrench position.
Two other things that you may want to consider:
1. PCI compliance. If you're involved in the storage of payment information, you may want to look into these regulations.
2. Regarding point 3 of your mission "...only integrating with specific partners to lock you in (e.g., Stripe Billing only working with Stripe Payments)...". I think there are many companies out there who want to integrate with more than a single payment provider (eg Braintree, Stripe, Adyen, etc) but run into this problem. It increases the cost of payments due to the "rent-seeker" charge method they use. Providing a design framework for how a single company could integrate with multiple providers could be beneficial. This is something I've considered trying to build, but I think it could fit with this initiative.
We're building an open-source alternative to Stripe Billing indeed, but we want to share our personal experience, hence the personal blog post.
We'll open our repo very soon and our public roadmap too. Thanks for adding the 2 points, we're working on 'a design framework for how a single company could integrate with multiple providers could be beneficial.' indeed!
Will ping you back here when we're live!
If you think you know them, pay attention to https://twitter.com/aotearoa_ben/status/1526786701750050817.
That's a law that changes the tax rate if
1. The purchase happens in Texas. 2. The payment cleared in a specific 2 day period. 3. The item cost < $100. The cost of shipping has to be included in the item's cost. 4. The item falls into specific categories that almost certainly don't match whatever categorization system you are attempting to use.
Seriously, calling out just one example, https://comptroller.texas.gov/taxes/publications/98-490/scho... points out that a kit containing school supplies qualifies as long as the cost of the qualifying school supply items in it exceeds the cost of the non-qualifying items. I guarantee that you don't have a category for that in your accounting system...
We transitioned to using an external tool (taxjar, IIRC, but might have been something else). Was a pain to feed them data to get back tax data, and.. so many exceptions and exemptions I had to learn about. Weeks later I get "all these numbers are wrong".
I dug in and went through logs.
"no, these are right. taxjar is adding x% on ABC and shipping".
"We don't charge tax on shipping".
"Well, taxjar is doing it, because it's how you're supposed to do it in this state".
"We've never charged tax on shipping"
"Well... you've probably been non-compliant for years then".
I think when they'd started 'computerizing' this back in the early 90s, shipping costs weren't taxed. Or... (IIRC) the amount of shipping cost taxed was proportional to the amount of taxable goods being shipped in the state (vs out of the state) and when they started, what they were shipping wasn't mostly taxable (like... some of the main items were taxable, but the supplies weren't, or something similar). And... if they were shipping to a non-taxable entity (non-profit) then the in-state shipping charge wasn't taxable, but they were supposed to pro-rate it based on the proportional distance in the target state, if that state charged tax on shipping, then track that and remit to that appropriate county/state authority.
These were some of the rules that a variety of people had tried to describe to me both what they thought was supposed to happen and what they thought the original software was doing, but... they could never find 2 examples to line up to confirm or justify those explanations.
Another example: certain items (chemicals) were taxable in some counties and not others, and might be taxed differently based on whether they were being 'used' by technicians vs delivered to a customer (again, different counties had different rules - and some of these rules were indeed confirmed by digging through taxjar and their support and state agency websites).
When this company started, it was one location in two counties in one state, and as they grew, the specifics grew (and also changed yearly) and what they should do became lost to everyone.
Moving things to an external hosted tax-as-a-service system was supposed to 'simplify' things, but it just introduced a lot of turmoil and headache, because routine numbers people were used to for years were suddenly replaced with "new" numbers which no one trusted.
If a person clocks in at 9:52M. The company only has to pay them from 10am. But in other states they must pay from 9:45am. There’s so much complexity in the us due to all states being different.
It is an interesting challenge though, to create software that is easy to maintain while capturing all these complexities.
Completely developed in the open, and with full test coverage.
This should be required to collect taxes. No API? No taxes. API down? Tax waived.
https://www.propublica.org/article/inside-turbotax-20-year-f... has a good overview of how companies that provide software for calculating taxes guarantee a market for their products by keeping the process of calculating taxes more complicated than it needs to be.
I have no words to describe what I just read. That's straight out of the TV show Silicon Valley.
The only way to reduce the pain instead of just shifting it around would be to massively simplify the source of truth - tax codes and laws.
Some tax regulations were considered illegal by higher authorities, but that won't stop a tax auditor in a local level charging them and potentially closing down your business.
I have to imagine government workers are already writing software to identify if we’ve all implemented it correctly, when the could have just given us the code/api up front.
Because of the jurisdiction issue, even if each local (and for each location all relevant governmental tax authorities) had a clear API, interacting with them all, and dealing with the interactions between them recursively, is itself a large problem.
And from what I've seen from smaller jurisdictions, the automation software is all about simplifying a human-in-loop workflow rather than full machine-driven rules.
I could imagine a world where the IRS ran the tax software for every taxing entity in the US. Each jurisdiction would have to configure their specs or potentially even codify the business logic. International would likely be tricky but it's similar to what is being done with ACH and other banking systems that are slowing moving to international. So, it's not impossible there's just not much of a push for it.
Nay. Shifting pain from thousands of projects to a single org obviously reduces overall impact.
If you believe the statistics on literacy and multi-lingualism, it's a tossup as to whether Chinese or legalese is more widely understood.
Source used (non-US) government api to do with legislation. The freaking ambiguity and cludges. Luckily they had a team of good people to help.
Imagine a US LLC setting up their chart of accounts to track IRS deductible expenses. Coding expenses into this 17-account schema in the normal course of book keeping means taxes can be filled by copying the account totals into the corresponding IRS form field.
But later, the business wins an federal contract which reimburses certain "R&D" expenses and possibly some "overhead" expenses. Now not only must every expense must be categorized with one of 17 expense categories for the IRS, but it must also be categorized with one of 3 categories for our federal contract purposes. But wait, a small business loan application may require profit and loss broken down by a different set of categories. New and local tax authorities may impose other categories.
One way to address this complexity explosion is with multiple instances of the same accounting software; each imports the same bank account statement, then justifies it against a chart of accounts. But many businesses will break out Excel and hope there aren't too many expensive errors.
If new regulation is inevitable, why does our software pretend otherwise?
I would also mention topics like: - Reporting (various data aggregation and audit reports) - High reliability, nothing hurts company more than unability to bill their customers - Rounding, this can be seen as a subset of "taxes" but it's much more complex - Locating user, also can be seen as a subset of "taxes" but user can have country A set in profile configurations, country B on credit card and country C geolocated from IP address - Timezones, issue everywhere but when we talk accounting it's super important - Talking to moronic payment gateway providers. This is the biggest ones. I would love to just flip Apple and Google like Linus flipped of nVidia. Proprietary, poorly documented, "find out yourself by weeks of experimenting" bullshit with no easy way of getting technical support even when you bring those companies millions per year. Things don't work, deprecate monthly or weekly and expect you to be always ready to make changes. Some implementation make zero sense at all with complete paradigm shift of handling payments like between Google in-app pay and Google subscriptions.
But as my SO says "Don't cry, it pays well"
And it's not "billing", it's Compliance + Bill Presentment (marketing!) where we are always trying to find the most profitable local maximum of explainability to Sales, Customers, and Management/Enforcement while also having hard checks to prevent or at least mitigate losses.
No one wants a Knight Trading excursion....
I came away from this with one big lesson - if you're considering a complex billing model, consider the engineering implications first. With most products, engineering feedback gets taken into account - often product proposes something, engineering breaks it down, product realizes that feature x is vastly more complicated than they thought and not worth the effort, and the requirements are changed to simplify or remove it.
The one thing that never seems to be true of is pricing models - that decision gets made at the very top and handed down with no chance for feedback. I think that if it the folks designing the billing system realized the costs, they might simplify things. If the complexity of your billing system means that 3% of your engineering team (plus additional folks in support and finance) is going to be working on it forever, but if you simplify it a bit you could keep 90% of it and only have 1% of engineering working on it, that might be a good tradeoff - after all, that leaves you more engineers to build features, which should drive additional sales. Unfortunately, that analysis never seems to get done up front and the cost is only understood after the billing system is deeply integrated into everything and would take an unpalatable amount of effort to change.
It doesn't work when the model is too simple or restrictive, which will simply result in the model being violated; you do need at least the ability to customize pricing for individual customers, and for specific invoices, but other than that the decisions (e.g. whether you apply invoice discounts as percentages or specific amount or both) can vary but you just need to pick any reasonable option and stick with it.
In the US, an invoice is just a weird PDF that you might glance at before sending off to your accounts payable team. But in other countries, especially those that use VAT style taxing systems, an invoice can be a legal document that expresses costs and taxes in a legally prescribed way for accounting purposes. Many countries have prescriptive rules over how you invoice, who can write invoices, when you can invoice, what goes on an invoice, etc. And every country does it differently.
Are you building a service that charges incrementally many times per period? Or even worse, refunds incrementally? You might be sending a truckload of expensive accounting papers to your customer every month, one per transaction. And each of those pages was printed using government prescribed software from oddball vendors you must use or else.
* Except (these aren't the right thing to do, but I've seen it in the wild many times):
- When someone in sales issues an invoice with a negative amount on line 3. Hey, if I take payments with invoices, why not give back?
- When someone agrees to settle up by providing free services and throws it on the next bill.
I worked briefly with a tax-saas, and to handle refunds I just... did a negative amount. It worked! Except.... I hadn't looked closely. Their API removed the negative. A charge of -$7.92 was just a charge of $7.92. Their docs indicated "we don't accept negative numbers - use a refund transaction instead" but... the API accepted the negative number, then changed it to a positive, and kept running. No error code returned. Really really poor experience, and I hope they've fixed that.
And if it is rejected you have to enter it all again by hand; editing isn't a feature.
Imagine you bill your customers monthly (the billable period), all the charges (usage-based features + subscription) will appear as line items of a single invoice.
This enables you to gather all the fees of a period into a total invoice, but still be able to provide granularity to your customers (breakdown of all the fees to be paid).
Thanks for the constructive feedback!
It's not EU governments or post-purchase issues you have to deal, most of the time - it's trying to persuade a large accounting department to pay you unaccountable money. If you can't follow their invoicing requirements, it's just not going to happen.
Do people ever stop and wonder WHY in nations like Portugal the informal economy is roughly DOUBLE of say the US? When every invoice must be available to the government, what actually ends up happening is non-conforming invoices either get made up by the customer or the money magically flows out under some other auspice.
Making it literally illegal to accept an invoice that is legal in the country in which you obtained it borders on logic even my toddler can understand is absolutely begging for bad business climate or tax evasion.
What is the mechanism for French government to subpoena a non-EU business to find this invoice? If the customer (instead of vendor) produced an invoice that matched their bank statements and non-conforming receipt, hypothetically, would anyone really know the difference?
- I am Frenchy (a French co) - Buying software from Ricky (a US co)
If I don't have a proper invoice from Ricky to give to my accountant at the end of the year, and I get a tax control, then, I'll be fined.
If the tax ministry notices large amounts of similar fines, they might look at Ricky.
So the risk is more on Frenchy (risk to be fined). But I'd say that Ricky is safe for a while, unless the invoiced amount are really huge.
That's why any US company wanting to do business in Europe either has an European subsidiary, or follows something that's compatible with EU invoicing schemes.
Yes which is why in nations like Portugal, where any business invoice may be subject to accountability to government, there totally isn't a massive informal (read: tax-evading) economy to deal with the fact that the government makes it literally impossible (at least in above words) to accept an invoice legal in the jurisdiction in which it was issued.
My company does plenty of business with EU businesses and we use a single standard EU VAT invoice without issue.
Also, the informal tax-evading economy relies on personal contacts and 'mutual understanding' - it's not easily accessible to a foreign vendor, and when foreign vendors do want to access that economy, it requires much more adoption of local customs than just making a different format of invoice.
Informal tax-evading (when the seller isn't evading, just the buyer) doesn't require personal contacts. This is just naïve thoughtless statement. It happens all the time. Example: business owner goes to Panama where they know no one. Buys a pallet of llama wool, seller gives a non-conforming receipt. Buyer goes back to France, imports the pallet as "cotton" and creates a fake invoice showing the Panamanian sold cotton. Seller then sells llama wool on the streets for cash, marking in the books that they sold "cotton" for significantly less. They then use the on-the-books "cotton" proceeds to buy a shit-wagon car or something, and fix it using the unrecorded cash on the side. They sell the now nice shit-wagon and record the proceeds as profit. Now all the money for the llama wool is accounted for.
Cross country tax-evading is only assured in this case to require personal contacts when the tax evading happens on _both_ sides. When it only happens on the EU side, there's no need for personal contacts on the US side.
Customs catches him (distinguishing cotton from non-cotton is surprisingly easy, so is forged documents for invoicing and freight papers, especially given that Panama is an unusual country to export raw cotton - or raw wool - from), he is now in prison for 5 years for tax fraud and document forging, llama wool is confiscated and auctioned off to legitimate sellers.
Not in the export/import business myself, just passed through customs enough to see just how truly stupid and vapid these officers are, combined with the truly insane amount of import/export volume each one is responsible for. In the US distinguishing types of fabrics and ensuring various legal substances that look vaguely similar actually are what they say they are is near the bottom of the priorities list. I don't have a lot of experience with EU customs (except see last paragraph).
I know people want to believe customs is some highly intelligent caring group of individuals who are sleeplessly guarding the country and the purse in a carefully planned and executed manner with a fine-toothed comb. Anyone with this belief has probably never crossed an international border. In practice it's more like the mental equivalent of a 4 year old set loose with a completely unreliable "drug dog" that alerts on a grandma who is ruthlessly interrogated while one guy with llama wool and 10 pablo escabars pass on the other side.
I can recall one occasion where I stepped directly from fucking IRAQ into EU and not a single customs officer even looked at me. Not one. Just a stamp without a single question. I could have been carrying literally anything and no one cared. I could have been coming from Iraq for any reason and no one gave a fuck what that reason might be.
It's not illegal to pay an US company with an US invoice in the EU. (HN likes to invent problems where there aren't, and companies do that all the time). It is not even limited to the US, most other countries can't/won't follow the format.
However there might be extra steps in adding those invoices to their accounting. For example, when import, the buyer will be charged the import tax (instead of having it added to the invoice)
The informal economy of Portugal has nothing to do with the above btw since that has nothing to do with American invoices (and you can bet German rules are not much simpler) - not saying the bureaucracy isn't, most of the time, stupid. The US also has it fair share of stupid crap that people have to deal as well but it is "transparent" to most Americans.
My team existed for the sole reason that the European business did not trust that the US payment services team understood European needs, and the "invoice is a legal document" thing was one of them. I spent so many meetings repeating to the US team that no, we could not switch to their invoicing as long as a new software release might retroactively change the template used to show already issued invoices (e.g. the registered office might change, or the VAT number).
We didn't have to deal with printed invoices thankfully, but we did ensure we produced each invoice once and stored it, so that the European finance team could sleep at night.
At the time Yahoo! had 8 different payment platforms. Some of those were due to weirdness around acquisitions and Japan (which was a joint venture), but apart from mine I believe two others also existed because of local weirdness that the local businesses decided it was safest not to let Yahoo! US mess with.
At the time I left, after 3 years of that, we'd managed to get agreements to migrate a few bits and pieces to the US after they'd shown the understood requirements for specific features, but it was like pulling teeth.
Do you have any early guides or api docs to take a look at. Our use case.
1. Base service fixed packages (based on users with volume discounts)
Extra costs 2. Extra storage space usage 3. Add on services (fixed or usage billing) 4. Invoicing third parties (reselling content)
If you want to change invoice you have to make a special "correction invoice" because changing "real invoice" is a criminal offense - fun things :)
They eventually saw the light, but it was a long slog.
Initially "digital invoice" meant that you had to get the invoice cryptographically signed by the same government agency also in charge of literally printing money (or an officially licensed company) and then ideally send it using the now mostly defunct monstrosity that is De-Mail because it made guarantees about end-to-end encrpytion and sender/recipient authentication. Luckily this is now largely irrelevant and most companies just send regular PDFs via e-mail and/or make them available for download.
Neither seller cares how you handle your payments to the taxman, nor taxman cares about customer payments.
The second part regarding credit/debit invoices is again a property of "append-only double-entry bookkeeping". You fix wrong credit/debit entry with reversed debit/credit entry.
While it is a bit inconvenient, though, I don't think it is a bad idea. I am pretty sure it helps a lot making the life harder for people evading taxes, and to some extent the availability of a standard machine-readable format makes it easy to do accounting. My wife sells videocourses online and I developed a thing that automatically issues invoices as soon as people buy a product (with some human supervision, mainly to avoid the machine doing havoc is some exceptional situation happens; normally it is just "click a button to send the invoice"). The invoices are then automatically available to the accountant for doing accountant things.
You can find the XSD here: https://www.fatturapa.gov.it/en/norme-e-regole/documentazion...
In my experience invoicing, and really all 'printable' documents is the land of "oh someone did something wrong put X on the document". And then again and again and again... And "this is two pages long that's too much" and on and on ...
It's bike shedding insanity with no ideal or end in sight. Every change is arbitrary with no real measure of success.
I swear the only time anyone LOOKs at these documents is to bitch about them.
Oh man and just as if I summoned it someone just sent me a ticket about something on an invoice.
From the perspective of someone who works with the people who process the invoices, putting "X on the document" and "two pages long" can be a huge deal when you have to deal with hundreds of invoices or when "X" can mean that an invoice must proceed down a different processing path. These aren't bike shedding concerns; they affect the actual processing of the invoice. In many cases, these concerns affect the validity of the invoice, meaning that the recipient's obligation to pay (or the associated deadline) is not triggered.
And doing something wrong on an invoice is never bikeshedding. An invoice is a legal contract, and in the E.U., an invoice is an unmodifiable document.
It sounds like you simply don't understand why they're doing it and you just assume that they're trivial because you have no knowledge of the domain.
Off the top of my head, I can thing of a dozen reasons for why they'd require something to be added, then removed, then added, etc., from one invoice to the next.
I don’t know why you would think that considering I work with them and you do not…
Then why not list them right here, right now?
- Changes prompted by compliance needs: business type (which governs the type and level of compliance required in most jurisdictions), changes to invoicing laws, changes to tax compliance laws
- Changes prompted by business needs: changes to the company's PO/invoicing/expensing process, changes to the company's financial, ERM, or payment processing software or service provider, one-off large expenses that aren't adequately handled by the existing data form; changes to financial modeling
- Changes prompted by marketing: new logo, design, or template
- Changes prompted by legal: new court rulings, etc., affecting legal's approval of the existing template(s)
- Changes prompted by employees: form is too complicated, form is not complicated enough, form doesn't capture my department or project's needs
The problem with an all inclusive form is that it is too complicated for most vendors and they end up filling it in wrong (requiring several rounds of review by the payor company), but too simple a form and you end up needing to make a lot of changes like the one the parent was talking about.
You could do multiple templates, but that just multiplies the work, and makes it more difficult for employees to figure out which template they should be using, and so they'll all end up using whatever random template they find first leading back to the One True Template and you're back to square one again.
And yes, all of the examples above are based on real life examples.
At least we are in a situation where the seller is almost only subject to the laws of the seller's country. There are some exceptions, but one can mostly deal with those (the EU has plenty, the good news is that if the seller is not also on the EU, most do not apply).
Also, I haven't seen a country accounting system that made accounting papers inherently expensive. Some make refunds very expensive, but not the papers. It's more a matter of badly designed software and processes.
We're working on making the 'CPQ' : Configure Price Quote journey simpler, and this includes the PO dead end. We're on a mission to make the pricing stack more open and customizable for companies (hence the open-source angle), and we're starting with the billing system. Join the journey, we'll share our repo soon, and hopefully we can address these painpoints faster with the community contributions too!
I guess that's good for employment numbers? There's really two ways to create jobs, innovate or regulate. The US and the EU have apparently chosen very different paths. With every country having it's own special snowflakes laws, there's a reason EU founders always try to come to the US first.
Seems like the real world is just choc-full of Liskov Substitution Principle violations :P
* Sales is payed (partially) off commissions, so they do their very best to sell whatever the customer wants, not what the engineers know how to bill, or that product management has prepared. It's really hard to push back against a contract with revenue 10x your yearly salary "just" because the new features to bill it don't fit together with existing features.
* It's very hard to keep track of which customer pays for what, exactly, and is entitled to which service. Easy for standard products, hard for bespoke products, nearly impossible for shared services (it took us >9 months to sunset a shared HTTP proxy that was in use by just 4 customers that didn't even pay for it...)
* Legacy contracts: imagine 25 years iterating on product design, and keeping the old contracts running (except when the customer leaves on their own). Some of the old contracts are hard or impossible to find, nobody wants to spend their time rummaging through them to find out what exactly a customer is entitled to. A year ago we still had a customer that still pays for 8€ per GB HTTP traffic (web hosting), because that was the going rate back in the days
* Usage-based billing: half of our customers want to be flexible and only pay for what they use, and the other half uses SAP and for them, invoice amounts changing from month to month are the pure horror. So clever sales people invent hybrid models ("pay for what you use, but fixed price until a certain threshold, we'll notify you when you get close to reaching the threshold"). Another source of complexity.
* Usage-based billing: it's basically a topic for the whole company, most departments provide services that needs metering, but metering is usually an afterthought to the architecture, hard to verify for somebody on the outside
* Usage-based billing: for big customers you cannot put all the details on the invoice (our mailing service provider is limited to 99 pages per invoice...), so you need separate reporting to drill into the details. Fire and brimstone will rain down if reporting and invoice disagree on some numbers...
This just scratches the surface. Customers that want to split their bills along cost centers that don't align with access control tenants, different electricity prices based on location, the list is endless.
Reporting for revenue is a hard game, indeed. Spent a lot of time trying to reconcile revenue streams in the same company we ended up building the whole billing system. It’s also not easy because metering is not something that is predictable nor easy to make it fit into a finance report.
I see the positionning of the CFO right next or below the CEO as a mechanism to let the CEO care about “big pitcure” money flow, while having someone in contact with reality guide decisions and veto stuff that won’t fly in cost/revenue terms.
> There was once a programmer who was attached to the court of the warlord of Wu. The warlord asked the programmer: ``Which is easier to design: an accounting package or an operating system?''
> ``An operating system,'' replied the programmer.
> The warlord uttered an exclamation of disbelief. ``Surely an accounting package is trivial next to the complexity of an operating system,'' he said.
> ``Not so,'' said the programmer, ``when designing an accounting package, the programmer operates as a mediator between people having different ideas: how it must operate, how its reports must appear, and how it must conform to the tax laws. By contrast, an operating system is not limited by outside appearances. When designing an operating system, the programmer seeks the simplest harmony between machine and ideas. This is why an operating system is easier to design.''
> The warlord of Wu nodded and smiled. ``That is all good and well, but which is easier to debug?''
> The programmer made no reply.
https://en.wikipedia.org/wiki/The_Tao_of_Programming
I own a copy; it’s great.
Which is older, the oldest continuously supported operating system, or the oldest continuously supported (computer) accounting system?
I believe it predate operating systems by a few years. (Wikipedia claims the first OS was the GM-NAA I/O, produced in 1956.)
DO NOT bill against your business logic entities. They change, and doing a COUNT at the end of the month won't catch all things which changed or which cost you money during the month. Instead, figure out what you bill for and and when you do that thing, record a row in a DB or into a steam of that "billable event".
Reconciling billable events is much easier to do, and it's tolerant of all the weird edge cases (such as a support person hard deleting data when they should soft delete, which would otherwise throw off your end of month counts).
There's a reason AWS (in general) can produce a log of everything you did which cost you money. It's painful, but it's less painful than the alternative.
Or you can log them into tables that get processed into other invoice tables.
This also lets you keep grandfathered customers around as long as you want.
You have an obvious start/stop checkpoint in the log, processing can be done in batch after hours, no information ever gets destroyed, etc.
Mutable database rows that are used for accounting purposes should sound dangerous to anyone trusted with these systems.
On a similar note, make sure you think about bitemporality as well. In other words, “effective_at” vs. “”created_at”. You might learn that due to a bug, you didn’t record a billable event, and you need to insert it into the history (say, to put it in the correct billing period). But setting the “created_at” in the past brings a bunch of issues; it’s confusing and you’ll soon hit issues where lying about created-time trips you up. (Migrations, reasoning about what version of the code applied what events, etc).
Fowler has good further reading here: https://martinfowler.com/articles/bitemporal-history.html.
This way, if the price changes, the item description changes, the images on the item change, etc, you can still have a record of what was actually sold and how much to refund a return etc.
Usage based billing sounds like a nightmare though.
Taxes in the USA are a nightmare too. I have a friend that lives on a street, and for one side the tax rate is 7.5%, and the other side its 9.25%. Literally no company we tested got this right. Even Amazon.
I truly believe in usage-based billing, so I do think pricing are getting more and more complex over time.
Taxes are a nightmare for everyone also (in Europe too...)
My advice for anybody thinking of building their own subscription management system is to stop and go look at something like Zuora. You do NOT want to get stuck maintaining some home grown subscription management system--it will never be better than Zuora and you'll always be playing catchup, leaving your company unable to quickly react to the market. All the time spent adding functionality you can buy from a third party is money that you could have invested in your actual product. And worse, the fact you couldn't quickly react to the market because some key feature was missing in your homebrew thing is literally leaving money on the table.
That's really great that Ford uses Zuora. But for someone trying to work with an inhouse system that doesn't map to what a system like Zuora can handle... you now have technical work and a lot of political and non-technical work, and a lot of internal coordination to get all relevant people on board to commit to a change.
* We don't remit taxes for customers because that would be impossible to build in-house, but everything else was achievable.
* We have tons of logging, and generate transient Quotes on our end, then check that the expected charge matches what the payment provider actually charged.
* Each Quote is reproducible, so if any bugs come up we can give the "create quote" code the same input and see what went wrong.
* What helped when building was hourly tasks running in the background to make sure customers who should be billed were being charged, and make sure we didn't double charge any customers, and then alerts so we could find and fix those bugs.
I would do it again to support both PayPal and Stripe, but it's definitely a source of complexity for the code base.
One interesting open source project was https://killbill.io/
PS: Since I built a billing and invoicing startup .. check it out: zenbill.com
This means that business events or user operations generate state transitions, which eventually are implemented as database transactions. The event log can be stored and inspected.
The end user terms are encoded in the state-transitions. Contract obligations are encoded in the state of the database.
As for calendar issues - operations need to be performed over "real calendars". It might be practical to "materialize" the business calendar into discrete units for this purpose.
In the framework of state-transitions, actions such as "price this with that" are implicit in the state matrix, and shouldn't be called from APIs. Rather, API calls should be used to produce reports which are print-outs of the "current state". Webhooks in this framework are attempted state-transitions. User interactions generate events, and requests for reports (for example invoices), but as little as possible in the form "do this with that".
My opinion currently is that there is little value in sharing code (libraries/frameworks) for these sorts of solutions because the code is trivial, application/language/domain-dependent, and coupled to everything else in the system.
But design patterns, templates, and concise examples are hugely valuable because they help illustrate the complexity and solutions up front.
Also note:
> Our open-core version is forever free. We will introduce paying add-ons in the future, with a consumption-based approach.
We definitely want to address usage-based pricings, but also all the pricings that are hybrid between usage and subscriptions (ie the vast majority of software pricings). Feel free to reach out if you have more question: anhtho@getlago.com
Account receivables is also another nightmare. We would get checks that randomly show up at corporate for no reason from insurance carriers but then when the carrier realized their error instead of handling it as a separate process they would deduct the check amount from whatever invoice ( or even across a range of invoices! ) from us they felt like. We literally had a bucket called "magic money" these random deposits went into that we would use to fill in the A/R gaps from insurance carrier insanity. There was no connection between magic money and whatever invoice they decided to short change us on so it was just a hope-for-the-best process.
We're building Lago so that the right questions / decisions frameworks are asked during the implementation, so it's like a forcing power embedded in the product.
Our experience is, unless the Product/Biz team has been exposed to billing, they will never specify in a way that is precise enough, so the engineers who implement billing will have to think/assess/decide themselves.
But equally important is to cut back engineering's enthusiasm for abstraction/unification": In my experience it is better to have 2 distinct, tested implementations against one pretty generic data model than trying to coerce both into one. Intellectually one can, but "oh so great billing concept" #3 or #4 will break ones neck.
I don't really think anything about billing is hard, it is just incredibly dull if done the pragmatic way.
I tell folks working in this area to accept the illogical nature of it all and to be ready for all sorts of arbitrary last minute changes. As the article points out understanding time plays an important role here too - when do different tax treatments take effect is an important question to answer.
That said, there is a logic to most of it, but not in a way that allows you to come up with general heuristics or universal abstractions.
That's why attempts at "no-code" systems where biz/accounting people can design their own system are a perennial fad, but as these systems are developed they always just end up evolving into weird domain specific programming languages.
It's also why any sufficiently large company ends up implementing their own in-house accounting system and/or customizing some byzantine monstrosity like SAP which is really just another way of implementing your own.
Normally sales people are deal-driven, and if promising the customer "14% off on the first three transactions on every rainy Tuesday, unless Monday was also rainy" gets the deal closed, then that's going in the contract.
No better way to ensure the contracts are reasonable, than making sure the person negotiating the deal feels the pain of billing it
Government tariffs covered billing rules for each leg of a call including rates based on caller's local time of day. Converting bridge time to local time via area/country codes(only data available) and time-zones is quite a mess. Add DST calculations to that per locale to reconcile negative duration legs or legs that should be 1.5 hours but show up as .5 hours just makes it cloudier.
If your system repeatedly gets it wrong, you could be in trouble for billing fraud or tariff violation. No joy in that job.
I worked on a piece of software that bridged billing data from IBM mainframes out to the web. The hoops that had to be jumped through to get data out of the mainframes and the number of hacks and kludges involved was legendary. Likely everyone here has paid many bills in their life on websites that used that software. At one point I knew the vast majority of my personal bills had gone through it. Credit Cards, Utilities, Gas cards, etc.. Stuff everyone had to pay with paper before the web.
Decades ago - the computer involved was an honest to goodness System 360 (which is to say, not a 370, Z, or any of the the code compatible 360 successors) - I spent an evening trying to figure out the service charge on my bank account. I couldn't get at all close to the figure on the statement by applying the bank's schedule of charges. I took it into the bank, and got bumped by the teller to a department manager, and thence to a VP (small town; small bank), whom I told, "I will pay the service charge if you can explain to me what I'm being charged for." 2 hours later he gave up, having not gotten any where near an explanation, grinned, and said, "I can fix this." He set some flag on my account, and I never again paid service charges at that bank. Yep - that flag remained set on my account for nearly 20 years, through multiple software and hardware upgrades, until the bank was acquired by a big, expanding regional bank, and I left for a different, entirely local bank.
PEOPLE
On a more serious note though billing can be hugely complicated and I am really glad I do not have to deal with it
Multiple currencies, trial periods, when in the process do you ask for the creditcard, country specific taxes,... And the worst: invoices.
While paddle.com takes care of many of those things - I was shocked after realizing how much work this is going to be.
Please think carefully about those things before you quit your job to work on your "Micro SaaS" or "unicorn startup".
For selling a SaaS worldwide, Stripe quickly becomes just as much of a nightmare. Taxes, billing, chargebacks, subscriptions, web hooks, different payment methods, VAT validation, card expiries, legislation, changing laws, SCA flows, and my favourite: SCA iframes that become non-responsive on browsers in a certain language. It is a never ending source of edge cases.
Don't ask me how I know... My biggest mistake with my business was that I vastly underestimated how much work it is to accept payments. In my case, it proved to be more work than building the product itself.
I'd like to offer a free trial. While they have support for this, a user needs to enter his credit card right at sign up for trials to work.
So in the end I have to build a hybrid solution where my system gets tied to theirs. Still I don't have full control over all e-mails that are sent to the user. I'm not 100% happy, but better than not charging money.
The complexity in billing, invoicing, payments etc are _not_ the stuff that computers or systems do. The complexity is and always will be in the stuff humans do.
Just because a tractor is used for farming doesn't mean a farmer should build her own tractor.
Now, back to the article's topic, the real question is: is there an off-the-shelf billing product available out there?
There are lots and lots of OTS billing systems. They might work the way you want or need. Billing is the classic "I wanted wheel, except I wanted a round one" kind of problem. Right now, I'm looking for a system that will let sales invoice whatever they put on paper, and be good a structured, recurring billing that is easy to hook up to my SaaS platform. Oh, and Bob in sales wants to be able to edit the invoices after they are issued...
But yeah, there were all sorts of nasty complexities -- this was a usage based system, but there were (optional) tiers (pay less for higher tiers), as well as certain time segments that were discounted. And then there were cases where multiple accounts could share a set of data and get a discounted rate and/or only one of the accounts would be billed. And on and on....
One lesson I drew from this experience was that, when discussing things with finance to try to figure out what the requirements were, it really helped to have pictures. I would create these elaborate usage/time graphs, and color in portions to reflect different billing/pricing scenarios. These made it much easier to discuss the current/future behavior of the system. Because saying "really, this is simply the integral of a usage function over time" didn't fly.
90% of your use cases are covered, and with excellent UX and docs to boot. But that sweet 10%!
---
For a concrete example, Stripe has a very nice "subscription" abstraction that makes it easier to operate a SaaS business. They also offer a nice "tax" product, that doesn't do filing for you, but does offer tax computation if you code your products appropriately. Sounds great!
But if you want to sell 2 subscriptions to the same customer, and those 2 subscriptions are fulfilled in 2 distinct taxable regimes (say, you're shipping widgets to NYC once per month and to a vacation home in NH once per quarter), you're screwed because Stripe only allows 1 shipping address for customer for tax purposes, and each subscription references this address when computing tax.
We went through support, which filtered through to engineering, and the eventual answer was "no plan to change this anytime soon", which is difficult to interpret any other way than "good luck". :)
There is no clean or even acceptably-dirty solution to this problem, tragically. So as a business we can't allow customers to transact with us this way. Oh well.
There's another, similar, story around creating draft invoices to give people quotes before checking out. The dashboard API has a function to modify the line items of a draft invoice, but it's not a public-facing API and there's no short-term intention to make it public-facing, so one has to build an ugly workaround.
This effectively means one needs to maintain internal models for subscriptions, products, taxes, invoices, invoice items, products, shipping addresses, on and on. And Stripe becomes a somewhat-dumb processor for these models (although the state machine for subscriptions is admittedly very valuable).
I'd love to "offshore" everything to Stripe, status as the source of truth for data included, but it's not feasible in a surprising frequency of scenarios.
Caveat that every payment provider is like this, Stripe is much better and actually improves over time.
There is a similar "not a public-facing API" limitation with configuring multiple invoice email addresses for Stripe customers (works in the dashboard, but blocked in the public API, where you can only set a single email per customer).
I'm actually quite surprised Stripe does this, since their initial reason for being was developer-friendly payments. But oh well, maybe I should take a look at an alternative (like Lago) sometime.
But it does not make sense for PLG companies, who start with self-serve and layer sales afterwards: the investment is not worth it, and it creates 2 different systems: self-serve and sales-driven billings, not great for the user experience (and for the internal processes!)
One of the most important things I’ve learned about working with third-party systems is that there has to be some amount of surrender to their way of doing things. Essentially, the price of using Stripe is not being able to use invoices as a quote, and that has to be articulated to the business. If that’s not acceptable, then the business should be considering the requirements in aggregate when evaluating a solution, rather than deciding on Stripe and then gradually re-implementing parts of Stripe internally because they aren’t right for the business.
(Obviously that’s easier said than done, and for engineers it’s a fun challenge to make something impossible possible, but it’s almost always a mistake and as engineers it’s our responsibility to articulate the burden of owning workarounds.)
Effectively, relationship is no longer user -> stripe customer, but user -> shipping_address -> stripe customer.
We decided the cost of implementing and maintaining this solution wasn't worth the benefit to the business. A credit to your later points.
We came up with a solution that allows us to ship one-off things to arbitrary addresses with correct tax calculation, and if a customer really wanted to setup a subscription to a distinct address, we could implement it in the future, hackily, with this primitive.
We have this same limitation on another payment platform. We just change the concept on our end to "payers" and allow an account to have multiple payers. When the user adds a payment method, they can either select that existing payer or, if they need to change the name or address, they create a new one.
The provider we use had an invoicing product, but it has several misfeatures that just make the product unworkable. We absolutely have to be able to enforce first-in-first-out invoice payments. Users will pay the May invoice before the April invoice and pretend like they are all paid up. I'm not sure why they didn't consider this with a subscription-invoice product... users paying the invoices out of order was a big problem because users saw it as an exploitable vulnerability they could use to get a free month here and there.
When you look at usage-based billing, even if a billed feature is linked to consumption, it needs to be related to a subscription given the fact that this feature can be priced differently for several plans. Every company has a singular pricing model in mind and it's hard to have an all-in-one model fitting all the needs. Even when we built internally the whole system for a fintech back in 2016, we had some edgecases creating conflict inside the company.
But I'd push back on the 1 subscription per customer idea. Consider any physical fulfillment business, where you have an optional annual membership (amazon prime, costco, whatever) and N subscriptions of varying frequencies for physical products.
It doesn't make sense to roll these into a single subscription on the backend (unless the periodicity happens to line up perfectly, and even then there's a heated debate, since you still probably want them to be separate charges).
You can go down the route of creating N "virtual" customers for each "real" customer, but it doesn't really solve the problems, because these N customers have a complex relationship with each other that you need to encode and manage.
Hence, the name of this post :). It's definitely not easy!
Take a simple example: selling clothing in a retail store. Sounds simple right? You have an inventory of items and someone pays for it. Should be a simple transaction, right? Well, now you're dealing with:
- How do you determine the price? There might be a retail price but there are sales, discounts, overrides for various reasons (eg defective goods), etc;
- You have to handle different payment options eg cash, credit card, debit card, gift cards, store credit. Payment methods can be mixed;
- You have to handle returns. These generally have to go back to the original form of payment. Some goods might not be eligible for a return. What if the payment is split between store credit and a credit card? Where does the balance go?
- Identity of the person making the sale or handling the return;
- Authorization procedures for returns and price overrides.
Everything is complicated.
As an aside, this is one reason why I'm so bearish on Web3 and smart contracts in general. The edge cases are so complex and possibly unknowable that codifying these, particularly on an immutable blockchain, to remove the need for human intervention seems doomed to failure.
Take Web3 identity. If you lose your password there are methods for recovering your account. The first is a reset password option. That may be insufficient and there'll sometimes be a human avenue to recover your account. Now consider a Web3 identity. I've seen various incarnations of these such as authorizing other identities who with consensus can recover your account. Well, those identities can be compromises so you've just added an attack vector. Alterantively that recovery mechanism may become insufficient as people lose identities themselves.
If these problems weren't complex they'd be solved problems.
Totally agree with you on web3 identity and its edge cases, it's just billing ^10000 in terms of complexity.
I think for 'billing', usually people (especially non-tech people), don't understand the different blocks of the pricing stack: authorization, pricing grid, billing itself, invoicing (they often only see end-result: a receipt or invoice), payment gateways, dunnings, accounting etc). This post barely scratches the surface of it, but we intend to write and deep-dive more on each of these building blocks.
I do think the potential for errors is currently real (people mess up simple wallet transfers all of the time) but it's reasonable to assume that big things and making them easy for normal people to use is going to take time.
It's easy to forget how ridiculously early into crypto humans still are. We're only about 14 years into the invention of crypto. We're only about 7 years since the release of Ethereum. The general public barely understands what crypto is other than having some conception of it as some kind of Internet money. Most people cannot discuss what a blockchain is, what smart contracts are, or why oracles are important. And forget the general public for a minute: even smart technical people on HN can barely give a good description of how all of these pieces work and fit together.
That's an axiomatic and therefor meaningless statement. Knowing something is hard to solve doesn't mean new tech won't bring improvements regarding the problem.
It doesn't follow that having an immutable ledger means it has to be used for every single application. That would be like claiming trains are useless because they can't climb stairs. They don't have to do that, we keep walking the stairs just like we've always done.
Immutability is great in situations where you want to record that something has taken place and don't want disputes about this. Some of the things you mentioned can be automated to some degree. Take returns as an example: You sell the item with a unique token that makes it identifiable and everything else can be automated, including chargebacks. We already have this to some extent with bar codes and RFID tags, but it can be taken to the next level with a non-fungible and non-falsifiable token that's connected to a smart contract which automatically returns money, does the accounting, updates the inventory, reports the tax data and so forth if authorized by the right person. As the owner you can easier monitor and analyze the flow of money, better detect fraud and come up with a myriad of improvements. Let's assume we have autonomous delivery trucks soon, then the entire supply process could be automated, including the billing as soon as the product is accepted. With something like the Helium Network shipments can be tracked, so there can be no dispute over where they are located.
For company owners who really doesn't want to spend much time at the office you could go a step further and even set conditions for disputes like contracts triggering a certain action after a suppliers products have been rejected too many times because of quality issues, or vice versa is a customer keeps ordering and returning products from a factory. The contract could switch to automatically ordering from a competitor.
I'm a solo founder not interested in dealing with either the complexities of these billing issues nor with the taxes issue. But my product would easily be of interest globally and the API's I've built have more interenational customers than global customers so I suspect the same may be true for the product I'm building out of those apis...
My guess: Because basically nobody wants to develop one of these as a hobby FOSS project. The whole idea is collecting people's money, which when you write your own code for public benefit, you basically don't do :-)
> We quickly decided to add plans, and 'pay-as-you-go' features
So, the author talks about things getting more complex, beyond their expectations. But - why did they not expect this? I mean, they know that companies pricing changes over time; that payment models change, including rates, cadence, use of internal credit etc. We all know this without ever having worked on a billing system.
> Qonto is a 'neobank' ... Fintech unicorn
So they were expecting billing to be easy for a bank-like endeavor? Hmm.
> #1 - Dates
The date problems per se are quite mundane and any non-trivial software system which involves timed activity deals with them. The billion period problems are not really date problems and are a recap of what OP had already written about. They are also problems most of us are well aware of from our personal experience as people in modern society ("Does my bus pass expire a day after I bought it, i.e. 24 hours later, or does it expire at the turn of the day = at midnight?")
etc.
Stored procedures (ca. Oracle v6) suddenly sound a walk in the park.
Perhaps when you first enter the domain it seems like a lot because you are unfamiliar with it, but such is it with all things.
Also, these guys have an economical incentive of convincing you it's hard. So keep that in mind.
Public roadmap: https://github.com/getlago/lago/projects/2
Request features here: https://community.getlago.com/c/feature-requests/11
We also have a fully hosted version if anyone is interested. Looking forward to receiving your feedback!
I've been building parts of a billing system myself over the years and I'm considering removing the remaining external parts, because I'm not happy with their limitations (Braintree's subscriptions are really limited, and absolutely nobody does correct invoicing for my situation (EU/Poland)).
Also, in a B2B SaaS you will end up with customers on totally different payment methods. Not every business has or wants to use a credit card. So you get annual billing with proforma invoices and the complexities involved with switching from invoiced billing to credit card billing and vice versa.
To note, this option only absolve of _some_ of the complexity and burden. You still have to understand:
- what problems the billing platforms is solving for you, how they're solving it (does it match the process you need ? -> you also need to know how it should work) and what to provide them to make it work.
- what problems they are _not_ solving, and you need to deal with on your end
A good service provider will help you at least on the first part, and educate on the parts you need to be aware of, but there's always dark corners that easily get overlooked if the business owner didn't do their homework.
But in the same breadth still ... F** Xfinity
1). Don't try to build it yourself, no matter how simple your case. Your needs will ALWAYS evolve to be more complex AND keeping up with the thousands (or more) of individual regulatory regimes in the world is a massive task.
2). At any given point in time, you need to know the state of everything required to rate, bill, and invoice at any other point in time. (i.e. bitemporal). As a consequence, critical data effectively becomes immutable, auditing is possible, and allows pre/post views of things after corrections.
3). Once you start to get into tiered/volume discount, combo discounts, providing multiple services, etc. complexity rapidly increases, so observability into how exactly things are calculated is of paramount importance. This one is probably pretty obvious, but missed a lot. How many times have you received a bill with little to now information about how things were calculated? It's possibly a business decision, but just as likely the process is a black box and not readily observable.Don't build a billing system. Or at least not a precise one.
Have a price list as normal, but just guestimate the user's bills based on any easily accessible information. Eg. For a hosting company I might just count how many VM's they have active when the billing script runs . If the user isn't happy with the total, give them a button to correct the total and pay that.
Spot check what the user's pay, and warn then ban any users which are clearly deliberately paying substantially less than list price for your services.
Someone's going to reverse engineer your heuristic and min-max it. At that point, you'll need to defend your heuristic against edge cases. And then you may as well just make it official.
Also, B2B customers are allergic to "pay what you feel"-type schemes.
Billing, like taxes, can be broken down into relatively simple smaller problems. You solve all of those then do a bit of maintenance. In return you get something close enough to all-you-can eat on the cheap! Automation is wonderful!
"Lago is backed by Y Combinator"
"The Open Source Stripe Billing Alternative"
Why would YC invest in both Stripe and a competitor of Stripe?
I had felt this pain vaguely before but actually seeing it everyday at my current job makes me realize there are engineers suffering on both sides of software billing.
Usage-based billing is incomplete without its suite of reporting, consolidation and visualization tools. No wonders there is a cottage industry of cloud computing consultants and SaaS solutions focused on helping you to make sense of your AWS/GCP/Azure invoices.
Eg. An SMS on an off peak time is cheaper. If the user makes 100 or more calls a month a special rate. How do you charge data per plan? If the user roams to another country there's an entirely new set of complex billing rules.
You might then realize that accountants have been using event sourcing to solve complex problems for hundreds of years.
Very senior developers will nearly always default to juggling updates to database records for everything. I cringe because it is often the source of serious gaps in domain language between developers and business operations people, leading to unreliable systems that are very difficult to reason about.
Immutable ledgers are your friend.
That shit is hard. There are so many ways to fuck it up because you didn’t know what you were doing. The list of unknown unknowns for a billing system are huge unless you are an actual expert in billing, which you aren’t and neither is anybody in your team.
This is so great. I personally haven't had to deal with this problem, but I've worked at a number of organizations (and heard about many others!) where this sort of business logic had to be implemented. It's just reinventing the wheel. I shudder to think how many companies have implemented a system for managing recurring subscriptions.
We have things like Ruby-on-Rails, Django, Laravel, and many others to take the bite out of building web applications. They keep us from having to reinvent the wheel.
We need similar open source frameworks for common business use-cases - billing, subscriptions, order/purchase management, and so on.
Sometimes it's weird to remember we're in the stone age of technology - all of this is a few decades old at most, and even early predecessors don't go back more than 70 years. Human history goes back tens of thousands of years. There's so much yet to come.
You mean like an ERP? There's 2 open source ones I can think of:
- Odoo, which is the bigger one, mostly Open-source (CRM / sales / subscriptions / invoices / webshop / inventory / purchase) but some enterprise-only modules (Rental, Field Services), I work with this software daily as an Odoo Developer (Customizing Odoo for customers' needs).
- Erp-Next, which is completely open source as far as I can tell, through my limited testing it seems to be less advanced than Odoo currently.
EDIT: you can even check runbot.odoo.com for some test-environments which are automatically built, where you can test/experiment, login is always admin:admin
Shouldnt that be 50kw used in total?
Now we know who's writing all the bugs :)
Source: https://github.com/kdeldycke/awesome-billing#finance
No. No, spelled F-U-C-K-N-O.
Should you compare and contrast off the shelf solutions, test one or two, and slowly expand its use to cover more of your business? By all means, yes.
Should you develop an in-house custom variant after that? Still no. For billing or anything else.
Even a pure tech company like Google should not vertically integrate everything. You quickly end up wasting time making horrid systems instead of generating your main business revenue.
"Oh, but our needs don't align with the way the industry standard third party systems work." That's a symptom of the disease. You won't cure it by enabling the virus to proliferate even further.
People who don't execute I.T. low level changes mistake computers for magic. Yes, computers can do a lot. But even the tiniest feature takes an incredible amount of focus, cost, development time, maintenance time, headaches, and gnashing of teeth to accomplish. The magic is that it looks easy, from the user's point of view.
Let's put it bluntly: We can draw a direct line from third party systems with multiple businesses as customers, with cost savings. It's far more likely to be a complete waste of resources to in-house a new software system.
Maybe you don't have a name for the kind of system you want or need. That's okay. Keep assuming it's probably already out there, and continually search for it. Googling is cheaper than building.
"But we can build it better than them because..." Please, that's the fallacy of exceptionalism. Remember, every company wants to believe they're truly pioneering, talented, and logistically capable of doing all the things. Don't spread yourself so thin. Keep the lights on. Grow your customer base. Everything else is a pipe dream.
"But the third party systems suck."
Yes, they very much suck. And yet the cheapest path to success is pushing them to un-suck their product. Not starting from scratch to create a second, even more immature product.
Frankly, computer systems are already quite flexible. You can often bend an existing one to fit your needs. This happens with Jira, for example. They customize it until it can't function at all anymore. But this sad extension situation is still cheaper and more effective for conducting business than invention.
If you truly believe a new kind of system needs to be built, go and start a new software company. Or pay a software vendor to do it. (Software vendors, just take their money, smile, and nod.) But for Warren Buffet's sake, don't spend a dime on a new line of code if at all possible.
I come here for a refuge from manipulation- a place where people can share and debate things on their own merits. Content marketing is a not-so-sly way to insert one’s business interests into these types of conversations I enjoy for the purposes of driving metrics, sales, and series A’s.
I hear that taxation and billing and collection are complex. Try to implement WA State destination based sales tax for digital only goods. Where is the relevant tax district? It doesn’t align with zip codes or congressional districts. Municipalities move and reshape the landscape like rivers do. Is a billing address representative of the location of the purchasing party for tax purposes? We took this to a tax judge in 2014, and their final response was ‘do your best’.
I would love to read a blog post with such a war story- about parsing the tax rate soap API into a relational database that can not slow down a checkout page. About how time zones affect rate calculations. Where rate file updates accidentally modified past transactions and how to model data to prevent that. I don’t want to hear something is hard so you should buy my services. Not here. Not on a call to my cell phone. Not ever.
Business leaders need to determine which complexity they want to their developers to tackle- what is the most valuable for profit growth and risk mitigation. If the audience is business leaders, appeal to them directly, and take it somewhere else. If it’s to appeal to software developers, it shouldn’t be attached to ulterior motives. We see through it and we resent it. To the extent that we have influence over purchasing decisions, I’m likely to punish companies who regularly engage in this behavior.
Content marketing is different than creating useful content as a marketing strategy. I’ll raise Digital Ocean as an example. Their sysadmin content is top notch, and I’m happy to give them business for it out of gratitude. That is my decision. Nobody is trying to convince me of how to invest my or my company’s resources.
We get it. Auth is hard. SMS is hard. Billing is hard. None of it is core to most companies core business. If you really want to make your point, show me some code to prove it. Or spend your content budget on something that helps people. Just don’t think you’re going to win me over with a sales piece on HN. You’re not. I’ve already forgotten your name, and if it does come up I’ll have nothing but a negative memory to recall.
Sorry not sorry.
In the 00's, I was part of an effort to get the State of Oregon to consider open source software. A 40 million billing system debacle was part of what drove this effort. The details are not important, save to say the ASA paid big to stall the legislation and that worked. It was stalled, and word got out that future bills would be toxic. So, that's over, but a great idea remains!
I am putting it here, because someone, somewhere should do it. Will save us all a lot of money and do a great public good. And it's likely to pay a fellowship of some kind very well over the next couple decades. I'm on a different project and need to finish it, and I'm likely to be paid well on that, so...
Instead of paying the likes of Oracle, or some consulting firm to write and support the billing system, the State should do it. So far, I've seen several bad contracts in my State where the State has a right to use the software, does not own the code, must pay for support, and on it goes. This is quite expensive, and frankly, it didn't work well.
Instead do this:
The State sets up an organization. This org is public, and those who work there are public servants, or some similar arrangement. The key thing is the billing system gets written on open code, uses open data, and the resulting software is owned by the public, through the State essentially. This code gets published on Github under an appropriate open license.
Fund this org, paying respectable developer salaries, and get the system written, tested, deployed, whatever is required. This funding comes from public tax dollars, and I would suggest the lottery as a primary source.
Once the initial system is complete, and let's say it's for water. Once it's complete, the system can be published and the whole thing becomes an export for the State that does it.
Any municipality anywhere can do one of a few basic things:
They can build and deploy the system and handle billing themselves.
They can hire the Org to help them do this, contact for support, whatever.
They can hire private entities to deploy the system.
Changes can come from anywhere, but the organization is best suited to implement them. This can be on a contract basis, funded reasonably in any number of ways.
The organization continues on to develop other public works tools, each funded by public dollars, each displacing some expensive thing or other, and each saving the people quite a bit over the course of say a decade.
Wash, rinse, repeat.
The end product is a nice pile of public works software needed by tons of municipalities most all of whom are paying hard for it now and they pay hard for basically everything, and continue to do that. The public could make these investments once and benefit for a very long time.
Perhaps more than one State wants to be a part of something like this, or whatever State gets going on it first, might want a few organizations, depending on what is to be done and the scope / scale required. Either way, the Organizations provide jobs and consulting / support and basically own the public works they develop and are funded to maintain those software works.
An advantage over contracts and proprietary license is cost, particularly when the likes of Oracle are involved (and no judgement here and no calling out as I'm only saying Oracle as a well known example among many out there and I just feel the public works can be done without Oracle and the cost associated with similar entities). Having the data be open is another advantage in that it can be used in all sorts of ways, and the public records can be made available to the public in simple, effective ways.
Mix in smart API's and the whole thing becomes very high value and an attractive base target for all sorts of public software development having to do with the basic machinery of society.
None of this is particularly sexy, but it does pack a big punch in terms of modeling how a more modern, digital society could employ open code, open data to get the work we need done, done in a cost effective, efficient way.
And that's it basically.
Over time, this first, or core organization will accumulate considerable domain knowledge. And it would become the basis for a lot of other work. Ideally, others started up in various places would all garner the knowledge needed by domain:
Utilities, water, power, internet (where it's actually municipal, and we know that's not popular with the big telecoms...), garbage, etc.
City functions, parking, various permits, anything that has a process that could be automated and or just performed on open code in simple, direct ways that are lean, consistent.
Vehicles: Registration, and all that tends to be tied into these systems. In my state, we've got vehicle registration, drivers licenses / ID Cards that link to facial biometrics (I don't like this, but that is another discussion), and it's also voter information central with a signature, party affiliation, address, and everything needed to manage the voting rolls, and execute Vote By Mail nicely, and in a trustworthy fashion. Oregon has done well on that front. Citizenship records are in this pool of software too.
Health Care: Right now, again in my State the Medicaid system insures people who are diabled, are poor, perhaps wards of the State. This is currently on Oracle, and has been such a mess it's ended up in court. I don't even know the state of it all right now, but it's a lot of money and a lot of grief.
Now, this won't be popular with a lot of us here, myself included, as doing stuff like this will displace some revenue streams companies and people depend on. However, a longer view looks considerably more attractive. The jobs created are needed. Tie this stuff into education, and it's a perfect "farm" training ground for up and coming developers to work on important projects and get good experiences.
A lean and mean digital civics is attractive and necessary for a lot of reasons. The case for doing it on open code and open data is super compelling and compliant with the basic ideas behind public works anyway.
As citizens, we get to experience our taxes actually getting more work done and at a lower cost. And yes, spending will move toward other areas where it's possible to extract revenue, and that's an ongoing problem. Real good can be done, and that's what I'm writing about.
Companies wanting to or needing to interoperate with the State government can do so employing everything from really old school, courier, papers, and the like, through to very new school, all digital, everything operating in ways we know can work well given the incentives are where they need to be and the organization is funded as it needs to be.
The second and third order effects could be very significant! Standards fall out of this kind of thing, and devices, protocols, and all manner of currently diverse and expensive to maintain systems could be made leaner, meaner, have much longer service lives and the knowledge needed will be out in the open and available to the people who need it.
Obviously security needs to be a consideration. And that's no different than what we have going on right now. The big difference is an open society can be built on open code and data and having it be funded in ways that maximize the value to the public while also keeping costs where they need to be, appropriate makes a ton of longer term sense.
In a scenario where this is all closed, the idea of competition is often cited as a reason to do it private, but the realities are corruption and other harsh realities tend toward scenarios where lock in type solutions favor the public getting the lowest value possible for the highest dollar amount possible. We see this over and over and over. And that's why the ASA (American Software Association) opposed the effort in Oregon and another one in Texas with such zeal it was kind of amazing really.
Should we do it in some fashion as I've hinted at here, the opposite becomes true. Initial value for the dollar might not be a whole lot different from whatever contract + proprietary software delivers. Over time, the trend will be toward getting very high value for the dollar.
Regarding Standards... Think center of gravity. Systems like these can have very attractive start costs. And cost of change will vary, and likely be made higher by current players wanting to leverage lock in to preserve revenue streams (and who can blame them?). Fair enough, but as more gets done, that center of gravity will prove compelling, and we all get the benefits working in a similar, more compatible way provide.
When we all require a resource or process, and let's just take power or water for a moment... Running these things at profit puts incentives in the wrong places. Maximizing profit is not the same as maximizing the public use value. And where that is wrong, everyone takes a small hit, and those tend to add right up.
Maximizing public use value can end rent seeking type arrangements that cause more grief than expected. Maintenance is one, tech debt is another of many that come to mind. Where these are out of public view, they tend to get ignored and risks accumulate, until there is an event, and suddenly, those risks play out, and we the public are faced with a large bill, and that's an old story, no need to say more.
I'll end with the belief this seems to be an excellent way to employ open code and data. And one of the big problems we find out there with open code and data is failure to work on "dull" or "uninteresting" problems. How many times do we have to find out that project X is being used by everyone, and nobody really owns making sure it's going to make sense to continue to use it?
Now I will just stop there. That's it. Some municipality somewhere would LOVE to get a State grant to get this started, and some State somewhere really needs this kind of thing enough to bite on the idea, and there are a ton of messes to clean up.
So maybe those should just start getting cleaned up!
(and because municipalities don't expose their tax rules as types)
Then again, if you make it too customizable, you end up with current single sign on solutions.
To do it "correct" you've got to have people who truly understand accounting concepts like double entry bookkeeping, financial reporting, cash vs. accrual, financial regulations, auditors, etc. Unless your company's product is billing, good luck hiring an engineer who knows this shit.
But trust me from experience. Don't build your own billing system. There are several good ones out there that will grow with your organization. Buy one.
Buy a billing system, it'll save engineering hours having to deal with it and if you choose right they'll scale with your company.
For example if your homebrew thing doesn't handle price changes, that might take you quite some time to build in functionality. That time spent is time where you couldn't quickly react to market changes, which is money left on the table. Had you bought something that supports a simple use case like "change the price", you'd hit a couple buttons and boom... your product now sells for a different price.
A companies billing system is one of the most important systems in the company. It is literally how money gets into your pocket. If you build it yourself, you will sign up for pissing away a ton of time writing code that literally will let you collect more money from your customers. Had you bought something, you'd just fucking go do the change right away and collect more money from your customers almost instantly.
There was a really nice and simple to use billing software that was built on stripe. I used it for a handful of products after I decided to never roll my own again. It is gone now. GoDaddy bought them and shut them down.
I think Stripe checkout is a valid solution now if you don't mind sending your customers away from your site, but I haven't played with it since they rolled it out.
I completely agree with you, billing is not the most complex process of all, medical services may be of higher complexity.
Our post was meant to highlight the discrepancy between the perceived low complexity (lots of teams who have never done it think it's simple) and the reality, with our own experience building a fintech.
I think for some processes (medical services? I'm not an expert in this field to be honest) people might suspect it's going to be challenging.
That was our intention. Basically we think no B2B saas should be building billing themselves, unless they are 300% sure their pricing will always remain subscription based only, and very simple (same amount every month).
Hope I helped clarify!
A couple days later we got an angry call from a customer whose card we'd maxed out. The final stage of the process was to adjust their bill by the amount that they paid, but there was a sign error in that adjustment so instead of lowering their balance by the amount paid, we increased it. As a consequence, for a user with, say, a $50 balance, we kept doubling the amount that we charged them each day.
Exponential growth is a powerful thing.
This isn't OP trying to claim it's the most difficult problem to solve from an engineering standpoint. Merely why it's hard.
Of course engineers solve problems, hard or not. That's not really being debated here is it?
It is the way.