Problems with homemade billing systems
getlago.com
getlago.com
My question is, how do I know that the OTS solution would actually be much better? For example, OP describes some tricky migrations from one type of billing to another, or complex grandfathering schemes - how would you have any guarantee that your third party solution would be able to support those? At least if it’s in-house you can implement the needed functionality eventually - if it’s third-party it might just be flat-out impossible.
I also feel that people often discount the cost of up top decisions that people below don't like.
You could say the same thing about literally any 3rd party software or service you purchase. "Why use cloud services to do anything?" etc.
Sure this thinking will hold for some things, but chances are that for something as standard as billing you're not going to be implementing some revolutionary feature that gets you more customers. On the other hand, if you mess up billing because you rolled your own system that doesn't have many standard options that a mature OTS system would offer it can lose you potential customers, because for months at a time you're stuck building the most mundane and standard things like an annual plan that could be used to better appeal to buyers.
For billing, a one of the most standard things ever for a business, you simply take a small amount of time to survey the OTS options and compare features. Any you know what? There's an excellent chance that buy actually getting a comprehensive look at features offered by such options you will learn about the types of things you will absolutely want to do in the future. Things that, in retrospect, seem like obvious oversights, such as an annual plan. You'll see features like that in your survey and say to yourself "Oh, yeah, that's something we may definitely want in the future, lets put that on our requirement list for a final choice."
If the third party solution cannot support the billing system devised, or will support it for a large cost, companies are more likely to choose things that are more in the box.
It gives that illusion. The reality is not so clear cut. There are lots of nuances to a capability that's not just a box check. It could be half-done, i.e. certain bugs in certain scenarios. It could be slow, unreliable or something else altogether.
There's also: https://www.businessinsider.com/former-vp-claims-salesforce-...
Bottom line, complexity is expense regardless if you build or buy. Companies need to factor the systems and operations impact of complicated pricing and billing models into the decision making process.
And as a developer, I love being able to hand certain people over to vendors who are used to dealing with those certain people. Hell, if we picked the right system, those vendors will have useful domain knowledge I don't have.
You could say the same thing about literally any 3rd party software or service you purchase. "Why use cloud services to do anything?" etc.
Sure this thinking will hold for some things, but chances are that for something as standard as billing you're not going to be implementing some revolutionary feature that gets you more customers. On the other hand, if you mess up billing because you rolled your own system that doesn't have many standard options that a mature OTS system would offer it can lose you potential customers, because for months at a time you're stuck building the most mundane and standard things like an annual plan that could be used to better appeal to buyers.
Survey the main OTS options and see what they can do. It will probably give some ideas like "Oh yeah annual plans, we'll probably want them at some point". At the very least you do this at the outset when you're small. Worry about complex billing requirements that won't be part of vanilla options in most OTS systems once you're big enough & complex enough to require them. Otherwise you're worrying about a Maserati problem.
The business (employees in the accounting dept and/or consultants) do a "fit-gap analysis" when evaluating potential software: https://www.google.com/search?q=fit+gap+analysis
For the "gaps" of missing functionality, look at either :
(1) customizations via extra programming or extra add-ons.
(2) eliminate that software as a viable choice and move on to the next OTS software for evaluation
> At least if it’s in-house you can implement the needed functionality eventually -
The vast majority of Fortune 1000 businesses replaced their "homegrown" accounting and payroll systems they built in the 1960s/1970s/1980s -- with COTS ERP systems like Oracle Financials and SAP R/3 because their internal IT teams never got around to the "eventually" option.
There's an anecdote (I think from the book "I'm Feeling Lucky") about a Google's early startup years where an employee said they "needed to buy SAP" for accounting. The co-founder Sergei Brin was dismissive, "why do we need to buy that when our programmers can build it?" Well, he was 20-something at the time and naive about the complexities of financial accounting software for a global business. He must have eventually realized the true scope of the problem because instead of building in-house accounting software, Google bought Oracle Financials. About 20 years later, they migrated from Oracle to SAP.
How do we jump to this conclusion? Did he realize the true scope of the problem or was it just left to someone else once Google grew in size? It doesn't even explicitly say how Oracle Financials came to be at Google.
> because their internal IT teams never got around to the "eventually" option.
I doubt that's really the case. Same with how cloud came to be. The sales people promised massive savings i.e. bonuses for management and here we are.
You delivered extra value for customers (making you more competetive) thanks to some particular ability of your previous (even manual) billing rules? Kiss them bye bye.
With the power to modify anything, people will make decisions and choices with serious consequences and benefits that may or may not be tangible, or worth the squeeze.
I’ve spent most of my career working in and around government, and poorly scoped or framed legislation costs billions because it drives customization and development. The business and engineering impacts of those decisions are often not appreciated.
If you work for a corporation, and you have a off the shelf billing system that fundamentally does what is needed… you have a chance of using ROI or some other metric to save the (expensive, hard dollar) customization for when there is real value.
The other thing is that sometimes these custom systems create customer problems in the name of “helping”. I’ve had many times where some legacy SKU or billing model, left in place to make my life easier instead made things much much more difficult down the line.
E.g. one thing that worked well in the systems I've worked on was to break things into small state transitions, very firmly document the allowed states and their transitions, and log every transition to the database. Wherever possible (anywhere that didn't interact with external API's, and sometimes even when they did) we'd aim for state transitions to be idempotent. When they weren't, we'd seek to reduce the non-idempotent call to an external dependency to just that call as a separate state transition.
When something very occasionally went wrong, we could trace things in detail, and pinpoint it, and most of the time we could safely replay transitions around whatever went wrong until we could see what had happened.
It wasn't difficult to build these systems that way, but it was tedious, and something where it's easy for people to be tempted into shortcuts.
As for scaling, you spit out events for rollups, and can shard invoice generation. We handled millions of revenue on a machine far less powerful than my laptop today. Scaling really would not have been up there for me.
To #3, maintaining old pricing isn't generally that hard. Very few people drastically change how they account for usage, which tends to be the most complicated billing scenario. They tend to change amounts and periods and thresholds, which should be data, not code.
With respect to a team, I agree. One of the places I worked on billing was Yahoo. At the time it wasn't just one team, but several - my team (responsible for Europe) existed largely because the European finance and product teams didn't trust the US payment services team to take their requirements seriously enough and so wouldn't let them near the European premium services... Billing isn't something you want to do yourself unless it's either a core competency or you're big enough to have to deal with that kind of bullshit.
(As for size of teams, I've built a billing system with 2 people, but I've also worked on billing systems managed by 50... You really want to have a good idea whether you're likely to end up towards the former or latter before you go ahead...)
I copy the entire product into the order. The thing one buys should be the thing on the screen when the decision is made. Poor pictures, description with typos, wrong categories, old/wrong pricing etc
One issue with it is that it often ends up forcing you to build out a lot of backward compatibility into your audit system as things change.
> We considered implementing an off-the-shelf billing solution but there was nothing flexible enough and the switching costs were too high. Algolia also tried to migrate to Zuora before backing out and rebuilding their billing system for the fourth time.
Most (all? I have yet to find one) off-the-shelf systems are not geared towards everything businesses want to do with their billing. If you take the author's advice and choose one of these systems early on, you will have the exact same headaches. Either the system simply won't let you do what you want to do or the system /will/ let you do what you want to do with custom or hack-ish solutions that reproduce the custom-solution problem, but in someone else's system. Those that implement broad swathes of billing functionality are so complex, they make /everything/, even the most basic stuff, hard.
BTW, when I say I know exactly how the author feels, all of what they described we encountered and implemented. We even looked at a migration to Zuora and came away with the same conclusion (also: really freaking expensive). We even had a new product that we setup on a third-party billing system, and we ran into the flip-side of the problems; we were not able to implement some billing functions we needed and we had to migrate off.
They would have to solve the same problem in a generic way. (add 10 more years)
> we had to migrate off
Adds to the fun doesn't it? My missing features hacked around previously exploded into an almost impossible puzzle.
1. Grandfathering
2. Upgrading across different length memberships (monthly tier 1 to yearly tier 2)
3. Crossgrading across different lengths (monthly tier 1 to quarterly tier 1)
4. Offering free trial to tier 2 when user has tier 1.
5. Promotional offers to non-paying users (half off first month)
6. Downgrades
7. Applying coupons to accounts (customer support wants to offer a specific user a free month of service when user has a quarterly subscription: billing schedule change)
8. Differences in tax regulations across countries (VAT vs whole value tax)
9. Differences in display prices (in some countries displayed prices should include tax)
10. Handling currency exchange rate changes.
It's just one of the things that might lead me to opt for 'buy' in 'build v buy', but... it's hard to trust the sales people involved in the process as well. And... taking the time to do real and full analysis... is time people often don't want to pay for.
And... once you're "in" to a COTS system... you're almost never leaving. The companies know that, the salesfolks know that, and have a lot of incentive to handwave away concerns, or just outright lie. Once you discover the lies... it's likely too late to switch.
Yeah. This problem is real. It’s so expensive to get out that we have had to make business process concessions because the COTS ended up not supporting, what I consider, basic functionality.
We only found out years later on the basis of changing other business processes.
And vendors will bleed you dry if they can. What some vendors consider to be “customizations” is insane.
It's not wise to outsource parts that are essential for your business, because: 1. you depend on 3rd party service for a getting paid 2. your operations cost may rise or the service may change terms/functionality
While out sourcing is a risk, it is also an opportunity to offload the hard work. You need to figure out what is worth doing yourself; what you outsource completely and ignore; and what you out source and audit.
Accounting - including billing, is normally something you should out source and audit. Let someone else worry about all the hard and weird rules. However you need to audit it because if you do not someone corrupt will steal your money and you are still required to pay whoever you owe money after that happens - this is one way your company can go bankrupt.
Many "startups" who are hungry enough to provide good value for money switch to "enterprise" pricing when investor money run out and your operations costs may rise a lot or you get "discontinued". Look what Atlassian did with self-hosted versions.
Also for tracking/logging layers it may be beneficial to mirror the data you're sending to 3rd party providers to your own db/logs, because you won't get data locked when switching.
So my point is that you need to outsource smartly and keep in mind that the 3rd party dependencies increase operations risk.
BTW that's why any user request shouldn't depend on 3rd party resources in a blocking matter (e.g. synchronous JS from vendors' CDN). You might pay more for your hosting but if you factor the service disruption risks you may be better off for mission critical parts.
Not convinced any 3rd party product is going to have the flexibility to support ongoing innovation. If anything it's more of a case for in-house expertise and resource. Find some folks who want to be the billing guys!
Disclaimer: I built a large in-house billing system :-)
perl billing system: https://github.com/fastmail/Moonpig
Would have nothing to do with your or someone else's decision. And Fastmail may or may not have made the right decision. It's hard to say. There are no simulators or time machines to gauge the impact anyway.
The most likely answer is preference and available skillset in the area. I've seen managers that absolutely want to self build and those that never want to maintain anything as much as possible (by offloading).
I work on card transaction processing and it's absolutely more complicated than you can imagine looking at it from the outside. It's a vast, distributed peer-to-peer system where trust is built into liabilities outside of the network itself. You want to build a system that is correct with regards to some specification in order to protect peoples' money... but the problem is that there is no specification that enumerates every possible transaction state because while there are "typical" sequences of messages to handle, your system also has to handle unexpected sequences that can't be specified: there are no guarantees that systems on the other end are emitting properly formatted events let alone behaving as expected. Card payment handling systems have to have the ability to manually correct and adjust state by humans; reconcile with external systems (because yay, payment systems in the US aren't based on instantaneous settlement until the roll-out of FedNow is complete), etc.
I've watched many businesses walk straight into, it's just X, how hard can it be? Only to watch their ARR shrink and stress levels rise when a project they thought would take a month turns into a year-long journey of discovery, reflection, and increased head-count.
People tend to get pissy about money. Almost as much as water plant scada systems and railway safety. Maybe moreso.
And if you're dealing with money, you're already inherently doing cloud stuff, because you're talking to the payment services, plus you already by definition have money to pay for a real billing system...
It seems as crazy as people on r/wallstreetbets advising their parents on how to put their retirement money into hand picked stocks or something.
You have to be mindful of some edge cases, but if the scope is kept narrow, it's doable. The added benefit is the flexibility to fit it to your use case. And with Python, you get many high-level libraries, e.g. for decimal calculations and PDF generation.
>> Now, when someone asks for advice about their billing system, my answer is clear: DO NOT build it yourself.
Moving from using CGI.pm to CouchDB/PouchDB about 10 years ago made it much easier to manage users and their data, and implement the UI. It also moved most of the workload to the user's web browser and made the app much faster for users.
CouchDB was practically designed for this. Early on their developers used "invoicing" as an example for using CouchDB.
But... if you have a marketing team that keeps moving the goals it wouldn't matter what you built the app with, it's going to take time to build and debug it. Could be that CouchDB/PouchDB could make that easier, but my experience is developers who've only been using SQL may get frustrated with it.
I was not in the billing team but we definitely followed rule #4 of the post: "You must be prepared to staff an entire team".
And yes, #1 "Pricing changes all the time and billing needs to follow".
About #2 "Your billing system needs to scale with your user base", we had to go from 0 to 3 million customers in 9 months to be viable. We were not typical, we made it.
No idea about #3 "Grandfathering causes headaches" but there were probably many headaches, that one and others.
Maybe there are 20-something people out there with enough world knowledge to make one. But it's not a safe bet at all.
Now, on a business level I acknowledge that frequently the monetary costs of the billing system are negligible compared to the business advantages of the "correct" billing system. My complaint is more that the scribbled note often ignores what the rest of the company is doing for billing, e.g, "yes usage based billing is great but do you really need to bill on furlongs per fortnight when the rest of the company is billing based on seats or some basic, easy-to-understand resource allocation?", ignores that maybe there's an easy thing that's close to what you said but since it reuses all the existing stuff you can have it in two weeks instead of four months, etc. Do you really need to issue a discount coupon that takes off a percentage proportional to the length of the customer's first name, unless the customer is in a jurisdiction where that's illegal in which case we roll a die, unless the customer is in a jurisdiction where that's illegal in which case we give them $10 off their first $100 and take off every prime-numbered dollar after that? Because I've got code that just takes 10% off their first month right here. Do you really need to bill on the first of the month unconditionally when the rest of the company bills based on subscription start date, or vice versa? Even if you're not too worried about the costs of paying 6 months of developers, I bet you are worried about those 6 months of opportunity cost.
I can't even count the months of delays caused by treating napkin scribbles as carved stone I've personally witnessed, and I'm only tangentially involved in billing over all.
And somehow managers who are deeply familiar with costs and benefits and tradeoffs and make sensible decisions all the time just become the most obstinate customers when it comes to billing. No matter how small the tweak proposed and how many months forward it may bring your release date to do something just slightly different (and consistent with the rest of the company's existing policies), they will go to the wall for their billing deviation, even when it was frankly clearly nothing more than a whim or a transient thought at some point by somebody somewhere of no strategic or marketing consequence.
It isn't just engineers who underestimate the complexity of billing until it's too late; it's everyone, really. "Just" print an invoice turns out not to be so simple.
It's crucial to bridge this gap in comprehension for more effective decision-making and strategy alignment between the marketing and billing departments. By doing so, we can ensure that both teams are on the same page and working together seamlessly to drive the company's success.