Things I learned while developing a billing system
arnon.dk
arnon.dk
But we found out the client devs (web, mobile apps) had severe difficulties and wanted it removed. Turned out they all implemented currencies wrong. Either with hardcoded decimals or with localised in and outputs that would break if a client changed their locales.
So now, my favorite best practice for any financial data handling is that: ensure your system can handle Bitcoin (8 decimal places) and festival tokens (missing currency symbol, zero decimals). Anywhere this leads to trouble is a red flag and will probably cause trouble later on. Now at least you are aware.
I think I get the issues with the other two, but curious about the first thing!
For German, I'd assume it's just about the special characters (äöüß), ie testing that encodings are somewhat correct (at least beyond ASCII).
It will do a good job of pointing out places where you haven't built in adequate flexibility for longer words. It's not perfect of course, there are times when a translated word is shorter, but it is a nice real world test of the basic flexibility of your page layout.
In [8]: "ß".upper().lower()
Out[8]: 'ss'Edit: Found the right search term. "Non-Decimal Currency". It's a thing, at least historically.
- An even 30 days = Easy to explain and do math with. Adding/subtracting 1 month can leave you in the same month or skip February entirely! Anniversary dates bounce around from month-to-month. Maps poorly to yearly calculations.
- However many days >this< month has = Harder to explain and do math with. Months without 31 days can get skipped when adding/subtracting 1 month. Recurring events get pushed away from the end of the month to cluster around the beginning of the month. It does result in a steady anniversary date for edge cases.
- Closest numeric day, but prev/next month (e.g. Jan 31 -> Feb 28): Easy and hard to explain and do math with. It's great for ensuring events only ever happen once per calendar month, but adds extra ambiguity (e.g. Jan 31 > Feb 28 > March 30? ... or March 28th?)
- 30.4... = Just no... except for the few times this is right and you need consistency to avoid unfairly comparing 28 days vs 31 days.
The consequence is some very surprising things the first time you see them. Some billing systems simply avoid doing work past the 28th of each month (either doing it a bit early or a bit late). Some just embrace the weirdness (whichever you flavor you pick) and you get used to the quirks (e.g. end-of-month lulls and start-of-month spikes).
I can only imagine needing to toss in standard/daylight timezone switch in there. Billing code really is the pinnacle of developer pain. It mixes the arbitrariness of special case business and customer rules with the absolute horror of time and date math.
SELECT CreatedDate AT TIME ZONE 'UTC' AT TIME ZONE 'Eastern Standard Time' FROM MyTable
You use the standard datetime libraries so you don't have to deal with all that complexity. Only, they automagically muck with the data, including irrelevant precision, such as time and timezone and daylight savings time for a calendar date.
And for the sake of completeness, your application allows the customer to specify a default timezone (usually that of their headquarters or something) and individual users to specify their own (useful if they work remote or in another office or something).
So now you have your database's timezone (and datetime handling), your server's timezone, your application's timezone (and datetime library handling), the customer's configured timezone, the user's configured timezone, their client's own timezone handling (which may be different if they're on a business trip in a different location), all screwing with the date of something that shouldn't even have a time, timezone, or daylight savings in the first place.
That's an endless whack-a-mole of bugs. And trying to just use UTC everywhere possible only gets you so far.
Add a month to the epoch time? Can't do it because that doesn't adequate indicate what day we think it is. "select '2021-01-30'::date + make_interval(months := 1)" says Feb 28? Done!
Do they do this? I don't know.
Even if they could do this in a single expression, there are absolutely scenarios where two date-arithmetic operations may be performed in two separate places in code. If those operate on the same logical date value, then we will see the same commutativity issues.
[0]: My sibling comment on this: https://news.ycombinator.com/item?id=26707864
For handling the different lengths between the months, I've considered a few approaches.
1. If re-bill is supposed to be the Nth of each month, that simply means N-1 days after the 1st. If that falls into another calendar month, fine. So if someone asks to be billed on the 30th of each month, they will be billed on Jan 30th for January, Mar 2nd for February, Mar 30 for March, and so on. I'd make sure the receipt explicitly says "Re-bill for January", "Re-bill for February", etc., so that bill the see on Mar 2nd hopefully will not confuse them, nor will seeing another bill later that month.
2. If re-bill is supposed to be on the 29th-31st, for February it takes place on the 28th (29th in leap years). For other months, if re-bill is supposed to be on 31st, it takes place on the 30th in April, June, September, and November.
3. If you initially order on day 1-21 of the month, your re-billing is the same day ever month. If you initially order on day 22+, your re-billing is on the same number of days before the end of the month as was your initial order. For example, someone who orders on May 30th would re-bill on the 30th in 31 day months, the 29th in 30 days months, and the 27th in February (28th in leap years). If you ask for a change in fixed billing date, you can do so either as Nth day of the month or Mth day from the end of the month, but N or M must be <= 28.
This may have unintended consequences. Some examples we faced around this:
1. Stress on our APIs when 15,000 invoices are created in the span of a couple of minutes. We had to build more queueing mechanisms around this. Some services just can't handle more than 100 calls a second.
2. Getting rate limited by MasterCard
3. Some payment providers thinking we're brute-forcing them
I'm just saying this needs consideration when you reach a certain scale.
Brunei weekends are Friday and Sunday. Those poor folks work Mon-Thurs, and then go back to work Saturday!
In general, however:
Don't fall victim to one of the classic dev blunders -- the most famous of which is "don't roll your own crypto" -- but only slightly less well-known is this: "Never write your own billing logic when business is on the line!"
> 1. If re-bill is supposed to be the Nth of each month, that simply means N-1 days after the 1st.
I have a credit card that takes the much simpler approach "Your payment due date is the Nth of each month. N can be any value from 1 to 24."
This seems to basically match your approach 3.
If you ask how I think you should bill ( :p ), do it on a fixed day and don't try to do rebilling at a fixed interval. If the day somebody buys your thing is unsuitable for fixed-date billing, the answer is prorating their first period, not interval billing.
I still don't exactly know how to handle this. If you start an account that compounds interest continuously, does the daily interest rate change if one year is longer than another?
Looking back I'm amazed at how easily companies got off the ground by hiring eager (but exceptionally green) young web devs to write sensitive code for ordering/sign-up, recurring billing, and more.
I like "730.5 hours" (usually rounded to just 730). It's a good fit for services billed by the hour or minute. I use it regularly to answer "how much per month" for AWS stuff. But yeah - it fits in there with your "Just no..." category.
Jan 27 + 5 days = Feb 1
Feb 1 + 1 month = Mar 1
Jan 27 + 1 month = Feb 27
Feb 27 + 5 days = Mar 4 (or Mar 3 in a leap year!)
Jan 27 + 5 days + 1 month = Mar 1
Jan 27 + 1 month + 5 days = Mar 1
Jan 27 + 5 days + 1 month = Mar 4 (or 3)
Jan 27 + 1 month + 5 days = Mar 4 (or 3)
Then, please consider that your argument that ignoring February is not a good way to approach any calendaring library....
I had specifically worked on functions like EXTRACT to pull out different parts like ISO Week numbers, as well as DATEADD and DATEDIFF functions which do date math. It is exceptionally unpleasant!
One solution is to distinguish between a "datetime" and a "duration". You cannot convert a duration to days (or really anything). You can add or subtract a duration from a datetime, and you can convert the difference between two concrete datetimes into days.
And that's actually how some libraries already implement it.
So the answer to "how long is a month" is "it depends, and I can only say if you apply this duration to a concrete time".
It might make some of the math/algorithms a bit painful but I think you could guarantee correctness with the above approach.
The issues that arise are then a question about how to translate them to journal entries, which will require accounting experience, and maybe your chart of accounts needs to be revised, but the core system should be fairly stable, and voiding an invoice by issuing a credit note comes automatically, as you’re working with an append-only ledger.
The issues about prepaid plans should also be handled, as payments for services not rendered yet should not be recognized as income, but instead kept as a liability: The OP mentions their system is used by 15,000 customers, so I would give each customer their own account (in the chart of accounts).
The main technical issue is what datatype to use for money, the rest are problems solved by following general accounting principles, though I fear that a lot of programmers out there are reinventing the wheel (in suboptimal ways).
This entirely depends on whether you are operating on a cash or accrual basis. Both approach are valid (at least in the USA) and cash basis is often used by small businesses.
Though the IRS does limit the types of businesses that can do cash-basis accounting, it would be businesses without inventory, and who does not offer credit to their customers, e.g. a hairdresser would probably use cash-basis accounting, but more complicated businesses would not, certainly not a business that needs its own billing system :)
I fumbled and bumbled my way to this realization while trying to build a billing system intended to tolerate all sorts of invoicing and reversal scenarios.
More precisely, it doesn't matter what the nature of the billing is, ie recurring (monthly, quarterly..etc) vs one-time; You want to build a system around invoicing for discrete items and resolving those invoices against various criteria (payment received, credits issued, cancelled plans...etc).
How hard can it be???
<goes away and starts writing a new javascript framework from scratch that addresses precisely _one_ of the billing scenarios, and depends on 17,000+ npm libraries, including leftpad.js>
I've built a billing system for a hosting platform. Back when all I knew was MVC (and the then common index.php ballofmud). I wish I had known about eventsourcing then, because so many problems that I spent weeks on, would have never occurred, or been solved in hours.
Eventsourcing comes with its own downsides, requires your mindset to change (esp hard in a team that has been doing relational databases or MVC for years), and is convoluted. But it is a very good fit for a large swath of problems. Financial ones the most.
I believe double entry accounting can be described as following this architectural pattern (despite predating it with hundreds of years).
Double entry accounting has both a ledger and a journal. The ledger describes the actual transactions, the journal groups multiple transactions and assigns a “why”, it can also link the journal entry to an invoice, receipt, credit note, or user who caused the journal entry to be created.
So the journal is your event log (not the ledger).
> Eventsourcing comes with its own downsides, requires your mindset to change
Similar to double entry accounting: The learning curve I would say is the “chart of accounts”, how to express everything as ledger transactions, be it tax, fees, discounts, credits, prepayments, etc.
But it can all be done, and once you understand the system, it becomes trivial and extremely flexible.
A rule of thumb is that any number in the interface should come from the ledger, e.g. if you issue an invoice and give your customer a discount, there must be a ledger entry corresponding to that discount.
At first it seems like it's so easy and straightforward but when you are looking at thousands if not millions of bills a year and the myriad methods you need to compensate for to ensure its accepted by large firms.... well. I can say nothing surprises me anymore. Including hardware level errors causing an issue no one expected with a final number.
And so on and so on. You don't realize it until you suddenly realize why it was such a bad idea to have one point of failure for everything.
Half of them will reject you for being wrong, the other half will reject you for being right because it doesn't line up with their floating point errors in excel.
Subsequently, whenever interviewing clients for requirements I'd mention this and it usually resulted in padding their specs.
This is fine as long as everybody follows ISO4217. In my experience many 3rd party systems _mostly_ do it and of course you get bitten by the edge cases.
For example, Stripe decided HUF doesn't have 2 decimals (I know that's how it's used in practice today but we're talking standards and system interoperability here), or that ISK subdivides into hundreds ("cents") in contrast to the ISO tables. Compare https://stripe.com/docs/currencies with http://currency-iso.org/en/home/tables/table-a1.html. As Stripe notes for UGX, backing out of this is nearly impossible because of the subtle break in backwards compatibility. And who knows how other payment providers deviate in their own way.
For my own sanity and that of my coworkers (e.g. easier analysis for extracted tables in the data warehouse) I strongly prefer avoiding this headache and use decimal instead. You can do the appropriate rounding for a currency in your money object as for example https://www.martinfowler.com/eaaCatalog/money.html does.
For example: 1 ea @ $0.00589 Normally this is when ordering in the thousands or millions of small things, but you need to record that fractional size.
The world is not straightforward, allow an admin to fix/change anything and when they get tired of making some change then code that new path in the system. Working with or writing systems that do not allow admin overriding is painful.
> Money isn’t always decimal
> A common wisdom in database design is “never using floating-point numbers for money”. Some recommend using the MONEY datatype, while others tell you to use DECIMAL.
> Both of these are wrong. Sure, yeah, in the US and most of Europe, money is decimal.
> That’s certainly not the case in Japan – you can’t charge 2500.50 JP¥.
The author has a very odd definition of "Decimal". Integers are most certainly decimals. The fact that in Japan you can't charge fractions of a a yen doesn't mean that Decimal isn't actually an excellent choice to represent that currency, because it certainly can represent any yen value you would need. The whole reason you don't want to use float is because it cannot represent certain valid monetary values.
As others have pointed out, storing values as "the smallest division", e.g. pennies for USD, is just wrong, because there are many contexts where you have to represent fractions of a cent <insert Superman 3 joke here>.
Whereas 10000 USD cents is a valid value.
Integer storage of money is my favourite as well to be honest. It's easy to work with and reason about, and it just makes decimalization a display issue.
I mean, in the US it's actually pretty standard for accounting applications to use a scale of 4 (4 units to the right of the decimal point, the default for MONEY), even though people are never charged fractions of a cent.
Simplest example someone gave is gas station pricing, in which prices often end in 9/10ths of a cent - the price is calculated that way, and only rounded when the customer pays.
> For example:
> $100 US may be stored as 10000
Eh, the US dollar is often subdivided further than that.
Here's four examples, in different industries:
http://cdn.radioiowa.com/wp-content/uploads/2011/12/gas-pump...
https://aws.amazon.com/s3/pricing/
https://finance.yahoo.com/u/yahoo-finance/watchlists/most-ac...
https://www.digikey.com/en/products/detail/microchip-technol...
For example, if you're paying #3.089 a gallon at the pump, and you pump exactly one gallon, you don't pay $3.089, you'll pay $3.09, and neither your account or the account you're paying into will need to know or care about the tenth of a cent difference, because our monetary system doesn't really deal with denominations that small in transactions.
3.089 * 10,000 = 30890
3.09 * 10,000 = 30900
Also reminds me of the plot of Office Space. Which is a bit humorous since they screwed up their own scamming scheme in similar fashion.
To bring this back to the original comment and the items it was referencing, store money (real money, like accounts) in the smallest possible subdivision (cents for USD), but that doesn't need to apply to pricing, which is a potential amount of money. Potential money needs to be changed into real money at the time of a transaction, and real money cares about quantities it's possible to have, and it's not really possible (or at least useful, in most cases) to have less than the smallest possible subdivisible amount of a currency.
You're entirely correct though that you can't assume that cents is enough to accurately model everything to do with a business that works in USD, and ignoring that will result in problems like you showed.
(I used the regulations as a way to ensure _all_ calculations happened on the (Java) backend that other people were responsible for, given Javascript's documented mathematical insanity meant any client side calculations couldn't possibly pass compliance...)
For instance, with proration. Take each line item, prorate it, round it, then add them all together. Totally reasonable.
Now add all the line items together, prorate the total, and round it. Also reasonable, but there's a decent chance the number is different by a few pennies because of rounding differences. If you chained more calculations the differences would compound.
Whichever way you choose, somebody is going to whip out their calculator and tell you that you did it wrong (but only when they come out better the other way).
This applies anywhere you're doing multiplication or division. Discounts, proration, taxes, "cashback", "store credit dividend", whatever.
I'm glad to see he also has the same thought about floating points. Also:
> Happily, Moonpig did not have to deal with multiple currencies. That would have added tremendous complexity to the financial calculations, and I am not confident that Rik and I could have gotten it right in the time available.
This is one of the things we deal with which is quite challenging.
And it only schemes the surface.
There's also PO-based vs. CC-based payment which have different sequencing of the details. You have one I run into all the time: Net-30 vs. Net-60 vs. Net-90. As well as linkages to quotations and terms. B2B takes all this to another level.
I've personally been working with billing systems for 2+ years and have felt all those pains.
Common, and in my experience totally wrong. It's the most pervasive cargo culting I've experienced amongst developers, where people with 0 experience with financial applications will recoil if you argue against it. In my time developing applications for front office at an investment bank, floating point often works best.
Of course in your case though, for a billing system, the method you describe is obviously the right one.
For front-office risk and pricing calcs, speed of calculation matters most. Hence IEEE754 float64.
Fixed precision decimals really only matter for middle and back-office, for trade confirmation and settlement.
Front office - S&T (sales & trading), i.e. revenue generating activity. Usually includes any quants/quant developers actively working on things that make money
Middle office - operations. handles settlements, confirms, and generally anything related to the post-trade flow that is "after the trade is booked"
Back office - accounting, legal, engineering, IT support, HR. Anything that isn't a revenue center that also isn't even tangentially facing revenue generating operations
Care to elaborate?
Maybe finance doesn't, but billing does.
Is that similar to what you do, or are you handling that differently?
Floating point ARE a major source of errors across everyone that use them. ARE infectious. ARE semantically wrong. ARE not made for financial calculation.
ARE WRONG.
Period. Just because under a lot of discipline (or luck, or just "assume" is working but nobody have checked, or work before but how knows if today?) not make it a good choice for financial/money.
Is the same error when people think old String types can be used in the unicode world, instead of have a proper type for that.
Luck help a lot. But is not something to be proud about.
They're not a major source of error when you're implementing a model which is already inexact to a far larger degree than the problems caused by floats. Black-Scholes does not perfectly price an option, your bootstrapped curve is not a perfect predictor of market conditions in 28 years time. These are the problems faced in front-office finance, the error is already so far beyond 1 + 0.1 not perfectly matching 1.1 that it's not worth caring about. I've worked on applications where users just wanted to see numbers to the nearest 100k so they could model out a few trades they planned to make over the phone.
When you're working on a retail banking app where you need to track customer's balances, or an accounting system, or anything like that, then sure, floating point would be malpractice. That is nothing like any of the applications I've worked on in the financial field.
I have worked in finance for my entire career, and everyone uses floating point arithmetic for dealing with money on the modelling side (sometimes $0.01 = 1.0, sometimes $1 = 1.0, depends on the institution/currency/convention/context, but we have to deal with fractional money anyway).
They do NOT use it on the accounting/back office side. That would be a spectacularly bad idea. Those systems are designed for accuracy to the penny.
If the application is doing financial modeling and estimations, where only 1-3 decimal place of accuracy is needed, then floating point is the right choice. It greatly simplifies the app.
If the application is doing accounting, payments and billing, where people expect accuracy to the penny or more (e.g. to 1/10000 of a penny), then it needs to use a Decimal type.
Accountants frequently spend all day in Excel. Excel uses floats for all computations (doubles, to be precise) [1].
Now, in all fairness, an accountant and developer's relationship to the numbers is rather different.
For an accountant, the risk of floats blowing up is largely mitigated by the fact that they have a close, intimate relationship with the actual numbers.
The responsible accountant should always deliver numbers they have personally reviewed, while a developer is usually automating a process, generating numbers that have yet to touch a human eye, so there is rather less room for error.
Still, it's simply untrue that one should never use floats for money. Many of the cases where floats would generate bad results are also problematic for simple alternatives. For example, fixnums are simply 'floats that can't float', so you need to be able to guarantee a fixed range ahead of time.
Understanding basic numerical analysis is unavoidable to writing correct code.
[1] https://en.wikipedia.org/wiki/Numeric_precision_in_Microsoft...
That does not invalidate the general advice to avoid floats for money (which is probably why you were downvoted).