Fintech engineering mistakes (2022)
startupwin.kelsus.com
startupwin.kelsus.com
Time synchronization. It is incredibly important in fintech applications for a number of reasons:
1. *Transaction Ordering:* Financial transactions often need to be processed in the order they were initiated. This is especially crucial in high-frequency trading where trades are often made in milliseconds or microseconds. A small difference in timing could potentially lead to substantial financial gains or losses. Therefore, accurate time synchronization ensures fairness and order in the execution of these transactions.
2. *Security:* Accurate timekeeping helps in maintaining security. For instance, time-based one-time passwords (TOTPs) are widely used in two-factor authentication systems. These passwords are valid only for a short period of time and rely on synchronized clocks on the server and client side.
3. *Audit Trails and Dispute Resolution:* Timestamping transactions can help create a precise audit trail, which is critical for detecting and investigating fraudulent activities. In case of any dispute, a detailed and accurate transaction history backed by synchronized time can help resolve the issue.
4. *Distributed Systems:* In distributed systems, time synchronization is important to ensure data consistency. Many financial systems are distributed over different geographical locations, and transactions need to be coordinated between these systems in an orderly fashion. This requires all servers to have their clocks synchronized.
I am sure there are even more fields where this is relevant.
How clock synchronization protocols can experience unaligned network partitions, where the ledger database cluster is able to continue running, but now with the risk of unsynchronized clocks and far-future timestamps, which can in turn then lead to money being locked up in 2PC payment protocols.
We therefore spent considerable effort [0] on clock synchronization in TigerBeetle, not for the consensus protocol—we never risk stale reads or take a chance with clock error bounds—but rather simply for accurate audit trails and to keep inflight liquidity from being locked up if transactions take too long to get rolled back.
[0] https://tigerbeetle.com/blog/three-clocks-are-better-than-on...
And then the banks send you info in batches and out of order :) This happened to us more than once. So the team responsible wouldn't settle/cancel a payment for X even if bank said so. They would read a few other sync batches yo make sure that nothing else changed for X.
Because "Financial transactions need to be processed in the order they were initiated" is a must :)
You don't represent money as (<currency>, <decimal amount>), you may want to store it as (<currency>, <decimal amount>, <timestamp>) to be able to apply the correct exchange rate post-facto, and not have to deduce from the transaction history.
It also helps with cross-checks/consistency verification.
Also, preserve the original timezone in the timestamp. It can save many headaches down the road.
— If you’re using JSON to pass around monetary quantities (eg. from the frontend to the backend), put them in strings as opposed to the native number type. You never know what the serializers and deserializers across languages will do to your numbers (round them, truncate them etc.).
— Start recording the currency of the transactions as early as possible. It can be a separate column in your table.
— Use TIMESTAMPTZ. Always.
But since this is not a human visible field, this is probably irrelevant, right?
https://ux.stackexchange.com/questions/9105/international-cu...
So we put the ISO code at the end and ignore whitespace. But including a prefix sign as with `€1.234.567,89 EUR` would break it.
When you’re using JSON to pass around datetime data, Use ISO8601 date and time with offset info, always.
e.g. "transactionDate": "2023‐06‐28T15:55:22.511Z"
I think that's just sufficient for all use cases everywhere. I've been a software engineer for almost 25 years and, of all the universal truths I've encountered, implicit and unintended changes to datetime offset are the ones I've seen at every single job.
So this technically isn't ISO8601, and certainly isn't "ISO8601 in a string", which is an _interchange format_ between application with potentially very different runtimes. I don't really recommend treating ISO8601 dates as mere strings, unless you intend to pass them through without even looking at the contents.
I refer to https://learn.microsoft.com/en-us/dotnet/api/system.datetime...
Use types that support ISO8601 with an offset, always.
The docs even say that, even if many are not aware:
> consider DateTimeOffset as the default date and time type for application development.
https://learn.microsoft.com/en-us/dotnet/standard/datetime/c...
TLDR: Also, which timezone is used (not quite the same as offset) really does matter--UTC is great but you can't use it everywhere.
________
One of my favorite simple examples of this "here be dragons" for the new developer: Any system that schedules a future calendar-event.
Such events are typically pegged, implicitly if not explicitly, to a particular timezone or geographic context. For example: "The company's Virtual Summit will occur on November 2nd at 1PM Elbonian Xtremesunshine Time, hosted out of our central Elbonian headquarters."
In that scenario, it is impossible to know for sure how many seconds-from-now it will happen until the moment actually happens! "2023-11-02 13:00:00 EXT" is actually a contract or spec for recognizing a future condition, one that will shift if/when the relevant nation/province/city simply declares their clocks shall be set differently.
So if the Elbonian government alters their daylight-savings switchover to occur earlier on 11-01 instead of 11-06, then the summit just moved. Even if you scheduled everything UTC all along... Well, now the summit is overlapping lunchtime for everyone in Elbonia, so it moved from their perspective.
Good catch!
As you note, for some uses this _does_ make a difference and tracking which one you have can in these cases be important.
Ha, good one! One can only hope the Elbonian Parliament will have the sense to abolish this stupid Xtremesunshine time and observe one time zone year-round.
I'd go a step further and prefix the strings with an ISO currency code ... to stop someone from just feeding it into their languages int to float converter and assuming that's ok. Only custom built (hopefully safe) converters will work.
I think correctness and precision requirements of financial transactions outweighs any devex concerns. Lots of banking and trading APIs do exactly this: pass your currency fields as strings.
By modelling your systems you will learn what the important failure modes are and you will get better at designing systems that are resilient and efficient.
Card payment systems are fairly unreliable peer-to-peer messaging systems. Be prepared for a lot of complexity. Using an event-sourcing architecture is really useful here for that "auditing" requirement and for debugging transaction state when the network sends you messages in error, out of order, or they forget to retry themselves when they promised to, when merchants send bad data, when POS systems do weird things, etc.
Various platforms use: UTC, the user’s timezone as set in their dashboard, the user’s detected timezone, the timezone of the server that is generating the reports.
Aggregating or reconciling data from different platforms can be a pain if the timezones aren’t clearly indicated.
I’ve had bank statements that didn’t agree to CSV exports of the same data because the servers generating the two reports were in different timezones.
The transactions were in the late evening on New Year's Eve, central US time, but that was already the next year by UTC, so any readout of "transactions for 2022" would not include them.
[1] e.g. https://www.irs.gov/taxtopics/tc301
1) https://learn.microsoft.com/en-us/dotnet/api/system.datetime...
For example, an invoice due on Friday is probably actually due by close of business (5pm say) in the timezone your business operates, and if created at 11pm it would be processed the next day (or even on Monday, don’t get me started about business day calculations).
Solid advice, and must come from painful burns. I've been preaching from the same book for a few years now: a timestamp without timezone offset is worse than useless.
Or as the DB expert in previous job so eloquently put it... Timestamp with zone tells you when an event actually happened. A timestamp without zone or offset is equal to wall clock time inside a windowless room, which itself is in an unspecified location somewhere on the planet.
We only had a dozen issues due to it so I believe it worked pretty well compared to the war stories I heard from comparable companies in the fintech space.
1. Never record an amount without its currency.
2. Reconcile all your data all the time. Never let any data go unaccounted for.
3. Maker-checker is a powerful concept. Embrace it to the fullest across your system.
4. You will be dealing with all sorts of non-standardized financial integrations. A lot. Think adapter pattern as early as possible.
5. You will be answering to multiple regulatory agencies. Create boundaries between them within your system and reduce the surface of compliance as much as possible.
For “using floating point data types”, it’s even worse; you often need to use strings, for example if you need to store a bank account number “01234567789” will have the leading zero stripped if you use a numeric type.
“Updating transactions” would be better phrased as “use an append-only log / evented architecture”. (Also, “use a double-entry ledger” is probably the most valuable advice I could have sent myself prior to getting into FinTech.)
“Be careful with retry” should be more strictly “use idempotent operations” and link to the canonical Stripe article on idempotency keys (https://stripe.com/blog/idempotency).
Another important one to think about is bitemporality. “Created at” vs “effective at”. Not obvious at first and you’ll have some painful migrations if you don’t build it in. Fowler has a good overview here: https://martinfowler.com/eaaDev/timeNarrative.html.
Edit to add - the advice that maybe using a NoSQL database is pretty bad IMO. I’d advise in the opposite direction - use SERIALIZABLE isolation in a SQL database. Read up on your Aphyr blog posts before trying to do anything distributed. Be paranoid about race conditions / serialization anomalies. If you eventually hit performance issues you need to think hard about what anomalies your access patterns might be subject to. (Obviously HFT won’t use serializable SQL).
This goes for any "number" that is actually an identifier. Phone numbers aren't really numbers either for example.
Rule of thumb: if it doesn't make sense to do math with it, it should probably be a numeric type unless there's a very good reason.
It is worth noting that many of these methods don't prevent tampering, they only make it visible when you look for it at which point the fact you are explicitly looking for it (rather than having been alerted to a potential issue) might imply it is too late.
The number one thing they crave is immutability. But when that's not an option, tamper-evident comes as a close second.
If your language has a dedicated type for monetary amounts, use that. (1)
If it does not, but you can make a value object to represent, e.g. amount and currency code, then do that.
If however, your language does not have a dedicated type for monetary amounts, or one cannot be trivially built or retrieved as a package (2), then you should ask yourself if it is really a suitable language for financial tasks.
1) https://learn.microsoft.com/en-us/dotnet/api/system.decimal
2) https://github.com/shopspring/decimal https://www.npmjs.com/package/ts-money
Adyen uses integers [2]
Square uses integers [3]
So not sure that the industry would agree with you
[1] https://stripe.com/docs/api/prices/create#create_price-unit_...
[2] https://docs.adyen.com/api-explorer/Checkout/70/post/payment...
[3] https://developer.squareup.com/reference/square/objects/Mone...
a) On web api, so this is by definition data interchange, not "in a language".
b) with a currency code to cross check it.
c) implicitly "in cents" which needs further documentation - does this mean "pence" when the currency is GBP? How does this work for for JPY? BHD?
d) cannot represent fractions of a cent or penny.
So: this int plus currency code plus docs is OK but not great, a lowest common denominator format for interchange, requires further documentation and cross-checking as "100 USD" does not mean 100 bucks and the conversion is currency-specific, and cannot represent all values.
I wouldn't refuse to convert _to and from_ this format at the edges of my application, for data interchange, but the conversion to something clearer and richer IMHO should remain there.
Payment APIs are far from the only use case in fintech, for interest calculations you do have to care about fractions of a cent or penny.
In fact, if your case is "I call the stripe api" ... are you sure you're a fintech and not an online store? Get back to me when you have to interop with FiServ, MasterCard or SAP.
Point missed, they use that format _on their client apis_, which tells you nothing about what they do in the code that handles it. Now it could be the same internally, in which case the reason is "there's crap code everywhere"
Erm, did you see the quotation marks around "in cents" ? It's a _quote_ from the docs at the link. The link _that you gave_.
The issue is not "do you or I understand the difference", it's "does stripe" and "do stripe users, after reading that?"
It is simplistic, like the int, and is not making the point that this is great.
Three additional 'mistakes' to prevent when dealing with money representations:
1. The definition of a currency might change. For example, some years ago Iceland decided to change the exponent of ISK from 2 to 0. Currencies have different versions.
2. As a FinTech you probably have integrations with many third parties, they don't change their exponents for a currency at the same time. Keep track of what third parties think the correct exponent is at any point in time, and convert between your representation and their representation. Otherwise, you'll have interesting incidents (e.g. transferring 100x the intended amount of ISK).
3. At first you think that counting minor units as an integer is enough, and then you need to start accounting for fractions of cents because sales people sold something for a fee of $0.0042 per transaction. If your code rounds all these fees to $0.00 you don't make any money.
3 - Indeed, the currency type that I referenced "is appropriate for financial calculations that require large numbers of significant integral and fractional digits and no round-off errors"
Next to that, ensure transactions are immutable, and ensure reporting is idempotent: Close your periods and generate reports for them, when you regenerate the report it should be identical.
This is more domain tips. A lot of technical tips hold for non fintech too.
If you want an example where consistency is not important, you might be able to overdraw from your bank with your ATM card. The bank is happy for this to be inconsistent, since they can charge for the overdraft.
If you are an engineer who doesn't know what a mantissa is, stop two minutes for a cool ride https://lashewi.medium.com/storing-currency-values-and-float...
Others (like me) will cry because we know of multibillion-dollar fintechs that still struggle with this.
The one I'm thinking of didn't even have cents for a long time. After a pretty heroic migration effort they added cents. And they did it properly. But within weeks folks were using floats all over the place for money, leading to flaky tests and all kinds of other errors.
I understand the sentiment and the potential issues, but it's really kind of domain dependent.
Not that fix nums are wrong, but they are far from safe.
1. You might need a different number of "implied decimals". Two decimals is enough for many currencies, but some are three, quite a few are zero, and a few are a janky 1-decimal model. And that doesn't even go near, say, pre-1971 GBP structures.
You'll have to put scaling logic on every interface with the outside world.
2. Even if you do it perfectly, it's going to change. For example, the subsidiary unit on the Icelandic Krona is being removed in a lot of financial APIs right now. You can either change your records, or change your scaling logic, but you've now got an inflection point where you can't reason about the behaviour before the changeover from the current code.
What you need is a decimal type, where you can have 6.33 dollars, 2.167 dinars, or 500 won, and have them all retain fidelity. Sadly, few popular languages provide it.
One more from me:
- Payment/order processing systems typically involve complex logic.
- One of the best ways we managed to keep complexity in check was to model these as state machines (with the state itself being persisted to DB)
I’m sorry I’ve drank too much coffee this morning.
Here's something small but I was doing a SQL interview one involved a a card_tx table with the amounts stored in integers values in cents. I immediately noticed this and chatted with the interviewer about this. My assumptions were wrong, and the floating point arithmetic was a simpler reason than what I was thinking.
Anyway you can display you have thought about relevant business problems to where you are interview engineers tend to have a strong reflex to engage the person more.
You raise a fair question.
Coincidentally, one of the reasons we picked Zig was for how readable it was, and strikingly so, even for high level programmers who might not understand systems programming or C. Because Zig reads like TypeScript, and we were working in payment switches where the majority of programmers could read that. This particular switch, in fact, had this same business requirement, that programmers should be able to at least read the systems language.
But generally, our experience has been that people who understand C will understand how to maintain Zig [0]. Zig's toolchain is also more accessible, and across all platforms. Zig's compiler is already being used by Uber for hermetic builds.
It's also easy to learn. You can pick up Zig in a week and be comfortable in a month. Zig has a simple grammar. I love how, when we have someone join the team, we never have a discussion about how to learn Zig, as if it's a difficult language to master (like C++!). Rather, there's excitement around learning the language, even ahead of starting at work, and within a day or two they're committing.
We made this decision for TigerBeetle in July 2020, and didn't take it lightly. We had already followed Zig's progress for 2 years by that point, and many factors were considered [1][2][3]. C was the other contender, given that we had to handle memory allocation failure.
The crux of the decision, then, was whether to invest in a systems language of the last 30 years, or in a systems language of the next 30 years. A distributed database is a big investment. It made sense to invest for the future. If anything, it would have been a colossal business mistake to have picked C or C++, which would have crippled our development velocity.
Furthermore, for TigerBeetle's design goals, especially w.r.t. our adoption of NASA's Power of Ten Rules for Safety-Critical Code and thus static memory allocation, Zig made (and continues to make) the most sense.
We also liked the efficient performance culture surrounding Zig, with talented game developers like Michal Ziulek and Stephen Gutekanst, and embedded programmers like xq, Matt Knight, Jens Goldberg and others moving to it. These industries (gaming, embedded) are often a good litmus test of where systems programming is at.
More details (our thinking on Zig through the lens of safety/performance/tooling/ecosystem/hiring/marketing) here:
[0] https://kristoff.it/blog/maintain-it-with-zig/
[1] https://docs.tigerbeetle.com/FAQ#why-is-tigerbeetle-written-...
[2] Building a Distributed DB in Zig: https://www.youtube.com/watch?v=0pHHb6ONTbw
[3] TigerStyle! (How to Design Safer Systems in Less Time): https://www.youtube.com/watch?v=w3WYdYyjek4
Our TigerStyle talk is worth a quick watch, if only to get a feel for TB's safety and testing techniques indepth.
It's certainly valuable to verify protocol specifications using formal methods.
However, compared to TLA+, we use Deterministic Simulation Testing, which enables us to test the system as implemented, not only as specified.
Doubles can exactly represent all 16 digit ints (IIRC) which is good enough for most use cases, you can catch the out of range cases (as you should do with ints as well)
If you use long ints you must track the decimal precision along the value which is not always trivial if you use mixed currencies.
Long ints are not guaranteed to correctly roundtrip through json serialisation / deserialisation
Doubles are easier to handle in the frontend
Currency math is different enough from regular math that you need special operator functions anyway so it’s not like ints are easier to handle either
The United States dollar is famously subdivided into cents, but is also subdivided into 'mills' (one thousand to the dollar).[0]
The Mauritanian ouguiya is divided into five khoums.
The Madagascan ariary is divided into five iraimbilanja.
The Maltese scudo is divided into twelve tarì, which are divided into twenty grani, which are divided into six piccioli.
Historically, such currencies were ubiquitous. For example, prior to 15 February 1971, the pound sterling was divided into twenty shillings, each of which was divided into twelve pence, a system that originated with Roman currency and was used throughout the British Empire.
Exchange rates are typically quoted in terms of the largest unit, whereas integer representations of currency would need to be done in terms of the smallest unit, so extensive information about currency structures would need to be used to correctly represent exchange rates. Floating point or binary-coded decimal representations are consequently much better.
This is exactly the issue with using floats where an arbitrary precision decimal with proper rounding is really needed. Easily solved with a good library and if your languages supports it, type, but it's really easy for a dev in a hurry to not use the library and roll some a=b+b*c_rate code that forces some type conversion. The rounding rules often are tied to contracts, and a subtle bug that's off a few mills here (total problem created $2.33) and there can lead to audits (total cost of audit $14,800) that cost a lot.
It might not be simple to implement, though. Off the top of my head,
* Can you confidently say that $100>¥10 in an offline environment?
* You'd need to support different bases for splitting units to deal with eldritch values such as farthings.
* Where is the line drawn? Is a Bitcoin a valid currency? A gold Krugerrand? A bundle of stock shares? A bushel of apples?
All of that said, I would love to see a world where I could write code like:
money balance = money.lumber.grade.kilograms(8);A major problem money has is that it isn't a unit in the sense we usually take the term to mean. We expect, for instance, that translating one unit to another with a suitable level of precision should be translatable without loss back to the original unit, but that's not true for money, even ignoring transaction costs. If "US dollar" is a unit, it is a unit that technically stands alone in its own universe, not truly convertible to anything else, not even other currencies. All conversions are transient events with no repeatability. But that is very inconvenient to deal with, and with sufficient value stability of all the relevant values, often it's a sufficient approximation to just pretend it's a unit. But if you zoom in enough, the approximation breaks down.
For that and similar reasons, while you could theoretically write that line of code, it would be implicitly depending on a huge pile of what would in most languages be global state. It would be a dubious line of code.
At a bare minimum, it should be a reasonable fixed point type that correctly handles rounding and intermediate values. So a dollar amount like 123.45 times a rate like 0.3450 doesn't exceed 4 decimal places but intermediate values are extended so we get correct rounding. The destination should probably determine the number of places. That bare minimum wouldn't stop you from comparing yen to dollars, any more than a floating point representing mph stops you from comparing it to a value representing kph.
But there are times where we need to track prices to the nearest tenth or hundredth of a cent. So it should be extensible so that 123.456 dollars * 0.3450 winds up at a correct round/decimal places.
You also don't need always-on, real time currency conversion. You could have a conversion type, operator, or method that does safe conversion based on the value I give it. So if I estimate that Yen are about 130 to the dollar, I can just use that. If I happen to write an application that queries a data provider and can populate that in 'real time,' that's up to me.
If you really wanted, you could find a way to create new types that represent currencies that aren't part of the basic implementation. That might mean you need to specify some things like the representation for different locales, or the default number of digits.
Money is never converted. It is exchanged. Trying to solve this is like trying to solve the question "Is "100 USD" > "1 LAPTOP".
When you turn 100 USD into 90 EUR, you didn't convert it. You exchanged it. You bought EUR, at a price given to you by someone or something exchanging it. This could be a bank, a well-established currency office, or some dude on the street. There is no real difference between all three of those: The third party gave you a price, and now has more USD and less EUR, whereas you have more EUR and less USD.
There are various entities publishing standardized average rates which are calculated after the day closes, based on a variety of datapoints they have access to. Those are often used in eg. accounting, to establish the "real" value of something you bought in a currency you don't often use, but it's not true conversion.
If you have, as a datatype, a currency becoming another, there is ALWAYS a "rate" attached to this. So the question "$100>¥10" you asked above requires more data, it should be "$100>¥10 @ 144.28". ANYTHING else is a terrible leaky abstraction. Don't do it. Source your rates automatically from a single source if you like, but make it explicit.
Anyway, a "Money" object really is just this: A precise decimal object, with an ISO currency code. The latter simply being a short string among an included, limited set.
Yeah, one more reason for it to be a different type.
> Where is the line drawn?
It's not.
As a rule, measurement unities have an absurdly bad support from computers.
https://hackage.haskell.org/package/safe-money-0.9.1/docs/Mo...
It’s not even clear what the second currency is. US$100 is over 14000JP¥ or 725CN¥.
If you’re offering me to wager whether $100 is more than ¥10, I’ll take the wager that it is.
In EU prices to customers are required to be comprehensive of VAT, so a price is € 60,00 included VAT 10%.
But in an invoice/receipt you have to explicit how much is the net and how much is the tax, so 60 / 1.10 = 54.55 and VAT is 54.55 x 0.10 = 5.46 which makes a nice 60.01.
You may be tempted to round down the 60 / 1.10 = 54.54 and have VAT 54.54 x 0.10 = 5.45 but this makes 59.99.
0.1 + 0.2 <= 0.3 // sure looks true to meAt least until you need to add things, at which point you need Kahan's algorithm.
Same for currency.
If you don't know then there's no point but the idea is to stop you comparing prices from London close to new York and so on
As admin I can still recreate the whole insert table with modified data, etc.
By which time hopefully interest bearing CBDCs will show up, and make all these mindless intermediaries sitting between my wallet and someone elses wallet obsolete.