TigerBeetle – A Million Financial Transactions per Second in Zig
youtube.com
youtube.com
TigerBeetle is still alpha and we're moving to beta now with automated fault injection testing of our consensus and replication. The project has been a year since ProtoBeetle in July 2020. Regarding multiple regions, we have some novel Viewstamped Replication optimizations that we hope to write up soon and document in our repo.
However, our motivation for managing clock faults was not because we wanted to do leader leases. We don't risk stale reads so we don't do leader leases, i.e. where the leader can operate as a lone ranger without consensus from the cluster for a period of time. We're not comfortable with what could potentially go wrong when stale reads are allowed to become part of the fault model.
The motivation for having some kind of clock fault tolerance was rather so that TigerBeetle can clamp the system time to cluster time for reasonably accurate transaction timestamps for auditing purposes, and for reliable timeouts of our two-phase commit ledger transfers so that liquidity is not locked up forever.
In other words, the only reason we synchronize clocks across the cluster is so that a leader with a far-future clock can't push the cluster's monotonic clock out into the far future for unrealistic timestamps or to delay transaction timeouts (or expire them too early if a new leader's clock is stuck in the past).
At present, TigerBeetle can already be used to limit debits/credits against an account to not exceed the credits/debits of the account by setting
account.debits_must_not_exceed_credits = true
If you wanted to rate limit transactions, you could create something like a "prepaid" account where you periodically top up the account, and then let transfers drain it out.
Would these address something like what you were thinking?
TigerBeetle also has support for something called Linked Events where you can submit a batch of either accounts, transfers or commits and where you can then chain a subset of events within this batch together by toggling a "linked" flag on each, so that these subset chains succeed or fail atomically together, much like what you can do when submitting IO through io_uring: https://github.com/coilhq/tigerbeetle/blob/main/src/tigerbee...
We didn't go into our accounting features too much beyond the big idea because we wanted to focus more on the technical aspects for this talk.
You want to promote your micropayments and therefore give your customers for example $1 as a gift and they just need an email address to sign up on your platform.
Once they have $10 in their account they have to at least give you their phone number/address before they can trade more money. Now you have 12 customers sending money to customer A who has now $12 in their account. In the meantime customer A sent out $1 to somebody else.
Depending on who sent, got the money first you have breached the $10 limit.
And I don’t grasp how this use case can be met with double bookkeeping besides using a single synchronous ledger (or maybe virtual accounts to do reconciliation). If you now have customer A maybe in USA and customer B on another continent I am even more clueless about latency. :)
If the goal of the policy was to prevent customer A from staying under the KYC threshold by constantly shifting received gifts elsewhere, then I think it should be possible to account for this by giving customer A two accounts, one that they receive gifts into (capped to be no more than incoming 10 credits), and one that they send from (capped to be no more than 1 outgoing debit).
KYC would then be required before customer A can receive any more gifts or be transferred into a full-duplex account. Can you spot any bugs in this approach? There might also be better ways to model a chart of accounts for this scenario. And if you have any more scenarios you can imagine and send through, I would also really like to work through them to see what we can learn.
Regarding latency:
If payments are global, then I believe that any system would need to incur latency at some point when updating both accounts if they are homed in different jurisdictions.
At this point, is your question about global cross-continent latency differences between active Paxos replication vs passive Multi-Paxos replication?
Or are you more comparing the latency of a strict serializability RSM like TigerBeetle vs the latency of something like a byzantine fault tolerant payment system that uses cryptographic signing and sharding but which cannot do things like payment-versus-payment?
I was thinking that maybe my use case is maybe something which should not be solved in the ledger directly, but merely the ledger should offer an API to fulfill transactions in the ledger. And the Api consumer has to provide an idempotency key to perform the transaction . But then again I would assume that the ledger would have to provide an endpoint what the current balance is (the sum over all transactions). And when the api consumer makes a decision depending on KYC and current balance, whether to try to record a transaction on the ledger, in the meantime the balance could have changed. So maybe one could send an idempotency key and the balance which was used to decide whether to fulfil the transaction. Or a more generic version: an api consumer could send an idempotency key and a smart contract compute kernel on a transaction request. And the smart contract gets evaluated on the fly with the live data inside the ledger.
I smell like I am inventing a database and SQL here. ;)
Regarding: “ KYC would then be required before customer A can receive any more gifts ”
Where would the validation live? Inside the ledger as a contract per customer across their accounts? When customer B tries to send money to customer A and A fails: what will customer B see? An immediate error or an asynchronous message
The back-of-the-envelope calculation for the latency per 10,000 transfers would be:
RTT from leader to the fastest follower +
ms to send 1.2 MiB over a 1 Gbps or 10 Gbps network +
ms to fsync 1.2 MiB on NVME
The cross-region replication can be asynchronous (configurable) which is also how I believe LMAX do it, or you could have your backup region be a few hundred miles away as with AWS regions.
"But then again I would assume that the ledger would have to provide an endpoint what the current balance is (the sum over all transactions)."
Exactly. One of TigerBeetle's contributions is to colocate data and policy across a chart of accounts. It's different from a "ledger" in that TigerBeetle is not only accounting books but also accounting policy. The system of record is the optimal and safest place to make decisions, and this avoids the overhead of distributed transactions. This provides a 50% throughput improvement just by cutting out the unnecessary interim query. Instead of amplifying network queries to make decisions away from the data, you design a chart of accounts with simpler primitives to control transfer flows between them.
For long-lived transactions that span multiple external systems, TigerBeetle provides two-phase transfers as atomic primitive, which is what many payment systems use, as I am sure you would know. These two-phase transfers also support 256-bit SHA256 hash locks, e.g. ILPv4 conditions, which I think could be used to support some interesting logic outside the database.
This all enables the application layer to be completely stateless and scaled shared nothing. You could literally spin up a hundred API instances in front of TigerBeetle, all shared nothing and ephemeral.
"When customer B tries to send money to customer A and A fails: what will customer B see?"
An immediate error: "exceeds_debits". This can trigger the KYC user flow to be queued in an external message broker service. You can see exactly how this all works here: https://github.com/coilhq/tigerbeetle/blob/main/src/state_ma...
Maybe this link is also interesting for you
Ahead of performance, we are especially excited about TigerBeetle's safety fault models, where we think there's an opportunity to break new ground, with all the new storage fault research that's been coming out the past two or three years, especially from UW-Madison.
Zig has been fantastic, in particular for Direct I/O alignment and static allocation, and it's only getting better with things like the self-hosted compiler. Having done a bit of security work in the past, I really like Zig's unique approach to safety, for example checked arithmetic enabled by default in safe builds, all the explicitness around return values and exhaustive syscall error handling, and of course comptime. The speed and ease of the compiler itself and cross-compilation tooling is excellent and the readability of Zig is also remarkable.
Huge props to the Zig Software Foundation for what they are achieving as a fully open source community funded project.
More performance allows you to trade expensive hardware for cheaper commodity hardware to gain lower operational cost (e.g. a cluster of three Raspberry Pi's) while still having more than enough headroom to handle volume spikes safely (e.g. Black Friday).
With digital payments, payments are also becoming more global, with higher volume and lower value, but existing infrastructure is not typically designed for this high volume low value performance profile (e.g. millions of payments at a fraction of a cent).
Finally, high throughput allows you to easily ingest huge amounts of historical data into TigerBeetle in a few seconds.
The design actually made this kind of throughput fairly easy to achieve, even for three small office laptops. The hard part is all the safety work that we're doing to make this hopefully many times safer than existing systems.
[1] "Coil’s Mojaloop Performance Work 2020" - https://docs.mojaloop.io/documentation/discussions/Mojaloop%...
Thanks also to Loris Cro for hosting us on Zig ShowTime. We had a blast! :)