TigerBeetle raises $6.4M to power the future of financial accounting infra
tigerbeetle.com
tigerbeetle.com
https://github.com/tigerbeetledb/tigerbeetle
Also, if there are any Africa-based devs here (but of course, you're welcome to come from wherever you are) we're running a systems programming conference with Derek Collison of NATS, Jamie Brandon of TigerBeetle and HYTRADBOI, Andrew Kelley of Zig, and many other great folks.
Next week, Feb 9th and 10th in Cape Town. Maybe we'll see you there!
Although at least one of them, Will Wilson's, will be an event-only talk. :)
Adding a few thoughts off the top of my head.
From my experience I can say that money transfers are rarely, if ever, done atomically between two end user accounts. i.e., <Account A debit, Account B> being one operation. Instead the money first moves from Account A to a pooled account, some business checks/operations take place, and then from a pooled account (could be same or different) to account B. The reason to do so is to run a host of business checks which could take quite some time, in some cases even a day or more. Which is why the two phase account transfer semantic of "Reserve", "Settle" scales so well. One can build all kinds of higher order business flows on top of them. Subscriptions, scheduled payments, and so on.
Account transfers are subject to a bunch of velocity limits such as daily/weekly/monthly count and volume of transactions. It depends on a bunch of factors such as user's KYC levels, fraud rules, customers' age in the system etc., An account starts off at a lower limit and those limits are gradually increased as the account ages in the system. This was a big pain to pull off in scalable fashion. The running counter of limits ends up being bottle neck. For example, a popular merchant account doing a few million transactions a day. Subjecting that account to velocity limit checks million times will mean that merchant account will become a hotspot.
Maybe I should do a blog post sometime, just to give a structure to my brain dump :-P.
I'm sure the team would be keen to talk to you!
And yes, you could say that the industry does suffer from some version of NIH. The concept of ledger after all is thousands of years old and yet software engineers don't bother to learn end up mucking up the implementation.
I attribute it to software engineers who are not willing to step back and learn the domain of money and payments. Whenever money is involved one should only have to look how professional accountants maintain their books and handle it, learn from existing products (such as Stripe[1], Modern Treasury [2]) before running off to roll out a half baked version of double-entry book keeping. The system kinda works initially but when the requests increase the errors begin to accumulate very fast and the businesses end up unable to trace millions of $$ or worse. I've seen it happen way too many times.
[1] https://stripe.com/in/guides/atlas/bookkeeping-and-accountin... [2] https://docs.moderntreasury.com/reference/getting-started
TigerBeetle does support two phase reserve/settle semantics. I think we saw a similar problem to what you saw, and wanted to make that primitive first class.
Would love to chat sometime, joran@tigerbeetle.com
https://twitter.com/sw_unscripted/status/1584695054563954689
- io_uring support on the std lib
https://github.com/ziglang/zig/pull/6356
- a cross-platform io_uring based event loop
https://github.com/tigerbeetledb/tigerbeetle/tree/main/src/i...
https://tigerbeetle.com/blog/a-friendly-abstraction-over-iou...
Thanks team
phil@tigerbeetle.com if so.
Same offer/request goes to anyone here who has built or is building this sort of system in-house. Would love to chat.
Just flipped back to the docs again, think I've found it: 'Set user_data to a "foreign key" — that is, an identifier of a corresponding object within another database.' -- This is all well and good, but if you're having to write to this other database at the same time in order to store said other data, doesn't it make your ledger a bit pointless? I'm just struggling to see the use cases. Can anyone help me "get it" ? (For the record, I work with ERP/accounting systems in my day job)
tldr; we anticipate you'll have some database with all your customer information. The metadata fields inside of TigerBeetle currently can be used to link back to rows in your existing database. Or if you don't need an external database and your metadata fits into our metadata byte fields, you can use them as you like.
Just want to say that I'm not trying to be negative - you're a long way down the dev road, and must have thought about all of this, so I want to read about it! :-)
Advantages for using Postgres ( assuming a double entry schema[1] ) and you're using Postgres for your main app db;
- You can do financial transactions inside business db transactions, and roll back both atomically
- Adding up numbers is one of the things computers are best at, and Postgres can easily handle a huge amount of financial transactions
- Re-use PaaS Postgres hosting/scaling/clustering/backups
- Easier integration with the rest of your app with foreign keys to relevant records relating to the financial transaction
- Easy integration with BI tools given Postgres is well connectable
This gem does literally billions of dollars worth of financial accounting for various companies at scale: https://github.com/envato/double_entry
It's dated, the API is a bit messy and needs work, as it was initially written 10+ years ago, but for a web based app I would choose a v2 of it over a non-postgres ( assuming you are using Postgres for your app ) solution.
No experience with the linked gem but the description states
> While this gem acts like a double-entry bookkeeping system, as it creates two entries in the database for each transfer, it does not enforce accounting rules.
And that seems like a far cry from things like "rejecting transfers if the accounts involved would exceed their net debit or credit balances".
> And that seems like a far cry from things like "rejecting transfers if the accounts involved would exceed their net debit or credit balances".
The READEME needs to be updated[1] for clarity. It doesn't enforce "accounting" rules, however The DoubleEntry gem has rules to optionally ensure a balance cannot go negative, and it also has an allowlist of allowed transfers ( which are defined by source account, dest account and code ).
[1] https://github.com/envato/double_entry/pull/216/files
> I think they key part is "distributed financial accounting database". For anything that didn't need to be distributed to handle massive volume of transactions I'd also pick Postgres, but I can imagine there's a limit where that wouldn't work, and as soon as you need to do this in a distributed database things get exciting.
That is why I qualified my statement with "you're using Postgres for your main app db".
For every financial transaction you'd likely have many more business logic level transactions, and you would want your business transactions to be as consistent as your financial transactions. No point storing a financial transfer if you can't match it up with a purchase on your business db.
https://blog.journalize.io/posts/an-elegant-db-schema-for-do...
I hadn't seen that gem before! My comment on most accounting tools is that they ignore the concept of "normal" account balances, which means they don't really think about "debits" and "credits". This is fine mathematically, of course, but makes it awfully difficult for accountants to understand how the software is supposed to behave!
We're running on mid-sized Postgres right now, but find that clickhouse (or duckdb) are something like 100x faster once we do GROUP BYs on 5M+ rows.
Incidentally, I just came across Journalize. But I'm struggling to understand what it's for. It might be helpful to say - either here, or on your homepage - who or what it is competing with. I'm probably not the exact target audience, but it seems to do more or less what Xero does for us (but I don't want to rule out that I just don't understand the homepage).
If your business generates a lot of transactions, to the point that you have a full time accounting or engineering staff whose job it is to wrangle 200MB excel files or complicated sql queries, then Journalize is for you.
We’re basically BI tailor made to create outputs that are relevant to accountants (we know what “depreciation” is supposed to look like) and let them take a monthly ledger entry and drill down to a specific user’s impact on the GL and tie it to the line of source data.
In this way, we’re doing the data engineering work, helping accounting teams have BI type tools to set up accounting rules, and giving transparency to “why” an aggregate number is correct.
TigerBeetle is a new database.
It's designed as a replicated state machine. You will one day be able to swap the state machine (double entry features) for another state machine, and leverage all the fault-tolerance and performance of TigerBeetle.
Also, the choice of zig is great (yay no memory allocation), but this is not the first database to do that. Many older databases did as well. There are advantages to dynamic memory management. But, moreover, postgres, et al, are written in C as well. Zig is not magic. Thus, I don't understand the differentiating factor here. Sure, zig is nicer, and sure, this choice of language makes it more likely to be performant than many new databases, but it sounds like you're trying to compete with the likes of Postgres, and this is then a less differentiating factor.
The paper “Protocol-Aware Recovery for Consensus-Based Storage” is orthogonal to choice of state machine, since it's at the consensus/storage engine layer underneath. So custom state machines benefit from this.
While I love C, and while C was the other choice for TigerBeetle, I wanted to pick a modern systems language to invest in for the next 30 years. There are several advances in Zig around safety: bounds checking, checked arithmetic, error handling, better tooling, cross-compilation, comptime, enforcing memory alignment through the type system, ability to handle memory allocation failure etc.
The big difference compared to Postgres is that TigerBeetle has better performance for the domain (TB can be far more aggressive with group commit since financial transactions in TB are only 128 bytes), a stricter storage fault model, and is a distributed database with automated leader election and fault tolerance. TigerBeetle is also designed to be deterministically simulated, like FoundationDB, so we can test to a high standard. For example, by injecting the theoretical limit (in terms of the consensus protocol) of corruption on all machines, including the leader, and then test that TigerBeetle self-heals.
Cool project!
No formal proof at the moment either but it's something we're thinking about.
Everything is fully deterministic, time is externalized, so our simulation can run very fast and explore a lot of random permutations of events (which is quite effective: https://www.researchgate.net/publication/322133078_Why_is_ra...).
This isn’t as thorough as model-checking or proving a formalization of the algorithm, but it has the benefit of working with the real code. It is a bit less real than Jepsen (as we essentially mock all syscalls), but is quite a bit faster.
Do you have any thoughts on how working on TigerBeetle with Zig is? It seems like for a high performance DB rust would be a natural choice. And how's the development experience without a zig-analyzer?
zig-analyzer is great — much further along than I’d expected at this point. Though, I am still curious what the end game IDE support would look. https://github.com/ziglang/zig/issues/3028 is a big component of that.
Formance is awesome. I've had a few fantastic calls with Clément. I think we see it that Formance operates at a layer above TigerBeetle, whereas TigerBeetle provides the underlying ledger infrastructure. So they are complementary.
That's man-millennia of painfully evolved application code, catering for tax legislation of many countries and The Way Accountants Want It, and database performance/serializability is a fourth-order concern.
Remember kids, novel is not necessarily better
Congrats to the team!
I've tried it out for a couple toy projects and it had showed great potential.
Congrats to the team!
Would love to catch up for a chat sometime.