Why use nosql for a ledger?
Each “journal” will use 2x4 WCU for the write to the table and 4 WCU for the global index updates (a total of 12 WCU for each journal).
I do have a few issues with the structure though. They seem to use LSIs to add a timestamp (instead of just using KSUID or similar for the ID), which ends up limiting each partition to a maximum of 10GB (a limit that doesn’t exist if you don’t use LSIs).
10GB might seem like a lot per account (which seems like the PK the partition is based off), but even at a conservative 200 bytes per transaction, that limits a single account to 50 million transactions forever.
Coupled with the fact this only seems to support literal double-entry accounting (not 2+ balanced transactions per journal item), you could quickly hit that limit if you are processing multi-item invoices for instance where items end up being allocated to different sales accounts.
Although if you were processing multi-item invoices you’d also have to work out how to make this transactionally-safe yourself, as this only guarantees ACID-compliance of a single DR/CR transaction.
Unfortunately, this wouldn’t necessarily be as easy as just allowing extra entries to a single DynamoDB transaction as those are limited to a max of 25 (and as covered, each entry requires 4 item updates).
DynamoDB isn’t actually a bad idea if you need an ACID-compliant database with read-after-write consistency. But I think the model needs to be rethought a bit to deal with some of the other design considerations DynamoDB forces on you.
There are definitely improvements to be made. This is the first version, I wanted to get it out and start getting feedback sooner. This is my take on each point:
For some reason Dynamodb was not sorting the partition with KSUIDs string, so I added the LSI. This is probably something I did wrong and is just matter of fixing the query or the insert and move back to using the PK.
Adding more than one balanced transaction per request is on the roadmap, but this will be limited to the DynamoDB transactions API limit of 25 items as you mentioned (25 / 2*2 WCU per tx = 6 transactions per request).
Each entry only uses 4 items in the worst case (seeding a new currency in an account). In most cases (existing balances/currencies), there will be only 2 new items.
I think even more improvements can be made. And the goal of open-sourcing is to get more heads thinking about how we can do it :)
But in a nutshell: availability and scalability. Yes you can achieve this with relational DBs, but it will be quite complex, while No-SQL gives it by design. All while ensuring all the goodness of relational data modeling, consistency and queries. The only requirement is a good table design: https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1s
https://github.com/decimals/sequence/blob/master/src/decimal...
100 Eurocents equal 1€
So you do not have any rounding involved
Also if a request comes in twice there are no requestids involved. Therefore it might be possible to generate another transaction it seems.
There are great blog posts from Stripe engineers about these request ids to make requests idempotent with Postgres databases. I highly recommend reading them.
I'm working on enabling idempotent requests, should come out next week. I'm using the same pattern used by Stripe as well (big fan). An optional `x-request-id` header that will ensure only once processing if present. This is very important to enable consistency over the network with other services using the ledger.
Here is a piece I wrote about how it's achieved: https://decimals.substack.com/p/things-i-wish-i-knew-before-....