Show HN: An open-source financial ledger API written in Clojure | Hacker News Reader

Show HN: An open-source financial ledger API written in Clojure

github.com

125 pointsandriosr33 comments

Hi everyone,

I have been iterating a startup since the start of the year. And I'm using Clojure for everything I do.

The first product was a ledger service, but it ended up not fitting well as a managed service. We pivoted to a different idea and decided to open-source the ledger project.

It brings years of fintech experience, with the most common problems faced at scale for a ledger solved into the design.

I hope this will be helpful to some of the companies that are struggling with this kind of system.

Let me know if you have any questions :)

Why did you go with a non-relational database?

Interesting question. No-SQL doesn't mean non-relational. DynamoDB is a relational database. It's up to you on how you design the table to accommodate your relations.

This is a great presentation about this: https://youtu.be/6yqfmXiZTlM?t=1447

DynamoDB is a key-value store. Key-value stores are not relational databases [1]

The link you posted is a really good resource for someone who wants to start using DynamoDB though.

[1] - https://aws.amazon.com/nosql/key-value/

You're right. DynamoDB is not a relational database. But data in DynamoDB can be relational just as in a regular relational database: https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

That's really stretching the definition of a 'relational database'.

You can create relationships on flat text files. It doesn't mean that you have a relational database now.

Many have fallen for that marketing video stunt and later regret it.

Sequence has interesting applications for longer-term work I'm on, and/but I'm Baader-Meinhoffing [0] re more-immediate small-biz accounting ledger needs: I spent a good chunk of yesterday in gnu-cash and Frappe books [1] working to import/tag/script Chart of Accounts and transaction info from two banks while aiming to avoid (new-to-me) UI, code base, or Generally Accepted Accounting Principles rabbit holes.

Sequence is clearly an API and I ought stick with consumer packages @ immediate needs. I'll spin up the container to check it out in general; Please let me know if I'm mistaken / you recommend specifics it'd be helpful for me to check out and feed back on -

[0] - https://en.m.wikipedia.org/wiki/List_of_cognitive_biases#Fre... [1] - https://github.com/frappe/books

You're right. Sequence is a ledger API. These tools will give you higher abstractions on top of their internal ledgers. I think you are on the right track. If you want something extremely simple maybe the Ledger CLI [0] is worth looking at. Try to leverage these higher level abstractions. You don't need deep accounting understanding to import/tag/script a few transactions.

[0] https://www.ledger-cli.org/

Nice clean code! Thanks for open sourcing. I was a bit hoping to see how you were calculating amortization/interest charges in Clojure, but it appears to be only a pure ledger. Out of curiosity, were you using hashing to verify transaction chains? I saw code for MD5 hashes, but not where it was used.

Thanks! I have interesting ideas for amortization/interests, maybe they can get added at some point. Yes, the hashes are the ids used for Avoiding double-spending along with dynamodb conditional put. I’m on the phone now, will follow up with code once I get to the computer

Amazing work there. I have looked at your code and it is very nice.

Fun thing, I am currently building a similar ledger in clojure. The task is much harder then expected. My goals are very different then yours; I would like to implement a video game around a fictional financial system. Consequently, It is very important to me to implement the concept of "interest rates".

My solution is over engineered power 25. Nevertheless, you might be interested into it. It is based on datomic. https://gitlab.com/didiercrunch/skirmish/-/tree/master/src/c...

Wow, this is very interesting! Will look into it, I have plans to add interests to Sequence at some point, this will be of great help.

I have seen your issue about datomic. To be fair, I am bit disappointed about datomic in a real life situation. If I were you, I would try to implement a postgres backend instead of a datomic one.

Interesting take. I think Datomic is a low hanging fruit. A lot of people get excited about it and is fairly easy to do. That being said, I saw quite a few examples of companies migrating from it. Complexity to operate for simple things being the biggest reason. What is your experience with it?

Nice work! Very well done

Why use nosql for a ledger?

I was wondering as well. How does that work with eventual consistency? Shouldn’t there be double bookkeeping so you don’t generate money out of the air?

Also it seems like the amount is of type number. Ideally you would calculate in Dollarcents or Eurocents (I.e. solely use integers) like

100 Eurocents equal 1€

So you do not have any rounding involved

Also if a request comes in twice there are no requestids involved. Therefore it might be possible to generate another transaction it seems.

There are great blog posts from Stripe engineers about these request ids to make requests idempotent with Postgres databases. I highly recommend reading them.

Not sure about how Clojure treats numbers, but in DynamoDB these are treated as Decimals and don’t have floating-point rounding errors.

Does it allow to save values with floating point imprecision such as 1.000000001? I don't think the issue is never in the database but the service that uses the API to send values. If such value is allowed, it will cause headache and errors when the consumers of the API make mistakes. I think the general practice is to store them in integers, just for the sake of being foolproof.

Yep. Integers they are: https://news.ycombinator.com/item?id=24435148

You pass “numbers” to DynamoDB as strings defined with the type N. As long as you aren’t using a floating point initially, DDB isn’t going to add any floating point imprecision itself.

Two great points. The amounts are actually integer, the same pattern used by Stripe and others. See it here: https://github.com/decimals/sequence/blob/master/src/decimal...

I'm working on enabling idempotent requests, should come out next week. I'm using the same pattern used by Stripe as well (big fan). An optional `x-request-id` header that will ensure only once processing if present. This is very important to enable consistency over the network with other services using the ledger.

It maintains double-entry consistency of balances with DynamoDB. I uses DynamoDB (or any No-SQL) conditional puts. You can think of it as a update + where of SQL databases. But infinitely more scalable and available.

Here is a piece I wrote about how it's achieved: https://decimals.substack.com/p/things-i-wish-i-knew-before-....

Is DynamoDB baked into the product or could new adapters be written to use different key/value stores? Things like leveldb for local-only ledgers or even an adapter on top of PostgreSQL's jsonb field might be useful.

It was designed to support multiple DBs. Adding a new adapter should be pretty easy. I will add the first for Datomic in the coming weeks. This should pave the way for the next ones (I will add some docs on this). I really believe on it supporting many DBs.

just created an issue on Github to keep track of this :)

Was my top question as well, but note the underlying DB is Dynamo, which supports transactions, and a quick peek at the code looks like it's using them as intended (I don't know closure and it's been ages since I've used Dynamo so take with a grain of salt).

https://github.com/decimals/sequence/blob/master/src/decimal...

It uses DynamoDB and transactions. Each double-entry appears to be 4 items, 1 for each balance update and 1 for each transaction.

Each “journal” will use 2x4 WCU for the write to the table and 4 WCU for the global index updates (a total of 12 WCU for each journal).

I do have a few issues with the structure though. They seem to use LSIs to add a timestamp (instead of just using KSUID or similar for the ID), which ends up limiting each partition to a maximum of 10GB (a limit that doesn’t exist if you don’t use LSIs).

10GB might seem like a lot per account (which seems like the PK the partition is based off), but even at a conservative 200 bytes per transaction, that limits a single account to 50 million transactions forever.

Coupled with the fact this only seems to support literal double-entry accounting (not 2+ balanced transactions per journal item), you could quickly hit that limit if you are processing multi-item invoices for instance where items end up being allocated to different sales accounts.

Although if you were processing multi-item invoices you’d also have to work out how to make this transactionally-safe yourself, as this only guarantees ACID-compliance of a single DR/CR transaction.

Unfortunately, this wouldn’t necessarily be as easy as just allowing extra entries to a single DynamoDB transaction as those are limited to a max of 25 (and as covered, each entry requires 4 item updates).

DynamoDB isn’t actually a bad idea if you need an ACID-compliant database with read-after-write consistency. But I think the model needs to be rethought a bit to deal with some of the other design considerations DynamoDB forces on you.

Wow, that was a great review, thanks! I agree with most points.

There are definitely improvements to be made. This is the first version, I wanted to get it out and start getting feedback sooner. This is my take on each point:

For some reason Dynamodb was not sorting the partition with KSUIDs string, so I added the LSI. This is probably something I did wrong and is just matter of fixing the query or the insert and move back to using the PK.

Adding more than one balanced transaction per request is on the roadmap, but this will be limited to the DynamoDB transactions API limit of 25 items as you mentioned (25 / 2*2 WCU per tx = 6 transactions per request).

Each entry only uses 4 items in the worst case (seeding a new currency in an account). In most cases (existing balances/currencies), there will be only 2 new items.

I think even more improvements can be made. And the goal of open-sourcing is to get more heads thinking about how we can do it :)

I wrote a piece that touches a bit on the why: https://decimals.substack.com/p/things-i-wish-i-knew-before-...

But in a nutshell: availability and scalability. Yes you can achieve this with relational DBs, but it will be quite complex, while No-SQL gives it by design. All while ensuring all the goodness of relational data modeling, consistency and queries. The only requirement is a good table design: https://www.youtube.com/watch?v=6yqfmXiZTlM&t=1s

this is pretty cool, love it

Reply on news.ycombinator.com