Why you might want a domain-specific database like TigerBeetleDB

You nailed it!

And this is in fact our design for TigerBeetle, an Iron Man suit that you can put any state machine business logic into. You get global consensus protocol and local storage engine, with all the performance of TigerBeetle, and it's a really nice experience writing your own business logic inside of that. You can even test everything using Deterministic Simulation Testing.

Long term, we want to extract this into a library. Reading the TB source, you'll see all these abstractions are something we're thinking about.

Aren't things like LMDB and RocksDB sort of modular parts you can build databases around?

Not just databases; this is a failing in our tooling more generally. "All" you need to do is code something flexible (in contxt, a feature-rich database), describe the (maybe probabilistic) things you know at compile time, and partially evaluate the first program optimally according to those constraints.

It's the same idea behind the Futamura projections, and unless there's something special that makes tailor-made databases significantly less diverse than arbitrary programs I doubt you'll see solutions here that allow you to mix and match very many components until the general case is also solved.

We do that for Cheméo, we use SQLite (or if the customer want PostgreSQL) as data store and query engine where it fits but some special indexes for some particular queries like a similarity search[0].

The advantage is that you can use the battle tested SQLite/PostgreSQL (companies love this "standard") and on top your domain specific engine which is stateless (we rebuild the indexes directly from the database, this way we have only one single source of truth and backup is easy).

[0]: https://www.chemeo.com/similar?smiles=CCC%28CCO%29CCC

https://github.com/tigerbeetledb/viewstamped-replication-mad...

* Very slowly changing validity requirements. Things like (optionally) preventing overspending are things that are pretty intrinsic requirements of the domain (depending on the context, obviously many banks allow overdrafts so they can charge you a fee for spotting you the difference). The validation space is relatively small such that you'll want to allow most transactions that aren't blatantly wrong

* Emphasizes correctness and ordering over everything else

* The transaction rate is relatively inelastic, that is there aren't huge swings on how many transactions are requests per second such that dynamic scaling is not a desirable property to have if it means being less reliable/correct. Conversely, it's a known slow operation that most people are fine with not being instant. Ex. I'd much prefer my rent payment take 10s to process and be more or less guaranteed to be correct than it take 1s and not.

That's why it's domain-specific. You can optimize for the things you care about in your domain most while reducing operational complexity to the minimum viable to meet those demands.

It's amazingly well-focused and solves what I assume are hard, limiting problems that the existing financial infrastructure is not equipped to support.

As you zoom in, you will see differences. For example, TigerBeetle's data structures are all cache line aligned, and we use static allocation etc. The storage fault model is significantly different.

We use the same testing techniques though. FDB are pioneers in the space.

We do in fact have some extremely advanced testing infrastructure, even going as far as using a deterministic Linux hypervisor to do coverage guided fuzzing of our compiled binaries from the outside in. At the same time, we fuzz from the inside out, also Deterministic Simulation Testing, with a ton of assertions (literally a 1000+ and counting) as a force multiplier for fuzzing. Our experience in all this, is that while Zig is early in terms of timeline, the quality is nevertheless extremely high.

Andrew and team know what they're doing. They've got some of the best people in the world in their respective fields. For example, Frank Denis of libsodium heading up Zig's crypto.

Beyond this, we do the basic things like consciously restrict our use of language features to stable features only (which may be why our experience has not been the same as yours with respect to compiler bugs?), and invest in our own I/O stack around io_uring instead of depending on the std lib, which we know will churn.

Two of our team are Zig core team members and we sponsor the Zig Software Foundation to invest in the ecosystem. Don't forget also that Zig's ecosystem is really C's ecosystem so there's an escape hatch there.

Also, TB is not yet production ready. We'll ship our production release when we're confident that our TB binary is safe, or able to shut down if it detects any safety violation.

At the end of the day, databases are a big investment. It's important to think about the next 20 to 30 years. C would absolutely have been the wrong choice for that future.

As you say, the future is bright for Zig, and we're convinced that Zig is the right choice for a database that follows embedded coding standards with respect to memory. For example, TigerBeetle has to handle memory allocation failure, and only does static memory allocation so we never malloc or free after startup.