SierraDB: A distributed event store built in Rust | Hacker News Reader

SierraDB: A distributed event store built in Rust

tqwewe.com

74 pointstqwewe13 comments

Very excited to see how this progresses! Honestly, it's always a little surprising to me that event store architectures aren't more widely used. The article is extremely correct about why:

> there's absolutely no clear cut way of approaching it for new projects

That's definitely my experience - there's no open source "batteries-included" event store where you can just `docker compose up` and start sourcing your events right away. (Maybe KurrentDB née EventStoreDB might offer something like this? But unfortunately, it has a weird license and feels heavily pivoted towards SaaS). And if you want to implement it yourself, a lot of the writing about event stores comes from the the Enterprise Design Patterns™ world.

Yes exactly, I heard from an existing KurrentDB customer that the weird licensing change was actually a deal breaker causing them to move away from KurrentDB despite the migration pains.

I think a community, open source built project in Rust has been a missing piece I can hopefully help to solve.

I've wanted to do this for the last 7 years and never got around to it. Couldn't be more proud that you put in the grit to bring this to fruition!

> Here's where SierraDB diverges from traditional distributed databases: reads don't require quorum.

Would we say this is divergent? Cassandra, DynamoDB, and many others allow you to specify the consistency of reads at the request level.

> Here's where SierraDB diverges from traditional distributed databases: reads don't require quorum. Instead, each event stores a confirmation count in its metadata. When a write achieves quorum, a background process broadcasts this confirmation to all replicas, updating their local confirmation counts. This means any single node can serve consistent reads without network round-trips - a massive performance win.

I have no context outside of this blog post, but this seems actually divergent from the typical definition of consistency given its not linearizable. What systems benefit most from this low latency stale-but-ordered consistency guarantee?

Does it also have snapshot capabilities, as mostly over time it becomes very difficult to replay events due to the shear amount of them.

I haven't put any effort into any kind of snapshotting capabilities yet, since I won't want the scope to be too large and there's often ways of designing your system in ways where replaying isn't a big issue (the db scans are fast, designing aggregates to have a smaller scope/less overall events, etc).

But with increased resources, I can see some solutions being considered around snapshotting. For now the goal is heavily unix philosophy inspired: a really fast and purpose built event sourcing database

This looks really cool!

Have always wanted to dip my toe in EventStoreDB/Kurrent but this looks super intuitive and nice to use. Especially like the js projections, can imagine it being really handy in prototyping or ad-hoc reporting.

Hi.

I feel like I've been using XTDB as an event store?

https://docs.xtdb.com/intro/installation-via-docker.html

SierraDB looks closer to Rama than XTDB https://blog.redplanetlabs.com/2024/01/09/everything-wrong-w...

XTDB doesn't currently solve the problems of user-defined projections (via stored procedures, triggers, Incremental View Maintenance etc.) or multi-partition scaling.

In memory partitions, yeah?

It's persisted to S3 storage, but SlateDB feels like it might sort of have some fit, maybe, as a scale-out distributed LSM-tree. https://slatedb.io https://news.ycombinator.com/item?id=41714858

There's an old 404 post too that looks like a reasonably on target introduction: Why SlateDB is the right choice for Stream Processing. https://web.archive.org/web/20241102212325/https://www.respo...

Interesting point about SlateDB - I've been thinking about how different architectures handle event sourcing and stream processing. SierraDB's append-only model with fixed partitions is really compelling for event sourcing, but I'm curious how it compares to something like SlateDB when you need more general-purpose streaming capabilities. Do you think the trade-offs between these approaches are starting to converge, or are they solving fundamentally different problems? Also, SierraDB's use of RESP3 is smart - anything that reduces client complexity is a win in my book.

Was a bit surprised to see this end in a docker instruction rather than k8s given the emphasis on multi node, replication etc.

Makes sense to me, lowest common denominator across people who deal with containers is probably basic docker usage. So providing instructions with docker lets people using Nomad pick it up as quickly as people who are using Kubernetes, or any other runtime.

Reply on news.ycombinator.com