Getting Started with Event Sourcing in Postgres

We rather prefer to have a subscriber who re-runs the projections which are purely functional, and dump their state.

This mandates that you have some way of querying relevant events from the underlying store. Sometimes the best store for this is a DBMS, sometimes it's something simpler like CAS linked lists in Redis or similar (lists of hashes pointing to keys, similar to git's ref/tag model)

Either way, this idea that you subscribe to a stream makes reconnection and place-holding (cursor) semantics difficult, and simply calling as-frequently-as-makes-sense your update function is simpler to reason about, and helps highlight performance issues by making it easier to see the perf of frequently called short lived processes vs dealing with any VM bloat of your choice of language's runtime managing long lived subscribers.

Caveat: naturally this is all very subjective, just my 2 ¢

Those are always the best kind. Real, live, messy, production code. You should open source it. We will learn something

That's a bit like saying Historians know everything, since every known fact is in the past and therefore history :P

I think the point is that the events are kept and can continue to be used as the primary source of truth, rather than used and discarded.

https://martinfowler.com/eaaDev/EventSourcing.html

This is an article I have from 10+ years ago and its still roughly the same way I do it now. I've never needed to scale it beyond something like Postgres or Redis.

A better option is to only include the changes in the event payload, in which case you don't need to know the original state and you can simply project them all in order - and if one makes the next impossible, it's first come, first serve. Again, though, depending on your data model, generating an event payload that only includes the changed elements may also be a pain.

Indeed. For postgres it's much better to use https://www.postgresql.org/docs/current/static/logicaldecodi... to convert the WAL into a consistently ordered (commit time, with no issues w/ non-discrete or out of order clocks) stream of changes.

What is depends on is how much horizontal scale you expect to have. At a small scale, you can use something like postgres as a centralized broker of sequence. A single postgres cluster can sequence a lot of records in a hurry, no problem.

At a larger scale you might need to decompose your domain into multiple, independently ordered aggregates. Now you have multiple choke point brokers, each of which can handle a lot of records, but which don't block each other.

Step it up another notch, you can use a consensus algorithm (paxos or raft) to get a cluster of machines to agree on the state of the sequence, and perform optimizations such as assigning blocks of sequence by shard and node.

I recommend not over building your event storage architecture until you measure your actual needs. There are cheap ways and huge expensive ways, and building too much is an easy way to make your project fail. Also, one of the nice things about a sequenced set of events is that it is pretty easy to replay into your new, faster event storage later once you've had the happy problem of too much success.

Eventsourcing requires really, really detailed analysis of your boundaries (and no small bit of experimentation) to get right, even when you've done it before. It should really only be applied to the areas of your application that are going to see a real benefit from it. My proxy for measuring this is "how much money does this group of features make for me?" If the answer is some version of "none" or "none, but its a requirement to have it anyway" then I don't try to apply ES to that area of the application. I never use ES on user records anymore, for example.

I'm sidestepping your actual question, but mostly because your use-case is something I consider to be a bit of a canard. To answer it though, as others noted, "it depends"—I model projections with actors, which means all events DO come through a single process handling events sequentially (and this is a very common way to do it.) But because I've got tons of these actors (one per stream that's actively running), there's no real bottleneck—this is also one way you guarantee ordering within a stream.

Higher up in the stack, at the eventstore level, you'll probably have a pool of processes handling writes and routing events to the correct handlers/projections, but this is an implementation detail that can vary widely. The advice to not get bogged down in the details early on is good—the thing i see almost every developer new to ES do is get mired in the basic "how do i implement this" question, and that's really not what's going to kill your project. Not understanding nearly as much about your business and the software boundaries in your system is where most projects go wrong.