Types of events in event-driven systems
blog.frankdejonge.nl
blog.frankdejonge.nl
Calling them RESTful events is a good term as I’ve pitched in much the same fashion. Every time a system responds to a create/update/execute request, drop an equivalent message into an event stream. Initially these can form the basis of a streaming ETL to whatever systems you use for analytics, but as new internal systems are brought online (e.g a new CRM system to help sales manage all the leads signing up for your successful product) the integration is already there, waiting. Just add a new consumer to the existing stream.
Overtime, you end up with a “data mesh”. If you look past the marketing of the term, it’s basically just means each application in your company should publish both sync (for real-time usage) and async (for data sharing between systems of record) interfaces.
To solve that issue you need a cooperating destination for your writes which handles de-duplication. Then you get into the weeds of causality, ordering and idempotence. Ugh.
For some real world examples see the AWS Kinesis documentation which says that any application using Kinesis must be able to handle duplicate records [1].
> There are two primary reasons why records may be delivered more than one time to your Amazon Kinesis Data Streams application: producer retries and consumer retries. Your application must anticipate and appropriately handle processing individual records multiple times.
[0]: https://en.wikipedia.org/wiki/Two_Generals%27_Problem
[1]: https://docs.aws.amazon.com/streams/latest/dev/kinesis-recor...
This is really not that different than using ETL jobs to move data about, but taking a streaming approach vs a batch approach.
The event stream layer isn't where the sync problems have arised in the systems I've worked on. It's the "commit transactionally both to your database and the event stream" part. Not a lot of systems are built to be ready to roll back the database change if the event publish fails. Or to be able to handle duplicate events if you error the other way.
Just need to make sure to not mess up causal ordering between events because of out of order retries, if such things are important for your application.
E.g. what if one event is something like "credit here, debit there" - you need to process it sequentially for both sides!
Ultimately, whenever you are pushing data between systems you can end up with inconsistencies which why it’s important to clearly define systems of record.
It doesn't give you any guarantees about "at a given time" consistency, though. Maybe your queue or your event processor got backed up. So how real-time do you need both sides?
These are, of course, problems that people have solved to varying level of satisfaction for most use cases! But I've seen systems built by people who saw the blog post about "have both!" and then didn't think about all these cases and then... it gets messy.
The pattern I usually see is wiring this the other way - use an initial event as the trigger to write to the DB, while also streaming the DB's binlog or equivalent back as events. Now the risk is that another service gets a "too fresh" view but this is usually less harmful than a stale view. Things listening to the binlog events need to process them idempotently, but this is usually not a major complication since most queue designs will require you to be prepared for that anyway.
Don't we call that just "a WAL"?
What exactly a WAL contains depends on the specific storage, but it's commands at least as often as it's state - probably more often since you want your WALs to be tiny and they often also facilitate rollback.
In general you wouldn't want to emit a fat event until the data is firmly committed to its system of record which is usually not the event system itself; in that use case it will be a kind of 'write behind' log.
If I am using an event as a "record" of an event, I tend to store a "before" state, and an "after" state. In many cases, I can actually store entire serialized objects. This allows, for example, the ability to recreate an object state/context, simply by re-instantiating the stored object, and using the reanimated instance to replace the current one.
Sort of a RESTFul event.
In my work, whenever possible, I like to use a "RefCon" pattern. This is a classic pattern, that I learned from working with Apple code, but I know that it's a lot older.
We simply attach a "RefCon" (Reference Context) to an event. This is usually an untyped value, that is supplied to the event source, when it is set up, then, to the event target, when the event fires. These days, it's easier, as closures/lambdas often allow you to access the initiating context. In the old days, it might have simply been a single-byte value, that could be applied into a lookup table, to allow the called context to re-establish.
I recently ran into the need for one of these, in an SDK that I wrote, where I didn't. I'm still kicking myself. I think that I didn't supply it, because I thought it might have been too complicated for some consumers of the SDK.
In the six years or so, since I wrote that SDK, I have been the only consumer.
The events that a subscriber is interested in are decorated with the exact data the subscriber specifies at subscription time, avoiding an API call while providing exactly the level of richness the client requires.
I’m a little surprised that Graphql subs aren’t more widely talked about.
I'd further advance that
- Trigger events and RESTful events are roughly the same thing, just a question of what you choose in latency vs. size vs. schema flexibility space. We even have events in our system whose schema inlines data below a certain size but links above that.
- There is a fourth type: windowed reductions over domain events, e.g. the "changelog" in Kafka Streams. This bears a similar relation to domain events as the RESTful event does to trigger events.
With regard to your first point, in terms of communication, they serve very similar integration patterns. I imagine you've standardised the consumption of these events, making it transparent if the event is linked or attached. Is that correct? In such cases, there's one difference, which is the out of sync state; the situation where the state has been altered after the signal was dispatched but before it was consumed. Is that something you deal with?
Yes, a pretty thin layer that returns the data if it was attached to the message or makes an HTTP request if not.
> the situation where the state has been altered after the signal was dispatched but before it was consumed. Is that something you deal with?
Our most common use case is trying to keep local caches of e.g. control plane data up-to-date, so retrieving something more recent than existed when the event was produced is usually a bonus. In the rare cases it is a concern we make sure the external data link is unique (e.g. into an S3 bucket with an expiration policy a bit longer than the event retention time).
> For me those are only internal and not meant to be communicated, rather they are there to speed up reconstitution of an event-sourced aggregate. I might be misunderstanding your point tho.
I think you've got the point. As these are implemented in Kafka Streams they are also an event source themselves which can be consumed like any other topic's messages.
Note that "sensitive" in this case can mean a persistence/retention boundary and not a security boundary.