Hacker News Reader

Howdy! Author here ^_^ I'll respond to a few items because, even though I haven't touched the system in a few years, I could still rant endless about the mistakes I made building it. Deep scars were acquired!

Firstly, to make sure we're talking about the same thing, where are you setting the bar for whether or not we can call something "Event Sourcing"? For instance, just to clarify, in our system events were indeed the source of truth. At the heart of everything sat a ledger, that ledger was stored in a database, and events were added to it as you describe in points 1-4 (hand waving away minor differences). You got a consistent, up-to-date state of an aggregate by replaying the ledger in order and applying all the events. So, to my understanding, I'd call that textbook. However, the problem may lay in our actual definition of the word.

>You don't need to cache the write model state anywhere, as the state is in events

I'd have to understand a bit more about where you're coming from with this one. While, yep, the state is in the events and you get the latest state by playing them back, that materialization is not free. If you're doing it for a single aggregate, then it's generally not a huge deal at small scales. However, once you need to make decisions against more than one thing, or compare things, or query things, that materialization cost becomes prohibitive, and you've got to set up materialized projections. With our tech stack, that had to be an eventually consistent process, and what got us into event streaming, and which ultimately caused most of my ugly crying.

As someone who's evaluating event sourcing for a real business use case, I really appreciated your article. I wish more people published their event sourcing war stories, rather than most resources being exclusively from consultants selling snake oil.

I think the point OP is making is actually the same as what you conveyed in your article--several of the worst pain points you experienced aren't inherent to Event Sourcing, they're due to a particular implementation of Event Sourcing. The problem is that that implementation is pretty much the canonical one sold by Martin Fowler et al.

I can think of a few specific ways in which event sourcing could be different than what you experienced:

1) Distributed computing is always hard. Event Sourcing is not a magical cure-all to distributed computing woes, but it's sold as that, and the result is that people go into a distributed event-driven architecture without being prepared for the engineering challenges that await. Eliminate distributed computing, and many problems simply go away.

2) Eventual consistency is always hard, but Event Sourcing doesn't inherently require eventual consistency, even for cached projections. If your projections are pure functions folded over the event stream, then caching and updating the materialized projections with strict consistency is straightforward. In the same transaction as inserting a new event, you run your projection function once on each of the materialized projections and the new event (`state = projection(oldState, event)`). This is only possible when you're not trying to create your projections in a distributed fashion, so see #1.

3) Language choice makes a big difference for reducing the cognitive overhead of updating projections. Your compiler should be able to tell you every place where you need to update a projection to account for a new event. Preferably, you should be able to define a hierarchy of events so that it only yells at you about places where the event is actually relevant.

Thanks for answering! The bar is as I described. The events are the source of truth, only if you're using them in the write model as a basis for the state rehydration. If you're using materialised view, even though it's built based on events, then you outsourced the truth to other storage. If you're doing a pattern that you're just using events to build up the materialised view you use for the write model logic, that can lead to using Event Streaming tools like Kafka, Pulsar, etc. And, as you said, this is a dead-end. Read more on what I mean by getting the state in Event Sourcing system: https://event-driven.io/en/how_to_get_the_current_entity_sta...

Of course, rebuilding a state from events each time you're processing command may sound dubious. Still, event stores (even those with relational DBs as backing storage) are optimised to read events quickly. Typically, reading event 100 events are not an issue for them. Also, for a typical LoB application. The temptation to use Snapshots is strong, but it should be treated as an optimisation when there is huge performance requirements. They may get out of sync with events. They may be stale, etc. I wrote about that in: https://www.eventstore.com/blog/snapshots-in-event-sourcing. The critical part is to keep streams short. That's again a difference between Event Streaming and Event Sourcing. In Kafka, you may not care how long your topic is, as it's just a pipe. Event stores are databases, so the more events you have, the worse. It impacts not only performance but also makes schema versioning harder (more on that: https://event-driven.io/en/how_to_do_event_versioning/).

It's easy to fall into the trap, as using a streaming solution is tempting. They promise a lot, but eventually, you may end up having issues as you described.

Still, that can be said on any technology. Event Sourcing has some dark parts. Also, it's essential to highlight them, but it's dangerous to present other tools and patterns issues related to ES.

Event Sourcing by itself doesn't directly relate to eventual consistency, type of storage, messaging, etc. Those are implementation details and tradeoffs we're chosing. Each storage solution has it's patterns and anti-patterns. Relational databases have normalisation. Document databases are denormalised. Key-value stores have strategies for key definition. Event stores also have their specifics. The most important is (as I mentioned) to take into the account the temporal aspect of streams, so keeping them short. I agree that there is a huge gap missing in knowledge sharing. I'm working on the article recently about that aspect. Here's a draft: https://github.com/EventStore/blog-articles/blob/closing-the....

I think that you highlighted well what can happen if you use the Event Streaming solution as a basis for the Event Sourcing solution. But unfortunately, the title and description state that it's about Event Sourcing, which is unfortunate, as it's highly misleading for people who are not aware of what Event Sourcing is and may suggest that it's much more complicated than it's in reality.

See also my other comments below in the thread, where I commented also in more details on other parts.

You're right that there is a distinction between event sourcing within a service and events as coupling between services. But as someone who architected a service that used event sourcing internally and did not use event streaming, I feel the author's points still stand. I also currently work on a very complex application built on event sourcing principles, and again, the same challenges are present.

The reality is that event sourcing brings a lot of challenges that have to be grappled with. It's a great paradigm, but it's not easier than in-place mutation. At least, not without a bunch of tooling.

I re-read the article after this comment, and I would have to disagree. The article does never attempt to explain "what" recent sourcing means to them, so it's hard to know for sure. However, they do mention populating state from an event log which contains meaningless events, so I have to assume they _are_ talking about event sourcing. What leads you to believe they are taking about just streaming, and not sourcing?

They were probably also sourcing their state from the events. However most of their problems come from sharing the events between the modules/services which is not a part of Event Sourcing.

I think that they were just building the stale read models, and used them as the write model, which created the whole confusion. Regarding the sharing events between module, it's one of the most common and the most dangerous mistakes. It's a leaking abstraction that will eventually create a distributed monolith. It has only downsides of monolith and microservices, without the upsides. I wrote longer on the topic of internal and external events, and how to model them: https://event-driven.io/en/events_should_be_as_small_as_poss...

What you are talking about is CQRS, which is a very valid pattern, and pairs well with event sourcing, but is not necessary part of event sourcing. You don't have to split your read and write models for events sourcing.

Could you elaborate, I think that I'm missing your point.

I am confused when you say sharing events isn't part of event sourcing. How does a service populate it's state from other services event source if it can't access it's events?

Did you read my article that I linked, or my initial explanation of the Event Sourcing flow?

> How does a service populate it's state from other services event source if it can't access it's events?

Because it's the source of the events and its own system of record.

If an event sourced app wants to share events, it should not be re-using internal events but creating new items intended for distribution just like you would with any other distributed system (thrift/protobuf over Kafka).

Could you provide an exact quote? I haven't been able to find any usage of "populate". I found:

`the raw event stream subscription setup kills the ability to locally reason about the boundaries of a service.`

`you have to talk to the people who will be consuming the events you produce to ensure that the events include enough data for the consuming system to make a decision`

`wire a fleet of services together via an event stream`

etc.

Plus the drawing that's at the beginning.

This clearly states that the author doesn't understand what's Event Sourcing and tells about Event Streaming. Event Streaming means:

1. Producer pushes/produces/publishes an event to the queue. In this sense, an event stream is a pipe where you put an event at the end and read it on the other side.

2. Then, you have a set of subscribers that can react to those events. The reaction may be triggering the next step of the workflow. It might also be updating the read model state. That happens with eventual consistency. You also cannot guarantee atomicity or optimistic concurrency, as tools like Kafka doesn't support it. As they were not originally built for. They let you effectively publish and subscribe to the messages.

3. Because of that, if you want to have the write model materialised from an event (or even "populated"). Then you don't have any guarantee if it's stale or not. You need to fight with out of order messages and idempotency. Thus all of those issues that are described in the article.

I'm sorry to say, but the author used the wrong tool for the wrong job. If this article be titled "Event Sourcing is hard if you're confusing it with Event Streaming" then it'd be an excellent article. But, in its current shape, it's just misleading and making the wrong point, repeating just common misunderstandings.

You can read more in:

- https://domaincentric.net/blog/eventstoredb-vs-kafka

- https://www.eventstore.com/event-sourcing

You seem to be saying that event sourcing and event streaming are mutually exclusive.

In almost all cases of event sourcing I have seen, event sourcing is event streaming, but more. When you say event sourcing, where do other services source from, if there is no stream? Yes, event sourcing typically uses a database instead of a real "queue", but it still functions like a queue. You still need consumers to subscribe to events, in order to populate state, as timely as possible.

They're not mutually exclusive, but they're orthogonal. They're different patterns that happen to integrate with each other. Event Sourcing is about durable state stored and read as event, Event Streaming is about moving events from one place to another.

Event stores can integrate with the streaming platforms, to publish events and move them forward. Most of them have even built-in implementation of the Outbox Pattern that allows to publish events. They're usually called subscriptions. They can be used to build read models or triggers other flows, or route events to streaming platforms.

The essential point is that for read models, you don't expect to have the same guarantees as for write models. It's fine to be facing idempotency, even out of order, as read models shouldn't be used (in general) for business logic validation. In Event Streaming, or models are stale. In Event Sourcing, write model is not stale, you can be sure that you're making a decision on the current state. Thus, most of the points mentioned in the article, are just from struggles to use event streaming solution as tools for storing events. Which is not the issue of Event Sourcing or event stores per se, but using the wrong tool for the job.

An app using event sourcing does not need to be consuming events from an external messaging service nor does it need to be publishing messages to one. It could very well be receiving requests over a synchronous REST API from an HTML/JS UI.

I believe the point OP is trying to make, is that integration with a messaging service/queue is a separate concern from what event sourcing solves but that the internal event pipeline often looks very similar to a networked one and so the two solutions are conflated.

I wrote the summary of my thoughts on this article: https://event-driven.io/en/event_streaming_is_not_event_sour...

BTW, does there happen to be a discord channel for this topic? ;)

There is a Slack channel: https://join.slack.com/t/ddd-cqrs-es/shared_invite/zt-m3vf3a...