Event Sourcing Is Hard (2019)
chriskiehl.com
chriskiehl.com
99% of the time when ES sounds like a good idea, the answer is to just use Postgres. Use wal2json and subscribe to the WAL stream - if you really need time travel or audit logs or whatever, they'll be much cheaper to implement using WAL. If you need something more enterprisey to sell to the VP-suite, use Debezium.
Event sourcing sounds so awesome in theory and it is used to great effect in many demanding applications (like Postgres! WAL = event sourcing with fewer steps) but it's just too complex for non-infrastructure software without resources measured in the 10s or 100s of man years.
We had a guy who spent two weeks writing a report script that collated data uploaded into S3 and wrote out another data lump into S3 after some post processing then sent this data lump to a guy via email. This entire thing had a bunch of lambda layers to pull in the pipeline, a build pipeline in jenkins, terraform to deploy it. The python script itself was about 200 lines of the equivalent of spreadsheet cell manipulation basically.
Only after it was implemented we found out it was only run every 3 months and took the original dude 5 mins to paste it into his existing excel workbook and the data popped out in another sheet instantly. This was of course a “massive business risk” and justified a project to replace it. What it should have been was a one page thing in confluence with the business process in it and the sheet attached. But no it now needs a GitHub repo full of shite and 2 separate people to understand every facet of it who require 3x the salary of the original dude each. And every 3 months, something is broken or it pukes an error which requires someone 3 hours of debugging and reverse engineering the stack to work out why.
Fundamentally, event sourcing, as the above, is the outcome of seeing something shiny and covered in marketing and wanting to play with it on any available business case rather than doing a rational evaluation of suitability. There are use cases for this stuff, but sometimes people don’t know when to apply the knowledge they have.
As for ES itself, I have resisted this one big time, settling on CQRS class functionality which is quite frankly no difference to query/mutate database APIs using stored procedures in PL/sql I dealt with in the dark ages. People also don’t have the architectural skills or conceptual ability generally to rationalise the change on data consistency on top of that or the recovery concerns. I would rather find another way to skin the scalability cat than use that architecture.
I wrote a code generator that wrote all the code in the end. Now I maintain a shitty code generator.
You can’t win.
I think you eventually run out of fucks. I have.
Sometimes the ability for an engineer to fetch the history of a business object out of a datastore and explain what it means checks an "audit trail" requirements box. Sometimes showing the history of a business process in a UI, with back-in-time functionality, is a game changing feature. If so, using event sourcing internally to the service is a great way to ensure that these features use the same source of truth as other functionality.
Where you get into trouble is when you realize that event sourcing will let you distribute the business logic related to a single domain object to a bunch of different codebases. A devil on your shoulder will use prima facie sound engineering logic tell you it is the right thing to do. The functionality relates to different features, different contexts, different operational domains, so keeping it in the same service starts to feel a bit monolithic.
But in practice you probably don't have enough people and enough organizational complexity to justify separating it. What happens is, a product manager will design an enhancement to one feature, they'll work out the changes needed to keep the product experience consistent, and the work will get assigned to one engineer because the changes are reasonable to get done in one or two sprints. Then the engineer finds out they have to make changes in four different systems and test six other systems for forwards compatibility. Oops. Now you have nanoservices-level problems, but only microservices-level capabilities.
If you aren't tempted down that path, you'll be fine.
For an interactive application it's an absolute nightmare when you don't have read-your-own-writes support (including synchronization for "indexes"). Async background updates aren't always appropriate or easy to do.
few questions: - what was your biggest ES system that you worked on? - how many people worked on it? - how did the ES system particulary solve your problem? - what was the tech stack?
Thanks
40 people approx work on it in Asia, maybe around 100 globally at the raw dev level.
I feel it's a sort of false good idea for our particular problems. Clients trade ether at the 10ms latency for high value orders or sub-ms for latency-sensitive ones. They conceptualise what they want to buy and a bunch of changes they want to apply on that bulk: for instance, change in quantity, price limits, time expiry and try to make money by matching a target price either by very closely following a prediction curve or spending as little time on doing so (giving us only a constraint and asking us to fit it by doing our own curve fitting algos).
The tech stack is pure java with a kernel of C++ for networking, with as little external dependency as possible. And no GC beyond the first few minutes after startup (to prealloc all the caches). Everything is self built.
How does it solve the problem: it simply is the only way we can think of to over optimize processing. If you want to fit a millisecond or sub millisecond target (we use fpga for that), you must cut the fat as much as possible. Our events are 1kb max, sent in raw tcp, the queue is the network switch send queue, the ordering has to he centrally managed per exchange since we have only one final output stream but we can sort of scale out the intermediary processing (basically for one input event how to slice in multiple output events in the right order).
I'd say it doesnt work: we run into terrifying issues the author of the main link pointed out so well (the UI, God, it's hard, the impossible replays nobody can do, the useless noise, solving event stream problems more than business problems etc), but I d also say I cant imagine any heavier system fitting better the constraint. We need to count the cycles of processing - we cant have a vendor library we cant just change arbitrarily. We cant use a database, we cant use heavier than tcp or multicast. I'll def try another bank one day to see how others do because I m so curious.
I worked for a large US bank, and had close contact with those who worked on the asset settlement system. I don't have as deep insight as you do, obviously. But the general architecture you describe sounds very similar. Except they clearly used message queues and treated them as message queues, and not event streams / sourcing / whatever. They used a specific commercial message broker that specializes in low latency.
Disclaimer: I work on Debezium
[1] https://debezium.io/blog/2020/02/10/event-sourcing-vs-cdc/
[2] https://debezium.io/blog/2021/11/30/debezium-1.8-beta1-relea...
Do you know how to model your events using a type system?
Can you be bothered to implement a Serialize and Deserialize method for each event type, or simply use a JSON serializer?
Do you know how to make your favorite language compress and uncompress things to disk on a streaming basis?
Do you know how to seek file streams and/or manage a collection of them all at once?
Can you manage small caches of things in memory using Dictionary<TK,TV> and friends?
Are you interested in the exotic performance possibilities that open up to those who leverage the ring buffer and serialized busy wait consumer?
If you responded yes to most or all of the above, you are now officially granted permission to implement your own WAL/event source/streaming magic unicorn solutions.
Seriously, this stuff is really easy to play with. If you follow the rules, you would actually have a really hard time fucking it up. Even if you do, no one gets hurt. The hard part happens after you get the events to disk. Recovery, snapshots, garbage collection - that's where the pain kicks in. But, none of these areas is impossible. Recovery/Snapshots can again be handily defeated by the mighty JSON serializer if one is lazy enough to succumb to its power. Garbage collection can be a game of segmenting log files and slowly rewriting old events to the front of the log on a background thread. The nuance is in tuning all of these things in a way that makes the business + computer happy at the same time.
Plan to do it wrong like 15 times. Don't invest a whole lot into each attempt and you can pick this up really fast. Try to just write it all yourself. Aside from the base language libraries (file IO, threading, et. al), JSON serializer and GZIP, you really should just do it by hand because anything more complex is almost certainly wrong.
IIRC it took another data scientist and engineer under a month to turn the raw WAL logs into an audit log interface with pretty SSO avatars and weekly reporting. Someone from the devops team with DBA experience implemented time traveling staging DBs with continuous archiving and point in time recovery from production in the same time. Someone else later improved it so PITR used full backups created from a filtered WAL log so devs could select which parts of the production DB they copied over instead of each babying their own staging cluster that took days to rebuild. The whole project ended up giving us all of the benefits of event sourcing using standard, well tested tooling for a fraction of the cost.
We've thrown around the phrase "not invented here syndrome" so much that we've over corrected - as humans are wont to do - and now the younger generation thinks architectures like event sourcing or infrastructure like Kafka are a better solution than to just consume replication logs over a TCP connection to one of the most popular open source databases on the planet (not directed at the GP but my former coworkers :)). I'm starting to wonder if I've reached the age where I sound like the adults in the Peanuts cartoons, except the sound vaguely resembles "Get your resume driven development off my lawn!"
Debezium can also be used without Kafka; either via Debezium Engine [1], where you embed it as a library into your JVM-based application and it will invoke a callback method you registered for every change event it receives. That way, you can react to change events in any way you want within your application itself, no messaging infrastructure required. The other option is using Debezium Server [2], which takes the embedded engine to connect Debezium to all sorts of messaging/streaming systems, such as Apache Pulsar, Google Cloud Pub/Sub, Amazon Kinesis, Redis Streams, etc.
[1] https://debezium.io/documentation/reference/stable/developme...
[2] https://debezium.io/documentation/reference/stable/operation...
I like some of the things in Event Sourcing very much.
However, whenever I ask "Why not just use PostgreSQL as the event store until I'm 5 orders of magnitude bigger?" I never seem to get a really good answer.
1. You get the events from the stream (that represents all the facts that happened for the entity),
2. You apply it one by one in the order of appearance to build the current state,
3. You run the business logic and, as a result, create the event,
4. You append a new event.
You don't need to cache the write model state anywhere, as the state is in events. Event Sourcing has its issues and troubles, but the stuff described in the article apply to tools like Kafka, etc. They're not tools for Event Sourcing but Event Streaming. They're designed to move things from one place to another, not to be used as durable databases.
I've made a list of articles like that: https://github.com/oskardudycz/EventSourcing.NetCore#this-is...
I'll repeat it again: this is not Event Sourcing.
Firstly, to make sure we're talking about the same thing, where are you setting the bar for whether or not we can call something "Event Sourcing"? For instance, just to clarify, in our system events were indeed the source of truth. At the heart of everything sat a ledger, that ledger was stored in a database, and events were added to it as you describe in points 1-4 (hand waving away minor differences). You got a consistent, up-to-date state of an aggregate by replaying the ledger in order and applying all the events. So, to my understanding, I'd call that textbook. However, the problem may lay in our actual definition of the word.
>You don't need to cache the write model state anywhere, as the state is in events
I'd have to understand a bit more about where you're coming from with this one. While, yep, the state is in the events and you get the latest state by playing them back, that materialization is not free. If you're doing it for a single aggregate, then it's generally not a huge deal at small scales. However, once you need to make decisions against more than one thing, or compare things, or query things, that materialization cost becomes prohibitive, and you've got to set up materialized projections. With our tech stack, that had to be an eventually consistent process, and what got us into event streaming, and which ultimately caused most of my ugly crying.
I think the point OP is making is actually the same as what you conveyed in your article--several of the worst pain points you experienced aren't inherent to Event Sourcing, they're due to a particular implementation of Event Sourcing. The problem is that that implementation is pretty much the canonical one sold by Martin Fowler et al.
I can think of a few specific ways in which event sourcing could be different than what you experienced:
1) Distributed computing is always hard. Event Sourcing is not a magical cure-all to distributed computing woes, but it's sold as that, and the result is that people go into a distributed event-driven architecture without being prepared for the engineering challenges that await. Eliminate distributed computing, and many problems simply go away.
2) Eventual consistency is always hard, but Event Sourcing doesn't inherently require eventual consistency, even for cached projections. If your projections are pure functions folded over the event stream, then caching and updating the materialized projections with strict consistency is straightforward. In the same transaction as inserting a new event, you run your projection function once on each of the materialized projections and the new event (`state = projection(oldState, event)`). This is only possible when you're not trying to create your projections in a distributed fashion, so see #1.
3) Language choice makes a big difference for reducing the cognitive overhead of updating projections. Your compiler should be able to tell you every place where you need to update a projection to account for a new event. Preferably, you should be able to define a hierarchy of events so that it only yells at you about places where the event is actually relevant.
Of course, rebuilding a state from events each time you're processing command may sound dubious. Still, event stores (even those with relational DBs as backing storage) are optimised to read events quickly. Typically, reading event 100 events are not an issue for them. Also, for a typical LoB application. The temptation to use Snapshots is strong, but it should be treated as an optimisation when there is huge performance requirements. They may get out of sync with events. They may be stale, etc. I wrote about that in: https://www.eventstore.com/blog/snapshots-in-event-sourcing. The critical part is to keep streams short. That's again a difference between Event Streaming and Event Sourcing. In Kafka, you may not care how long your topic is, as it's just a pipe. Event stores are databases, so the more events you have, the worse. It impacts not only performance but also makes schema versioning harder (more on that: https://event-driven.io/en/how_to_do_event_versioning/).
It's easy to fall into the trap, as using a streaming solution is tempting. They promise a lot, but eventually, you may end up having issues as you described.
Still, that can be said on any technology. Event Sourcing has some dark parts. Also, it's essential to highlight them, but it's dangerous to present other tools and patterns issues related to ES.
Event Sourcing by itself doesn't directly relate to eventual consistency, type of storage, messaging, etc. Those are implementation details and tradeoffs we're chosing. Each storage solution has it's patterns and anti-patterns. Relational databases have normalisation. Document databases are denormalised. Key-value stores have strategies for key definition. Event stores also have their specifics. The most important is (as I mentioned) to take into the account the temporal aspect of streams, so keeping them short. I agree that there is a huge gap missing in knowledge sharing. I'm working on the article recently about that aspect. Here's a draft: https://github.com/EventStore/blog-articles/blob/closing-the....
I think that you highlighted well what can happen if you use the Event Streaming solution as a basis for the Event Sourcing solution. But unfortunately, the title and description state that it's about Event Sourcing, which is unfortunate, as it's highly misleading for people who are not aware of what Event Sourcing is and may suggest that it's much more complicated than it's in reality.
See also my other comments below in the thread, where I commented also in more details on other parts.
The reality is that event sourcing brings a lot of challenges that have to be grappled with. It's a great paradigm, but it's not easier than in-place mutation. At least, not without a bunch of tooling.
`the raw event stream subscription setup kills the ability to locally reason about the boundaries of a service.`
`you have to talk to the people who will be consuming the events you produce to ensure that the events include enough data for the consuming system to make a decision`
`wire a fleet of services together via an event stream`
etc.
Plus the drawing that's at the beginning.
This clearly states that the author doesn't understand what's Event Sourcing and tells about Event Streaming. Event Streaming means:
1. Producer pushes/produces/publishes an event to the queue. In this sense, an event stream is a pipe where you put an event at the end and read it on the other side.
2. Then, you have a set of subscribers that can react to those events. The reaction may be triggering the next step of the workflow. It might also be updating the read model state. That happens with eventual consistency. You also cannot guarantee atomicity or optimistic concurrency, as tools like Kafka doesn't support it. As they were not originally built for. They let you effectively publish and subscribe to the messages.
3. Because of that, if you want to have the write model materialised from an event (or even "populated"). Then you don't have any guarantee if it's stale or not. You need to fight with out of order messages and idempotency. Thus all of those issues that are described in the article.
I'm sorry to say, but the author used the wrong tool for the wrong job. If this article be titled "Event Sourcing is hard if you're confusing it with Event Streaming" then it'd be an excellent article. But, in its current shape, it's just misleading and making the wrong point, repeating just common misunderstandings.
You can read more in:
In almost all cases of event sourcing I have seen, event sourcing is event streaming, but more. When you say event sourcing, where do other services source from, if there is no stream? Yes, event sourcing typically uses a database instead of a real "queue", but it still functions like a queue. You still need consumers to subscribe to events, in order to populate state, as timely as possible.
Event stores can integrate with the streaming platforms, to publish events and move them forward. Most of them have even built-in implementation of the Outbox Pattern that allows to publish events. They're usually called subscriptions. They can be used to build read models or triggers other flows, or route events to streaming platforms.
The essential point is that for read models, you don't expect to have the same guarantees as for write models. It's fine to be facing idempotency, even out of order, as read models shouldn't be used (in general) for business logic validation. In Event Streaming, or models are stale. In Event Sourcing, write model is not stale, you can be sure that you're making a decision on the current state. Thus, most of the points mentioned in the article, are just from struggles to use event streaming solution as tools for storing events. Which is not the issue of Event Sourcing or event stores per se, but using the wrong tool for the job.
I believe the point OP is trying to make, is that integration with a messaging service/queue is a separate concern from what event sourcing solves but that the internal event pipeline often looks very similar to a networked one and so the two solutions are conflated.
Because it's the source of the events and its own system of record.
If an event sourced app wants to share events, it should not be re-using internal events but creating new items intended for distribution just like you would with any other distributed system (thrift/protobuf over Kafka).
> The idea of a keeping a central log against which multiple services can subscribe and publish is insane.
This really doesn't mean "Event Sourcing" to me, it sounds like enterprises that have decided Event Sourcing == Kafka (or some cloud-hosted IOT-branded variant) and treat the central broker/coordinators/confluent-cloud crap as "the central log"
To me, the fundamental idea is that changes are recorded in a meaningful format, not the result of changes; What I mean by "meaningful" is important: I don't think SQL statement replication (as implemented by popular SQL databases) constitutes "event sourcing", because the only part of the application that typically consumes this is the SQL database, but if you're storing JSON in a log (or better: some useful binary format) then you're probably doing event sourcing.
With this (perhaps broad) view of Event Sourcing, I would say I have been building all of my database applications for the last 30 years as "event sourcing", and I've never had any trouble building applications (even large ones!) using this model: It lends it self to an excellent user-experience simply by letting the UI "in on it", and the article seems to recognise this:
> Event sourcing needs the UI side to play along
If your user makes a change, and for whatever reason that change takes time to do, it is better to tell the user you're going to do it and then do it rather than wait for the HTTP request (or TCP message or whatever) to "complete" the transaction online because network always fails and bad error messages (the default!) cause users to do stupid things.
But when you use Google Cloud's UI, you can see how nice this can be: You make a change, you can look in the event log to see your changes, you can see if they've been processed or not, and you can examine and share the results simply by sharing the results pages (instead of having to make screenshots or other ghastly things).
I think for many applications this is worth it, but there aren't good tools for event sourcing (in my opinion) so it may be for a lot of applications (especially the ones I don't work on) the juice just isn't worth the squeeze -- but to me, this suggests value in developing those tools, not in ignoring this amazing thing.
You're right. Here's a list of posts[1] about Event Streaming which claim to be about Event Sourcing. The discussed article is in the list.
[1]: https://github.com/oskardudycz/EventSourcing.NetCore#1319-th...
It is frustrating to argue whether "Event Sourcing" is good or bad if we have different definitions of what it is, but I don't know of a better name for the thing that I think is good, so it is helpful to point at a body like this to say this is what I mean.
For a long time, I thought about mailing lists: Back in the 1990s we had mailing list applications that you would send email to and they would do various things based on the subject line or the recipient (To) address or something like that, and so you would "join the stream" (subscribe to the list) with one email, and "leave the stream" (unsubscribe) with another; You could publish a new message (it just had to fit in an RFC821 datagram) and processing would both distribute those messages and build "views" of that data in archives accessible out-of-band (such as on the web or in gopherspace). Sequencing messages could be done by simply refusing (tempfail) messages received out of order, and persistent errors would be surfaced (eventually) in the form of an NDR which could itself have a process running on those messages.
I think it is a small thing to imagine a lower-latency email queue, changing the datagram or to manipulate the log (queue) independent of normal processing (such as removing PII -- or in our case, spam messages!) and to create other "processors" for messages for administrative tasks (like setting up customer accounts, or shutting them off for non-payment!) with a little bit of shell scripting, that if you had this kind of experience, most of what constitutes "Event Sourcing" probably doesn't seem very hard, and if you haven't had this kind of experience, that these things may dominate the design (e.g. Confluent+anything) and lead unfairly to a bad impression about "Event Sourcing"
That's not to say I don't think there are hard parts to these kinds of architectures, just that I think those hard parts are usually worth it.
A good example would be a tree inspection application where jobs can be pushed out to mobile inspectors who might not have phone signal but can still collect the data. Once synchronised, views are simply additive. More data can be added to cases simply by adding events.
I would absolutely not use ES for a hotel booking system because that data needs to be as accurate as possible in the moment and we've had record locking techniques for over 40 years.
Seems a lot of people want to try ES without really considering whether it maps to the real-world domain of the problem they're trying to solve.
I've followed this rough pattern to build a rock-solid offline-first core for a personal application: https://flpvsk.com/blog/2019-07-20-offline-first-apps-event-... It's got some pain points; in particular it's quite difficult to evolve the schema once your app is out in the wild. But event sourcing is very amenable to TDD, and once all my tests were passing I had a nearly bulletproof solution for my current schema.
Strangely, offline-first functionality is usually not what event sourcing is pitched for. Maybe it's just a a legacy of its enterprise origins, but I often see it sold as a solution for enterprise applications where it's liable to cause more trouble than it's worth.
This isn't a great use case for Event Souring either. An inspector must be in front of the tree to inspect it, right? And they can see each other inspecting the tree.... Why would several inspectors inspect the same tree?
Assuming they inspect the same tree at different times, you can sync offline data at the row level, and let the last update win, provided that they're entering the same data.
It works for any model where the use case doesn't really matter about up-to-the-minute accuracy, but for case generation and collation over time, it really excels.
We found the ES events wayyy to chatty to push back out to mobile (in 2012) so we push the aggregates back out to the mobiles.
Plus, if you're capturing GPS coordinates, running it through Kibana for real time map reporting is really exciting to watch!
Synchronising data at row level when offline becomes online is extremely hard to get right and reliable.
With the Event Sourcing model, every new piece of data gets thrown into the bucket , with each new piece of data the view becomes bigger.
Why? Are the users in different time zones? If the time on their devices are synchronized, why is it hard to get right? I'm not trying to pick an argument with you, but trying to figure out if you've considered all the possible alternatives.
By updating at row level, you've instantly lost the ability to tell the user what time this data changed, unless you're throwing everything in an audit table and at this point you're already halfway to Event Sourcing.
ES gives us a convenient way to throw a pile of information over time and not have to worry about defining the final schema up front, we can do that when constructing the views of aggregates.
It seems to me you're defining a schema regardless, but with event sourcing your schema is embedded in your events rather than your database. And you're putting off the reconciliation to projection phase. I get it. But you still need to worry about writing transformations for your data to bring it up to date with your latest schema... That is often the most frustrating part.
Of course, with all these articles they miss out that you're not supposed to just do Event Sourcing, it's one part of the system, like we mix RDS and object stores depending on the right purpose.
Bugs arise and are solved in these informal systems, but I rarely hear people ask "how do we never get this particular class of bug again?". These questions are kept for researchers (either academic or corporation), never to reach the regular software developer.
If you don’t have strong ordering guarantees, it also means your state machine has to accept transitions from/to any state (eg comutative). Otherwise if ordering is important, your event log must have some ordering built-in (exactly once semantics) or you implement re-ordering and replay as past events arrive (a branching model).
> The bulk of these probably fall under "he obviously didn't understand X," or "you should never do Y!" in which case you would be absolutely right. The point of this is that I didn't understand the drawbacks or pain points until I'd gotten past the "toy" stage.
Most of what they discussed was indeed not great, but that was due to lack of understanding. I don't think anyone who does or promotes ES says it's _easy_. However, I would argue it _conceptually_ simple to understand (herein lies the fallacy people make simple != easy).
Like any pattern or tech that you are new too, it's going to be hard to build and operate in production (beyond the toy). Everyone takes for granted the decades we've had great tooling and operational patterns for state-oriented databases (mostly RDBMS').
And to be clear, event sourcing is "persistence strategy". What is mostly shown/discussed in the article is "event streaming" along with CQRS. They concepts are largely conflated. Once you understand, you won't be bamboozled.
Not to throw Confluent under the bus, but they have unfortunately written or promoted many articles for "event sourcing" use cases and this has confused everyone. You _can't_ do event sourcing with Kafka. It doesn't support optimistic concurrency control at the key level (where as something like EventStoreDB does natively or even NATS now with JetStream). Kafka if fantastic for what it is designed to do.. event streaming.
Is this article a fine summary?: https://developer.confluent.io/learn-kafka/event-sourcing/ev...
Also refer to the list of great resources @oskar_dudycz posted in another comment: https://github.com/oskardudycz/EventSourcing.NetCore#1319-th...
- Lacking support for key-based look-ups: when trying to materialize the current state of an entity, you cannot query Kafka efficiently for just those events (a partition-per-entity strategy doesn't scale due to the partion number limitation, which may change with KRaft in the future)
- Lacking support for (optimistic) locking: you cannot make sure to reject events based on outdated state, unless you apply work-arounds like a single writer architecture
Like a table, there can be an arbitrary number of rows (>millions), thus you could have millions of event streams. You can't have millions of Kafka partitions. The scope/scale of a Kafka partition is not designed for that small granularity.
Again, I have no issue with Kafka and I think Pulsar is superior in the event streaming/platform space. I also the Kafka ecosystem is impressive and I love listening Confluent Cloud podcast! But it is not designed for this use case.
And if you're using an immutable log (Kafka/Pulsar/Liftbridge etc.) fixing those mistakes is hard.
And given you're basically recording interim states of a state machine, how do you verify that event Z being submitted after event Y is a valid state transition?
Validating a record against a schema is trivial. But validating three consecutive events against the state machine? I'm not aware of a good way to do that cheaply.
I've done it before using apps that sat in the data pipeline, and maintained state to reject invalid transitions, but that can end up being GiB of state, then you have to persist that across restarts or horizontal scaling.
It's all doable, but it costs. Is the event streaming paradigm actually worth the cost? I haven't been convinced yet.
But, I'm a big fan of streaming DB change events for auditing purposes - but those are coming from software that already enforces valid state changes.
So maybe the answer is - every event has to be submitted via Postgres.
Anyway, pedantry aside, my point remains thus - it's hard to prevent invalid state transitions being submitted to an event sourcing datastore, and when devs can submit events directly, then these invalid transitions will occur.
Indeed. These are two separate patterns. They can have great synergy when used together but it's not a requirement.
> it's hard to prevent invalid state transitions being submitted to an event sourcing datastore
Apply DDD, use code review, and use stream level ACLs.
I then tried to apply this to a conventional request-response web service. It did not fit well. Everything was immediately significantly harder with only a marginal gain.
My rule fo thumb now is to use event sourcing when doing otherwise would be difficult. A justification based off of an improvement, rather than one solving an obvious problem, isn't worth the extra effort.
Is event sourcing about storing 1 authoritative, immutable, serialized narrative of events, or is it about tying together a pile of microservices?
I think the part where it goes wrong is where we try to subscribe to these events (i.e. push). This adds delivery semantics to the mix and forces a lot more constraints at vendor selection time. Being notified that something occurred is not part of event sourcing in my mind (though it is clearly one logical extenstion).
Don't get me started on explaining snapshots, why you don't need snapshots until replays take 1000ms, why replays won't take 1000ms until the topics are thousands of events deep, why the team won't believe that this isn't a problem, and why you will definitely need snapshots if you keep scaling.
One of my unwritten requirements for any system is being able to drop in inexperienced devs for maintenance...after I leave. Most people are likely to shoot themselves in the foot by not understanding (or even being aware of) database isolation levels much less grokking an event pipeline.
- The general problem with event-sourcing (at small scale) is that it forces you to have to think about and handle many things that can mostly be ignored with a more typical persistence approach. This overhead of complexity was one of the factors that (IMO) killed a company I worked for. Event-sourcing feels relatively "raw" in this way. However, this also provided a valuable learning experience as a developer.
- I'm curious about better ways to do event-sourcing. For example, using Datomic is apparently like a well-thought-out, pre-made event-sourcing system: https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...
- If you intend to embark on an event-sourcing journey, read Versioning in an Event- Sourced System https://leanpub.com/esversioning/read , even if only to better understand the complexity of this one aspect. I agree with the author of this submission that it is hard to fully grasp the complexities of event-sourcing without trying it, and hitting many road bumps along the way, since it's not an especially well-trodden path.
- Being able to redefine the interpretation of an event as the system evolves feels like a superpower at times. Similarly, being able to bring a new part of the system online, and have it based on a rich history of everything ever done in the system, is very cool. One of the major benefits of event-sourcing is how adaptable it is in systems that change over time.
- Beware CQRS and eventual consistency. This adds a lot of complexity and is not a prerequisite for event-sourcing. It's also telling that ordinarily-simple problems to solve, like preventing duplicate username registrations, are hand-waved away by CQRS proponents: https://web.archive.org/web/20191101010824/http://codebetter... . You will face this class of problems, and you may not be able to hand-wave it away so easily.
The reason I posted this was that I found it helpful in prepping for a workshop I was giving today - recording is not up yet but slides are here https://docs.google.com/presentation/d/1mAPYXNmRdDKepXylB91F...
> In effect, the raw event stream subscription setup kills the ability to locally reason about the boundaries of a service. Under "normal" development flows, you operate within the safe, cozy little walls which make up your service.
I wonder if this isn't more a case of the "cozy little walls" turning out to be a lie.
If your business requirements are that this user action has to trigger those half-dozen barely related background tasks, then your design will always be messy and complex, no matter if you use service calls, event queues, polling or whatever else.
In such a situation, I think a better question would be which design lets you most easily reason about the entire system - e.g., what is the dependency structure, how is the causal chain organized, etc.
Of course there are also "secondary" concerns, such as: Which design lets you make changes to the system easily, can you still easily develop and test things in isolation that can be isolated, etc. I think services and events actually have some upsides here (but I still may be in the "seduced" phase...)
I think this calls more for a better API/dependency management (and treating events as full-fledged APIs!) and a good tracing solution which lets you follow an action through different services (and event queues).
> You're free to make choices about implementation and storage and then, when you're ready, deal with how those things get exposed to the outside world. It's one of the core benefits of "services". However, when people are reaching into your data store and reading your events directly, that 'black box' property goes out the window
Again, I would absolutely treat events just as much as a public API as REST. Only because the causal realtionship is inversed doesn't mean you can throw away schemas, versioning, dependency tracking etc.
Also, the "from scratch" systems, seemed conceptually easier to not only understand, but extend, because you were forced to write code in an idiomatic and consistent way. Everything was an event (albeit, a "command" and an "event" confirming 'execution' of the "command").
For us, the entire working set would be able to live in memory. Larger objects would be left as pointers to the actual data in the log. Snapshots of working set would be taken on a daily basis in hopes of keeping recovery under ~5 minutes during service restarts or updates.
Exactly this. We took hourly snapshots (for a trading system), and could restart the system in a second or so.
Another nice side effect of an event driven system is that you start to think about what other data you can put in there. And then you realise you don't necessarily need a database any more, which makes the architecture of the system simpler.
There are so many benefits working with a data-driven system. Once data is in the log/ledger/whatever you call it, it's immutable, which leads to parallelism if you want it. Ensuring the system is idempotent also means you get (easier) failover as well.
This is precisely where we want to be.
We are already exceptionally minimal when it comes to database usage - SQLite is all we use today. If we are in a place where 100% of the business facts go to an immutable/append-only log, we can handle the entire persistence mechanism with a simple filestream write command. Not having to worry about mutating prior data solves a large portion of database engine problems automagically.
I'm not saying I hate the separation of write and read models, and all the challenges that brings (heck, the platform I'm working on now does that, and it's not a big deal), but it was massively bloated and overly complex for a system that would collect some data from users, submit it to a panel of third party services, and display results back to those users.
The reasons for going this route were complex, and not entirely clear to me, although I can pick on two: firstly, they'd run into performance and scalability issues with their existing CRUD-ish SQL Server-backed implementation and, secondly, Thoughtworks told them it would be a good idea.
I don't want to handwave this away but I've encountered enough insoluble SQL Server performance issues over the years to be confident those issues could have been solved much more simply (and the same is likely true for Oracle, Postgres, et al). I'm not suggesting that intractable performance issues don't exist on relational platforms, just that most times people think they've encountered one, they're mistaken.
Would I use event sourcing again? Maybe. But I'd need a really strong use case for it, and I'd certainly kick the tyres on any technical organisation that was using it before I joined.
I'm talking about indexes.
Yes, the index part of your database is amazing, because it constantly snaps itself into various states, and you know exactly why: because it needs to maintain itself in the same deterministic relationship with your models.
I think that if databases would support more complex and composable indexes, like providing the ability to index join queries, then application programmers would be more nimble and bug-free working with their data.
I've written more about this here: https://lironshapira.medium.com/data-denormalization-is-brok...
And while we're at it, the whole point of eventsourcing is that they're not supposed to model "internal" process state? They're supposed to be the ground source of truth; whatever the process has to do to mangle that into a model it can work with should be no skin off the events' back.
The rest seems very, very correct to my experience. Especially not knowing the pain points until you're past the toy level.
I think Event Sourcing is an amazing model for certain scenarios. Often, a hybrid approach can be better.
Has anything changed since then? Is there new tooling that resolves some of the pain points, or new techniques that change the picture?
I architected an ES system once. It was absolutely the right call, but we had to work through nearly every one of those problems.
Regardless of whether you do it well or not, engineers will find it cumbersome to work with and think that it's overengineered.
For shops that are being crushed by fluctuations in concurrency, challenges with the uptime and p50 of partner teams, don’t have much of a choice but to go event sourced, optimistic, and eventually consistent.
I’m certain a great enterprise IT shop with great leadership could mitigate the challenges of service growth and instability with some clever optimizations of their current stacks but unfortunately I have to meet a shop that is uniformly competent.
Event sourcing helps with the competency gap problem. If every team’s data is on the outside my team can work around your lead time for changes or general ops incompetence. If inventory is down, fuck you I’m still taking orders. The extra work of maintaining a projection of your inventory events makes good business sense because time kills all deals.
Yes, could google, but now am discouraged. If you're going to write one of these, at least link to the wikipedia page:
https://en.wikipedia.org/wiki/Domain-driven_design#Event_sou...
Then you can retrieve PII for a GDPR if needed.
Or, remove it entirely by submitting a "tombstone" record with key -> null, then log compaction removes it.
Which basically means, don't mix PII into transnational data.
PII is a similar but different US concept.
https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...
1. I don't think Event Sourcing sucks - I think we are lacking accessible technology for supporting it. 2. For most difficulties encountered in Event Sourcing, I would rather blame distribution taken to the extreme.