The event stream stuff makes sense in some specialized use cases (LMAX?) but for general purpose stuff? Oof, no.
EDIT: Also, don't hate me folks, but any article that references Fowler should be looked at with just a little bit of suspicion. Not hating the guy but he's blown quite a bit of hot air in his time.
Probably one of the largest technical blunders of my career (or at least the largest so far!) was pulling a team down into ES hell on a greenfield project.
Turns out, growing an event sourced application is fantastically hard[0]. Ours eventually became a twisted opaque mess which was terrifying to modify due to (a) the sheer volume of code it took to do so, and (b) eventually losing control over who read which events and where as the team grew.
Usually arguments go "we need an audit log so ES is a natural fit and we'll be able to iterate quicker because we can just re-project when we get things wrong, oh and testing is really straightforward too because we can just...".
You can have an audit log without ES. You're also going to hate versioning events. And navigating your tangled nest of events that you accidentally accumulated in the name of agile isn't the nirvana you were hoping for.
I believe it's better to have a system that responds to the requirements of here and now, leaving history behind, learning from mistakes going forward and accepting that yesterday is done.
In my experience, in growing software systems, you're more likely to have not to have even had the data/event at all, than to not have misappreciated what you already had. So event logs feel more like a bunch of baggage ready to trip teams up than a wealth of value.
Some features that would be needed:
* Automatic , schema based event and aggregate versioning
* A combined event and aggregate data store with ACID guarantees, schema validation and indexes
* event migrations (so you don't have to carry around old event versions forever)
* Event stream migrations that allow re-writing streams for purging data, gdpr cleanup, etc
* Copy on write based stream clones that can be used for testing with production data
As it stands, such a solution is nowhere to be seen. So you have to cobble together a working system with lots of custom work , which often ends in a complete mess.
If you declare the business process you want to support - either in code or in BPMN, you're automatically mapping out the events that are relevant to the business.
Having a system that takes care of executing this, and tying the pieces of this workflow together is one example where ES can shine IMHO.
[1]https://docs.camunda.io/docs/product-manuals/zeebe/technical...
I'm the kind of person who learns by my making my own mistakes (as opposed to learning from others', which is what smart people do)—and boy, did I learn a lot from the mistake of trying to build a general purpose application with event sourcing.
Theoretically, event sourcing is the way all applications should be built. I was first taken by this methodology by Martin Kleppmann's talk "Turning the Database Inside Out" (2014)[0]. Looking back, I think I had always subconsciously felt like there was something wrong with mutating state in place. You lose information. You lose accountability. After all, "accountants don't use erasers," and Martin's talk validated many of my obsessive-compulsive insecurities about data. As Charity Majors points out on Twitter [1]:
> ...while storing raw events may be "expensive" compared to any single optimization, all optimizations can be derived from raw events. The reverse is impossible.
And that's true. The table-stream duality [2][3] means that if you store events, you can later project those events into the exact perfect table for any use-case. But if you transform those events into a projection and store that projection, well, whatever table you store is data you have. You wind up having just an aggregate.
Unfortunately, the technical costs you incur when you attempt to store events instead of tabular data are tremendous. The problems and questions that arise are basically endless. Tons of things you wouldn't even think about with a relational database become complicated design decisions [4]. If there were tooling and frameworks that made this easy, it would be one thing. But, there are not!
Kafka Streams is very cool. If you already have a running business with well-developed models and a large team, event sourcing with Kafka Streams could possibly go quite well. But if you need to get an MVP off the ground, it's just not what you need. What you need is PostgreSQL, perhaps with something like Hasura or postgraphile in front of it. What you need is a lightning-fast way to iterate and flesh out your idea. If that's you, and you're reading this, and you're smart: learn from me—event sourcing: don't do it!
[0] https://www.youtube.com/watch?v=fU9hR3kiOK0
[1] https://twitter.com/mipsytipsy/status/1115537408705957888
[2] https://docs.confluent.io/platform/current/streams/concepts....
[3] https://news.ycombinator.com/item?id=23207123
[4] https://softwareengineering.stackexchange.com/questions/3801...
For me ES makes perfect sense in systems handing a workflow for you. Products like Zeebe[1] and Ubers Temporal[2] comes to mind.
These systems basically becomes your business transaction log and as such ES is just a great fit. (I have no idea whether ES is used in temporal though… never looked at the code)
Short answer is yes, we do, but it's abstracted away so you get the benefits (complete observability, retry/resume from failure) without the downsides (handwriting event sourcing logic and storage).
Long answer... it depends what your definition of event sourcing is. We shared some of our internal debate here: https://news.ycombinator.com/item?id=28149159
Event sourcing is an impt implementation detail for us, but it is not something we push onto our users. Still they benefit from it anyway. very much agree with your statement. happy to take followup qtns :)
my grandfather had the same words :)
> What you need is PostgreSQL, perhaps with something like Hasura or postgraphile in front of it. What you need is a lightning-fast way to iterate and flesh out your idea.
optimizing my team's incoming stream stack atm, event sourcing seems so enticing on paper...
Indeed. I don't doubt he's a very clever guy, but all of my most miserable gigs involved phalanxes of architecture astronauts endlessly bike-shedding every design session with the platitudes of Fowler et al.
I've just started reading some Fowler stuff. What hot air are you referring to? I would like to keep that in mind.
Like if all you’re doing is using the event to take the value from the UI and shove it into the model with setState and re-render then you made a two-way binding.
Wouldn’t it be helpful if you could tell React to just do this for you and save you the boilerplate? Boom you invented ng-model.
Of course, this can be lossy, but is it any more lossy than the non-event-sourced CRUD system which is essentially transforming historical events into "overwrite these fields" events that can only be used for their original purpose?
To me, the reason not to use event sourcing is more that the tooling and developer experience is just really poor - and so decisions like the above, while valuable to think through directly, rapidly become overwhelming. Visualizations of system state are only as good as the code you bring yourself. I'm hopeful that people building tools on top of things like https://materialize.com will start to change this though!
If you have bidirectional communication, you wind up contorting yourself into a pretzel to bury "state" into your event streams.
Maybe you care about all the events that have ever happened on your incredible journey to reach a list of the customers you have today.
Or maybe it would be sufficient to have been mutating a customers table all along. If that is sufficient, then it would be faster and more straightforward to have done so.