The Reactive Monolith – How to Move from CRUD to Event Sourcing

A valid question then is: "what do you gain over just using the table"? You gain a well-described model of your business domain, with very clear actions about what should happen on which real-world event, and an audit log. Whether that's worth it depend on your use case.

Secondary benefits include easily being able to share events for new apps or use-cases, business analysts really like them, and making it unlikely things go wrong because data is in an inconsistent state.

And in reality, the event log does live forever in the real world outside of your system. Attributes of your aggregates from last year are still valid for events related to last year, even if they're now deprecated or no longer in use.

CQRS/ES is about system design that evolves. It evolves in a much cleaner and easier way if you follow it.

Is it perfect? No. There are some gnarly problems related to, for example, GDPR's "right to be forgotten", but that needs to be solved across database backups as well.

A couple of months back, a QA review reported a bug - the log system didn’t work. Turns out it’s never worked. Nobody has ever done a rollback either.

classic event sourcing require playing your events in order to calculate state. I think that's a complexity. Just do normal state computation but write it into a new timestamped row

We were modeling well dynamics. And to calculate interesting properties of the well we needed sensor data, the previous state of the well, and a model of the well built by a petroleum engineer.

Unfortunately the model of the well wasn't always up to date (pump was replaced, etc..), so we needed to be able to replay all of the sensor data into the well when a model was retroactively changed. Event sourcing was such an easy pick because our events were very simple (15 fields of data) and our replay requirement was a showstopper.

But if you have the luxury to do consistent CRUD with a relational/graph, ACID database, then a temporal data model might give you much more leverage.

Essentially you think of your records as bookkeeping entries and go from there. You retain the ability to do ad-hoc relational queries, while not having to build core database functionality yourself.

The business eventually failed to due to this, due to slow implementation of simple features and many other issues with it.

I will never use ES due to this project, it's pointless, anything you can do with it, you can do without it.

The word ”traditional” have also been thrown around.

Few weeks later we were sitting against this famous reproductive endocrinologist who were explaining us our options:

- Yeah, going to a fertility clinic is of course an option. However, you should at first perhaps consider a more traditional approach instead.

- What is the traditional approach, some medication?

- People seem to have forgotten how to have sex these days.

However embarrassed we were at that time (BTW he then laughed and saved us from the awkwardness), trying the obvious, easy solution and taking care of problems as they come became a conscious thing afterwards. I love over-engineering, but asking myself and the other stakeholders "what are we actually trying to accomplish here" is the most engineer-like thing I do nowadays.

I'm not suggesting just winging it all the time, but looking for the easiest solution first, not last. That's all.

Some features that would be needed:

* Automatic , schema based event and aggregate versioning

* A combined event and aggregate data store with ACID guarantees, schema validation and indexes

* event migrations (so you don't have to carry around old event versions forever)

* Event stream migrations that allow re-writing streams for purging data, gdpr cleanup, etc

* Copy on write based stream clones that can be used for testing with production data

As it stands, such a solution is nowhere to be seen. So you have to cobble together a working system with lots of custom work , which often ends in a complete mess.

I'm the kind of person who learns by my making my own mistakes (as opposed to learning from others', which is what smart people do)—and boy, did I learn a lot from the mistake of trying to build a general purpose application with event sourcing.

Theoretically, event sourcing is the way all applications should be built. I was first taken by this methodology by Martin Kleppmann's talk "Turning the Database Inside Out" (2014)[0]. Looking back, I think I had always subconsciously felt like there was something wrong with mutating state in place. You lose information. You lose accountability. After all, "accountants don't use erasers," and Martin's talk validated many of my obsessive-compulsive insecurities about data. As Charity Majors points out on Twitter [1]:

> ...while storing raw events may be "expensive" compared to any single optimization, all optimizations can be derived from raw events. The reverse is impossible.

And that's true. The table-stream duality [2][3] means that if you store events, you can later project those events into the exact perfect table for any use-case. But if you transform those events into a projection and store that projection, well, whatever table you store is data you have. You wind up having just an aggregate.

Unfortunately, the technical costs you incur when you attempt to store events instead of tabular data are tremendous. The problems and questions that arise are basically endless. Tons of things you wouldn't even think about with a relational database become complicated design decisions [4]. If there were tooling and frameworks that made this easy, it would be one thing. But, there are not!

Kafka Streams is very cool. If you already have a running business with well-developed models and a large team, event sourcing with Kafka Streams could possibly go quite well. But if you need to get an MVP off the ground, it's just not what you need. What you need is PostgreSQL, perhaps with something like Hasura or postgraphile in front of it. What you need is a lightning-fast way to iterate and flesh out your idea. If that's you, and you're reading this, and you're smart: learn from me—event sourcing: don't do it!

[0] https://www.youtube.com/watch?v=fU9hR3kiOK0

[1] https://twitter.com/mipsytipsy/status/1115537408705957888

[2] https://docs.confluent.io/platform/current/streams/concepts....

[3] https://news.ycombinator.com/item?id=23207123

[4] https://softwareengineering.stackexchange.com/questions/3801...

I've just started reading some Fowler stuff. What hot air are you referring to? I would like to keep that in mind.

Probably one of the largest technical blunders of my career (or at least the largest so far!) was pulling a team down into ES hell on a greenfield project.

Turns out, growing an event sourced application is fantastically hard[0]. Ours eventually became a twisted opaque mess which was terrifying to modify due to (a) the sheer volume of code it took to do so, and (b) eventually losing control over who read which events and where as the team grew.

[0] https://chriskiehl.com/article/event-sourcing-is-hard

Usually arguments go "we need an audit log so ES is a natural fit and we'll be able to iterate quicker because we can just re-project when we get things wrong, oh and testing is really straightforward too because we can just...".

You can have an audit log without ES. You're also going to hate versioning events. And navigating your tangled nest of events that you accidentally accumulated in the name of agile isn't the nirvana you were hoping for.

I believe it's better to have a system that responds to the requirements of here and now, leaving history behind, learning from mistakes going forward and accepting that yesterday is done.

In my experience, in growing software systems, you're more likely to have not to have even had the data/event at all, than to not have misappreciated what you already had. So event logs feel more like a bunch of baggage ready to trip teams up than a wealth of value.

Indeed. I don't doubt he's a very clever guy, but all of my most miserable gigs involved phalanxes of architecture astronauts endlessly bike-shedding every design session with the platitudes of Fowler et al.

This article isn't really pushing people into an overengineered and disastrous situation only because it doesn't say everybody should follow it. But it's a really bad article for everybody to follow and people are reacting like if it was recommended to them.

But where it’s really needed, nobody would choose a novice. So novices learn to do event sourcing on systems that don’t need it, and proceed to make a huge mess. Most companies don’t have any kind of architecture review to prevent this happening - they have managers who naively trust the developers.

CV oriented development.

The problem is that the CRUD is only recording "What" change/mutation was done, but not "Why" by "Whom", and "When" it was done. With tailing WAL / CDC you can also capture the "When".

This can be partially mitigated by adding audit fields like:

  Why (reason for change):

  created_bc ("because") / created_reason   (a person opened an a/c, a new baby born, or a record was migrated from another datastore)
  updated_bc ("because") / updated_reason   (fixed typo in the address, or moved to new address)
  deleted_bc ("because") / deleted_reason   (closed an a/c, an end-user died, GDPR request, soft-deleted, removed by moderator, etc.)

  By Whom (i.e. user_id, support person, decision maker, etc.):

  created_by
  updated_by
  deleted_by

  When (event time) - with CDC you can capture all the intermediate updated_at events:

  created_at
  updated_at
  deleted_at

The industry is moving chaotically and sometimes getting stuck in a local minima for a decade. The innovation follows diffusion model, and it slowly builds momentum until the tipping point is reached.

So I do agree, that CQRS/ES might be the next informal industry standard, but we do need tooling, best practices, use cases and success stories.

Maybe we need a special programming language, a DSL, or a framework for it, or a special DBMS.

Another point: it's not that easy to build a correct, scalable and maintainable moderate-complexity CRUD app. Anything beyond a simple blog engine or TODO list tutorial can quickly become a mess. The tutorials are omitting error handling, and that where the juniors are learning from.

Last anecdotal datapoint: when building a moderate-complexity CRUD app, and trying to handle all edge cases/error handling/correctness, I realized that I'm building a poor man's CQRS, so I might do the real thing as well.