The Way We Are Building Event-Driven Applications Is Misguided

Another "forget this style, only use our new framework". What he forgets to mention, and what is probably not in the scope of "our new framework that does everything better", is that it is totally valid to use orchestration and choreography. Same as Confluent pushes the case to use choreography, this post pushes to just use orchestration.

That's wrong. From my point of view:

  1. use choreography for small to medium services (for example one domain)
  2. optionally orchestrate them on a higher level (for example between domains)

Also: You don't *need* to store all your events forever. This pattern, often pushed, comes with downsides as well. Maybe you have a good case to do so, but don't fall into the trap to just use this pattern because you've read that in some blog post!

Alternatively, you can also just go the route of those vendor blog posts and pay later for the lock in.

As with yesterday's "big data" article, the usual question is whether you have enough transaction volume to need all this stuff. If you can organize your web site so that most traffic is read-only files, and the heavy machinery only turns on when somebody does a "buy" or something important, the whole problem probably fits into a CRUD app.

There's a useful insight in there - fan-in and fan-out are hard. I hit this in a completely different context - asset handing for a game-type client. Events come in ordering the creation of object X using assets A, B, and C. It may take several asset fetches from the network to draw something. An asset may be used for multiple drawable objects. There's caching and concurrent asset fetching. A request to create X has to start fetches for A, B, and C (they might be in cache, though) and wait until those fetches complete. A request for Y might want B, C, and D. Items should be fetched from the network only once, of course. This is a messy coordination problem.

Problems like this, with fan-in, fan-out, and concurrency, map badly to the standard paradigms. Neither threads with blocking nor "async" help much. The problem can be visualized as set operations, but can't easily be implemented that way. A single thread event driven coordination loop works, but it's kind of clunky.

So I started reading the article hoping for a theoretical breakthrough on fan-in, fan-out problems. No such luck.

The set-theory approach is hard to do, but promising. Each object that wants something has a small set of the things it wants. There's a big pool of such sets. There's also a big pool of the items you have, which changes constantly. It's easy to express what you need to fetch and which objects are now ready to go as set intersection and difference operations. But you need representations for big sparse sets which can do set operations fast. Probably B-trees, or something in that space.

Microsoft Research fooled around with this concept years ago in a different context. The idea was to have a database which supported pending SQL queries. The query would return new results when the database changed such that the results of the query changed. The goal was to to support that for millions of pending queries. Financial traders would love to have that. It's a very hard scaling problem. Don't know how that came out.

Another (maybe more mature) alternative to Infinitic is Camunda, who have been pushing this rhetoric for over a decade.

My problem with both is that neither feel very modern, in the sense that they are tied to (in Infinitic's case) or strongly favor (in Camunda's case) Java-centric development, don't have a great developer experience story, and don't feel very cohesive with the language.

What I'd love to see is someone tackling this for smaller orgs where the above tradeoff in developer experience isn't worth the gains, i.e. orgs with <100 engineers. Something where you have a single source of truth for your processes and schemas, with version control, that directly yields strongly typed integrations & hooks into any language, with a great local developer experience and deployment story.

Camunda requires too many magic strings and schemas to be kept consistent across services for that, and Infinitic forces me to use Java or Kotlin which I don't want.

A "simple" solution might be to have declarative process and schema definitions in some DSL for version control, which auto-generates schemas, types etc in whatever language, giving me both strong types, intuitive local development, and clear deployment story through my existing CI/CD.

A workflow or “orchestrator” as defined in this article looks like another service. Unless I’m missing something, it’s a state-machine where inputs/variables are the events of other services, and where transitions trigger “commands”.

Don’t get me wrong, I see the value in pulling this logic into a central place/process in order to perform “orchestration”, I just don’t see the value in yet another framework. This is a pattern that pops up in every distributed system I’ve worked on, but it is just one of many.

In many cases, a synchronous request response end-point is all that is necessary/desired for a service tied to a customer interaction. In others, a service processing a queue backlog (with or without automatic fan-out). In others publish-subscribe seems the best fit. In others, a more complex state-machine managing several different events, transitions, and outputs.

Can you model everything as a state-machine? Yes. Should you? I would argue that you want a range of tools in the toolbox.

Both very technical, both with very little "sell" despite being given by a Camunda co-founder and a Temporal principal eng.

https://www.youtube.com/watch?v=zt9DFMkjkEA "Balancing Choreography and Orchestration" by Bernd Rücker

https://www.youtube.com/watch?v=EaBVzjtSK6A "Building event-driven, reactive applications with Temporal: Workflows vs Sagas" by Dominik Turnow

Buzzword architecture is awful, and having Kafka offer fake design patterns to reassure naive architects, such as a request/reply with small response timeframes.

Attempting to replicate REST with overcomplicated overhead.

This is the biggest issues I encounter with event driven applications.

Maybe I missed the party trick but under the hood a step function definition will be orchestrated between lambdas (the services).

The only difference to this pattern is you define event inputs and outputs on the service, as part of orchestration, which I feel is absolutely needed to know by consuming services (why wouldn't you, unless you're hoping to avoid object versioning / changes clashes of services by doing that, which is better served by you know, actual versioning tags...).

I don't know what type of cross-domain communication I need until I model something. Any given domain could have many published events. Any given domain could have many commands. Any given domain could have many subscriptions.

The business models of those domains come first. That dictates everything including APIs, data store, commands, and events.

This article is similar to saying all applications should use a relational database or all applications should use AWS Lambdas.

The most important step in any architecture is "It depends."

Notably the article calls out tracing, which is the bottom-up solution to this. Choreography functions by dictating a flowchart that events move through, so visible comes from that dictation. Tracing functions by watching what events move from which producers to which consumers, and surfacing that organic data.

I'm not sure I see the fundamental advantage of dictating these flows rather than observing and reporting on them with some kind of internally-standardized conventions on message metadata with some standardized tooling to consume that. They seem to end up in a fairly similar situation: producers can see their consumers and vice versa, both sides need to coordinate on changes.

I really don't think it's that much boilerplate, at least compared to the pain of dealing with a choreographer that's failing to scale or just generally bad (and which you now can't replace, because everything is hand-tailored to it).

The general advice I have here is to not use event-sourcing/event-driven architectures for everything.

Use it where the domain maps well to a process that happens over time. An order system is the canonical example. That doesn't mean your products, customers, accounts, etc all need to be event sourced/event driven as well. It will be fine leaving those managed by a CRUD architecture.

Keep things as simple as possible (but no more simple).

There are plenty of areas where an event-driven/event-sourced architecture will make things easier and others where it will make things way harder. I find if you need a lot of sagas/orchestration... you're delving into the latter.

(I make the distinction between event sourced and event driven as this: the former is about persisting and deriving state; the latter is about communicating changes to other systems).

It had some flaws, of course, but the general idea was you drew business processes in a UI tool (or wrote the corresponding code that generated the process, but drawing was easier) and the process was then implemented into messages queues/topics and different types of integrations as part of the deployment process.

Because it was standardised, you could plug in a product that visualised processes, showed where bottlenecks were, let you restart failed ones, etc. In retrospect it was pretty advanced.

[0] https://www.trustradius.com/reviews/webmethods-business-proc... - sorry for the puff piece link but the corporate website is so bad

My preferred form of orchestrator is a gatekeeper. A gatekeeper does three things only:

- maintains a table of which service consumes which events and why - copies events from the outgoing queues to the dedicated incoming queues according to the routing table (or controls access to the incoming queues via RBAC) - logs the metadata about the messages it has seen

It doesn't inspect the messages in any way, store them or maintain any runtime state at all.

You write:

There is no easy way to understand how a specific business process is actually implemented. The whole process is defined by how each service reacts to others. There is no central repository, and the implementation details of business processes are scattered everywhere.

Yes, indeed. There is no easy way because the total system architecture is implicitly defined by the interactions of the components.

There is currently no way to actually express the architecture of such a system as runnable/testable and running code. You have the options of either (a) documenting the architecture or (b) leaving it to be defined implicitly.

The advantage of Kafka is that we can use a well known battle tested solution as our backbone and write buggy software around it.

For example, "built-in observability of workflows states" seems to imply very invasive constraints on workflow/orchestrator implementations: maybe a net positive, but contrary to the promise of an "easy way to implement your business logic in full Java".

It's a queue, it's a stream, it's a journal, it's an event bus, it's a database.

It's also apparently a workflow engine now?

It annoys me so much how appealing this is to a median developer. "Wow I just get to learn this one tool and then I can use it for EVERYTHING!" is not something competent craftsmen/professionals do in any other field, yet in software it's all the rage.

"The way we are building event-driven applications is misguided"

Maybe the way you are! Don't overcomplicate/overabstract things so much!