Can someone share some long term event driven success stories? Almost everything you see online is written by consultants or brand new, greenfield implementations, curious how long these systems last.
The core orchestration of the system was done via events on the bus, and nobody had any idea what was happening when a bug occurred. People would pass bugs around, “my code did the right thing given the event it got”, “well, my code did the right thing too”, and nobody understood the full picture because everyone was stuck in their own silo. Event driven architectures encourage this: events decouple systems such that you don’t know or care what happens when you emit a message, until one day it’s emitted with slightly different timing or ordering or different semantics, and things are broken and nobody knows why.
The worst part is that software is basically “take user input, do process A on it, then do process B on that, then do process C on that.” It could have so easily been a simple imperative function that called C(B(A(input))), but instead we made events for “inputWasEmitted”, “Aoutput”, “Boutput”, etc.
What happens when system C needs one more piece of metadata about the user input? 3 PR’s into 3 repos to plumb the information around. Coordinating the release of 3 libraries. All around just awful to work with.
Oh and this is a very high profile piece of software with a user base in the 9 figure range.
(Wild tangent: holy shit is hard to get iOS to accept “do process” in a sentence. I edited that paragraph at least 30 times, no joke, trying every trick I could to stop it correcting it to “due process”. I almost gave up. I used to defend autocorrect but holy shit that was a nightmare.)
can you not just pick the original spelling in the autocomplete menu above the keyboard?
If you have some logic A and B running on user input, I wouldn't be splitting that across different services.
I can attest to this case study being 100% true. Our platform has been using EventStore as our primary database for 9 years going strong, and I'm still very happy with it. The key thing is that it needs to be done right from the very beginning; you can't do major architecture reworks later on and you need an architect who really knows what they're doing. Also, you can't half-ass it; event sourcing, CQRS, etc all had to embraced the entire time, no shortcuts.
I will say though, the biggest downside is that scaling is difficult since you can't always rely on snapshots of data, sometimes you need to event source the entire model and that can get data heavy. If you're standing up a new projector, you could be going through tens of millions of events before it is caught up which requires planning. It is incredible though being able to have every single state change ever made on the platform available, the data guys love it and it makes troubleshooting way easier since there's no secrets on what happened. The biggest con is that most people don't really understand it intuitively, since it's a very different way of doing things, which is why so many companies end up fucking it up.
Like I get the "message bus" architecture when you have a bunch of services emitting events and consumers for differing purposes but I don't think I would feel comfortable using it for state tracking. Especially when it seems really hard to enforce a schema / do migrations. CQRS also makes sense for this but only when it functions as a WAL and isn't meant to be stored forever but persisted by everyone who's interested in it and then eventually discarded.
Enforcing the schema isn't too hard ime. But every migration needs to be bi-directionally compatible. That's likely what they meant with "you need an architect and can't make major changes later on"
It's the same issue you've had with nosql, even though you technically do have a schema
eg. Your commands have to ALWAYS do the same thing else replaying the event log does not produce the same output and then you’re back to square one.
It’s usually easier / more useful to just use an audit table.
Yes, and I honestly think a traditional database that exposed this stuff would be a winner (but I guess it's too hard, and meanwhile event-sourcing frameworks are building better alternatives). Separating the low-level part from the high-level part that does things like indexing and querying has a lot of advantages: you decouple data storage from validation so you can have validated data without having to drop invalid data on the floor, you decouple index updates from data writes so your scaling problems get way simpler, you can get sensible happens-before semantics without needing transactions that can deadlock and all the crazy stuff that traditional databases do (secret MVCC implementations while the database goes out of its way to present an illusion of a single "current" version of the data, snapshotting that you can't access directly, ...).
I suppose you could have event sourcing in a purely isolated manner where the events aren't used for anything else, but you'd be severely limiting the advantages that come free with that.
Your example uses the same events for both so sure that can be done but doesn’t have to. I haven’t worked on a system like that personally so maybe it can fine.
But honestly I’m a bit skeptical since that removes services’ data sovereignty. Sounds like the recipe for “distributed monolith” architecture. And actually I just remembered another team at my company is ripping out their use of kafka as their data source on a green field project cuz it was a disaster, so skepticism emphasized.
I also tried doing it in a property setting, where profit margins were tight. The effort needed wasn’t worth the cost, and clients didn’t really care about the value proposition anyway. We pretty much replaced the whole layer with a more traditional crud system.
CRUD: Imperative. Client tells us to create/update a specific entity with some data.
In web or business systems it works well for some(!) parts. You just shouldn’t do everything that way - but often people get too exited about a solution and then they tend to overdo it and apply it everywhere, even when not appropriate.
Always chose the golden middle path and apply patterns where they fit well.
Event driven and CQRS "entities" made logic and processing much easier to create/test/debug.
Primary issues: 1. Making sure you focus on the "Nouns" (entities) not the "Verbs". 2. Kafka requiring polling for consumers sucks if you want to "scale to zero". 3. Sharding of event consumers can be complicated. 4. People have trouble understanding the concepts and keep wanting to write "ProcessX" type functions instead of state machines and event handlers. 5. Retry/replay is complicated, better to reverse/replay. Dealing with side effects in replay is also complicated (does a replay generate the output events which trigger state changes in other entities?)
Been running now for 6 years, minimal downtime except for maintenance/upgrades.
In the process of introducing major new entity and associated changes, most of the system unaffected due to the decoupling.
The focus when designing the system should be on the entities (Customer, Payment, Bill, Order, Inventory) instead of the processes (ordering, billing, fulfillment). I summarize that by saying "Nouns over Verbs".
The state of each of the entities is affected by the processes, but the effect happens from changes in other entities, Customers place an Order. Customers get a Bill for the Order, Customers make a Payment, etc.
The states of each of these entities is independent of the others and reacts/changes only as a result of two things, either an external "Command", or an "Event".
Commands are events that occur outside of the system boundary, usually visible as part of an API (if RESTful) that uses POST/PUT/DELETE or they are imperatives from one entity to another.
Commands are imperatives, Place Order, Pay Bill, Fulfill Order, etc.
Events are records of occurrences in the system, expressed in the past tense and are immutable. Order Placed, Bill Paid, Order Fulfilled.
Customers place an Order by POSTing to /orders (or potentially /customers/uuid/orders).
Events are generated from entities inside the system. (Order being placed generates an order_placed event).
The difference is that by focussing on the entities, and their state, independent of other entities, the entities can be created, tested, installed, evolved independently of other entities in the system.
The thinking about them is simplified and focussed, they are naturally decoupled because they can only find out about other entities by inquiring or affect other entities by generating a Command or an Event.
Any events they generate are processed asynchronously and can have multiple consumers.
(No stake in this one way or another, just curious.)
Chances are that system, and its outcomes are described very differently on a resume
These system are working fine, but maybe a common ground : * very few services * the main throughput is "fact" events (so something that did happen) * what you get as "Event carried state transfer" is basically the configuration. One service own it, with a classical DB and a UI, but then expose configuration to all the system with this kind of event (and all the consumers consume these read only) * usually you have to deal with eventual consistency a lot in this kind of setup (so it scales well, but there is a tradeoff)
The WAL is an event log, and when you squint at its internal architecture, you’ll see plenty of overlap with distributed event sourcing.
As is any web application, because the web (at least without sockets) is constrained into communicating events only.
Also, most local GUI applications, because people just like events better for it.
Our users are small-businesses with organisation numbers, and we mostly think of them as unique. But they strictly aren't, so we 'overwrote' some companies with other companies.
Once we detected and fixed the bug, we just replayed the events with the fixed code, and we hadn't lost any data.
Every use I've seen sent events after database transactions, with the event not part of the transaction. This means you can get both dropped events, and out of order events.
My current company has analytics driven by a system like that. I'm sure there's some corrupted data as a result.
The main issue being people just don't know how to build and test distributed systems.
It sounded kind of impossible, I said as much, and then proposed a different approach. The interviewer persisted and claimed that it could be done with 'the outbox pattern'.
I disagreed and ended the interview there. Later when I was chatting about it with a former colleague, he said "Oh, they solved the two generals problem?"
> Every use I've seen sent events after database transactions, with the event not part of the transaction.
Maybe this is what they were doing.
The point of the outbox pattern is that a durable record of the need to send an event is stored in the DB as part of the DB txn, taking advantage of ACID guarantees.
Once you have that durable record in your DB, you can essentially treat your DB as a queue (there's lot of great articles on how to do this with Postgres for instance) for some worker processes to later process the queue records, and send the events.
The worker processes in turn can decide if they want to attempt at least once, or at most once delivery of the message. Of course if you choose the later, then maybe your event is never sent, and perhaps that was the point you were trying to make to the interviewer.
They key takeaway though is that you are no longer reliant on the original process that stores the DB txn to also send the event, which can fail for any number of reasons, and may have no path to recovery. In other words, at least once delivery is now an option on the table.
Well then, hopefully you would have found it an unsatisfactory 'solution' and walked away from that interview too ;)
> Once you have that durable record in your DB, you can essentially treat your DB as a queue (there's lot of great articles on how to do this with Postgres for instance) for some worker processes to later process the queue records, and send the events.
Yeah but I already have a queue I can treat as a queue. It's called Kafka.
(P, K)
Their solution was to introduce another part into the system, "the outbox":
(P, O) K
Without the outbox you can get P without K or K without P, which lead to consumers out of sync with the producer.
This requires the consumer to be able to deal with repeated events to some extent, but you usually want that anyway, since an event can be processed twice even if it appears only once in the topic. For instance if the consumer crashes after processing but before updating the offset.
Right, which is why it's an unacceptable solution to 'transacting over postgres and Kafka', and why I wouldn't want to work for a company that wants me to believe differently.
And there's a better solution anyway: just K.
However I've seen some frameworks where you can do collision imperatively. For example
if (sprite.collide(tilemap)) {do something}
These are generally on smaller less taxing frameworks (in this case I'm referring to haxeflixel) but they do exist!
So we ended up using protobufs over a local MQTT broker and adopted a macro-service architecture. This suited the project very well because it had a handful of obvious distinct parts and we took full advantage of Conway's law by making each devs work the part where their strengths and skills were maximized.
We made a few mistakes along the way but learned from them. Most of them relating to inter-service asynchronous programming. This article put words on concepts we learned through trial and errors, especially queries disguised as events.
I think it works well when it's the only thing that can work.
- Producer and consumer are decoupled. That’s a good thing m right? Good luck finding the consumer when you need to modify the producer (the payload). People usually don’t document these things
- Let’s use SNS/SQS because why not. Good luck reproducing producers and consumers locally in your machine. Third party infra in local env is usually an afterthought
- Observability. Of rather the lack of it. It’s never out of the box, and so usually nobody cares about it until an incident happens
It sounds like your alternative is a producer that updates consumers using HTTP calls. That pushes a lot of complexity to the producer and the team that has to sync up with all of the other teams involved.
> Let’s use SNS/SQS because why not. Good luck reproducing producers and consumers locally in your machine
At work we pull localstack from a shared repo and run it in the background. I almost forget that it's there until I need to "git pull" if another team has added a new queue that my service is interested in. Just like using curl to call your HTTP endpoints, you can simply just send a message to localstack with the standard aws cli
https://github.com/localstack/localstack
> Observability. Of rather the lack of it. It’s never out of the box, and so usually nobody cares about it until an incident happens
I think it depends on what type of framework you use. At work we use a trace-id field in the header when making HTTP calls or sending a message (sqs) which is propagated automatically downstream. This enables us to easily search logs and see the flow between systems. This was just configured once and is added automatically for all HTTP requests and messages that the service produces. We have a shared dependency that all services use that handles logging, monitoring and other "plumbing". Most of it comes out of the box from Spring, and the dependency just needs to configure it. The code imports a generic sns/http/jdbc producer and don't have to think about it
I just grep for the event's class name.
The amount of times I've come across someone who's inserted SQS into the mix to "speed things up"...
JavaScript
When I say increased, I mean we want the best answer but there are some answers the bank can’t know. If someone has transferred money into your account from another bank but we don’t know that yet, optimising for absolute correctness is pointless because the vast majority of wrong answers are baked in to the process. We can send you a message and you might read it a day later. Unless we delete the message from your phone, we can’t guarantee the message you read is fully consistent with our internal state.
Frankly our system is much better than the batch driven junk that is out of sync a second after it has executed. “Hey you have a reward.” “No I used it 2 hours ago you clowns.”
Note this isn’t cope. In some cases we started fully sync but relaxed it where there are tradeoffs that gave us better outcomes and we weren’t giving anything material up.