Now when people argue “because decoupling,” I hear, “You don’t get as much notification that you just broke a downstream system.”
https://www.datadoghq.com/knowledge-center/distributed-traci...
Unless you have a single monolith, you’re going to face issues with versioning whether it’s event based or API based. In each case you can usually add new properties to a message, but you can’t remove properties or change their types. If you need that, create a new version.
The author does a lot of videos on the event sourcing topic. Event driven I get. It works well in several applications I’ve helped to build over the last 15 years. But event sourcing? I truly don’t get it. Yeah I get it’s nice in terms of auditing to see every change to an entity and who made it, or replay up to to change x on y date, but that really is a niche requirement.
I'm not sure what point is being made here. It's good that you can do that - but are you implying that that's not possible in an API-driven system?
It's not just about auditing, it's also about transactionality and atomicity.
If you want to withdraw $5 from your account, the traditional approach of locking, updating everything, unlocking (or in other words wrapping everything in a transaction) doesn't scale as well as the notion that you just record the transaction (event). Implementation-wise this withdrawal can involve, updating two accounts and updating the audit/account transaction logs. We also want this to scale since our bank has millions of customers all operating more or less concurrently. A distributed log (like Kafka) is easy to scale and easy to reason about. You just insert the transaction record and you have a distributed system that will scale and is easy to reason about.
Another driver/flavour for something like event sourcing is what some might call state-based or state-oriented programming. That is instead of modifying state directly you are synchronizing state via events. This lets you e.g. code state machines around those that can lead, again, to easier to reason about (and test) code.
EDIT: Also event sourcing would typically be eventually consistent. I imagine for some banking applications a stronger consistency guarantee might be required, e.g. to prevent you from withdrawing the $100 in your account multiple times.
I did work somewhere that used a Kafka stack in production. It wasn’t a compelling use case and they spent almost an entire year on infra and productionizing it. It left me extremely sceptical about anything “big data streams” related :)
Can't wait to read the incident report when one of the consumers successfully receives and reads the message but the other doesn't.
There are of course other ways to do auditability.
Event Sourcing + Projections provide a nice way to build multiple models/views from the same dataset. This can provide a lot of simplification for client code.
There are also other companies, which do the typical snapshot and roll up to the current time, when they start the services, that need the data without having access to the database.
That's not exactly an obscure feature exclusive to datadog. From the top of my head, both AWS and Azure support distributed tracing with dedicated support for visualization in their x-ray and application insights services.
When you've never grown out of a single node domain but you do event driven "because scaling" or whatever, you've shot yourself in the foot amazingly hard.
But people often forget there are trade-offs to everything and if you don't have these hard problems, you're giving yourself only headaches.
My pet-peeve is "decoupling" - it's treated as holy with only benefits and no downsides. But it's actually again a level of complexity - unless you need it, tightly coupled code will be easier to write, read, debug etc.
As an event producer as long as you follow reasonable backwards-compatibility best practices then you should be pretty safe from breaking things downstream. As a consumer, follow defensive programming and allow for idempotency in case you need to reprocess an event. Pretty straightforward once you get the hang of things.
That can protect you from "downstream can't even read the message anymore" but it doesn't help you with the much more common "downstream isn't doing the right thing with the message anymore" problem. Schema evolution is kinda like schema'd RPC calls vs plain JSON: it will protect you from "oops, we sent eventId instead of event_id" type of errors, but won't prevent you from making logical errors. In a larger org, this can turn into delayed-discovery nightmares.
A synchronous API call could give you back an error response and alert your immediately to something being wrong. The system notifies you directly.
A downstream event consumer may fail in ways entirely off of your team's radar. The downstream team starts getting alerts. Whether or not those alerts make it immediately obvious to them that it's your fault... that depends on a bunch of factors.
Events sent for readership you can’t control are ideally of the type «x changed», and the consumer must then fetch data on the relevant endpoint.
That or the company must have serious versioning policies.
Only make additive changes, don't change existing fields. For enums it's up to the consumer to ensure they don't fail when a new case is added.
Be very careful with including data (especially time/expiry stuff) in the message too. If you need to reprocess the event several hours later then it may no longer work or be stale. Rather than include the data in the message itself, we would include the database ID and then have the consumer query for that entry.
Tldr: ok to add things. Not ok to remove things or change things
I don't know how this could be true. Events are things - nouns which can be backed-up, replicated, stored, queried, rendered, indexed and searched over.
I generally like event-driven architecture, but I need to admit that debuggability is sacrificed where it matters most.
Remember "callback hell"? Assumption that a function call returns after running to completion requires rather specific synchronous cascading architecture, which WILL break in multithreaded code. Most of the multithreaded function calls will set a flag in shared memory and return early, expecting caller to poll.
If your API is based on single entry-point `invokeMethod(callee, method)` it is equally untraceable to event entry point `fireEvent(producer, event)`.
Which is exactly switching from function calls to event-driven architecture, and the problems with that are exactly the problems we're talking about.
You do not even need return-early (non-blocking) semantics for these problems to manifest. You can implement a giant string-keyed vtable for all methods in your program (or use a language with advanced reflection capabilities) and will have exactly the same problems. Namely there probably won't be tooling to match caller-callee pairs, which is the core issue here.
const
myEvent = 'myEvent',
target = new EventTarget()
target.on( myEvent, () => {
console.log( "It's easy to introspect well-organized code." )
})
target.dispatchEvent( new Event( myEvent ))That's not something I have to remember or forget, it's a simple habit that is as natural as importing and referencing a function.
As a general rule, numbers and string literals should never be hardcoded. Internalizing this should be a base expectation of any high-performing team member.
And as a consumer, many independent tasks can be triggered by the same event.
I'm working on a system right now and because of events, it's very easy for me to write a handler for when a certain type of record is created in the database. My feature depends on knowing that new record was made so we can send some emails and do other things.
The people that wrote the code that creates the record, didn't have to do anything to support the feature.
But I agree that it's not the right solution for every problem. But there are certain problems it solves really well.
Right up until you need to change something about the event because the business logic it represents has changed. Then you suddenly need to track down all the systems that have been relying on it, including that one that nobody knows anything about and always forgets exists because some guy decided to implement the service in erlang and nobody who ever touched it even works at the company anymore.
First, if your event (or whatever) changes enough that there are inter-component breakages it means engineering requirements must have changed and tracing dependencies of requirements is organizational thing.
Second, you either do trunk based development and constantly break downstream or do leaf based development and have constantly out of date core dependencies. In any case, that's release version management, which is again organizational thing.
Don't take it into consideration and you're fucked.
Source: previous "seniors" didn't take it into consideration, they left
Same issue as microservices: there are people who want to use the paradigm but not do the investment in monitoring/tooling.