Show HN: Learn what DDD, CQRS, and event-sourcing are all about
docs.wolkenkit.io
docs.wolkenkit.io
I believe that the core principles of DDD/CQRS/ES are flawed and are the cause of a lot of pain. If your team plows forward and ignores the pain, you will have a bad experience working there. If your team decides to fix the problems with the architecture (usually in a way that's rather specific to their situation), and still insists that they're doing DDD/CQRS/ES, then you will have a great experience. The same can be said of many approaches to SQL database design.
When I joined Lokad, most of the code had been written by a strong CQRS advocate, and there were significant pain points caused by this approach that would have been nonexistent with a plain old SQL architecture. The author himself did a retrospective that covers a few issues: https://abdullin.com/lokad-cqrs-retrospective/
Instead of dropping everything and moving back to a more conventional SQL architecture, we dropped everything and moved to a solution which, if I were to describe it as DDD/CQRS/ES, would have me branded as a heretic by the DDD/CQRS/ES community: it does fit the definition, but it does not follow the best practices. There is a single data model shared by both commands and queries. Commands can sometimes return data. Each (micro?)service has access to exactly one aggregate. There is no uncertainty on event ordering, and no eventual consistency at the aggregate level. Materialized views for each aggregate are kept in-memory at all times. All of these intentional weakenings of the CQRS architecture were driven by pragmatic needs, and made with knowledge of the consequences.
"There is a single data model shared by both commands and queries." - Nothing wrong with that, especially in simple cases.
"Commands can sometimes return data" - Well commands can succeed or fail. Greg young himself admits commands are really synchronous, and you need to communicate that.
"and no eventual consistency at the aggregate level." - This is the recommended approach. Between aggregates and it becomes loser.
"Materialized views for each aggregate are kept in-memory at all times" - Thats some advanced CQRS usage there.
But a messaging system for events.
If i'm using event sourcing use a dedicated event sourcing database with catchup subscriptions etc
Can you recommend any?
Commands are by definition synchronous and return a success/failure.
One of my favourite read models to use is an in memory model providing I can keep all of the data in memory (and its reasonable for the query patterns).
when you say "materialized views" - do you really mean the current state of an Order for example ? and by "in memory", is it being persisted to a database ?
Then, we would have a materialized view that would be a single C# immutable object (canonically called `State`) that contains a dictionary of every order, and maybe a few indices (such as "identifiers of orders currently waiting to be shipped out" or "list of orders by customer"). Everything fits in memory, so every query is simply a bit of C# code that traverses the in-memory graph of objectd rooted at the `State`.
A command emits events, which are then applied by the system to create a new `State` from the previous one. Events emitted by other instances of the application are also fetched by the system every so often, and you can request a "force catch-up" to guarantee that the `State` you access takes into account all events appended so far (so that you're certain to see the results of your actions, even if the load balancer bounces you from one instance to another).
Here's what we use: https://github.com/Lokad/AzureEventStore
Having read the article: veeeeeery little of what is mentioned is related in any meaningful sense to CQRS...
Horse-before-the-cart framework design is always bad. Encapsulating untested theories in frameworks instead of lifting them from operational designs, always bad. Building frameworks as a learning exercise, bad. Abstracting away data decisions and then trying to provide opinionated answers to storage at the framework level, very bad. Conflating logical system architecture with supplier provided products (ie cloud services), is framework _cancer_.
That is simply not a reasonable starting point to provide value and scalable systems, and the end result was the same as all other similar attempts: an A for effort with a result that was at best 'close, but not quite...'
To be clear: I do not intend this as criticism leveled at the author or the company. The article is great. I very much agree with his analysis though:
> Lokad.CQRS was created with a very short-sighted design approach in mind, a reusable LEGO constructor...These days I'd try to limit the damage I inflict upon the developers and avoid writing any widely reusable frameworks
This is where all the issues lay... Premature framework development causing up-front decisions untethered to practicality based on conjecture and prognostication instead of experience in a challenging domain insufficiently understood by the framework architect.
From a high level perspective: neither CQRS, nor framework development, nor application development were sufficiently groked before being baked into a long-term production commitment. Those are organizational issues, not CQRS issues. In the same vein, coming from such an environment and basing judgment of DDD or CQRS on anything that came from that environment won't hold water.
The architectural problems you're talking about, the core principles you think are flawed, work stunningly well in other places that have not made the same underlying mistakes. CQRS advocates != CQRS experts != framework experts.
It's just fundamental aspect of asynchronous programming, conveniently forgotten trying to sell solutions.
Edit: I would argue that you can't write a system which is open (several domains) and believe that you will have entirely independent events throughout the system and claim that your system will be "bugfree"/correct.
So this means for your problem to be correct you have to use the correct tools to solve that problem. If you think that everything (all problems) can be decomposed to independent events, which per definition can happen in any order, you can safely use ES. But if they have any dependency on each other you need ordering, and ordering in a distributed system is a hard problem. And add to this is that somewhere some guy from the business will come up with an actual requirement where you must synchronize and if you then have solution which inherently where this is not possible you are screwed.
This is true for both reading and writing in that system. Most problems are dependent otherwise it won't solve/do anything, there are usually a purpose with most systems.
While you can have systems where the events are entirely independent, structurally independent, such as JS event loop and red/blue methods (just locally independent), or it doesn't matter like plain application logs, this is rarely, if not impossible between domains if you want a correct system since you won't know if all events have arrived yet to tigger another event, if there's any dependency between them, which there will be between two domains.
DDD is good but not in this context, CQRS is trying to partially to solve above problems, but not needed and is just bloating. It does fit better with C# since the (weird) implementation of async/await and potentially with JS for the same reasons. ES do have their use cases where they fit very well but not with the practical impacts of DDD.
> user.login({ login: 'user', password: 'secret'});
> const isUserLoggedIn = user.isLoggedIn();
If you look at this purely from an API design point of view, this stinks: by forcing the command to return void, the code ends up twisting its design to return the result of the command through an artificial channel (the state of the 'user' object). In the end, we are still interested in "the result of the login attempt", and the easiest way to do this would be a function that returns a value:
> const isUserLoggedIn = user.login({ login: 'user', password: 'secret' });
So, what's this about? about DDD, CQRS and event-sourcing, or about a particular Javascript framework? Those two aren't the same at all (even though I suspect there is a little overlap).
The last 1/3 shows how to apply these concepts using wolkenkit, which is a specific framework.
When we created the brochure it seemed important to us to first explain the concepts in a neutral way, because we want to help people gain a better understanding of these concepts, no matter whether they are going to use wolkenkit or not.
You might have an easier time getting people to read it when they don't get the impression "it's just another javascript framework".
Is this really the case? I was always under the impression that everything regarding financial transactions needed to be transactional.
So financial transactions are transactional, but there's no guarantee that every node has a fully up-to-date state at all times.
Which is very annoying because the article about CSS can't get it's UI right.
They call Svelte "The magical disappearing UI framework" and indeed, their fonts look like they are written in an invisble ink about to dry up.
Some bank transactions, for example, do not have that luxury because it might allow 2 possible answers to the same query if the eventuality has not been met.
Some trading transactions don't have that luxury either because the milliseconds that you waited for the consistency a stock might have changed its price dramatically.
It really depends on the Domain if the time to wait for the eventuality is acceptable or not.
Rarely do you have an up to the microsecond view of the markets (unless you are located appropriately and pay significant amounts of money). There is always a delay in the information's accuracy and relevancy.
A domain that can't handle any delay in consistency is atypical in my experience. Either way you can do DDD/CQRS without eventual consistency.
Yes you can do DDD/CQRS without ES but CQRS is not needed, it's just a technical construct not solving any verifiable problem. DDD on the other hand is more sane and to some extend, verifiable.
I live in the US, and if I got to the ATM and get cash on my account, I can go to my iPhone app and see the transaction and the balance updated.
Yes, there is a nightly process, but not everything is a nightly process.
"Rarely do you have an up to the microsecond view of the markets (unless you are located appropriately and pay significant amounts of money). There is always a delay in the information's accuracy and relevancy."
Do you know what a hedge fund is?
Eventual consistency is not all that scary, try putting on a business hat instead of a technical one. Its about risk management.
You may also have two independent items of information that have no consistency relationship or requirements.
That's why eventual consistency "works" so "well" with NoSQL: because there's an overlap between using NoSQL databases and not requiring actual consistency.
You can't get rid of eventual consistency, obviously, but you can set up your system so that your code expresses rules and features in an "immediately" consistent abstraction, and the system then translates those rules and features into the eventually consistent world, by applying tactics and techniques that you would have otherwise applied manually. It's a really natural approach for programmers: automate the translation from immediate to eventual, instead of doing it yourself every time.
Of course, most of these tactics come at a cost, either of performance (we won't acknowledge the creation of a gizmo until we've propagated that creation to every server capable of rendering the list of gizmos) or of availability (the registration of your username can't be acknowledged unless the persistent consensus reflects that you're the only one who registered it). But that's fine: most of these things aren't critical enough to make it worth dealing with eventual consistency manually, and when they do become critical, it's time to go down a level of abstraction and apply immediate-to-eventual translation tactics that rely on additional assumptions about what you're doing in order to improve performance or availability.
Well it's all about errors (and subsequently about control and correctness). Even if you have two transactions you are able to undo the other without anyone else (readers of the system) know that it was there because something else happened just before yours. A transaction usually happens in an atomic fashion, which makes it possible to back out on something which is no more correct so you have to fail it somehow to someone which does have the ability to recover, the one who initiated the call.
Without being too philosophical, most, if not everything depends on synchronous state (fex events themselves), computing just gives the illusion of that things happens immediately. Eventual consistency is cheating, and you can do it where the events are entirely independent. Otherwise you will have side effects when events are not ordered.
JavaScript have an "asynchronous" coding style, where (red/blue) methods are put on event-loop as "events". But the events needs to executed in order (synchronously by a thread) for the program to behave correctly so you give the methods attributes (red/blue) for them to execute in an linked order. So even the asynchronous behaviour in JS relies on synchronous features for it to be correct!
Edit: Had a second look and it's layouting software
I'm one of the authors of wolkenkit, so if you have any questions, feel free to ask here :-)
Naming things is hard but after watching UBER Inc destroy the pronunciation, spelling and etc of a German word, I'm always cautious about unusual names.
(Also because, as now I'm hung up on the name before giving your product a fair chance)
> That‘s why we have built wolkenkit, a semantic JavaScript backend that addresses three important aspects of software development, and that empowers you to build better software faster.
There's a bunch of things wrong with this claim, "a javascript backend" sounds like an alternative JS VM to me, a backend for javascript (I presume it's meant that this is server side JS not client side?) I also always get my back up when I read "build software faster", surely it ought to be about maintainability when it comes to the perks of DDD/CQRS.
What is your reasoning for AGPL - it's a no-go for
a) indie developers because they can't afford the enterprise stack which is presumably not AGPL
b) enterprises because immediately expect a huge price tag
And last but not least: Why is this not platform agnostic (e.g. only node)?
In a client-server system like a web API where you want to respond to the client as fast as possible having to reconstruct your entity from snapshot + some history can take some time. So you end-up with corrective events and responses which are "we have noted your request, poll us to know the result when it's done". The worst is the ES system is already in use in your RDBMS if you use one. Most CQRS + ES demonstration are done with the simple happy path. Rarely something like user A changes the role of user B. User B was saving changes on a document they can't access anymore: what happen?
Now with software on your PC or with a constant link to the server things feel a lot more useful.
For your example, here are two approaches depending on your implementation and your tolerance for edge cases and eventual consistency:
- you can enter into a saga which would do a two phase commit, ensuring that the user is indeed authorized when the edit is made. All possible states are captured in your event model and so your event history will always show what has happened
- you can load the document aggregate when you process the command and query it at that time, throwing an exception if the user is not authorized and then you can notify the user or send another command. In this case, your event model does not have to include events that describe the violation of domain rules
The first option gives you the guarantee you are looking for, but comes at the cost of complexity. The second option will give you the correct results unless a user is deauthorized before the edit command completes. Some systems process all commands in serial and so it would not be a problem. Some systems partition their commands to distribute the work and so this edge case can occur.
But you can select what makes most sense for you.
So basically don't introduce race conditions that aren't inherently part of the domain. This seems like a modelling failure to me.
Either you are really good and get things right without much help, which isn't often the case, or you end up in two other directions.
You over-engineer with DDD/CQRS/ES.
You under-engineer without them.
Most developers I met preferred the last solution, probably because they end up with their "own" mess and not the mess of some process.
And I have to admit, the only people I met, that used these techniques, were some "digital transformation" consultants who made their living by selling this stuff to big corps, which isn't too convincing either.
CQRS/ES definitely seems like a correct foundations for a distributed system, I just don't think we have the right abstractions to properly model this without wanting to stab yourself in the eye.
Still, it should be noted that Fowler warns that CRQS adds significant complexity and advises great caution about using it. See https://martinfowler.com/bliki/CQRS.html
* I think it's in this talk https://www.youtube.com/watch?v=JHGkaShoyNs
If your schema is any more complicated than a plain stream of characters, and you pick CQRS for your app, then you are an overpaid, bored, architecture astronaut consultant looking to pad his resume. Nevermind the multi-million dollar dumpster fire of an app you spearheaded and ditched out on before completing (and that got cancelled, of course, a complete loss.)
If I ever saw CQRS on a resume, I'd unconditionally pass, you're not setting foot in the door.
This can be as simple or as complicated as someone wants to make it.
It could be as simple as having commands operate on a domain object, but the queries returning a fully populated view object from some custom sql.
I like queues as much as the next developer. Queues let you scale performance Mt Everest and deal with concurrency.
CQRS as an architecture is like queue abuse.
I mean you can use them, but all CQRS means is a separate code path for queries and commands.
In my system I have command objects, they operate on domain objects which contain some business logic.
I have query objects which just execute custom sql code which returns a populated view.
This is useful because views often display specific data from multiple domain objects.
This system allows you to bring back the entire view in a single query and nothing else, rather trying to retrieve multiple domain objects which are really designed for business rules rather than queries.
Often yes, people do starting adding queues, materialised views to CQRS which can make it complicated.
You don't say?? You don't say??
Sorry to be flip, I hope you can tell that I've been burned. Badly. Thanks for your detailed response. You obviously care and are not a stupid person nor do you come off sounding vain.
But, you're not selling me on CQRS at all. What you've described is just classifications and organization for types in a sane sounding UI design that happen to have similar roles as types in a CQRS system.
If you were selling me on CQRS, you'd start by talking about data flow. Something like, "it's a realtime, live multiuser system where all commands are queued, transformed to system events, and those events are committed to log (via another queue, we have to synchronize after all) that stretches back into time. We can replay those events, rewind time, events are merged ..." And it goes on and on into the stratosphere of architecture astronaut-hood...
My problem with your response is I honestly can't trust what you say here until I do a code review. Is your actual code pathological, or are you whitewashing the constant twizzling and niggling you have to do to keep your ship afloat? How bad, honestly, do you have to work to add a new feature? What's your velocity like? I want to see how much work and layers there are in your code to do data type transformation.
Data transformation is where El Diablo lives, because data transformation requires lots of people to agree and business terms to meet code - it's the unholy marriage of garbage disposal, forest fire, and whirlwind. CQRS seems to invite needless data transforms that other architectures do not require, and that's the pukey taste in my mouth when I have discussions about CQRS.
But that isn't CQRS. That's event sourcing, a complicated implementation of event sourcing.
CQRS can work on a standard SQL system, you can design the database using a standard normalised database.
All that has to be different is that commands and queries are separate.
A simple implementation;
A command can literally be
var command= new AcceptOfferCommand(db);
command.Execute();
var query = new AvailableOffersQuery(db);
var viewModel = query.Execute()
Inside the query method you may just perform a raw sql statement which grabs data from multiple tables, populates a viewModel and returns it. Rather trying to retrieve multiple domain objects from multiple repositories and then shuffling that data into a result..
That's one way of doing simple cqrs. It can be simpler than this. But this code just made it obvious.
CQRS allows you do very complicated things, but it doesn't mean you have to.
My main point in replying to you was that you weren't selling me on whatever brand of CQRS you think you've got. Why? It seems like you're atomizing your business logic into a fine mist, why is that good? And you still haven't talked about all the myriad transforms you do: where's the payoff? Does YAGNI apply in your case?
Plus I can run the usecases from a console frontend for quick testing without much setup.
I don't really do any transforms other than mapping domain objects into the dB or database results into a query result.
Why do I need it? Because I practise DDD, and domain objects are not designed for queries. To populate a single view you have pull multiple domain objects from repos. This can be slow and adds complexity.
Query objects allow me to query the database using standard SQL or stored procedure and put the output into query result using only a single query. These queries can as complicated and optimised as they need to be.
Meaning I can avoid all DDD abstractions that I don't need for querying, but I do need for enforcing complicated business rules on the command side.
The act of starting an order does not look like an order. Asking for an order cancellation does not look like an order. Using GraphQL to pull an order summary with report-specialized drill down capabilities does not look like an order. These specialized domain objects should be modeled individually. By trying to encapsulate all these facets in a single object what you get is tight coupling, compromises, domain impedance, logical errors, and slow development.
In context: One of the root principles of DDD is that each domain represents its entities as is befitting that domain. Domain entities should be meaningful in their domain and _not_ corrupted by every niggling redigestion of the same facts through tens, or hundreds, of simultaneous conflicting views... Reporting concerns should be first-class citizens when solving reporting, for example.... View concerns and high-load systems often require denormalization, materialization, transformation, and replication that has very little to do with the user of origin while also _destroying_ the premises of traditional ORM driven RMDB interactions.
At scale, under load, while supporting multiple N-tiered applications and their accordant services, unified domain models break down. Aggregate boundaries map these epistemological distinctions.
CQRS on top of that is just good engineering. In the same way modern car-tunnel builders build two separate tunnels for traffic in each direction, CQRS isolates low-frequency user interaction from high-frequency reporting (or vice versa!), allowing independent scaling and direct delivery of features without impossible coordination activities between teams or finding silver-bullet technologies that may or may not exist.
That said: these are Enterprise solutions to Enterprise issues... Scale is the reason they exist. If you aren't looking at a horrible mess around a unified domain model then these solutions might not apply. If you aren't dealing with multiple simultaneous data delivery platforms (mixing SQL server and Oracle and Mongo DB and S3 and DocumentDB and DynamoDB all at once), these solutions may not apply. If you're not balancing multiple teams working on multiple products that have a constant need for harmonic data exchange, these solutions may not apply. If you're not juggling BigData, if you're not dealing with complex business logic, if scaling up an extra ten thousand users isn't something you deal with, if you're not supporting multiple client types, and so on: these solutions may not apply.
GraphQL and Falcor are interesting, and I like the sort of query-as-befits-the-view aspect to working with it. Taking a wait-and-see on those.
You can code it up in minutes, and most of the code is pure plumbing to connect to datasources. If that sounds complicated to buy independent control over data exchange then you're simply not exposed to the complex multi-layered and distributed applications that demand a structured approach to the issues CQRS handles. Unquestioned and implicit assumptions are great for small potatoes.
Domains should model domains and reflect the mental models of the people who work on, and with, the system. If that is experienced as complicating the only conclusion is that you're working with subject material neither complex nor interesting enough to warrant a considered architectural approach.
For someone who deals with computing (currying, monads, SOLID, etc), you should know that trying to meaningfully translate superficial domain complexity to generalizations of code quality or design quality of any given solution in all domains is laughable.
Your sales technique makes your project sound like torture.
CQRS boils down to a single principle: two one-way roads make for easier road maintenance and improvement that one two-way road.
Two individual channels are too complicated to code? Torturous? Inherently badly designed? Hard to add new features too? Unclean? ... Crazy talk, unsupported by reality. In fact: CQRS specifically addresses all these points you've brought up through its decoupling.
Those kinds of prejudiced assumptions do not hold water, cannot be logically supported by the underlying design nor in-production systems, and reveals only domain ignorance.
Successful engineering around distributed Enterprise systems tells a clear story: those maintenance and improvement concerns are predominant at scale.
Opinions may vary... Informed opinions on this matter do not.
Truly, one look at this archetypal wonder, (which certainly every programmer worth his salt has already studied its types and patterns at length), and all doubters shall melt away. If there ever were a more killer app demonstrative of the state of the art, it should be the muse of comp-sci journals or even popular culture!
A toy Microsoft CQRS spike on github from a couple years back does not count, neither does a book reference. I want a canonical example of something real: deployed code to users, thriving and successful. I'll be a happy to learn something new.
My suspicion however is you're all talk - a super-consultant who sold some CQRS work and has an axe to grind because you have co-workers reading this thread, including some of the client's devs. You arrogantly bragged to them and then waded out to sea because you thought me an easy mark to prove your claims to expertise on the subject.
I am prepared to gracefully concede to my betters.
I suppose I crave simplicity in code, with as little magic as possible, and with as few maddening descents into "not-invented-here" coding adventures when open source modules and frameworks exist that could do the job.
My use of the word "clean" refers to Uncle Bob's definition of clean code, not some scientology thing. He preaches the necessity of writing well-tested, high quality code via TDD. You mentioned SOLID earlier, which is another Uncle Bob-ism, and I figured you would get the clean reference. Not Uncle Bob's, but YAGNI's another acronym he likes to throw out there - YAGNI comes to mind whenever CQRS is mentioned. I bet YAGNI would be your worst nightmare if anyone around you had the courage to utter it. Are you a bully or do you just have no scope control on your projects from your business? Hey coworkers of bonesss, challenge everything this guy says with 'YAGNI', and see how he reacts!
Yeah, so anyways as I said in my other reply let's see your large scale code example that necessitates CQRS. Impress me with the high quality design and implementation, I'll do a code review.
> At scale, under load, while supporting multiple N-tiered applications and their accordant services, unified domain models break down. Aggregate boundaries map these epistemological distinctions.
I mean, I get it, you're talking about DDD. You've got context-bounded models. I've explored all this for over a decade. I apply the concepts daily (as mandated by our architecture, not my choice.) In the SOA days we had canonical models that are exposed on the service interfaces and domain-specific models that were representative of data coming from data sources or were internal application specifics and we always mediated between the models. It's nothing special. I sometimes think, if we all talked a little more we could make our terminology more consistent - but I get it, there's domains and different definitions and types will get mapped.
What you wrote is too pendantic, though. Epistemological! You're writing web apps, c'mon you are killing me!