Show HN: I wrote a book on implementing DDD, CQRS and Event Sourcing
leanpub.com
leanpub.com
You can access a free sample of the book on the webpage in different formats.
I worked on the book for four years as a side project. In 2016, I had worked in three different projects that applied either some or all of the concepts. The last one was the main motivating factor for the book. It was a startup that was settled on DDD, CQRS and Event Sourcing, even though the developers had limited knowledge.
Together, we learned, experimented and applied as much as we could. We read "Domain-Driven Design" by Eric Evans, "Implementing Domain-Driven Design" by Vaughn Vernon and whatever we could find from Greg Young. With the knowledge we picked up and the experimentation we did, we felt in a good position. Still, there were many unanswered questions on how to put everything into code.
In 2016, before leaving the startup, I first thought about writing a book. Effectively, I wanted to create the guide the startup would have needed in the beginning. When I started, I was overly confident about my knowledge. However, throughout the years, I learned many details. Today, I know that I could not have written it in 2016 at once.
The final book is different from what I envisioned it to be at first. My initial goal was to write a short book with approximately 150 pages. The published version has now over 450 pages. However, I consider this a good thing. The book provides enough context to be read without any prior knowledge in DDD, CQRS or Event Sourcing.
I've written a blog post that covers my motivation on writing the book in even more detail: https://www.alex-lawrence.com/posts/why-i-wrote-a-book-on-dd...
I'd be happy to discuss your thoughts on and experiences with DDD, CQRS and Event Sourcing. Regardless whether it is about their combination, their sometimes counterproductive conflation or one of the concepts. Finally, if you have any input or feedback for me, it would be highly appreciated!
My experience is that most supposed implementations of DDD are problematic because the focus is on the infrastructure, plumbing and implementation, rather than on the real challenge which is modeling a complex problem domain.
This is the exact opposite of Evan’s intention as stated in his book.
Because there is a general inability to model a complex system into an object model as is the basis of his book, focus falls onto programming constructs and things like “ubiquitous language” which pre-dates Evans and is self evident to most.
About 80% of CQRS/event sourcing projects I review or consult on are unsuccessful. In most cases this is primarily due to them being applied where they shouldn’t be, by (inexperienced) teams viewing these techniques as “silver bullets” to success.
I should note that I am mainly involved with line-of-business type systems, rather that projects focusing on computing infrastructure and the computing and data sciences, and therefore speak to the former.
It is important to note that my book is not an alternative to "Domain-Driven Design" by Eric Evans. Rather, it is an introduction to DDD, an explanation of CQRS and Event Sourcing and an implementation guide. Also, it is not explaining how to approach and model complex domain problems. I wouldn't consider myself experienced in that. Everything I worked with so far was rather trivial.
I can very much relate to your experience with projects that applied CQRS and Event Sourcing. In fact, the longest project I worked on was a negative example. The patterns were used as part of the system architecture. Their use was never justified with any reason specific to the domain or the business. In the end, some parts of the system would have been better off with a completely different architecture.
I already have convinced project teams to not use Event Sourcing when they thought they needed it. Also, I often hear it being conflated with Event-Driven Architecture.
I think that software developers too often underinvest in modeling a domain in favor of propping up an imagined silver bullet.
I wonder if there’s an easy way to hire for good behavior?
For CQRS, when you say “ it is due to them being applied where they shouldn’t be” can you provide more insighs?
In what cases do these architectures make sense and don’t ? What are the trade offs?
A few questions stand out:
AFAICT implementing tactical DDD involves a lot of boilerplate code. That seems to be consistent with what you've written in your article on writing the book, mentioning "...we even built a Node.js framework as byproduct." If tackling DDD in Node requires so much work on something that isn't the businesses' core domain, and there's a lot of surface area for it to go wrong, why do it in Node?
As someone who writes predominantly TypeScript, static types feel like a much lower hanging fruit for alleviating some of the pain in tackling rich domains. As you've written the book in JavaScript, I'm curious whether you used plain JavaScript on the professional projects?
Also curious to hear from someone who has done tactical DDD in Java or C#: do you find a good supporting framework essential?
I fully agree with you that (static) types are inevitable for tackling rich/complex domains adequately. Personally, I would often recommend the use of TypeScript over plain JavaScript.
As for the book, when I started working on it, I was using plain JavaScript. Over the course of the past four years, there were times, when I considered to rewrite the DDD parts with TypeScript. My current plan is to add an appendix or integrate additional content about (static) typing.
While types are an important aspect for the Domain layer, I think the rest of the book works fine without TypeScript. Its absence might even help to keep the code example more concise.
> I'm curious whether you used plain JavaScript on the professional projects?
Yes, in both relevant projects we were using plain JavaScript (or CoffeeScript). We had many runtime type checks and a high test coverage to overcome the lack of static types. TypeScript would have definitely been the better choice.
Probably the most frustrating part for me is tackling domain events and the implementation of the aggregate root. How have you tackled broadcasting domain events when an aggregate changes in a way that might be of interest to other bounded contexts?
About the Domain Event publishing, I'm afraid I don't understand the question. What challenges did you face? Is it about event-sourced aggregates?
What is the mechanism for it to do so? Does the aggregate root retain the events and decide when to dispatch them (likely after successful persistence to a datastore)?
Once ready for dispatch, what do you use to handle the events? Some kind of DomainEvents class ala https://udidahan.com/2009/06/14/domain-events-salvation/ ?
This is the boilerplate I'm referring to.
The article from Udi Dahan shows Domain Event handlers that "will be run on the same thread within the same transaction". This implementation is not suitable for handlers that affect other transactions. Udi explains that "you should avoid performing any blocking activities, like using SMTP or web services. Instead, prefer using one-way messaging to communicate to something else which does those blocking activities." What he is referring to as "one-way messaging" is the actual event publishing in my opinion and must guarantee event delivery.
In my book, the example implementations store all occurred Domain Events together with the Aggregate state. This is because it resembles the later use of Event Sourcing. There is a separate component that watches for changes in Aggregate data and publishes new Domain Events. After publishing, the events are marked accordingly.
Regardless of where newly occurred Domain Events are retained, it should be persistent. Many times, the events are stored in a separate table inside the same store as the affected Aggregate. Later, they are retrieved, published and either marked or deleted. This is called a Transactional Outbox: https://microservices.io/patterns/data/transactional-outbox.... The approach ensures that both the Aggregate change and the request to publish an event happen within the same transaction. The actual publishing happens in a separate one. This way, you get guaranteed event delivery or more specifically "at least once" delivery.
Libraries and frameworks can help you with different things, but they can also get in your way. If the CQRS module of Nest.js seems useful for a specific scenario, I would give it a try.
CQRS can be implemented in many different ways. Even more, a lot of software is applying this pattern inherently to some extent.
If you separate a software (or a part of it) into a write side and a read side, the two become independent. They both only interact with the synchronization mechanism. Therefore, they are free to use their own set of patterns, libraries and even technologies. Many times, the two parts are operated as individual programs/services and can scale autonomously.
Personally, I never encountered a situation with implementing CQRS where I really would have needed a framework. In fact, I contributed to a Node.js framework for implementing CQRS and Event Sourcing. In hindsight, I think the framework was a mistake and should not even be used.
My book explains how to implement all the concepts, patterns and mechanisms without any third-party code. The goal is not to use the custom code in production, but to understand how everything works.
While this is not necessarily an argument against a printed version, note that this book works best when reading on a computer. It comes with the full source code bundle and a small utility that allows to execute all code examples interactively.
Also, it is likely that there will be updates to the book in the near future. I'm already considering to write an appendix. By exclusively providing an e-book, iterative improvements are easier to do and the readers get all updates for free.
Nevertheless, I may consider to offer a printed version later on.
This could lead to a dramatic shift in how technical books are created.
Imagine a new website that didn't offer videos or text or tutorials, but actually was integrated with all of those things together?
So in your case, the book can be read or listened to, as you work on the example implementations. Using a little ML, you could probably even offer suggestions to roadblocks.
Just got sidetracked on this thought.
I'm a DDD guy, but have never personally implemented CQRS and event streams. I love the idea of separate read and write streams, but what is the best mesh of tech to do it well? (rhetorical question)
I'll buy the eBook. Thanks for working on this.
That said, as a Go/Javascript person I really appreciate another book on the topic. I appreciate the effort and I am going to buy it.
This has various benefits but the biggest one being you get to play with the design of the 'microservices' before actually committing to making them microservices allowing you to make better informed decisions. The harder part is that it does require stronger discipline and if not done correctly, a harder transition to microservices (e.g. pulling out the components)
In fact, I did the same in one project. Also, while I am not covering Microservices in my book, the Sample Application uses a similar approach. The different contexts integrate via event communication. At first, this is an in-memory implementation and everything is running in a single process. Later on, it is replaced with a filesystem-based variant and the individual contexts are operated as individual programs.
Do you use c# ? I let only application layers talk to each other for now. Also, i have some issues finding a decent saga to use that stores it's state ( eg. Something like the library Mass transit)
While I read about the possibility of doing both at the same time, I never encountered this practice in a project. How I see it, there is always only one source of truth. If you apply Event Sourcing, it is the event log. Any other representation should be seen as derivation/projection.
The one long-term project where we applied Event Sourcing showed quite some challenges you can face with this pattern. For me, the most challenging aspect is the versioning of an event-sourced system. When the Domain Model evolves, it might necessitate changes to events or the introduction of new ones. Greg Young even wrote a book on this topic: https://leanpub.com/esversioning (I haven't read it though).
I am aware of the costs of the approach, but would like to know the real world problems it solved for you.
The project was about building a SaaS for structured meetings, both for on-site and remote use. It involved topical areas such as collaboration, moderation and presentation. The model defined that every meeting consists of six steps, where each is facilitated with one specific method (out of multiple). There were also additional functionalities such as a collaborative whiteboard and audio/video communication.
I briefly mentioned this in another comment: With regard to software architecture and pattern use, the project is a negative example. CQRS and Event Sourcing was prescribed on a system level. Regardless of conceptual/technological boundaries, every part made use of it. This decision was by no means reinforced with domain- or business-specific arguments. However, for us as developers, it was a great learning opportunity.
In terms of significant benefits for specific contexts, I can only think of the whiteboard part. However, even for that, it is about a feature we never got to implement. At some point, we wanted to add a "Undo" functionality. With Event Sourcing, you can simply append events that represent the reversal of previously persisted state transitions.
Independent of domain-specific aspects, I experienced three universal benefits. I hate to say it, but one was in fact reporting. Another benefit is the increased possibility to solve concurrency conflicts in a domain-specific way. Third, an event-sourced service is an ideal foundation for sending notifications, such as when publishing Domain Events. In the project, it helped us to build a real-time reactive User Interface. Of course, I'm not saying you necessarily need Event Sourcing for that.
How big is the sample application in the book? I've read a lot about DDD but things just don't seem to click for me, partly because example are trivial. For example, I read "Architecture Patterns with Python" (https://www.cosmicpython.com/book/preface.html) but just didn't find it practical. Some random thoughts I had while reading, in no particular order:
* what if domain logic can be optimized in the database?
* how do joins across multiple tables work?
* in Django I've come across plenty of examples where you have property method on a model that goes and performs additional queries. For instance, something along the lines of `my_post.get_latest_comment()`. Should this instead be somewhere in the service layer?
The book contains numerous standalone examples throughout the chapters, which are all executable. At the end of each chapter, the explained concepts are applied to the Sample Application context.
The Sample Application is a greatly simplified task board. Users can register, login, create projects, manage team members and work with a task board. The task board is essentially a collection of items, grouped into three columns. Individual tasks contain details such as title, description, status, assignee, etc.
While there are some constraints/invariants in the Domain Model, I would consider the Sample Application fairly trivial. It does not represent a complex domain problem. One reason for this is that I personally did not encounter any highly complex domain so far. It would have been weird if I tried to explain something I am not really experienced in.
> what if domain logic can be optimized in the database?
IMHO, the question is a bit too generic. The generic recommendation would be to not put domain logic into the database. However, one would need to look at what the domain logic in question is. Also, what does "in the database" mean? Are we talking about code that is being executed in the database environment, such as PL/SQL?
> examples where you have property method on a model that goes and performs additional queries
This sounds a bit like the Active Record pattern or the use of an ORM, where one "model" is somehow linked to another. Again, a generic answer would be that domain objects should stay free of infrastructural concerns. If you have posts and comments and need to get the latest comment on a post, this represents a use case. Use cases are executed by an Application Service. This part is allowed to query any number of domain objects / data sources. If you would be using CQRS, you would likely have a denormalized Read Model that contains comments per post.
> Working through the book requires advanced knowledge in JavaScript and in Node.js.
Do you have any recommendations on various Javascript / Node.js education or training resources that would make good precursors to your book?
Furthermore, any general advice for the uninitiated?
Similar to what zeroc8 said, I heard that people liked "You don't know JS". Also, I agree with joshxyz that you can learn a lot from the MDN and the Node.js API docs. Especially MDN is a great resource for learning specific concept. As example, look at their Promises tutorial: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... Furthermore, I also think the publications from Axel Rauschmayer are worth checking out (exploringjs.com).
The section in the Preface of my book might sound a bit daunting. Even if you don't know JavaScript, but have experience with at least one other programming language, that might be fine. I think the most challenging aspect is the asynchronous programming part, which is being used extensively due to Node.js.
- nodejs.org api docs for nodejs stuff
- exploringjs.com for ES spec differences and progression
- caniuse.com for checking browser support
- there are bundlers like webpack and transpilers like babeljs but those are a little beyond the basics
"You don't know JS" is a great book, and it's essential for someone who actually writes and works with JS all day. You'll pass most JS questions if you interview for a new job. At the same time, in my opinion, it's just a book full of gotchas that shouldn't even be present in good codebases (because most developers don't know JS :)).
I’m the author of Event Horizon, a ES/CQRS toolkit for Golang: https://github.com/looplab/eventhorizon