Data-Oriented Architecture (2020)
blog.eyas.sh
blog.eyas.sh
Success with these efforts is unlikely in larger teams. You need to put the most wizened chef in your data modeling kitchen and have them serve the business stakeholders until they are leaving 5/5 reviews on yelp. Design of schemas by committee is how you wind up with N of them and up to N^2 bullshit protocol/mapping/service layers for communicating between them.
You can iterate this problem space with Microsoft Excel. You do not need to write any code to figure out the most difficult part of most software products.
Completely true and something that I wish more people understood and practiced! Most engineers (even seniors) want to immediately start cutting code. I think the saying is "Weeks of coding can save you hours of planning".
I believe this! If you can format your data in excel in such a way that people can easily answer their questions with pivot tables and vlookups, you're most of the way there to a good schema. And it's a much lower cost to produce some test workbooks in excel for people than building and populating full databases using new/prototype services.
The article here calls it Data-Oriented Architecture once, and then proceeds straight to DOA. Trying to convince people to switch something as fundamental as their software architecture is difficult enough without calling your idea "Dead On Arrival" from the first paragraph.
However, the term does occasionally slip into external meetings and interactions with people. It's not a good reflection of the company for sure.
[0] https://en.wikipedia.org/wiki/Run-time_infrastructure_(simul...
Of course any traditional relational DBA would love this article. But relational databases are an architectural anti-pattern.
Don't start with the data.
Start with the domains.
I disagree. I think relational databases are such a mature technology that you should definitely start with them if you possibly can. They are flexible enough that they can accommodate a lot of different applications. In addition, relational databases are performant and scalable enough that unless you reach MAGAF scale, they will likely work for you.
Let the domain decide.
This is such an inflammatory and ignorant statement that I don't really have any other answer other than to call it pure bullshit. Relational Databases are probably one of the few popular areas of our profession that are not only backed by formalisms and research that tell us to do it correctly, it is also proven to work in practice: it is the de facto backbone of most financial, commercial, government systems and, heck, even military, but also even of modern FAANG-style big tech.
Sure, they're boring. And it takes time to learn. And normalisation might has its costs but we can know them all upfront, and we know where and how to break them. The (currently trendy) horizontal scalability is indeed non-trivial with them, but vertical and horizontal-via-sharding are easier than most ad-hoc solutions. But the boredom pays off in guarantees, and to learn you have 50 years of mature material to rely on.
(This is not to say that non-relational databases are useless. Both have its uses, but they're both only really useful if you properly understand them)
Complex enterprise systems are complex for reasons beyond the technology. Often due to processes and bureaucracies. Those complexities often end up turning into "essential complexity" in systems, and Domain Driven Design is not able to reduce such essential complexity, and nothing can.
Nor is Domain Driven Design incompatible with relational databases, especially the tactical part, which is often the major selling point of DDD. It is entirely possible to model domains within databases, and is entirely common.
Also, DDD is not a guarantee of success. If anything, it has not proven itself nowhere near as much relational databases have. As much as the original idea of DDD is good (again, especially the tactical part), it I see it often become rationalisation for the same old architecture astronautics that cause excessive incidental complexity. For people wanting architectural complexity, I can completely understand the distaste for something simple like "keeping it in the DB". It completely negates all the fancy technical patterns prized by developers. This is not to discount the original idea and techniques of DDD: merely saying that the excuse "relational databases can be bad if not well done" applies as often, if not more, to the practice of DDD.
So it's easy to succumb to temptation and take the easy path (I don't want to add three tables to keep it normalized... let's allow this column to be null, the applications can check... another field for the second phone number, we'll see about the third one... make your queries non-isolated or we could have a deadlock...) with apparently good results, and then when technical debt comes home the architecture amateurs can blame the tools instead of themselves.
I've seen this way too many times to back off of my assessment. Relational models do not, in the long term, enable an agile business model. It is an anti-pattern even if it takes years to reveal itself as such.
The problem always lies on excessive abstractions and patterns on the backend side. Which really isn't solved anywhere by any strategy other than keeping things damn simple.
Your descriptions of "impedance mismatch when converting relational models to business models" and "automating mappers that leads to a hardened and immovable design" are a precise example of that: OOP patterns muddying the development process and causing problems in unrelated parts of the system (in this case the database).
Of course, keeping things simple and programming in a way that fits the relational model is incredibly difficult for developers that have been trained to believe that complexity is the answer to all problems future and past. So the resort for those people is to keep pestering management to throw the baby (relational databases) with the bathwater (relational mappers and other related OOP patterns).
And when people needed the transactional data in a relational model, we streamed events and that was read only in some other tool and the core system didn't need to know about any of that.
Been down this road. Saw a fork in 2015 and haven't looked back.
It sounds like you put ORM hell behind you by getting rid more of objects than of databases.
> Not only were we able to build faster (schemaless is so much easier on change management), but also offers the ability to add features at a frighteningly fast pace
What kind of change management? Frightening clients with API changes? Eluding code reviews and formal release processes?
I didn't say relational databases are bad. I said it was an anti-pattern. In my mind, relational databases should be mostly used in data warehouses, reporting, and analytical systems.
Transactional systems are better suited to document databases and CQRS.
For most applications, having a really good relational data model, and controlling data updates through a service/microservice will give most of the benefits of "object oriented" design without any of the drawbacks. The service will act as a "live object", data integrity will be preserved, data science modeling will be easy, and the application can be designed using whatever tomorrow's preferred platform of the day is, or even some low code stuff for simple applications.
Say what bro? Surely this is sarcasm. Relational databases make up the foundation of most of the services you use today. Even poorly designed relational database schemas will work in production.
In my opinion what you suggest would be a good solution... But try to pass that by mos. Manager for which (in most companies) we are already late to compleate the newly received feature/requirement
A sound architecture (admittedly unlikely) would help the kind of incompetent organization you allude to to maintain order and fail later.
A key thing like about this architecture is it gets some of the benefits of microservices (codebases can be split, services deployed and managed separately, low coupling as a default) but without the downsides of fragmenting the data store, which causes a lot of the issues in a pure microservice architecture.
Once you fragment the data store you now have to fully compensate for the loss of transactional integrity at the data layer. It also forces a coherent data model across all the services - no more "service X" has a slightly different model of a data type to "service Y" - there literally is only one definition of the data type.
So I see it as a reasonable midpoint between monolith and microservice that gets most of the benefits of microservices without the most severe drawbacks.
DOA seems to include many applications accessing the one database - which is how it's been done for decades, even before RDB, I guess it's in reaction to SOA.
What is here referred to as the central data store is the model.
The services are the views. Just like views, they do not hold onto any actual application data, that must be stored in the model.
The services must not communicate with each other, just like views (and controllers/ViewControllers) should not communicate with each other. Instead they communicate via the store, just like in MVC, where the model is updated and then the model notifies other views of changes.
(The last part is the one that a lot of the alleged implementations of MVC get completely wrong, leading to the problems that they then blame on MVC. Which then leads to them creating new patterns to "fix" the "problems" of MVC that would be solved by just implementing MVC correctly).
So yeah: good stuff. Nice to see how good architectural patterns are widely applicable.
I like your overall analogy but you lost me (as you predicted lol) with that last part. Views notifying other views (probably via listeners and event handlers) sounds more like the MVP pattern to me. But yeah overall I like your take on understanding the article.