Monoliths are not dinosaurs
allthingsdistributed.com
allthingsdistributed.com
"Complexity can neither be created nor destroyed, only moved somewhere else."
The drive to simplify is virtuous, but often people are blind to the complexity that it adds.
Okay, so your microservices are each very simple, but that made the interactions and resource provisioning very complex. What was the net gain?
The correct solution depends on the circumstances. There are excellent uses of microservices. There are excellent uses of monoliths. There are excellent uses of monorepos. There are excellent uses of ... (wait never mind monorepos are just better).
Understand what is ESSENTIAL complexity and what is INCIDENTAL complexity. Simplify your system to remove as much incidental complexity as possible, until you are left with the essential and be satisfied with that.
The quote would probably be more accurate as:
> "ESSENTIAL Complexity can neither be created nor destroyed, only moved somewhere else."
That is the more precise adaptation of Kelsers Law.
Obviously you can always add also superfluous complexity :)
Often a developer will see something big or complex and see it as a problem.
But they should consider whether this matches the size/scope of the problem being solved
In professional software development projects, specially legacy projects, often times the complexity is not justified by the problem. There is always technical debt piling up, and eventually it starts getting in the way.
Often times means not always -- what would you say some projects are doing right so that their complexity is justified by the problem?
It's one thing to shift complexity to a different group/team, such as managing microservice infrastructure and orchestration that can specialize in that aspect of many applications/systems. It's very different to create those abstractions and separations when the same people will be doing the work on both sides, as this adds cognitive overhead to the existing team(s). Especially if you aren't in a scenario where your application performance overhead is in eminent danger of suffering as a whole.
The net gain was composability of microservices, distribution of computing resources, and the ability to marshall off implementation details. Just because those requirements were routinely ignored in the era of monoliths doesn't mean the complexity wasn't essential or didn't exist.
What microservices give you is a hard boundary that you cannot cross (though you can weaken it, you cannot eliminate it) between modules. This means the internal state of a module now has to be explicitly and more deliberately exposed rather than merely bypassed by lack of access control, or someone swapping out public for private, or capitalizing a name in Go. If there's any real benefit of microservices, this is it. The hard(er) boundary between modules. But it's not novel, we've had that concept since the days of COBOL. And hardware engineers have had the concept even longer.
The challenge in monoliths is that the boundary is so easily breached, and it will be over time because of expedient choices rather than deliberate, thoughtful choices.
I just doubt that people who don't have the discipline to write decent modular code will do any better with microservices. You will end up with a super complex, hard to change and hard to maintain system.
True remedy in both cases is refactoring. So if team don't have time for refactoring in monolith, then switch to microservices would need to free up enough time for the team to start doing it.
Can that even with on a level of a single team?
One name for this is "Modulith" where you use modules that have a clear enforced boundary. You get the same composability as micro-services without the complexity.
Here's how Spring solves it: https://www.baeldung.com/spring-modulith
It's basically a library that ensures strict boundaries. Communication has to go through an interface (similar to service api) and you are not allowed to leak internal logic such as database entities to the outer layer
If you later decide to convert the module into a separate service, you simply move the module to a new service and write a small API layer that uses the same interface. No other code changes are necessary.
This enables you to start with a single service (modulith) and split into microservices later if you see the need for it without any major refactoring
The biggest challenge with monoliths is the limits of a single process and machine.
In other words, it sounds like you're speaking of a case where all services together consume terabytes of memory irrespective of utilisation, but if you can split it up into multiple heterogeneous services each will only use at most hundreds of GB. Is that correct, and if so, what sort of system would that be? I have trouble visualising it.
An infrusctructfre of this sort is meant to solve very hard problems, not to make regular problems much harder.
"For something to get clean, something else must get dirty"
and the corollary:
"You can get something dirty without getting anything clean"
You must remove complexity to make understandable code.
You can remove complexity without making anything understandable.
Every time I've had that argument at work, ultimately there's no agreement of what is essential. In one of the worst cases, the big fan of the concept was a CEO considering themselves very technical, who decided that everything they didn't understand was obviously valueless and incidental, while everything they personally cared about was really essential complexity, and therefore impossible to simplify... even though the subsystem in question never had any practical applications until the company failed.
So ultimately, either the focus on avoiding incidental complexity is basically a platitude, or just a nice way to try to bully people into your favorite architecture. A loss either way.
Not true. I blow.up a car. Complexity is destroyed. I rebuild the car from blown up parts. Complexity is created.
There's no underlying generic philosophical point about microservices and monoliths. What we can say is that microservices are not necessarily less complex than monoliths, but this relationship has no bearing on the nature of complexity itself.
Wouldn't the blown-up car be more complex than the non-blown-up car?
Typically I define complexity as a low probability macrostate meaning low entropy. So debris generated from the explosion has a high probability of randomly occuring, while a new car has a very low probability of randomly occuring.
Following this definition you arrive at a formal definition of complexity that is intuitive.
Imagine a tornado that randomly reconfigures everything in it's wake. The more complex something is the less likely the tornado is going to reconfigure everything into that thing. So it is very likely for a tornado to reconfigure things into debris and extremely extremely unlikely for the tornado to reconfigure everything into a brand new car. A tornado reconfiguring atoms into a car seems to be impossible but it is not, it is simply a low probability nearly impossible configuration.
Therefore the car is more complex then exploded debris.
Think about this definition because it also aligns intuitively with the effort required and technical complexity of any object. The more effort and technical complexity it has the lower chance it has of a tornado randomly reconfiguring atoms to form that object. Thus that object is more "complex".
Whatever your definition is, the quote saying complexity cannot be created or destroyed is fundamentally from any perspective usually not true. If you want to define it as simply microservices or monoliths it still doesn't make sense. Whose to say that when converting a monolith to microservices that complexity remains conserved? Likely the complexity rises or lowers a bit or a lot. Complexity doesn't remain the same even if you use informal and inexact fuzzy definitions of complexity.
Is this not a linguistic sleight of hand? There are billions of trillions of states we label with "debris" but only a few thousand we would call "car". So a specific state of debris, then, is equal in complexity to a car?
It is literally within the formal definition entropy. Debris is a high entropy macrostate, while a car occupies a lower entropy macrostate. There are less possible atomic configurations for cars then their is for debris. Each of these individual configurations of atoms is called a microstate.
A macrostate is a collection of microstates that you define. Depending on the definition you choose that definition has an associated probability. So if you choose to define macrostate as a car, you are choosing a collection of microstates that have a low probability of occuring.
The law of thermodynamics says that systems will, over time, will gain entropy meaning they naturally progress to high probability macrostates over time. So in other words, complexity is destroyed over time by specific laws of entropy.
https://en.m.wikipedia.org/wiki/Entropy_(statistical_thermod...
The reason why this occurs is straightforward. As a system evolves and randomly jiggles over time it trends towards high probability configurations like "debris" because simply that configuration has a high probability of occuring. Generally the more billions of microstates a macrostate contains the higher entropy that macrostate is.
Through logic and probability and the second law of thermodynamics we have a formal definition of complexity and we see that complexity naturally destroys itself or degrades with time.
This is the thing that confuses people about entropy. It's definition is a generality based on microstates and macrostates you choose to define yourself. It's similar to calling a function with generics in programming where you choose to define the generic at the time of the call.
But even within this generic world their are laws (traits in rust or interfaces in c++) that tell us how that generic should behave. Just like how entropy tells us that macrostates we define will always trend towards losing complexity.
The heat death of the universe is the predicted end of the universe where all complexity is inevitably lost forever. You can define that macrostate as the collection of all microstates that do not contain any form of organization.
That doesn't really line up logically. :/
Debris is almost any configuration of atoms that are considered trash or unusable.
There's are much more ways you can configure atoms to form useless trash then you can to make cars. Case in point "you" can manufacture "debris" or "trash" by throwing something into a trash can. Simple.
When's the last time you manufactured a car? Never.
Seems like a case of you including and excluding things to support your "logic". Ugh.
But isn't that introducing incidental complexity? Not sure if you actually disagree.
I have absolutely seen complex code that was created (often for perceived “best practices” like DRY) which could be removed by simplifying the code.
I think a good solution to any problem needs to match it in complexity. I regularly use this comparison as a benchmark for solutions.
Of course, you can also see it this way: The complexity you remove from the code by making it cleaner is added to your team communication because you now have to defend your decision. (Only half joking.)
There has to be an overarching thesis that is well-formed (formed based on experience/tests from adjacent/related markets, key assumptions validated at reduced scale, strong backing by investors/founders etc).
During this vision formation/validation stage, keep things very lightweight and don't over-invest in prolonged tech platforms.
For example, if your product/service has an on-the-field operational part to it, then run that part with manual operations with pen/paper – pen/paper is extremely flexible and universally usable and survives all kinds of surprises in the field – don't wait to build a software solution to test your thesis in the field. Manual ops can scale quite well especially during learning phase (scale is not your goal, learning/validating is). Choose the use-case for your experiments carefully – optimizing for fast learning and applicability of that learning for next adjacent use-case you intend to expand to.
Once you get going, still build the tech systems without dogmas or baking strong opinions in too much. Keep your engineers generalists and first principles problem solvers and interested in solving both tech and functional-domain problems and encourage them to be humble and curious – because both the functional and tech world is constantly changing. Don't hire a huge product management org – every engineer/manager should be thinking about customer/product first. Over time, parts of your product are more mature and parts of your product are very nascent and still volatile. If your entire team is still thinking customer/product first and build tech to solve problems from first principles, then they should have found the right coupling/cohesion balance between different parts of the system to avoid shared fate, high blast radius or high contagion of that volatility/instability affecting more mature parts of the system.
I have to say that this short quote is not the whole story; for example it's ridiculously common for artificial complexity to be introduced into a system, like using microservices on a system that gets 1k users a day.
In which case, it is sometimes possible to remove complexity, because you are removing the artificial complexity that was added.
The problem is that the shops that move from monoliths to distributed systems are under the impression that all of their problems are now magically over "because microservices".
This sort of specious reasoning just shows how pervasive is the fundamental misunderstanding of the whole point of microservices. Microservices solve organizational problems, and their advantages in scaling and reliability only show up as either nice-to-haves or as distant seconds.
Microservices can an do make sense even if you have 10 users making a hand full of requests, if those services are owned and maintained by distinct groups.
Maybe, but after the next CEO comes in, those groups would be reorganised anyway :-/
Few companies maintain their org chart for a large length of time. My last place had the microservices maintained by distinct groups when I joined. When I left a third of the people were gone and half the groups were merged into the other half.
This is not an uncommon thing. Going microservices because you don't want to step on other peoples toes is a good reason, but it's an even bet that the boundaries will shift anyway.
That's perfectly fine, because microservices excel in that scenario: just hand over the keys to the repo and the pipelines, and you're done.
Succintly put!
It was kind of horrible, as VS would constantly take you to the IFoo interface definition instead of the DbFoo implementation when you were trying to debug. Not fun at all. It was much earlier in my career.
Probably the most complex system I designed was an inherited permission interface down to a component level through a byzantine mess of users/groups and ownership for a CRM tool in a specific industry. If I had to do it today, it would be very, very different and much simpler. Also much earlier in my career (about 20 years ago).
These days I've really enjoyed using scripting languages, mostly JS/Node and keeping things simple (I don't care for inversify for example). In general the simpler thing is usually easier to maintain and keep up with over time. Yeah, frameworks and patterns can make boilerplate easier but it's often at the expense of making everything else much harder. Vs just starting off slightly harder from the start, but everything is simpler overall over time.
Aside, been enjoying Rust as well.
I believe we do not have the right tools yet.
The main misconception about microservices is that people miss why they exist and what problems they solve. If you don't understand the problem, you don't understand the solution. The main problems microservices solve are not technical, but organizational and operational. Sure, there are plenty of advantages in scaling and reliability, but where microservices are worth their weight in gold is the way they impose hard boundaries on all responsibilities, from managerial down to operational, and force all dependencies to be loosely coupled and go through specific interfaces with concrete SLAs with clearly defined ownerships.
Projects where a single group owns the whole thing will benefit from running everything in one single service. Once you feel the need to assign ownership of specific responsibilities or features or data to dedicated teams, you quickly are better off if you specify the interface, and each team owns everything begind each interface.
If team A needs a new endpoint from team B, what would a typical dialogue look like under microservices and a modular monolith, respectively?
What microservices easily provide in this scenario that is far harder to pull off with a monolith is that with microservices the service owners are able to roll out a dedicated service as a deliverable from that dialogue. Whether the new microservice implements a new version of the API or handles a single endpoint, the service owners can deploy the new service as an extension to their service instead of a modification, and thus can do whatever they wish to do with it without risking any impact on their service's stability.
Never seen it work, don’t think I ever will. The only way microservices can ever work is if they’re extracted over time as the monolith gets fleshed out, likely over the course of years. And even they’re probably a bad idea.
The microservice syndrome begin once you start splitting services simply because you arbitrarily declare them too large.
Determining this at all takes skill. Determining it that far ahead of time requires so much skill and foresight that suggesting that people try is bad advice.
So I generally try to design with a separated authentication from the beginning. I'll start with a "dev" auth service that will just have a form you can fill out whatever you want for the "user" and permissions, then sign a token to get into your local/dev environment application. From there it's really easy to create a more robust system or wrappers for external systems (Auth0, Okta, Azure AD, etc).
In general, however, MicroServices is about completely separating a given context domain from other services in a larger system. Account Management is separate from Assets, etc. In practice this means you will have a much more complex orchestration, deployment and communications system in place, often with some sort of Distributed RPC, Queue or other layers on top. You also have to be much more concerned with communications between teams and service versioning and availability.
This complexity isn't less complex, it just shifts the complexity, which can help with larger organizations, but for smaller teams it can really bottleneck everything. This is why the general concensus is to start with a more monolithic codebase designed to still scale horizontally, then break off systems as the need arizes.
The exception being if you are in an organization that has already paid the overhead/debt of setting up for microservice orchestrations.
Otherwise, I generally agree... I'll usually take a monolith approach and break things off in ways that make sense. Usually starting with long running processes that can simply be workers off of queues. Sometimes potential bottlenecks that have higher compute overhead, such as passphrase hashing and comparison which is relatively easy to DDoS, but if broken off only effects new logins and password changes.
In order to DoS your typical site through passphrase hashing you would need to be:
- have a ton of valid usernames/emails of accounts that need to be checked (because a typical password check will rate limit by account) - send in a massive torrent of traffic from a ton of IP addresses (because a typical password check will be rate limited by IP, even more than typical IP based rate limiting)
While this is not impossible if you had those resources it still might be easier to just DoS the site though standard pages/ endpoints by sheer traffic.
It's only natural that a job that used to take 2 now takes 10 people because of those facts.
And I agree. We've traded safety and a few other things that could have been ironed out in exchange of massive gridlocks, headaches and colossal facepalms. Performance is only on par if you consider the hardware improvements, it's crazy.
You can no longer claim to have expertise in OpenShift, Redis, Kafka and RabbitMQ and machine learning from having implemented a todo app :-(
Then let's say half of our profession has less than five years of experience (though I would guess it's closer to two years) -- they see what the elders do and copy that, like any good apprentice would. Suddenly, it's all they know.
Wild conjecture, of course. I would like to see it studied!
There is no formal theory around system design so you get people complaining about how monoliths are error prone and way too complex and you get people complaining about microservices with the exact same argument.
The result is that with a period of about a decade the industry just oscillates between the two methodologies in an endless flat circle.
Monoliths will come in vogue again in the 2020s.
Another fun part is that people on the internet complain but without merit. I cannot judge some project because I usually don’t know the project. So might as well be that monolith or not team was having high attrition or there might be many other things that would affect project.
No. It goes further then this. We actually don't even know if one is better or not.
There is no formal theory that allows us to prove these designs. For example we have formal theory behind algorithm complexity that quantifies runtime cost but no such theory exists for engineering design patterns. We aren't even sure what to quantify.
But just because a formal theory doesn't yet exist doesn't mean one can be developed in the future. Until then these system designs will be like history... Always repeating itself.
Perhaps the real crux is that large teams don't build good software. Especially not from the ground up.
Faced with this serious problem impacting sales, the programmers thought about it and eventually found a solution. They added... SLEEP instructions. All over the code in key points, sleep(seconds), calibrated so the response time was about the same as on ye olde CP/Me machines.
Customer reaction: completely satisfied.
... I think we can draw some conclusions on the universality of human stupidity and incentives to act that way.
Slaves don't trade productivity for their lives, they trade time.
No matter how many trillions of times more productive human activities will become, slaves's time will still belong to the masters and very little of their life will belong to them.
I am already seeing this newly discovered love of simple, fast monoliths and the developer ergonomics they offer. What is old is new again - we are at the beginning of this cycle. The older generation learned that "complexity kills", the new generation is beginning to get it as well.
DHH has a couple of posts on the topic recently: https://world.hey.com/dhh/how-to-recover-from-microservices-...
The systems can be worse because of what other commenters mention, the financial incentives, but also the companies profiting from worse systems must ensure that expectations are gradually lowered amongst people who write software and that these peoples' "skills" are progressively dumbed down. That is how they ensure that status quo is maintained and that people will never "go back".
Meanwhile, hardware and networking improvements have been amazing.
Here's why I'd choose microservices for a large project:
1. People don't produce uniform code quality. With microservices, the damage is often contained.
2. Every monolith is riddled with exceptional cases in a few years. Only a few people know about corner cases after a few years, and the company becomes dependent on those developers.
3. It's easier for junior developers to start contributing. With a monolith you'd need to be rigid with code reviews, whereas you could be a little lax with microservices. Again, ties into (1) above. This also allows a company to hire faster.
4. Different modules have different performance and non-functional requirements. For example, consider reading a large file. You don't want such an expensive process to compete for resources with say a product search flow. Even with a monolith, you wouldn't do this - you'd make an exception. In a few years, the monolith is full of special cases which only a few people know about. When those employees leave, the project sometimes stalls and code quality drops. Related to (2).
5. Microservices have become a lot easier thanks to k8s and docker. If you think about it, microservices were becoming popular even before k8s became mainstream. If it was viable then, it's a lot easier today.
6. It helps with organizing teams and assigning responsibility.
7. You don't need super small microservices. A microservice could very well handle all of a module - say all of payments (payment processing, refunds, coupon codes etc), or all of authentication (oauth, mfa etc).
8. Broken Windows Theory more often applies to monoliths, and much less to microservices. Delivery pressure is unavoidable in product development at various points. Which means that you'll often make compromises. Once you start making these compromises, people will keep making them more often.
9. It allows you the agility to choose a more efficient tech/process when available. Monoliths are rigid in tech choices, and don't easily allow you to adopt a different programming language or a framework. With Microservices, you could choose the stack that best solves the problem at hand. In addition, this allows a company to scale up the team faster.
Add:
10. It's difficult to fix schemas, contracts and data structures once they're in production. Refactoring is easier with microservices, given that the implications are local compared to monoliths.
Point 7 is the most important of the technical considerations: just make your services big enough to make sense as a separate unit and small enough to not be another unchangeable monolith. Yes, it’s not sane to have 300 Lambdas that add one number to another talking to each other over network, so just don’t do it.
Microservices the “Netflix way”, with the huge graph of services, gives a bad name to the idea of factoring out modules into independently deployable parts, which always made sense. Kubernetes just helps with deployment, but how coarse or fine you factor is on you.
The simplified view I hold is this: if your main problem isn't a lack of engineers, but how to split work between hundreds, or even thousands of engineers, then microservices make a lot of sense; that's true at FAANG and some other companies. If your main problem is how to get the maximum out of a limited number of engineers you can afford to hire, then microservices is not the solution. The latter is by far the most common case.
1 & 8: This is something I've experienced first-hand quite a bit. When code quality starts slipping in service, it accelerates and inevitably leads to more hacks built on hacks. The containment afforded by service separation helps keep these hacks quarantined and minimizes the pollution and spread.
6. There's no ambiguity or question of who owns what when services/packages are separated and have distinct owners. Transferring ownership means transferring ownership of the service. In large companies with some teams working on secret projects, it's much easier to have viable workflows with service separation than with a monolith. Permissions management and so many other things are easier.
7: I think this is an excellent point. Another way of looking at this is as having multiple focused monoliths.
I'd also add that there's a benefit in blast radius minimization. Updating a library or merging in a risky change doesn't risk breaking literally everything all at once. One service can update something and you know for a fact that it can't break anything other than that service and its dependents.
We weren't doing Sun RPC, PDO, CORBA and DCOM everywhere for fun, yet with some rebranding, SOA should be the solution for everything.
Monoliths vs microservices is really a spectrum.
Microservices trade design complexity with operational complexity, to address the shortcomings & patterns of the modern IT sector, at both worker and management levels.
This was an excellent choice for consulting (companies like ThoughtWorks, who were pushing microservices) as it optimized their workflows and bottom line: the answer to every client's question of "what's the architecture?" was "Microservices". The companies pimping m.s. were internally optimized for this architecture - running a consulting shop where the time between "here is our project" to "here is our proposal" is approaching zero is profitable.
And the best part -- just like all those "GPT-processes" now being pushed by companies built around an api -- is that by the time clients encounter operational complexity of running micro-services, the consultants have long left the building.
Yup!
And it annoys me so much how architects/consultants get away with just proposing 'microservices' without being held responsible for all the increased operational complexity. It's left as something for devops to figure out later.
Instead, factor services along natural fault lines. These are areas in the solution that scale differently from other parts and can tolerate communicating over http or message queue.
It is fine to have lots of people work on a single service. We compose things using 3rd party libraries all the time. Just treat internal code a bit more like 3rd party libraries.
More like 3 people requiring 13 microservices
Once we've explored this to its logical conclusion the pendulum will swing back to something more logical. Of course, it will swing too far and the cool kids will only want to work on monoliths even when it doesn't make sense (kind of like what is happening now with SQL maximalism).
And at least with one of them, the pendulum has already started swinging back. The cool-kids (aka "javascript hipster devs") have re-discovered the concept of a monorepo and are pushing for it hard, making it seem like some sort of new concept with fancy tooling. It'd be funny if it wasn't so caricatured and a sad state of our industry.
Monorepo and monolith are somewhat orthogonal. The desire to use a monorepo likely stems from wanting to emulate Google.
But what do people say that leads down that road, in your experience?
Welcome to your new job as Architect for our software!
What are some of the main bullet points you have in mind
for our next major release... ?Is this something that actually happened? Not heard from some third party - actually has someone experienced this as a decision in a technical team? It seems... unlikely.
(Not saying it's the right way to do things, just the natural way to optimise if you need to keep 20 teams busy working on one system. I think the incorrect assumption is that all 20 teams need to be busy doing the same thing.)
This is why I suggest team size is not a valid reason to break a problem into microservices. The problem itself is the thing to analyze.
Multi-team ownership of a single deployable has downsides to tradeoff against the complexity of more service and harder boundaries.
I think people bring up an exaggerated idea of extreme microservices being forced on people when it rarely if ever happens. If not, I'm happy to hear actual first hand experiences.
Anyways, it's still not an example of this happening. It's a recollection of a letter that maybe implies this result.
That's not to say there's no relationship between team size and the applicability of a service-oriented architecture. Microservices are a way of drawing hard boundaries around blocks of logic. These boundaries come with cognitive and operational costs, so they represent significant overhead, but they are a tool for abstracting both logic and physical operations to the maximum degree possible (100% is never possible for a single application). In order to get a net benefit, you have to have enough engineers that they can be experts in a subset of services, and the interfaces to their peer services have to be reliable and stable enough that they can be productive without knowing anything about their internals. So while I agree with you there is no universal floor of microservices that makes based purely on team size, there definitely is a ceiling.
Is there though? Operating systems have 10K - 20K devs based on a casual internet search. Of course there are many services / processes but they are not communicating using an http API or message queue. Similarly, Postgres has 350 people. I don't think team size is a factor at all.
I don’t know how many of you are musicians, but many people are surprised to find that the tone quality of a guitar has way more to do with the person playing it than the way it was built.
(yes, this is a joke, a bad one at that. badum tish)
(Also a bad joke.)
Python has byte strings, Unicode strings, format strings, regular expression strings, and probably a few others!
Rust has str and String, depending on ownership.
Why I hate Rust
[1] https://developer.apple.com/documentation/swift/substring
Personally if I was "stepping up" from PHP and wasn't going to use something powerful (or didn't need all that power) I'd be stepping up into javascript and nextjs and stuff like that.
Framework solves a general problem, but we try to solve a specific problem, thus it will always be a mismatch, frameworks usually solves that with a combination of magic and opaque abstraction layers.
Frameworks works best for short running projects, long running projects tend to outgrow the framework.
Prefer libraries over frameworks.
1. Don't pick an architecture because it's all the rage right now.
2. Don't pick an architecture that mimics your org's structure. Aka don't fall prey to Conway's law: https://en.wikipedia.org/wiki/Conway%27s_law.
3. Don't pick an architecture that your team can't operationalize--e.g. due to lack of skills or due to business constraints.
* "I always urge builders to consider the evolution of their systems over time and make sure the foundation is such that you can change and expand them with the minimum number of dependencies."
* "There are few one-way doors. Evaluating your systems regularly is as important, if not more so, than building them in the first place."
This is useful advice to people who can make high level org decisions. But they don’t necessarily know what’s going on at a software level.
The people who do know what’s going on often have a very hard time getting buy in for refactoring. Many of them (rightfully) conclude it’s better to just make management happy churning out features.
You can't not fall prey to Conway's law. You can only choose how you organize your people and their interfaces.
A long time ago I remember hearing a good bit of advice that for a new project you should innovate the product or the tech stack but not both. I think this is similar: unless you have a lot of resources (people + political clout), don’t buck the existing org chart in more than one area if you can avoid it.
You’ll constantly be dealing with all the problems because nobody else will care.
The only counterexample to Conway that I have seen working is “away teams”:
https://pedrodelgallego.github.io/blog/amazon/operating-mode...
This is natural and even beneficial: it reduces communication and coordination costs, which is the costliest and hardest part of any organization or system. Perhaps that sometimes lands you in a suboptimal local minimum, but escaping that is usually quite costly in the short term. Think about how often top-down reorgs actually generate value.
> If you hire the best engineers....
Guess what, there is no broad consensus on what "best engineer" means. I bet your org is rejecting really good engineers right now because they don't know what Kubernetes is. Same goes for literally any other technology that has been part of a hype cycle (Java in 2001, Ruby on Rails in 2011, ML in 2011; no the precise years don't matter).
> ...trust them to make the best decisions.
A lot of work encapsulated there in less than ten words. If you hire a bunch of people and tell them "you are the best", you think they are going to sit around and run the Raft protocol for consensus on deciding how to architect the system? No, each of them is going to reinvent Kubernetes, and likely not in an amazing way.
Microservices are often best deployed when there is a mixture of cultural and engineering factors that makes it easy to split up a system into different parts that have interfaces with each other. It has little to do with the superiority of the technical architecture.
----------------------------------------
[1] Looks like the article was written by the CTO of Amazon, which...surprises me a bit. Then again, from all accounts, Amazon's not exactly known as the best place to work; so maybe I'm right? In any case, anything written by Amazon is not directly applicable to the vast majority of small-to-medium companies.
Also, they never claimed technical superiority of microservices, quote:
> For example, a startup with five engineers may choose a monolithic architecture because it is easier to deploy and doesn’t require their small team to learn multiple programming languages.
No, but they are arguing against the strawman of the perceived technical inferiority of monoliths. Look at the title of the article.
I am simply calling out that strawman as such.
If they wanted to convey the message of "It all depends, use the best architecture for the job", they should reflect that in the title.
That’s weird. Any time I’ve seen anyone let their engineers engineer, those engineers have immediately started following the hype.
Literally nothing. Or at least nothing that actually works in any way.
In companies I'm familiar with projects are run by technical leads and product leads.
Managers don't interact with anyone doing actual work, which is probably for the best.
I've seen terrible spaghetti code apps where literally nothing can be refactored because of the model / view, God object dependency stuff all over the place.
If the need arises, you could move into SOA, and break out a couple of your larger domain modules. Continue following these same rules though.
If the need arises still, you could move into micro services, and break out more / all of your domain modules. Truly understand first whether you actually need this first.
But the "let's do micro services this monolith is old junk" trope, abandonwaring the codebase, building out a bunch of services without strong, fundamental domain knowledge, and then complaining when shit is expensive and fragile and broken- it's getting tiring.
It would be awesome if you could set up a parallel “monolith-stories.com” site with the exact same content, except for inverting the sentiment scale :)
Seriously!!
Like is it two pizzas per microservice or 10 microservices per one pizza.
Nearly spat out my coffee. Thank you for making my morning.
Increasing number of pizzas you get bigger teams.
Increasing number of microservices you assign more of them to a single team.
[1] Too lazy to keep looking right now
[2] https://en.wikipedia.org/wiki/Conway%27s_law
[3] https://www.reddit.com/r/ProgrammerHumor/comments/6jw33z/int...
As money becomes more expensive and as we inch further towards a massive economic crisis, companies that have allowed their R&D budget to bloat out of proportion with needlessly-distributed architectures are NGMI.
What even is a microservice?
There are other distinctions, borders and splits that are more important to consider.
Here's a koan that highlights a few of these considerations:
> If you run a kubernetes cluster on a single physical server, is it a monolith or is it a microservice architecture?
It’s not “monolith vs distributed”.
It’s “good monolith vs bad monolith or big ball of mud vs domain-driven design”
It depends on what your primary domain is, level of complexity, number and makeup of enterprise integrations, and more.
Some monoliths are very bad.
Some distributed systems are very bad.
My rundown is:
- is it a simple crud system? —-- monolith
Otherwise: - model it, identify bounded contexts, proceed accordingly
It's really only Amazon themselves that can afford to do so.
So Netflix doesn't really count as an example of how AWS is a practical option for video hosting.
Just pick the right architecture for the given problem. Sometimes it's monolith, sometimes it is not. The end.
Obviously doesn't scale but aluring anyway
The last hyper growth startup I worked at grew 10x in scale every 2-3 quarters for nearly 3 years at meaningfully large scale (millions of monthly transacting users). In that time, the number of different business-lines/categories and amount of functional flows and their intersecting/overlapping complexity also grew multiple folds.
So, we were adding whole new things and throwing away old things and basically refactoring everything every 18-months. Without knowing consciously, the superpower we had was our ability to refactor large live systems so well. In hindsight it became clear to me that our ability to do this hinged on a few different things:
1. A critical mass of engineers at both senior and junior levels understood the whole systems and flows end to end. A lot of engineers stayed with their own team developing strong functional-domain understanding. Similarly a good number of senior engineers rotated across teams.
2. The devops culture was extreme – every team (of 10-12 engineers) managed all aspects (dev-qa-ops-oncall etc) of building and operating their systems. This meant even very junior engineers were expected to know both functional and non-functional characteristics of their systems. Senior engineers (5-10 yr experience) were expected to evaluate new tech stacks and make choices for their team and live with the consequences.
3. Design proposals were openly shared and sought critical feedback. Technical peer reviews were rigorous. Engineers were genuinely curious to learn things, ask and understand things, challenge/debate things etc. Strong emphasis on first-principles thinking/reasoning and focusing on actual end-to-end problem-solving without being territorial or having dogmas was strongly encouraged and the opposite was strongly discouraged.
4. Doing live-migrations – we mastered the art of doing safe live migrations of services whose API schema or implementation was changing and of datastore whose schema or underlying tech was changing. We had a lot of different database tech migrations – from monolith SQL dbs, to NOSQL clusters to distributed SQL dbs and their equivalent in-memory dbs and caches.
Surprisingly, the things we didn't do so well but didn't really hurt our ability to refactor safely were:
1. Documentation – we had informal whiteboard diagrams photographed and stuck in wiki pages. We didn't have reams and reams of documentation.
2. Tests – we didn't have a formal and rigorous test coverage. We had end-to-end tests for load-testing and we had a small manual QA for doing end-to-end integration testing for critical flows. These came about a bit later – but trying to scale them effectively proved very challenging. But these were not seen as hurdles for doing refactors.
3. Formal architecture councils and formal approval processes – we didn't have these. Instead we had strong people to people connect and strong team level accountability – culturally people owned up their mistakes and do everything they could to fix things and do better next time. Humility was high.
Later, I worked at a large mature company with very large scale – and everything was exactly flipped on all the points above and any major refactors were a serious pain and migrations took forever and actually never completed. The contrast was very eye-opening and I realized in hindsight the above contributing factors.
He’s been ranting against cloud too.
[1] https://world.hey.com/dhh/even-amazon-can-t-make-sense-of-se...