Developers Are the Problem, Not Monoliths
codeboje.de
codeboje.de
Man, I agree so much with this. I've been on too many teams now which cargo cult _hard_ around microservices. In the best case, they think it'll solve the problems that plague the team ("we don't have to be responsible, because it'll enforce good boundaries!!!" (literally a quote)), in the worst case, people genuinely advocate for them as a "best practice" -- the latter group is far more terrifying.
Developers, myself included(!), tend to focus only on the upsides of a tech / pattern until they've lived through the downsides and earned their scars.
The most painful thing is that a lot of the time 'microservice oriented architecture' ends up not being made up of microservices at all, but is instead still a monolith that's been awkwardly distributed across a network, which has the nice property of being the worst of all available solutions.
I think the material point is that microservices can have drawbacks just as any other architecture or technology can have drawbacks. Whether it's microservices, or monoliths, or functional programming, or whatever language people are pushing instead of C, whatever it is, there will be drawbacks. And I think people are speaking to their annoyance at other developers for not recognizing the drawbacks of various technologies.
(Interestingly, this probably means that the people advocating for those technologies also don't really have a good appreciation of the appropriate uses of the strengths of the technologies in question.)
This sort of thing is a much overlooked aspect of YAGNI, and the related but lesser known concept of Reversible Decisions, which in a nutshell is: Any decision that can be changed easily, make it as cheaply as possible (time, emotional investment, etc. Chosing at random can work here, or quick show of hands without commentary - grandstanding is expensive). Any decision that will be difficult to unmake later? Play for time to allow more data points to come in before making the decision.
I kind of alternate between toolsmith and team lead, depending on the political landscape of the company. My philosophy is basically this. I say 'tool' below but it also usually works for processes (tools work better in times of high stress):
- Give people tools so they can do better (ie, make higher quality code changes)
- Make the tools foolproof, so they are more help than hindrance
- Apply social pressure (encouragement first, shaming for stragglers) to get people on board for using the tools
- repeat step 2 as new users find new problems
- Change development process to include the tool
So my first question is, is your plan to use microservices going to emulate this pattern, or are you just continuing a war of words by adding friction to the system? If you see people are writing bad code now, how much bad code that you don't see is going to be in microservices you don't care much about?
Are those microservices going to have an effect on your quality of life, the esteem of your team among other divisions of the company? Then you probably care what's in them and "out of sight out of mind" is a terrible strategy.
The 80/20 rule tends to rule, and the whole mess never gets close to being cleaned up.
"My methodology works until junior devs show up and don't know how to code"
"My methodology just works... except all the times it doesn't and when it doesn't, it's your fault because you did it."
That's what I got from this article.
It's development itself that's the problem. Software is really complex and things take a long time to build. The relationship between the time it takes for management to say "build it" and to actually build it is asymmetrical. It requires many people to build things of significance. Coordinating understanding of things between people is a hard problem. Gaining the experience to know why you should or shouldn't do something takes years. During that time you have to be working on things and contributing to code bases.
My only real beef with Fred Brooks is that he put the idea in people's heads that one or two people on the project are all important and everyone else is replaceable window dressing. A surgical team is a terrible structure for software development. It's much more like a team sport, where everyone should be constantly growing and reviewing past outcomes for new wisdom. It's not a perfect analogy/model because I also believe in the power of conceptual integrity, so there's a bit of a paradox in my line of thinking.
People should be able to add things to my code without me constantly meddling or taking over. Maybe I can clean it up to make it faster/safer, but my goal should be to be able to get things off my plate. If I'm indispensable, it should be in my ability to take on new areas of concern, not by monopolizing old ones and keeping my coworkers from getting promoted by making them look stupid.
He asked me the question, "So is it the team that works or the process?". That has stuck with me, especially since another mentor was fond of saying you haven't proven you know how to do something until you can repeat the results.
If you have a team with the right kinds of flexibility and the right set of boundaries (they are inflexible about things that tend to end badly), most processes can be made to work, because of all the things they do that aren't prescribed. Some of the hardest parts of development are breaking down big units of work into sane pieces with clear exit criteria. So far I know of has captured the essence of how to do that. If you can't get that part right none of the other stuff helps much, and if you can get that part right the rest is mostly background noise.
Programming is for people. I can count on zero hands how many times I've seen monoliths not end up... globbed together. It's fundamental to humans to like things broken up into little pieces with boundaries.
If monoliths don't work for people, then monoliths are the problem.
Or look at it this way: Even if monoliths are the problem, well, the people wrote the monoliths...
You are doing the same thing OP did: Fundamentally misunderstanding the purpose of programming. It's an interface for humans to communicate with machines in a way that makes sense to humans.
If monoliths always end up being unmaintainable pieces of garbage, then monoliths aren't really designed for humans or our tendencies. That means monoliths are the problem... they don't adapt to human nature.
1) You have tooling.
2) Probably also shared conventions and libraries.
3) Everything is easy to find and run on single dev machine.
And no, monoliths almost never end up that way as long as they're internally modular. Services are just modules with less sharing and more overhead.
How many monolithic 1,000,000+ line code basis have you navigated.
1) That's true of both. 2) That can be true or false of both. 3) That's covered by tooling?
> And no, monoliths almost never end up that way as long as they're internally modular.
Well your caveat is literally the problem this post is discussing. They usually aren't.
You don't have to prove to me that monoliths are bad. I get it. (You don't have to prove to me that spaghetti is bad, either.) But you seem to have missed my point.
Monoliths don't just happen. They didn't just wind up in your server room because the roof leaked during rainy season. They didn't get delivered by FedEx by mistake, and your people went ahead and signed for them. No, the monoliths are there because people wrote them.
Why did people write them? 1. Because they didn't know better, and just fell into it. 2. Because they thought that was the best way to architect something. 3. Because they didn't take the time and effort to make something more modular.
People cause the monoliths. To fix monoliths, you have to fix the people problem, or you're going to continue to get monoliths.
Breaking data structures apart often leads to it's own form of spaghetti.
I can say this based on four years working on a large multi-tenant service platform composed of several hundred microservices, held against a background of 24 years building software systems, including multi-million-line monoliths (good and bad).
If you're a small company, should you start with a microservices architecture? No. It is much harder to operate. Should you factor your monolith so it is easy to transition to microservices later? Yes. Your monolith will be of higher quality for doing so.
I'm not against micro-services, I'm against making everything a microservice because one can.
It's also an architecture that is now much more prone to failures, because while procedure calls essentially never fail, RPCs do. Or if they don't fail outright, have unpredictable latencies etc.
(I turned a microservices/SOA code-base into monolith back in 2003/2004. It worked wonders.)
Hmm...
1. Software applications are written with little error-handling on networking errors. During a network outage, such applications may stall or infinitely wait for an answer packet, permanently consuming memory or other resources. When the failed network becomes available, those applications may also fail to retry any stalled operations or require a (manual) restart.
https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
Infinite retries aren't "not really a problem". If the server isn't there or can't be reached, it isn't there or can't be reached. This is not a problem software/infrastructure can paper over transparently. You can try your best to mitigate, but that's hard, and often makes things worse.
I remember a colleague trying out CouchDB. His takeaway was that all error-recovery attempts by distributed CouchDB servers made things worse. So you could use it as a semi-decent single-instance DB, but mercy on your soul should you be so brazen as to attempt to use the distribution features.
> heavy investment in microservice tooling.
And what percentage of microservices is that?
> costs in terms of infrastructure and latency
>> Or if they don't fail outright, have unpredictable latencies
Hmm...
This can also introduce big problems with 95th-percentile latency, because the retries aren't free, they trade failure for slowness. In user-facing services this often isn't a problem, because the user eats the slowness and grumbles. But in services way down in the stack (eg. BigTable), it can be a big problem. High 95th-percentile latency can make services that previously weren't on the critical path suddenly part of the critical path; it can trigger timeouts in higher-level code; it can trigger retries in higher-level code; and it can cause large increases in load (with the possibility of further cascading failures) if those additional retries put some other service over its capacity limits.
TANSTAAFL. RPC failures still result in complexity, it's just that there are possible systems that can shunt that complexity into where it's less of a problem for the user. The only time you can outright ignore them is if you're a junior developer, though; anyone at senior/TL/architect level needs to think about them and be prepared.
Given the choice, I'd rather re-write 50 microservices in a heartbeat.
It's a function of people and constraints.
Folks talk a lot about managing complexity, but it feels a bit too tech-weenie and direct enough for the problem we have. The phrase I've been using is managing cognitive load, because it covers more than making decisions around microservices or null pointers. Our failure -- and hatred -- around managing cognitive load is what keeps taking great organizations and destroying them.
What happens if you have 1-5 really smart devs and give them freedom and a problem? You get a solved problem What happens if you keep adding really smart, capable people? You get a mess. Continue this process, you'll have burdensome process, a couple of new frameworks, an architecture team, and more. You'll get a huge amount of really super smart people -- all making a mess and pointing the finger at somebody else for being responsible.
After some consideration, I have come to a sad conclusion: most of us in tech do not want to do our jobs. Our jobs are to solve problems for people. After that, it's brutally keeping the cognitive load down for everybody that touches what we do. To put others first in this way and realize we're prone to making things far, far too complicated is humbling. It involves creating simple things that a first-year programmer might make. In short, it's boring. And who wants to be bored?
We have met the enemy and it is us.
Basically, just solving a problem once isn't going to keep large teams of people employed long-term, so people don't work that way.
Here's another way of looking at it: how do you know not to do your job?
It's a simple question. All true professions like lawyers or doctors, have clear guidelines for when their services are not appropriate. You go ask a doctor to use his medical skills to hurt somebody and they'll turn you down (hopefully!)
But not tech. You can throw 100 technology developers at anything. Doesn't matter if the problem is already solved, whether it needs solving or not, whether it's ethical to solve, or whatnot. Doesn't even matter (and this is more to my point) whether you can do it in one line of code or a million. In fact, a million lines would be better! Keeps everybody looking busy.
We desperately need better ethics in our profession. Code budgets can help some here, but there's a lot more room for growth.
Contracts (api, documentation for those apis, documented processes, etc.) are important to scale and greatly reduce communication load. Microservices aren't required to make that happen, but they sure help.
As you see it's easy to shift the responsibility. You have to question and justify every foundational or core decision, weight all cons and pros, and revise. But, well, if nobody is motivated or cares..
If developers gave a estimate (estimates should be ranges) and stuck to it (and delivered on it) then the managers would trust our estimates a bit more.
I constantly beat the drum for quality and the practices to enable it. but i agree with others, just the manager's actions are not the reason. As a dev I didn't let managers bully me into bad code, and as a manager just giving permission to do things right wasn't enough.
Experienced developers will know that spending time not writing code is where you really solve the challenges at hand.
It's not about patterns or arc' style, that is merely a tool to reach the goal.
It's about experience and knowing when step back and iterate, and take the time needed to solve something.
I often spent months breaking a problem down, and in that period of time I often realize that I was on the wrong foot from the beginning. So I iterate once again, slowly building confidence in a better solution. And in those months of tinkering I don't write production code that problem.
Quality work takes time to do, it's a oneway street that thing.
I think people don’t understand what a true enterprise monolith actually looks like.
In my own experience, the longest lasting codebases I've worked on or implemented are those meant to be thrown away that didn't have a lot of extra cruft for "pattern" sake. Add it when you need it, and push back on every feature you can push back on.
Also, organize by practical feature, not by type of file. If a feature only has a view and not a distinct controller or models, that's okay... making it easy to discover where the hell crap is makes it easier to maintain. That doesn't mean shove all of your models/controllers/views/components etc into 4 mostly mirrored trees, it means merge them into a structure where like needed/used things are together.
1. The tagline on this website is "Guiding Developers since 1869". Is that a joke?
2. If he is so against micro services, why does he promote three of his own pocket guides on how to do them?
Is there something I am missing?
2) He's not against micro-services. He's just saying that your problems won't go away bc you're building micro-services instead of monoliths.
PS: i would say #1 is a joke.
i'm trying to imagine a situation where it could be serious, but i'm not coming up with much.
The issue is I've never worked at a company without at least a few dumb and/or lazy developers. I've done freelance and trying to do it all myself, but its just too time consuming. So, I dunno, I've learned to accept rules like this just to curb the stupidity some.
It's all of them. They all contribute to a system, and the system is what produces the output. You cannot lay the blame on any single part of the system without understanding the interrelationship between each part.
A developer produces some quick and dirty prototype and it ships but is a bitch to maintain. Blame the dev? Maybe. Blame the PM? Maybe. Blame sales? Maybe. There was a deadline and the dev hit it, you can't blame them for shipping when the deadline wasn't relaxed (by the PM or sales). But maybe you can blame them for giving bad estimates.
Why are the estimates bad? Do they know how to estimate? Have they collected metrics on how long problems of various size take to be completed? Has the PM? Has the head of the software division set that as a goal to understand? If not, then some teams will estimate better than others because they took the initiative. But they probably weren't given the money or time to do it, they stole that time from their customers (which isn't wrong, but it's hardly scaleable).
Is the team junior? Have you trained them? Do you have a training program in your company? Mentorship? Do you send them to conferences? Don't blame the junior devs for the failure of the business to understand and convey the importance of training to PMs and devs.
A dev who feels they have no authority to say no will always say yes, even if it's outrageous. They saw a peer get fired or demoted for saying no too many times and don't want to risk it themselves. It's the system that creates this. So create an environment where the devs can say no to requirements. Help them to understand how to present their case when saying no. Or, better, teach them to say, "Yes, but...". "Yes we can deliver that feature, but you will have to delay delivery by a month." "Yes we can deliver that feature, but you will have to pay for database training or a DBA because we don't have that competency in our group." "Yes we can deliver that feature, and hit that deadline, but only if you descope these other features." Don't make it optional, make it clear what the tradeoffs are. Sure, devs can do this now. But many feel disempowered to do so. Create the culture within the business that empowers them to make these meaningful contributions.
A company that lets sales to scope all features and decide deadlines is setting itself up for failure (failure to deliver stable, reliable products on time and in budget). A company with a too conservative dev group that says no to everything will fail in the market place (failure to deliver what customers want). It continues from there. Don't look to one group and play the blame game unless that group is overly strong/weak within the organization. At which point you need to distribute authority/responsibility better and empower people to make meaningful contributions and decisions.
Examine the system and culture, then shift it towards one that can achieve the desired goal, or change the goal if you're unwilling to change the system, or fail.
You can change yourself though and inspire people to move with you. So, any change starts with oneself and not with others.
> If the biz says, “Jump,” many devs ask “how high?” or just do it. It is the same with requirements or workloads.
There is only so much push back senior, let alone junior, developers can give. Having "soft skills" helps a lot with this, especially when alternative solutions can be proposed or your force the people with real power to assign _actual_ priorities to work. (i.e. "I'm sorry I've not done anything for your project, the VP told me to work on this other project. Please consult with them if you believe yours is more urgent.")
> A recent example makes that more clearly. Take a dev team, who build a new app from scratch - greenfield. They chose a document store but modeled their domain objects in a relational way. Now X features and iterations later, the performance went downhill… Yet, no one admits the wrong architectural decision. But anybody complains about it - even the devs. Bad monolith.
There are also real consequences to making large changes to something like a datastore. The amount of time and risk is hard to calculate, and for a non-tech business hard to justify when there are other issues requiring developer attention.
I think what I dislike most about this post is that it continues to perpetuate the feeling that the developers and the business are on separate "sides" or otherwise the post ignores concerns from the rest of the business. It's really about trust. If you, as the developer go "this is going to take 3 weeks" and the asker goes "but I really, really want it in 1", the two sides need to trust each other enough to honestly discuss the discrepancy. Could some MVP be rushed out, but the follow up 2 (or 3) weeks _be scheduled now_ to clean up the rough edges, ensure formatting and full coverage test suites are finished, &c. If the developer doesn't trust the asker to be reasonable, and the asker doesn't trust that the developer honestly needs some additional time, negotiations cease and someone winds up unhappy and the business as a whole suffers from either a later product or additional tech debt.
I think most people, not just developers, have a hard time understanding that if you can or simply how to frame "non-productive" time in terms of the business' bottom-line or risk, you'll have much more luck (but no guarantees, some people just don't care about your opinions as their underling unfortunately, but I would say most do).
For instance, maintenance of a mechanical system leads to downtime and increased operational costs. However, that maintenance increases the reliably and operating life of the mechanical system.
For code, just building in a little time for each issues' estimate for cleaning, testing, review. Why is this important? Reviews help familarize other team members with the implementation of new features, minimizing the single point of failure. Tests help to spot coding errors or conflicting business priorities at a later date when the details of this specific product may not be fresh in everyone's mind. Cleaning up/formatting code enables more of the team to jump in and fix issues, again reducing the single point of failure. All of these together help ensure that the code is less likely to randomly "break"/not handle certain cases correctly as well as enable anyone on the team to take up a bug with a reasonable expectation that it won't balloon because no-one knows how the feature works.
Anyway, the issue has nothing to do with monoliths vs microservices and everything to do with the trust and communications between different parts of the business.
Thanks for pointing this out. This is critical. If each group feels they have a separate objective/goal that doesn't match the others (and the business as a whole) you'll get lots of problems. Communication breakdowns, months or years spent building the wrong things, etc.
An organization needs to establish its objective/goal and direct its internal entitities towards achieving that goal. A misalignment of goals will create divisions within the organization that means it will, overall, fail to achieve its objective (or only achieve some unsatisfactory result, like being late to market or of poor quality).
However, if you want to change a system or even build up trust, one can only start working at oneself. And that was my intention for the post, raising awareness that devs must change themselves and stop blaming other humans or even patterns for shit they (partly) created themselves. We can not change another person, we can only change us and maybe inspire others to join.
Same is with trust. People do only trust you if there is a good reason for it. And you have to build that reason up over time. For example, if I can't hold my ground with given estimates, how can I expect that the other person trusts me (in that point)? Those sum up. In the end, often biz does not trust devs and vice versa. But I can't change biz nor the other devs, only myself.