This isn't over-complicated, it's just complicated
benlorantfy.com
benlorantfy.com
However, the problem didn't have to be this complex. If someone would've asked some questions about the 'why' of it all, they would've been able to come to that conclusion. But nobody did. So the team was working on an over-complicated system for an over-complicated problem.
So, while I agree with the article that one might be under-estimating the complexity of a problem, it might be wise to look into whether solving the complex problem is actually necessary and whether 99% can be achieved by solving a less complex problem in a less complex way and accepting the solution isn't perfect.
I went for the "why", and found that this whole thing that swallowed 45 man-hours a week could be reduced to a very quick daily cron job.
I did that. My manager shrugged. I went to work elsewhere. It showed me how weird large corporations can get, but also how important it is to question requirements.
For example, I was asked to first guide and finally take over a project that went south. It was a simple control system. The person doing the work was unfamiliar with PIDs, I tried to guide them towards that solution, but it was unfamiliar territory for them and they thought they could do better with their own solution.
They ended up creating a very complicated non-functional mess, and were insistent that it was impossible to solve simply. 10 lines of PID later it was under perfect control.
For instance, I’m flying to the other side of the country with a team to do an experiment at a vendor’s location because the safety folks at my current location are inexperienced and are working with a contractor who has zero experience installing the appropriate safety equipment at my location and it has been stuck in discussion for about two years after the scope has increased an entire order of magnitude.
One person with sufficient authority and courage could have cut the Gordian knot a year ago and improved the efficiency of the entire project a good order of magnitude.
Funny thing is, I found it much easier to challenge the original framing with upper management than with my peers.
- (me) Hey, if we sacrifice this feature, we can ship the whole thing twice a fast.
- (ceo) Fuck yeah, lets do this.
vs
- (me) Hey, this seems to be solving a lot of the problems we don't have today, how about we keep it simple and leave only what matters?
- (engineer) This is best practice, simpler solution is an anti-pattern.
- (engineer) Google is doing it this way. Do you think you're smarter than Google?
- (engineer) But I already built it, why change it to something inferior?
- (engineer) My solution is very flexible, it allows changing database to a toaser with a simple configuration. It has unit-tests and mocks. Are you saying all of that was for nothing?
And yeah, it's mostly misaligned incentives. Solving complex problems makes engineers happy. When we don't have complex problems, we invent them.
Think of it as smooth ramp up | ramp down response to change, pressure, volume in industrial process control systems.
eg: https://ctms.engin.umich.edu/CTMS/index.php?example=Introduc...
The proportional part is similar to a "dumb" controller - spindle slow, speed up spindle. The Integral part will increase the rate at which the spindle changes speed the longer the spindle remains the wrong speed, and the derivative part of the controller attempts to prevent oscillations of spindle speed.
The author's point being that if you did give "the problem and all its subproblems" much thought, you'd agree on their proposed solution.
But there's also absolutely a problem of "giving the problem and all its subproblems TOO much thought".
That's how over-complicated, over-engineered, second-system effect, everything-and-the-kitchen-sink designs, and architecture astronauts' designs, come to be...
Sometimes you're just giving the problem and all its subproblems too much though, and solving the simple case will get you a Pareto 80% of the way there - or even 100% of the way there, and all the other issues you want to tackle are remote eventualities and edge cases you can just ignore without issue...
>Do not judge a solution as “over-complicated” until you fully understand the problem domain and every sub-problem that the business needs to be solved.
That's another issue: every sub-problem that the business wants to be solved is seldom every sub-problem that the business needs to be solved.
Requirements get blown up with BS all the time, to the detriment of business value, delivery times, etc. Sometimes by the customer (who just throws in stuff they think they need, or they might need someday), oftentimes by the sales people or resident architecture astronauts...
So, solving at an appropriate design level "the problem domain and every sub-problem that the business needs to be solved" doesn't mean your design isn't over-complicated. It might be properly-complicated for those needs, over over-complicated compared to what it should actually be, if the real needs where considered (as opposed to mere "business wants" presented as "business needs").
The difference in process I usually see is incrementally coding up a solution adding detail cases vs first mapping problem facets to concepts then factoring those concepts so the implementation has boundaries that follow the problem seams. Just because you can point to an aspect of the problem for every line of code that exists doesn't mean it's not over complicated.
"Over complicated" is either an assessment by someone who doesn't fully understand the problem OR an inability to see or even imagine that a simpler solution could exist. Simple doesn't mean easy (e.g. add an 'if' somewhere in current design). Understanding is not only about coverage. A thing that I recently encountered is decomposing a problem top-down, and implementing the solution top-down, which lower-levels getting parts of the problem at hand unnecessarily leaking into it. Once I suggested to think of decomposing the problem top-down, but built the solution bottom-up from that envisioning, all the while trying to name concepts of the lower-level things and keeping them as free of unnecessary details at each level, things cleaner separation of concerns with deliberate interface points.
It's weird because I'm not even that old, and this kind of planning or refactoring was definitely a part of the curriculum at my definitely-not-Ivy-League uni.
Even though NIH is a pretty toxic attitude, the opposite can be as well. You don't have to stray that far past hello world CRUD apps to find an areas where the available libraries are poorly suited, and the superior option is to roll something on your own.
It turns out that most of the time it does need to be that complicated. Especially with anyone who isn't just a raw junior developer.
If a legitimate feature is complex there's very little you can do. Apart from push back on the implementation specifics.
Quite often, code is too complicated due to it needing to interact with other problematic code.
So you could spend many times longer fixing all the underlying complexity and hopefully make it better.
Or just accept that it's not perfect, which while regrettably is the pragmatic choice (Until things get so bad it isn't)
So the design itself needs to be questioned.
Trying to make a "good" architecture when there are no sane way to do it in a good way is one of the deepest sins in my experience.
What a messy out of necessity implementation needs is seldom another layer of abstraction to mess with your head.
But realistically, just a handful of the real world scenarios we all face fall into this category of "least worse arch is black magic". I'd bet my house than 90% of the architectural problems we faced exist because of a poor process on understanding requirements or poor architectural design. In no small amount due to hese two things being very hard.
It’s incredible how simple things can be if you think about them creatively enough.
They often tend to be sort of bolted on, rather than integrated neatly by rewriting the thing to a new simple solution that caters to both requirement sets.
The problem is more like Katamari Damacy-code than spaghetti code.
Most engineers get excited about engineering stuff, that’s a good thing but in my experience, a ton of premature optimization can happen before you get your product in front of users.
Personally I think working prototypes that you can throw it front of a non technical person should always be the initial state of a project if possible.
It’s much easier for folks to grok complexity when it’s tied to a functional prototype and discussions come up about its limitations.
The leadership solution to overcomplication is not to say “you’re overcomplicating things” but to be a steadfast curious collaborator with them to encourage the person to see more clearly the challenge that they’ve obscured with needless puzzle-making. Easier said than done.
It requires overcoming the fear of being fired before gaining the required knowledge and experience.
My point being that when you put an arrow on a white board you absolutely and immediately have to be skeptical and assume that some data will have to move against the arrow, and that will drag some hidden complexity. Stuff like very far downstream passes in compilers will need the source file line numbers and some inversion system because they might need to report errors or emit debug symbols, and that can be complicated because the code might be unrecognizable at that point.
He lists some frontend considerations; a backend developer might have a list like: comprehensive observability, automated builds and deployments, infrastructure defined as code, developer function-as-a-service platforms, batch processing infrastructure, horizontal scalability, database recovery time/point objectives, compliance, and SLAs.
The truth is that these ARE requirements once a project gets to a certain scale, yet they have little to do with your core application domain. To a certain extent, it is inherently complicated.
But if you look at the "modern" cloud approach to these problems, it's full of additional complexity from the solutions themselves: k8s, ec2 spot market, vpcs, prometheus/grafana/jaeger, ELK, multiple environments, airflow, a database+replica+connectionpool for every microservice, terraform, jenkins, istio, lambda ... oy. It literally takes a team of people to manage the scaffolding to support the actual apps. Is that really the best we can do? I think there's room to radically simplify this scaffolding, while still meeting all of the advanced business requirements. In that sense, today's approach is over-complicated.
I think that this is the "money quote."
I wrote something to that effect a while back[0].
I've been at this stuff for a long time. I have definitely done my share of overenginering, and overcomplicating, but it's my experience that a lot of that, comes from underestimating the complexity of the problem domain, during the design phase, and going "Lucy and the Chocolate Factory,"[1] after starting implementation (when things start to come out of the woodwork).
The more experienced that I get, the more likely I am, to predict some of the issues, before implementation, and I have learned to develop in a modular, flexible manner, that allows me to modify things, down the road.
Right now, I am in the middle of a fairly major refactoring of an app, that is in "chocolate meltdown" mode, because of all the addons.
Rewriting the server interactions is a big, risky task, but we aren't shipping (one reason I don't like the MVP model), so it's entirely possible to get it right before our first real users get their hands on it.
[0] https://littlegreenviper.com/miscellany/problems-and-solutio...
You can often simplify code at the surface level, such as removing dead code, but structural complexity is really hard to get rid of, often impossible.
People are conflating "complex" with "difficult". Difficult problems can have simple solutions. Complex problems cannot have (correct) simple solutions.
This is different from "it's hard to figure out how to represent this problem and its solution in software". For a crude analogy, E = mc^2 is a very simple equation, but it was very hard for humans to figure out. In other words, it was extremely difficult — but it wasn't necessarily complex.
I disagree. What most people usually mean by "complex" in any context is some fuzzy not-really-defined thing they aren't really sure about. You're bonkers if you think most people's minds refer to a soft intuition for descriptive complexity when they say the expression "complex", even if the context is software
Having said that, I do agree with your original reply since you were thinking of that specific type of complexity
I didn't say that this is what people mean in "any context" but rather _specifically_ in software (see the quoted text in your post). I do agree that people are fuzzy about it; that's also why I also said that there is frequently a confusion between _complexity_ and _difficulty_.
Software is a subset of any context, which is why I figured it would have been fine to replace it with the more general idea
Basically entropy.
Now, if you can write another program that does the same, but can be compressed to a much smaller file, it is likely to have a more elegant approach. (Especially if both use the same language and style guide).
Another factor to consider, though, is the compression factor of a program. The more a program is compressed, the more repetitive it's likely to be. Such code tends to have less real complexity, even if it may look complex at first glance.
Sometimes, I need to add a lot of extra code to gain 30% performance.
Well, I suppose it is a bi-modal distribution:
Level 1: Naive implementation of a problem, using little code. Often full of bugs or uncovered edge cases, or very slow. Level 2: Most bugs and edge cases are covered, but the code base is enormous and still very inefficient. Level 3: The code is successfully refactored. An elegant solution was found, that removes a lot of repetitive code, while improving quality and performance (often by a lot). Level 4: The code is tuned to near-perfection. Most of the code is dedicated to hardcore optimizations, making it really hard to understand to a novice. Also, this level risks introducing bugs and vulnerabilities that are very hard to find.
There is the extreme case of the app that does 3000 requests to boot up that boots up in 40 seconds over a LAN and takes 40 minutes over the WAN. From one viewpoint the code that does all the round trips might be straightforward and the process of packing up your data so the app can get it all done in one request involves more hard thinking. Something that might be just "1 more request" for the first system may involve more rethinking of the one big request in the second system.
On the other hand, the second system is automatically much less error prone and you never need to think about it doing http requests when it is in a partially initialized state.
Sometimes even if a complex solution has benefits you have to weigh what the cost is. Will it be harder to maintain this code/process? Will someone new coming in ever be able to understand it without extensive training?
Sometimes complex problems require complex solutions, but it should always be a last resort after exhausting how it might be done simpler.
If you have a complicated solution and can't explain or justify that complexity, you can't necessarily expect others to just respect it and take your word that it's necessary. They might have their own opinion. And maybe their opinion is wrong! But if everybody's flying by their own personal tacit intuition and can't communicate about the actual technical issues, you're going to get disagreements. You're going to get a stand-off of everyone throwing their feelings at the problem, and the HIPPO will decide what is ultimately done.
IMHO Complexity is a function of the number of subsystems; while complicatedness is the function of the nature of those interactions.
(Which leads straight back to the worst part of it - why do we pretend complexity is what we want to estimate anyway? We all know it's time the managers at least are interested in. We don't have 'an appetite' for x amount of complexity in a two week period, we have time for a certain amount of stuff. Calling it complexity is just pretending not to be micro-managing time; we should estimate time (and call it that) but have an expectation that the estimates will often be way off and a culture that that's fine.)
Very true, and also often, in my experience, the cause of over-complicated implementations, by the time the previously-unacknowledged complexities could no longer be ignored.
So things are biased toward accidental complexity.(i.e. Over-complexification).
That and people adding subjective beliefs in their solutions.
That's also why the space keeps evolving.
I know of solutions that are more elegant than others, physically smaller, but those are often even less simple than the complicated solution.
> Complex problems exist...
Complex is an attribute for the solution, the problem could be solvable or not.