System design and the cost of architectural complexity (2013)
dspace.mit.edu
dspace.mit.edu
In the mid 2000s we installed Server 2003/8 on a whole bunch of physical servers in a data centre. This was for one of Australia’s ‘big four’ banks. Me and a team of less than ten, most of whom I still know, managed the entire thing. We were the ‘3rd level infrastructure’ team. Most of us understood most of the stack end-to-end, although there were some specialities.
In 2023 I work for an IT integrator. These days we use Azure, because Microsoft has fooled us in to thinking that it’s cheaper.
My current project is, essentially, a website migration. Not even a big or complex website. The project has been running for 6 months. I only joined recently so I can’t be sure but we’ve probably burned AU$2m. The schedule I’m managing has us doing the cutover in June. That is optimistic.
The architects are currently trying to work out how Azure [details] connect to Azure [details] while still [security] and being able to [complex integration].
Every day some new issue appears that the architects have to figure out how to work around.
No single person has a goddamned clue how the end-to-end thing fits together. Not one, not a clue. That scares me.
‘Architectural complexity’ has crept down to the level of infrastructure. It’s painful to see. I hope it stops soon but I am not hopeful.
but once it stops fitting inside one persons head then the thing is literally only possible to design by committee - it cannot fit together and perhaps should stop being called a single system
The magic is the enterprise message bus. It is meme crap until you actually need one. When it works, it really works. This is the only thing I've ever seen realistically tie together certain industrial environments.
The important part is where you draw up the system boundaries
You mean tens of years?
Anything past that though... maybe QNX down to the bytecode? It's at least elegant enough to suggest so.
You might argue that that isn't required to fully understand the system, but that argument is no different than saying you don't need to understand the hardware at a detailed level to fully understand your system.
Before, all the pieces of the architecture were in a couple hundred *.cpp files in one directory hierarchy. Now it's millions of files across thousands of directories each turned into a service run by a dedicated team, and none of them can see each other.
You can become an expert in all those pieces. But it requires actually using them all together, to discover the parts that don't fit and how to work around them. This is why the modern ideal of completely independent APIs is a terrible design. There is literally no way to know if anything works with anything else until you try to run it in production. Monoliths are terrible at scaling, but easy to understand. SOA apps are great at scaling, but impossible to understand.
Good luck on your migration. Changing the wheels on a moving tractor trailer always sucks.
I mean my use of bad metaphors kind of underlines the point we don't really understand his problem - but the large organisation that builds these is itself an example of such impossible to understand interactions - maybe we will learn from Azure etc and take those learnings into taking our own orgs.
The super bizarre outlier is the Open Source community, which doesn't really follow any organizational plan (except the corporate projects and foundations and committees that pretend they're open source). For whatever reason, Open Source succeeds despite no hierarchy, no clearly defined architecture, no particular strategy.
But then again, maybe that's why it works so well. An organization has a fixed structure, rules, priorities. It can fail, lose its intellectual property, its funding, be competed against. Open Source has no such limits. It's the wild west. Do whatever you want, whenever you want, and whatever works, survives. Lke natural evolution. An organic ecosystem, rather than a plastic one.
A) you can google or ChatGPT things more easily these days
B) there's a truck load of services and ways to connect and do things. It's hard to learn all of these or pick a best option for everyone
C) SSO + identity models + API permissions generally made everything more complex
D) Many things in Azure are changing often, so you don't want to get too invested in a particular "version"
So things got more in-depth while expanding your available options (which are usually just as complex)
Back in the bank days all of understood Windows, multi-tier AD, DNS, DHCP, SMS* (now MECM via SCCM), that distributed file system whose name I forget, file & print, Exchange, and all the other things that had a plugin to mmc.exe — we could do ‘em all. Oh and a handful of networking because that was also much simpler. Oh and all of the client configuration.
You could give me a server and a day and I could stand you up a basic infrastructure. I could know it all.
(*Though the maniacal way that SMS 2.0 did its magic with text-based log files shuttled in from the endpoint always drove me bonkers.)
RESTful APIs? Nope, we are going full-gRPC.
No hand-rolled OAuth - we are using AuthSolution1. Forget that, AuthSolution1 has a security vulnerability, we are going with AuthSolution2. Wait, AuthSolution2 won't work with cloud providers database offering... let's try AuthSolution3.
Let's use IFTTT. No, how about Zapier. But, we need gRPC and now GraphQL.
The databases are over-provisioned, lets look at serverless as a critical path item. Executive VP #2 thinks we should use a NoSQL solution.
BTW: Cloud provider X is dropping support for database version 1.1, we need to migrate to 2.0... or skip ahead to the just-released 3.0.
And, then the CEO comes and asks: "Can you spend some time trying to get ChatGPT to build this whole thing for us?"
Opened up an AWS app earlier in the week, asked wtf is this and how does this even work. Logged out and said not my problem.
I feel like the complexity and the knowledge pulled back and hidden by the layers of abstraction still must reside in a mind to make things work. I am troubleshooting issues in code that I can’t write or fully read, I ask how it work, what it relies on, what it writes to, what it’s intended function. By the time we get through explaining the problem to me, we usually come up with a solution or understanding of why it doesn’t work and what needs to change.
Some really smart folks writing this stuff but cannot keep it all or understand how these systems connect and work together. 300 lines of code can hit 10-20 different technologies and it made perfect sense when you wrote it, but when it’s broken…
This should not be a valid excuse to not understand your job.
> there's a truck load of services and ways to connect and do things. It's hard to learn all of these or pick a best option for everyone
Agreed, and I think this is where people go wrong. Giving devs the ability/permission to pick whatever random tooling they want is not a good path for maintainability. "It's in AWS, so we can use it." I'd go so far as to argue that if you want to use Managed Service X, you need to first successfully launch it on an EC2 with no other help than the official docs; otherwise how can you possibly hope to understand it when it goes wrong?
> SSO + identity models + API permissions generally made everything more complex
Fully agree, IAM is a nightmare. But again, if it's your job, that's not really an excuse for anything other than higher pay.
The level of abstraction that is commonplace these days is insane.
At $dayjob, there is a reverse proxy in front of a API gateway that is itself a load-balanced service. There are load-balancers behind it, pointing at a reverse proxy on a service fabric. Within that, some "architect" decided to add additional "mid tier" servers because that was the cool thing to do in the 1990s. Behind all of this is an Azure App Service, which is in turn a load-balanced server that includes a reverse proxy.
They want to move this to Kubernetes, so they can now have proxies seven layers deep, nested hypervisors running containers, and services ping-ponging across six zones (data centre locations) in two different clouds. That connectivity will go through a software virtual router platform, so it's not even direct point-to-point connectivity.
I pity the poor fool who will have to diagnose an operational issue in this madness.
No mere mortal will be able to make it go fast.
Look no further beyond those getting paid to find the reason such monstrosities exist.
I think i's more likely that it's something described in studies about human problem solving, which leads us to preferring overly complex solutions because we prefer adding something.
https://www.nature.com/articles/d41586-021-00592-0 (you may need Sci-Hub - paywalled)
> Adding is favoured over subtracting in problem solving
> A series of problem-solving experiments reveal that people are more likely to consider solutions that add features than solutions that remove them, even when removing features is more efficient.
https://www.nature.com/articles/s41586-021-03380-y
> People systematically overlook subtractive changes
https://www.scientificamerican.com/article/our-brain-typical...
https://www.washingtonpost.com/business/2021/04/16/bias-prob...
I think we also have forces in play that add to this bias. Or they are the result of it, for some of them. For example, when there is an easy and cheap or even free solution there is no incentive for marketing and selling it. It's hard to tell a customer that they don't actually need you, especially if it isn't a one-off, or that they should go somewhere cheaper and more simple. It is much easier to get paid for adding than for doing nothing or for subtracting, both as an employee and as a business.
It certainly looks that way when I look around. We have frequent complaints about a lack of skilled labor - but that's also because everything keeps getting more and more complex.
For example, once it was enough to once every half year update a piece of paper on each bus or tram or train station with the schedule. Now it's some highly complex networked computer solution and display to display when the next bus is going to arrive. Sure there is added value - but there also is a gigantic amount of new complexity for every small "convenience". When it comes to something like waiting for the next tram, specially in an urban environment with 10 minute schedule, the benefit of knowing if it's 2 or 7 minutes that you'll have to wait is negligible compared to the effort of providing and maintaining that service. (we have this situation in my town).
Of course there is added value for much of it, maybe all. But I see the complexity and size of systems I knew in a much more simple form when I grew up exploding.
The problem is, continuously educating new people to know ever more and to be able to handle more and more sophisticated systems has a continuously increasing cost to society too, since we don't live any longer than before. Never mind everything we do, from planning to maintenance, is ever more complex too.
In addition, the new system may not be as robust as the more simple ones of the past. It all works as long as it does. If we see another chip crisis for example, and I just read a pretty dark forecast for China/Taiwan, all those solutions that require working world-wide supply chains, here, chips from Taiwan, are a problem.
For instance, say Team A is used to receiving a report in Excel every week. If an upstream system is added or changed, Team A will fight like hell to keep their Excel report even if it’s no longer needed. Whereas “the system shall generate a weekly Excel report” is just another feature in the backlog and much easier to sell to the organization.
I just watched a simple four-building network get over-architected into a monstrosity involving multiple consultancies, vendors, and products.
Why? Nobody gets paid to not add things. Especially as a third-party, you get no money at all for doing nothing, even if that’s the best solution for the customer.
Worse still, in this case the vendor relationship was managed by a customer staffer who is themselves a direct contractor! If the work to manage external consultants dries up, their own contract won’t be renewed.
“You can’t convince someone of something if their salary depends on them not believing it.”
I had a doctor trying to make me pay less and come less. I had many more trying to sell me useless garbage, from network marketing vegetable juice to magnetic mats, after they did not find an obvious disease and concluded "it's in my head" (I'm fine now, reason was eventually found at a university clinic and solved for good).
As I already mentioned in my post, to actively market things you need to make money. The teams making he most money are able to do the most marketing and the most sales. Those pointing to free solutions are unable to hire sales people. There is no pharma company that can sell unpatentable already easily available substances, or only with minor improvements for packaging it more conveniently or in a cleaner form, the ones making billons can run lots of ads and sende out sales reps and give gifts and benefits to those making purchase decisions.
Web forum and word of mouth compete with large companies with dedicated and very sophisticated marketing and sales organizations for reach and ability to convince, and you can only make contracts with the latter.
The forces out there don't prevent you from "doing the right thing", they just slightly, or maybe not so slightly, favors those wo don't.
The chemical company that finds their product does not really help much and potentially has very severe environmental effects is much more likely to go out of business or to be bought than the ones lying about it and increasing the marketing budged and sales effort (see tobacco, oil companies and climate change research, etc.) .
"Let's do nothing instead of this" is highly disfavored by our system - and by our mindset. The people now found responsible for making and selling products and substances they already know are bad, if they were out of work, society would shun them and "paying them to do nothing" (even if it would be much better) is frowned upon by most of society. Everybody is forced to do something, anything. On the other hand, it's actually quite hard with all the basics long taken care of - for those who can pay. Or, the problems are so intricate it's hard to impossibel to solve in a private initiative, often even for billionaires, who too have to chose their battles. So we end up with lots of cheaters and people selling products, substances, services with little to no regard for consequences. They may be bad people, but our system creates a lot of pressure on everyone and people react.
Good luck as we all live in interesting times
> I pity the poor fool who will have to diagnose an operational issue
Why? They're getting paid, have a job, it's great
Pity the business owners instead maybe. But they're clueless and happy too?
Hmm. I start thinking these things are the natural course of events. Techies adding more techies and complexity, maybe just like middle managers can hire more managers and bureaucrats and make not the tech, but the organization, more complex (and thus better, from their own personal perspective?)
> Why? They're getting paid, have a job, it's great
Since they won't be able to diagnose the issue, except by pure luck, let alone fix the issue, they may be reprimanded and fired, even though they did nothing wrong. Not really great to tell their next employer that they were fired, or getting references.
A federal government department so yeah, that statement is accurate enough.
Unfortunately we all pay the bills at the end of the day.
All the cloud providers do that. You could say that peoples' judgment has been clouded...
More seriously, this is how they operate; complexity is a good thing to them, because it means more lock-in and ways to charge you in non-obvious ways, and the whole "consulting" aspect of the industry feeds off that.
In my situation, I probably will have to do that so there's a selfish reason there for sure.
I recently had a whole series of frustrating situations where I dug through rediscovering how old code / systems work to make small changes or to find out the small change was enormous. Really deflating stuff. It's not my fault but it can be so demoralizing. Feels like a weight on you... I was done for the day after both of those horror shows.
Then yesterday I had a 3 day project start and in 2 hours I ... did the thing. It was super flexible / powerful, handled errors gracefully, and easy to change / test. All because a year ago someone (well myself and anther person) took the time to simplify the original spaghetti code that originally existed and break it into more digestible functional-esque chunks. Dropping something "in between the chunks" (fancy technical terms here) was easy to do, test and read. Completely the opposite experience, it was energizing and fun.
As I age, my memory is getting worse and worse and I realize that quite clearly. Therefore, I always try to write documentation as I'm writing code, so that I can remember why I did something. It helps a lot so that 6 months later, I can do exactly that... but I also know that anyone else looking at my stuff will also realize why things are the way they are.
Sometimes I think I get some tasks done faster than when I was younger…
I've been saying for half a decade or longer:
"Going slower today means we can go faster tomorrow".
It took a long time for some of my team members to process this, but I believe they've all taken it to heart by now. The aggressive, rapid nature of a startup can make it very difficult to slow down enough to consider more boring, pedantic paths. Thinking carefully about this stuff can really suck today, but when its 3am on Saturday and production is down, it will all begin to make a lot of sense.
Having empathy for your future self/team is a superpower.
As an old C++ hacker, I'm waiting for the day when modern C++ shops read Accelerated C++ from Koenig and Moo circa two decades ago. Then, I could rejoice in someone anywhere writing C++ code that more closely resembled the python-esque C++ masquerading as pseudo-code in that book.
More sadly, I just keep seeing people emulate bit-twiddling from yesteryear when the compiler likely optimizes a fair bit of this.
The cyclomatic complexity scores in the paper look off by an order-of-magnitude but they may be better than the laugh riot I've measured in the last few years and my math may be failing me at runtime.
1. F1 paths are known in advance.
2. The major unknowns in F1 are your competitor behaviors.
Compare that to a typical startup: you're mostly riding in the dark on a track you see for the first time, and your major unknown is customer behavior.
So: the steps in the path are known in advance, but not the order, presence/absence, quantity, arrangement, etc. The major unknown is what you will do next. The best performers are highly reactive to, and involved with, the audience and colleagues (musicians).
(This ofc changes for choreographed performance)
As a full stack dev, I’ve got a stack of patterns (techniques) in my pockets to pull out for this or that situation, but I don’t really get to know which one will be the next one I’ll need. And I do my best work when I can get involved with the end users, interacting with them to grok their needs; and with my coworkers, so we’re a team.
Slow is smooth, smooth is fast.
I've crashed and burned startups while never comparing any of them to driving half-blind without my glasses at night.
Human sciences and user research provide excellent solutions to customer behavior. Like F1 cockpits, the risk scales with the domain.
F1 is not just a vector sport. If it were, math may be enough to win. Turns out F1 takes engineering, mechanics, and a driver.
However, viewed through a macroscope, F1 dynamics are closer to a cooperative game, as in software development.
While an unknown in software is competition, much larger unknowns are given by shifts in teams, machines, and their methods.
Turns out that comparing F1 and software development from 1970 to 2020 are remarkably the same story for much the same reason. Neither exist in stasis.
F1 and software development have more in common than is obvious from the grandstand.
It's hard to correct that sort of behavior (without being an actual manager that knows code and can spot bad architecture).
There’s also engineers that prefer doing these different kinds of work. They thrive on quick wins and kudos from founders. Early engineers probably need to be ok with bugs and edge cases that they’ll never go back and fix. Personally I don’t like doing work like that, but I’m definitely in the second class of engineers who needs systems to be modular, composable, and well defined.
"Going slower today means we can go faster tomorrow".
So I guess, relative to yesterday?
The idea is we identify the rough shape of what we are going to build and the components needed to deliver it (Linux? Terraform? K8S? HTML/CSS/JS? etc.).
Next we measure up what we can "take for granted" for the engineering skillset the organization hires for. Then we pick books, official project documentation, etc. that will act as our "reference." We spend our upfront time pouring ourselves into this documentation and come away with a general "philosophy" of an approach to the architecture.
Then we build the architecture, updating our philosophy with our learnings along the way.
At the end of the project, we commit the philosophy to paper. We deliver the system we built, the philosophy behind it, and the stack of references used to build the system.
What this means is I can take any engineer at the target level they hire for, hand them the deliverable and say "go spend a week reading these, you'll come back with sufficient expertise to own this system."
It also acts as documentation for myself for future contracts if I get brought back in. Prior to starting the contract I can go back in and review all of those deliverables myself to hit the ground running once I'm back on the project.
Do you have any starter resources to come up with principles for a system? Maybe something showing how certain principles lead you to implementation choices that would've been different under another philosophy.
* Small composable Terraform modules
* Don't manage IaC declarations alongside code in a polyrepo
* Direnv for managing env configs across repos
* Manage k8s using k8s manifests and not terraform files (kubectl provider gives us this)
* Delegate flux management to flux-cli
* Auto-unseal Vault to capture and protect the vault token
Then a list of recommended reading:
* Terraform Up and Running
* Building Microservices
* Site Reliability Engineering
This list is more tactical since we didn't build the system, we were auditing their current setup.
Can you elaborate on this one?
(And thanks for the interesting comments)
This, of course, does not apply to projects using a monorepo.
You still keep them separate but your monorepo tooling handles that separation. When multiple changes go out together, some being infra some being code, the tooling should be aware of those dependencies (just like any other) and handle resolving the infra first.
The muscle memory of devs in a monorepo tend to be different too. Folks are used to scoping their SCM changes to folders instead of working at the top level of the git repo (i.e. you usually don't find yourself doing a `git reset --hard HEAD~10` in a monorepo outside of a feature branch - other teams get grumpy when you blow away their changes on the mainline branch).
I make this distinction between polyrepos and monorepos for IaC because I've seen this advice result in folks splitting their monorepo into a birepo, or using IaC adoption as a driving reason to migrate their company to a polyrepo. There isn't anything wrong with the birepo approach, but it can be accomplished inside the monorepo all the same.
People I work with get very annoyed with me because of this, but I am obsessive about documentation for this reason. Sure, it requires a lot of tedious writing and screenshots, etc., but it has saved me countless times. I still can easily get back into things years later thanks to documentation.
The caveat is that when people who are not as passionate as you maintain a product and seemingly forget about documentation.
In the old days, documentation was a very strict requirement on many of projects I was involved with. Now, in modern agile projects, it’s an afterthought at best, despite having amazing documentation tools that we’ve never had before.
One thing I’ve done is to maintain a separate Git repo that only hosts documentation. This in combination with a simple UI that dynamically converts .md to HTML on-the-fly (or renders a cached version) seems to be a good compromise.
There might be sense in renting that machine from Herzner and sticking Ubuntu on it after all.
I think I can speak for many of us technical professionals when I say, been there, done that.
It's remarkable how "add feature X" can be a 1 hour task or a 1 month task depending on whether the system was designed to evolve and scale in that direction. But the non-technical management or customer just sees it as a simple request, and the (Nth) software developer is left to pick up the pieces.
An average software engineer will create an overly complex system.
If a skilled engineer can come in and create something that doubles team productivy, decreases bugs by 60% and improves retention across the org by 10x, that's huge! That's not a 10x engineer, that's a 100x engineer.
The biggest problem with a simple architecture and often why someone would complexify it up, is as a protection mechanism so you don't get some performance review motivated yokel doing a drive by to add in a feature and look like a rock star because it was so damn easy. NSS, that was the next step. You see this in open-source as well. Someone builds a beautiful foundation, and Steve rolls up with a submarine pull request to slam that cherry on top.
A complex architecture ensures that you can adequately gate keep.
The next aspect is you have to relentlessly say no to feature requests that literally destroy the system to add some new feature, the pressure to smash it to bits and make it a no longer simple system is such a perverse incentive that unless you have support from on high, every simple system will decay into a total mess, then the finger-painters will have moved on to some other system to predate on.
Anything specific that tends to be used often?
At 48 I've stopped fixing other people's things.
I almost said that I’m surprised they weren’t held accountable… but we all know better.
Why are folks taking the path of least resistance becomes an interesting question.
Similar to the expression "Up to scratch." In a boxing ring, there is a line in the ring that boxers must come to to begin the fight. To be "up to scratch" is to be ready and worthy of the fight, and from that, to be of acceptable quality or capability.
"Toe the line" most likely has it's roots in military tradition [1]. To toe the line is to stand at the line for inspection - it's still used that way today (I had to do it myself in boot camp).
Relevant, as in this context, it means being obedient to the hierarchy.
1 - https://en.wikipedia.org/wiki/Toe_the_line#:~:text=The%20mos....
Would you rather have a simple system that does nothing, or a complex / difficult-to-work with system that solves customer needs? It's easy to go too far in either direction.
And even if your framework is to use customer needs as justification for complexity, people will still use something like Kubernetes for an app with 10 monthly users, because it "helps us deliver features faster, have zero-downtime deployments and be scalable".
Most (non-product) engineers have a natural inclination to introduce more complexity, and will use mental gymnastics to justify it.
With respect, this is clearly a false dichotomy.
Creating the "simplest possible system" that solves customer needs is the entire point of this discussion. Avoiding as much complexity as possible brings enormous benefits. (Less bugs, less employee turnover, higher productivity, etc.)
Never accept the notion that systems must be "difficult-to-work with" in order to solve customer needs.
My point is that there are many, many developers who "relentlessly say no to feature requests that literally destroy the system to add some new feature" when the reality is that they do not want to put in the effort required to add the (meaningful) feature to the system.
They are choosing system purity and laziness over delivering customer value, which requires more work in order to keep things simple.
Just today, for example, I removed several thousand lines of code from a very complex form that no customer ever really used other than for testing. My boss told me "just nuke it". Life goes on.
As an engineer you just have to ensure you have equity in the small company before it's purchased and/or hope the big company buys your company rather than just copying it and fighting you in court for a decade.
You'll end up with a "I programmed a little bit back in the day" person who makes decisions based on how to currently keep their staff best utilized. Tools choices, boundary points, stacks, become chosen based on goals that are not rooted in design simplicity.
I've watched this story play out again and again and again at companies. Upper management will promote these types of individuals into lower/middle management because
a) they're deemed actually reassignable whereas a really talented Ux designer is obviously adding the most value doing that
b) people buried/invested in tech stack/design details are harder for owner/operators to relate to than individuals more like themselves.
Great delegation and engaging with community feedback productively are of course part of that strong management.
I agree that anything else is extra, but most of my difficult professional engineering decisions are tradeoffs between those first two things: I can build on an older system less well adapted to current needs and deliver more quickly, or build something that fits current needs better but takes a bit longer. I sadly don't have enough days in life where there's a better and quicker option!
Put differently, it's a lot faster to put another layer of duct tape on top of the last duct tape patch of a pipe leak. But every layer of duct tape you put on makes it more difficult to replace the pipe segment underneath. Developing judgement on when to do which is the (difficult) trick.
We deliver the duct taped one one time so the software folks can start their work, and then we build the non-duct taped one wheel it in a week later and quietly swap it out for the duct taped one.
I actually expected to find that it was a fake quote. Lately, I keep finding that many of the good quotes are actually fake. It’s like they actually survive on the value of the quote and, if we’re honest, whether someone decades or centuries ago actually said it is only tangential.
But thx for the correction!
You're speaking of ChatGPT!
I suspect this goes both ways. I have worked at a place characterized by frequent internal shuffles, and more than once spent ages reverse-engineering code when the original team could have handled the problem in a fraction of the time.
What is taught is not software-specific, but is entirely applicable to software, outside of the world of 'throw everything on the wall and see what sticks' as long as the venture capitalist can be shown growth at all costs. I wish more software developers were mindful of complexity and architecture.
What's the timeframe that teams should adopt? If it will pay for itself in three years, is that too long? What's a good argument to make here that will appeal to the bean counters that are used to thinking in terms of quarterlies? I personally am comfortable with "eventually/infinite" but that is a tough sell.
Capitalization vs Expense. Maintenance/refactoring is not capitalizable, and thus gets discouraged in businesses that care about P&L. New (capitalizable) projects/features are encouraged instead. What are some good ways to encourage maintenance/refactoring in this kind of environment?
Internally because the sense of accomplishment from seeing the new bit of functionality is tiny compared to the effort it took to get done. Organizationally it becomes harder to reward productivity because with no sense of how long something should take there's no way of knowing if an engineer is fast or not.
It's a productivity death spiral. Engineers slack off because there is no internal or external reward for doing good work. That slacking off slows development and then that velocity gets accepted as normal. Engineers continue to slack off against the new "normal" and establish an even slower normal to slack off against. That's how hours long tasks turn to days and then weeks and then months until the project dies.
What if you had the ability to build a system that was initially structured well enough such that it could be made to last indefinitely? From a cost/benefit standpoint, is the Ship of Theseus approach not the most ideal for a business owner?
Even for a developer, the notion that you have to constantly "trojan horse" your new product iterations into production via the existing paths means you will achieve mastery over clever workarounds and other temporary shims. Once you gain competence in this area, it is possible that you will never want for a new boat again. You start to get attached to the old one and all of the things it takes care of that you absolutely forgot about years ago.
If you could have that, it would obviously be amazing. Do you (or anyone) have that ability though? So far it seems the answer is no. People are just not smart enough to predict the future.
As a more nuanced answer, a system that is scalable enough to grow to 1000x the current load is usually way too expensive to build with the resources you have "right now". The best you can do is build a system for 10-100x the current load and hope you haven't forgotten any edge cases, but usually you encounter them way sooner than 10x the load. Building so that you can easily refactor your current system is the way to go, but even then you will sooner or later run into problems your original design did not consider.
It is better to have simple solution at start, then, after each breaking feature, whole thing needs to be refactored, because initial assumptions about that software might have been changed.
I think the key is that you accept that this is true, and that having a good architecture is a continuous process that never ends. You build the best architecture you can imagine, given the current level of knowledge. Then you have to be willing to refactor every single day after that, as you gain new knowledge. Too many people think that refactoring is only for special occasions, when things have gotten really bad. Every single PR can contain a small refactor. Small changes can accumulate and eventually lead to major architectural shifts.
In the end it mostly comes down to reducing complexity, but the goal is always allowing new features to be added as fast/easy as possible.
Because I'm lazy.
Quick throwaway code is orders of magnitude better (unless you're landing airplanes or building x-ray devices). Especially if you're consciously treating it as a throwaway.
The only way to build well structured system that lasts long is to have vast domain expertise, meaning having done literally the same thing multiple times in the past. This rarely happens in general, and pretty much never happens if you're innovating.
When something is "the same as some other piece of code with different variable names" it becomes it's own isolated solution, be it a library, framework, service, or software. We typically don't run our own SMTP servers, don't write our own databases and don't implement TCP/IP stack from scratch.
When we're not doing all those things, typically what's left is innovation: i.e. building something that doesn't have an existing generic solution.
quick hacking together of prototype systems delivers a dopamine high that is quickly reduced to zero when those systems need to be maintained and extended into an indefinite future
building systems that are well structured and not terrible requires knowledge and experience, but absolutely 100% does not take more time than building shitty hackathon prototypes
"There is no theoretical reason that anything is hard to change about software. If you pick any one aspect of software then you can make it easy to change, but we don’t know how to make everything easy to change. Making something easy to change makes the overall system a little more complex, and making everything easy to change makes the entire system very complex. Complexity is what makes software hard to change."
"Architecture controls complexity", and under-/poorly- designed architecture while superficially "simple" will give rise to un-intended complexity. Microservices are the du jour poster child here.
Anecdotally, the people I see advocating for spaghetti architecture are often utterly convinced they're solving some critical problem. Conversations with stakeholders then turn into fearmongering — often helped by the fact that most stakeholders don't care about architectural nuance — and it becomes easy to wave off any competing simpler architectures by deriding them as "taking on tech debt." In general, it's surprisingly easy to play corporate politics to bring a convoluted architecture into reality.
Good abstractions make simpler software. Leaky abstractions multiply the complexity of software by a lot.
Some respond to this by making "simple" software that dispenses entirely with abstraction. This ends up in a lack of architecture where complexity still multiplies, though perhaps less than the typical mix of mostly leaky abstractions and a few sound ones.
However, it's kind of the nihlism of software and throws away the opportunity for us to actually improve our craft... so I'm not all too interested in it.
In my experience the problem is that as with all things it's all about balance. We shouldn't throw away architecture entirely and write stupidly simple, quick solutions because they will be messy. But we also shouldn't over-abstract things so much that only the person/people who built the system can understand it and work with it. Both could have dire consequences for an organization, making building new features and delivering value to users slow, difficult and costly.
Once I finish reading this MIT paper, I want to dig further into exactly what makes software 'complex'. In my experience:
- too many layers of indirection - overly theoretical concepts not grounded in real world thinking - lack of documentation - bad naming - lack of testability - tight de-coupling - following 'best practices and patterns' without clear justification - trying to solve for problems before they exist
We should be building systems that are grounded in concepts that are easily understandable - which is exactly why Object Oriented Programming has been so successful. We write programming languages as a means of communicating with each other about program logic, why not do it in terms that we as users already understand in the real world?
I'm not against picking up some tech debt here or there if you pay it off.
A good architecture will allow you to make changes easily. A bad one doesn't. It's actually pretty simple, conceptually speaking.
If you believe that "late stage" companies make correct architecture choices you're probably incorrect. It's not about late stage or early stage, it's about knowing how to build software from scratch in a way that you don't hamstring yourself (and others) down the road.
The "McCabe cyclomatic complexity metric" doesn't predict the last two. Instead they use the "MacCormack" method (who supervised this PhD...), so I wonder how it differs?
funfact: they reference an evaluation of Mozilla'a refactoring (the one Joel said you must never do).
It's a PhD thesis, and it's long, detailed and academic (tautology), yet lacks an introduction, so:
The Missing Introduction
Instead of diagrams of arcs and nodes, they believe a matrix representation helps show complexity (vertical axis: using; horizontal axis used - or maybe vice versa - with dots at that location), called a "Design Structure Matrix" (DSM).
The key sections seem to be:
3.6-7 [p.34-42] about DSMs
5.1 [p.67-83] about complexity by the "MacCormack approach" [pasted below, from p. 70]
[5.1.2.2] Find all the indirect paths between files in the network by computing the graph's transitive closure
[5.1.2.3] Assign two visibility scores to each file that represent its reachability from other files or ability to reach other files in the network. [Visibility Fan In, Visibility Fan Out]
[5.1.3] Use these two visibility scores to classify each file as one of four types: peripheral, utility, control, or core.
They also have "network density" and "propagation cost" measures [p.76]
---
Aside: A PhD thesis takes some time to read before commenting. I wish there was a way to facilitate discussion of the submission, in addition to the title-based opinion and experience that we have now. The patient_hackernews experiment (https://old.reddit.com/r/patient_hackernews/) didn't seem to work out.
Perhaps just a re-submission (or a bump?) labelled "for readers only" 24 hours later?
1. Not considering how inevitable development issues and bugs will affect the entire system. That is, if a bug gets introduced here, how bad will the systemic effects be, and how do we prevent it? A severe case is if you start writing bad data from one component that then needs to be manually backfilled or reverted (especially if this then generates yet more bad data further down the line).
2. Not considering how entire-system failures will present themselves, and not having a good way to diagnose them.
3. Failure to develop testing (integration or whole-system blackbox) that catches the first case, failure to develop tooling (tracing, logging, synchronizing changes across components) that assists in the second case. Or instead adjusting the system so that the first or second cases are considered.
It’s easy to get stuck in a local optimum where a few old hats who understand the entire system are the only ones capable of predicting and diagnosing failures across components. It’s also easy to say that less skilled or new engineers just need to put in the time to get good, but it’s often the case that the old hats have a potentially-automatible procedure for narrowing down the problem or tracking where the issue came from, or that the benefits of separate components don’t justify the increased rate of bugs and time spent tracking them down to fix them.
I had a block of code that had been added to by others, making it a bit of a recursive mess with a handful of bugs that were hard to reproduce let alone fix. When I finally got tired of it all, I sat about to replacing the recursive code with some iteration, and ended up with DP code, which avoided a bunch of duplicate errors and the need for caching.
Without all the DP terminology, it's what other people would call building a (programmatic) plan and then executing it, rather than a depth first search algorithm. I used to use this quite regularly, but have only used it a couple of times on my current gig. Strictly speaking planning an entire action before starting it might result in a bit extra data hanging around because of building the data structures ahead of time, but at the end of the day it allowed me to eliminate a whole lot of steps that seemed necessary, and also remove duplicate warning messages. It wasn't necessarily faster than it could be, but it was way faster than I ever expected it to be.
Writing obvious code with clear actors and data flow steps makes it a lot easier to add features, and make performance improvements. Obscure code leads to more obscure code (qualitatively and quantitatively).
There are those hard-learned sutras that just make life so much easier down the line.
Down the line always comes (unless you are playing career musical chairs and are willing to gamble that you won't be around to have to deal with the code mess when there's a sev2-, but as they say.. karma's a b..."
Avoid premature optimization
TDD
Avoid complexity
Readable code
Documented code
Probably helps avoid most problems.
'Complexity is the mind-killer' - Linus Atreides
Changes in a stratified architecture are much simpler than in a layered design. (Changes in a layered design almost always mean changes to all the layers)
- The site of function calls to the site of the function's definition
- The site of class method calls to the site of that class method's definition
- The site of a class method definition to the site of the class definition
- The site of a subclass definition to the site of its parent class' definition
- The site at which a variable with a complex user-defined type is instantiated or accessed to the site where that type is defined. (User-defined types include structure, union, enum, and class.)
Then they compute the transitive closure of this graph.
Then they compute two metrics for each node by looking at the transitive closure graph (page 76, section 5.1.2.3):
- Visibility Fan In (VFI): how many other nodes have edges that go from the other node to this node?
- Visibility Fan Out (VFO): how many other nodes have edges that go from this node to the other node?
They observe that by looking at the VFI metric across various files, files tend to sharply cluster into either 'low VFI' or 'high VFI', and similarly for VFO (although some files may be high in one metric and low in the other) (page 79, section 5.1.3).
They then classify each file as:
- low VFI, low VFO: 'peripheral'
- high VFI, low VFO: 'utility'
- low VFI, high VFO: 'control'
- high VFI, high VFO: 'core'
They then find that 'core' files are the most costly, in terms of defect density, developer productivity, and probability of staff turnover.
You'll see these massive systems with event driven architectures and layers of nested microservices deployed to hand rolled K8S clusters with custom plugins and Cloud Native CI/CD systems strapped over the top of a complex repository setup and a complex metrics/tracing/logging stack to make sense of all the inter-dependencies and lifecycles of a request in this system. All of this for a system that is taking <100 RPS and likely won't see more than that for many years.
I'm not exaggerating when I say you could host the entire company on a single R620 (+ a cold backup) with PID1 managing a couple of bash scripts and a few Python/Go/Java/JavaScript processes wired up to an SQLite db and a nightly backup.
But their architecture choice does create jobs in our industry. These services need teams to maintain them. You have a K8S team, a DevTools team, a CI/CD team, teams for the API services, teams for the backend microservices, SREs to respond to incidents, etc. Those teams all need to be managed, coordinated, staffed, paid, etc. so you hire on the management and administrators to handle the size of the engineering organization responsible for this thing.
So instead of two 4-5 figure servers and a few $100 a month in hosting, plus a handful of well compensated engineers who keep this thing running, your engineering department is burning 6-7 figures per month in cloud costs and millions in headcount.
To be clear, some companies need these setups. I do platform engineering consulting and help companies build these exact org structures and systems I'm talking about here. But I only take those contracts when the work is justified. There is a spectrum, and very few companies fall on the end of the spectrum that demands these solutions.
Today, a majority of the companies in the American market should be on serverless offerings that scale to near-zero - simply to reduce the need to staff an engineering org to maintain the "infrastructure" under the system. You buy that engineering org from the serverless vendor with a support contract. I'm not setting a mom-and-pop shop up with an R620 because they'll be 100% dependent on a high-skill engineer for the rest of their existence, which isn't likely to yeild an ROI for them.
Past that you start migrating to VMs or bare metal. That'll get you pretty far. Modern computers are Very Fast, you can serve a lot of traffic from a single 1U slot.
Very few companies get to the scale where any of these FAANG architectures start to make sense.
If I were to take a guess, I think a lot of the FAANG stuff is salary chasing. To justify a $250k - $500k salary folks think they need to build these fancy architectures to demonstrate their value, and the junior engineers and management are all in because they're chasing the salary ladder and need to get this experience to get into FAANG. But the reality is, with back of napkin math, you can save a company _millions_ per year with conservative architectures.
For me at least, negotiating contract rates against that pool of savings is an unsolved problem.
Serverless winds up being complex, in my experience.
You can accomplish an incredible amount for a small business with Pages and a single monolithic function. Once you outgrow that, IMHO, it's time to consider moving to a server - not to a more "complete" serverless offering like lambda.
The migration path for this is pretty staightforward since the API surface area of a cloudflare worker is pretty small (depending on whether, and how, you are using Durable Objects, KV, D1, R2, etc. things can get a little harder). I can usually port a cloudflare worker to a Node.js service in < a day of work.
Other than some hairy legacy stuff here and there, it all just works and the run rate is exceptionally low.
When it occasionally breaks, it’s obvious what’s wrong and quick to fix.
We spend most of our time making it better.
Long live the 3-tier monolith.
The most practical measure of complexity is duplication. Are there two, or more, pieces of code accomplishing the same or a similar job? That is complex. The solution is to refactor the many parts into fewer parts.
characterized by a very complicated or involved arrangement of parts, units, etc.: complex machinery.
I would say you could have a software system with many interconnected parts, and/or with a complicated or involved arrangement of parts, without having duplicate code or duplicated functionality.
Further, sometimes 1 thing is overloaded in difficult-to-understand ways, and so there should be more things, not fewer. Sometimes there are many things that should be 1 thing.
There isn't just one good measurement of complexity, as complexity isn't inherent to things or systems in themselvesas, rather complexity is a feature of perception, which gets confused in all sorts of irreducible ways.