The collapse of complex software
nolanlawson.com
nolanlawson.com
What, you think "microservices" is a new invention? We called it distributed systems, and we really knew not to go there unless we wanted to decimate our productivity and sleep.
For example, I look at our software running on .net MVC and think as an Engineer, it simple and knowable but the Front-End Team are less worried about simplicity and more on flashy front-end stuff since that is their job. We end up bolting on a Front-end JS framework and complexity immediately ramps up by like 300%.
Are they wrong for wanting a better and more flexible front-end? Not necessarily, I mean all the other companies have cool stuff and if we don't maybe our company dies.
Ditto for lots of other examples...
I don't know why programmers think this way. We don't expect everyone to be a doctor, a botanist, a novelist, or a musician but for some reason we think anyone can be a programmer. That's just not the case -- programming is a skill like any other -- it takes some natural inclination, some training, and a lot of practice. Just like any other skill. A programming language is to a programmer what an instrument is to a musician.
Anyways, there's a reason programming should be made more widely accessible - it develops thinking skills and rationality when you learn it, as Mindstorms pointed out. Between the anecdotes in the book and things like [0], I think it's a travesty that we aren't pursuing this to its fullest extent, and instead try to teach children Python or JavaScript, two decent languages for software development but not exactly forgiving with beginners.
[0] https://medium.com/@stevekrouse/goodbye-seymour-cb712757264f
And like with these other things, you can make getting into them easier. With music, children start with simple instruments that no professional ever uses. And it's the same with programming, there are plenty of easier environments for children. But if you want to make programmers and you want to make musicians, eventually they have to use the real thing. My own son jumped straight into the deep end of Unity development knowing nothing because he wants to build something real. I neither encouraged or discouraged that environment and it's pretty unforgiving.
I don't think the solvable problems you speak of as are as solvable as you think they are. Also making software development out to be special both in terms of it's benefit to thinking skills and rationality and how it's merely some tools away from being professionally approachable to the masses is totally unfounded.
Let's say your software is in a financial company. Their software has to enable them to follow all the government financial regulations. Well, the government is following the larger societal "complexify to the point of collapse", and the financial regulations are certainly doing so. That means that the external behavior (the "business logic") of the software is insanely complex. You can't make that go away just by visual programming.
But maybe you're not in the financial world. Maybe you're just writing programs for internal corporate processes at some generic company. Well, your software is still subject to the complexity that builds up in the company processes. Again, the programmers can't eliminate that complexity.
Or maybe you're writing a customer-facing app - an external-facing web app, or an application that people actually install on their machines. Here you're at the mercy of the product or project manager trying to find new things for the app to do, and they still complexify the app to the point of collapse.
The problem isn't that programming is too complicated. The problem is that what we want programs to do is too complicated. Visual programming can't save us from that.
Just an example: around here, most people have a first (given) and a last (family) name. If I don't model that as separate, I have trouble interfacing with other software. If I do, I have trouble with people from other cultures that don't follow that convention. Storing both risks the data going out of sync. What's the "right" way to store person names? There doesn't seem to be a simple solution.
Another example: we model physical cables (both for power grid and for data transmission) in our CMDB. All works fine, until you suddenly have a Y-shaped cable with three connectors that doesn't fit into your data model. The real world always has these 1% of cases that don't fit the general pattern; if you focus on the 99%, the 1% make trouble. If you focus on modeling every case, you have 10x the complexity, even for the simple case.
And then there are things that are moderately complex and security critical, like password recovery workflows. We haven't really found a way to reuse these among different technologies. Like, if you once figured out the perfect password reset workflow with Ruby on Rails, and your next job uses Python + Django, you're back to square one.
If somebody has a good idea for how to tackle these problems, please let me know!
But where software does fail in my humble opinion is making it easy to pull in tried and true tested solutions to the problems that we do face, even if they are not as common. Because even though they may not seem common, I'm absolutely certain many face the same scenario.
The amount of duplication solving the same problems is insane. But this is not an easy problem to solve and I don't intend to trivialize it.
I think we need to come up with better tools. code sharing through libraries/repositories (like npm) is great, but it can't be the final solution.
Back in the day in Haskell I dreamed of a system where you could type out a type signature and a fully tested rated implementation would be imported from an "open source" service. You could import modules, functions, data structures, anything. But that vision is still a long ways off.
I think of that as optimizing the entire stack for what the customer wants of rapid/high-speed analytics.
The type signature of GPT-3 is `string -> string`
But if you look at like co-pilot for example, if that was given the ability to have type signatures serve as input you might get a lot more powerful results than what it does with raw text (which is very impressive).
But this comes down to type signature design. You can encode any function using simple types like int -> int which aren't very useful. Where Haskell shines is when using types to limit the scope of inputs & outputs. What I am getting at is that you can still write uninformative type signatures in Haskell, but it also gives you the power to write more informative ones.
I don't think Haskell is the answer, so please don't take that as what I am saying. I do think however using richer type systems could be a stepping stone towards a solution to this problem.
I feel like it wouldn't work if you could actually mathematically constrain the outputs to be syntactically correct either. That's probably one of those Gödel things.
Type signatures don’t tell you which should be the “then” clause vs the “else” clause in any conditional.
Don't try to force schemas onto schema-less data. Store the "name" as a JSON string/blob representing the various possible attributes (given, middle, family, title, etc) and provide a variety of functions for representing that data. IF you really need to do this at all (for an internal app, you probably don't).
> Physical links
Include an Hardware Asset FK in your Link M2M table. Model each binary link explicitly, so a Y cable = 2-3 different Links that point to the same cable Asset. Or you can have single Links with M2M inputs/outputs. But definitely don't model Y cables explicitly.
> Password recovery
What is so complicated about this, specifically? Python and Ruby are completely different languages with no guarantees for interop.
If you can't do that, you have more pressing issues.
Where are you getting "a dozen assumptions" from?
Our cables have stickers with cable IDs, which are stored in the CMDB (and can be used to trace the endpoints through patch fields, for example).
So if you model a Y cable as two physical cables, you have to both allow duplicate cable IDs AND multiple cables per port. Both of these have significant downsides in preventing data entry errors.
No. Model it as two abstract "Physical Links" or "Connections" which FK to the same physical cable Asset, which is uniquely determined by its inventory ID and/or manufacturer model+serial.
If a Link needs to know whether its part of a Y cable Asset, it can join the Asset table.
If a Y cable Asset needs to know what it's connected to, it can join the Link table.
Additionally, you have a nice clean Link table for doing graph traversal (get me everything connected to this impacted Asset by degree <= 2), and a nice clean Asset table for inventory tracking.
C and C++ libraries with light language-specific wrappers largely serve this purpose, in practice. It's plausible that, say, a PHP postgres client lib and a Node postgres client lib will share nearly all their code—probably as C or C++.
This doesn't get leveraged much aside from interfacing with daemons and sometimes for extremely complex e.g. media or crypto libraries, though. No-one's doing this for high-level software workflow building-blocks, like a password reset flow.
[EDIT] as for the "why", I suspect it's because the things it's used for are far simpler than the things it isn't. A password reset flow can potentially need to interface with lots of different things, some of which may be custom to the project, some of which may vary with run-time input, et c. If you cut it down to only the parts that could truly be re-used anywhere, with all kinds of points where you can hook in as needed... you've only made about 5% of the work re-usable, so it's just about pointless.
The Common Language Environment on VMS, or its lesser(but infinitely more available hence virtuous by effect) imitation, dot net CLR . FFIs, if you bought a good environment. Sounds like if open source implementation of good foreign interfaces existed, the OP's problem wouldn't exist, let alone system level interoperability.
Store both a unified name field and separated given name / surname fields and let the user manage both. Yeah, this risks going out of sync, but that's the user's problem, not yours. Yeah, three fields are technically more complex than one or two, but it produces less complexity down the line.
Falsehoods programmers believe about name: https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...
Then they leave that field empty.
https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...
As a practical matter, the HL7 FHIR data model for names can work well enough for the vast majority of applications in almost any country.
struct Screenname {
Matches<String,regex"\w+"> _;
// TODO: ensure it is illegal to ask for users' real names
// for now we'll just do the right thing unilaterally
}For how to model physical objects like those cables, go see how Grainger does it as a working practical model.
To really handle names properly, you need more context than the name in the presentation layer that many schemas take their modeling from can obtain. Government health care or similar widely-adopted encoding is sometimes Good Enough. If you want non-lossy exactitude however, then that's a much bigger scope (I'd be investigating a first-pass classifier with contextual hints taken from various geolocations, age, etc., that implies soliciting the name comes after what you normally solicit for input, and refining from there).
Password reset; this is why vendors like Okta exist to abstract authN away for us, and auth0 for authZ. Then there is the rabbit hole of what this abstraction leads to...
[1] https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...
The way past bureaucracies that didn't have the luxury of unlimited cheap complexity did: you set some rules and anyone who doesn't like them can either suck it up and follow them, or deal with the consequences themselves. Allowing users to specify arbitrarily complex requirements and never saying "no" is what gets us into this problem.
I try to work on the side in a spiritual successor of Foxpro/DBase (https://tablam.org).
I consider a mix of relational/array model fill a lot of ergonomics for this (and it was proved to be right by the family of DBase langs).
But what makes this much more complex today is the explosion on OS targets (Windows, Linux, MacOs, Android, iOS, Web), and the requirement to integrate with many other stuff, dealing with many formats (json, xml, ...), is harder to do UIs now and the base support is more inconsistent and mixed as ever...
So, is possible to make a simpler tool, I certain of it, but then the developer/user will say "ah, ok, so how this connect to Redis, GraphQL and Amazon Web Services, run this on Android, Windows, parse CSV, ..."
and that is what make this very hard at the end...
The fundamental problem of simplifying software is humans. Just consider date formats, time zones, and tax codes. Humans love to make complicated things.
My philosophy on this has been to take the complex human stuff and stick it in a black box. A professional feather in cap with this approach is called BladeRunner ( https://dl.acm.org/doi/10.1145/3477132.3483572 ) which radically simplified the distributed system aspect by putting all the gnarly glue and business logic in a V8 VM (JavaScript).
My next thing is a focus on board games where I have invented a programming language and have started to evolve a platform. It's called Adama ( https://www.adama-platform.com/ ), and I think it is pretty cool. The interesting thing is that the complexity of board games is exceptional.
I have a few clues to share. The first is that reactivity, which is found in excel, is a key to simplifying software as this makes the glue more automatic.
Another clue is figuring out bidirectional communication which relates to reactivity as a two-way street. However, this is primarily hard because we don't have great things off the shelf to deal with this beyond TCP. For more of a deep dive, check out https://www.adama-platform.com/2021/12/22/woe.html which talks about WebSocket.
My final clue is that you can't run away from state. So many people offload state because state is hard, and you have to contend with it. I'm building yet another database.
If you look from a person or user perspective - time zones or date formats are used in one location or in some context. People living in that context have it a lot easier because most of the time they don't care about other time zones.
I would say people simplify things but on local scale. If you want to go global that is your problem not humans that live in one place and use single time zone all their life.
1. becomes a post from which clever and ambitious people can build complex things. You invent docker to simplify application deployments and then somebody builds a n-dimensional microservice cloud on one side and starts commercializing new hardware architectures on the other. You don't remove the complexity, you just let it move around into new domains. (not a bad thing!)
2. is more temporal than you expect. The "finite and fundamental" problems of yesterday, today, and tomorrow are of different sets -- partly because (1) opens up new problem classes and partly because most problems are inescapably cultural and therefore subject to fashion cycles.
Not that I want to discourage you, but my view is that anything that makes software engineering simpler just leads us to tackling more complex problems until the complexity reaches the limit that people can handle.
So in that view, you can't succeed at making software engineering always simple. Instead, you can make previously intractable problems tractable.
Frameworks approximately do this, but for specific domains: they let you organize complexity into well-defined areas and put a pin in them, easing the cognitive load. Then you can handle those abstractions more easily. But yes, to your point, because frameworks typically only tackle a few levels of complexity, you still get that complexity back when you use the framework to advance the problems to the edge of what your framework is designed to address.
A recursive/fractal management of complexity will allow all levels of the hierarchy to feel similar, so you are never increasing complexity, only looking at a different resolution. I think the key to this is mapping out the fundamental organizational problems and how they relate to each other at different resolutions.
This is the 'induced demand' argument. As perfection seems impossible, and mistakes inevitable, this seems likely.
However, the more I look into issues, the more I realize that a very small number of errors introduced early on is what ultimately causes a plethora of them. Just fixing a very small amount of these mistakes would have untold effects on computing over the long-term. That is, if we can get over the initial switching costs.
> small team of engineers
It took 2 men to over-take thousands creating Unix over Multix. It took Linus Torvalds - just about alone - a week to create git. We have already seen this prophecy come true.
As teams get bigger, communication sales factorially, and the more people you have the more mistakes you make, which increases the size exponentially with each further mistake. What Unix and git showed is that when you put everything into a small team of engineers heads, they can work through the complexity enough until they can do it themselves.
> Frameworks
One of the things I realized a while ago is that pure/impure and library/framework have a decent mapping between the two. If you have a framework, you give it code and it acts for you, just like impure code. And so the problem we keep running into, is instead of inverting the flow like 'hexagonal architecture' says, we keep piling impure onto impure onto impure. Hexagonal says to not do anything of substance, keep the adapter clean, but each framework sure is doing something. Each layer on the stack we go up, the harder it is to get down. So now we have OS and applications and containers and k8s running micro-services that ends up being run in a web browser, when all we really needed was microkernals.
That sounds a bit drastic. Surely at worst the scaling is quadratic?
The ability to suss-out meaning behind those channels and come to a shared understanding pushes it to factorial. If A is speaking to B and C, A needs to also think about what communication is happening between B and C, and this is different to what B knows about C and C knows about B.
In a carefully balanced classroom you might manage to get everyone knowing the same things, and this will allow knowledge to scale. Think about how much better TV got once they knew that viewers would watch every episode. What happens in software is that specialization quickly comes into play such that this is impossible in any organization that doesn't use mob programming. Other people might as well be speaking a different language.
Next time you are in a Sprint meeting, think about how much you don't understand. Even as the team lead or architect who designs the entire system and whose job it is to understand everything it will be a shocking amount - it's black boxes all the way down. You'll claim that you can't know everything and that anyone who tries will fail.
And this is made worse as the only true way to learn something is to do it, and if you aren't actually challenging yourself on something, anything you learn without doing will just fade away - as Spaced Repetition shows.
The bigger the team, the bigger the software, the more inevitable the collapse.
Metcalfe's law is quadratic rather than exponential -- O(n^2), not O(2^n). And if n people all need to be aware of communication happening between every pair of them (which is a worst-case), then that should just add a factor of n, bringing it up to cubic.
(On an unrelated and less pedantically nitpicky note, one of the most valuable professional skills I've developed is an ability and willingness to dive into black boxes. It's remarkable how many bugs arise from the interaction of two or more pieces of software with no overlap between their developers. Debugging those can involve a long, agonizing back-and-forth between people who know how X works and people who know how Y works -- or it can be over in a jiffy if you can quickly familiarize yourself with the basic internal workings of both. Past a certain point nobody can know everything, it's true, but this doesn't need to be as crippling for productivity as a lot of people allow it to be.)
But that doesn't stop black-boxes from being created faster than I can fix them, unfortunately.
Multics did not have thousands of developers. Linus was just more practical about software development, was maybe better at building a community, and more importantly focused on x86 commodity HW which was a bet that only Microsoft also happened to make.
Thanks.
Actually, "combinatorial" or "exponential" would be better words to describe complexity explosion. The number of possible cases/scenarios/logic flows/etc grows exponentially as a system's complexity increases. It is a curse and a blessing at the same time keeping so many SWEs gainfully employed.
In a lot of human work (not just software, but any endeavor involving technology even as "simple" as building a tunnel or simply involving many people coordinating together), complexity emerges this "unknown unknowns" property. This has happened since time immemorial. If you don't believe me, go master how to live out a subsistence life in an agrarian area of $nation during the 1500's with their tools but modern knowledge, then try to convince yourself complexity did not exist back then.
All still manageable by humans but at continually higher and higher levels of complexity.
you seem to have thought about this a bit did you have any ideas of how things could be rearranged to a more recursive solution?
Similarly using an operating system/assembly is much easier than trying to control pulses of electricity yourself!
I think one thing missing from this line of thinking is that these are people. They have their own motivations and egos. They get sick, have families, take vacations, quit.
I think part of the way large teams and large companies are structured is a hedge against changing teams with highly variable capacity. It's hard to overstate how much it can hurt a project to lose a someone who has a ton of experience locked up inside their head.
The alternative is that I have worked for places that try to treat thier engineers like fungible tokens that can be shuffled and replaced at will. That environment feels extremely dehumanizing and demotivating.
There’s an awful lot of cultural baggage in coding. Many of the concepts that seem essential - powerful text editors, devils tooling - can be completely removed. It requires rethinking from first principles and being willing to upset the Apple cart, but I believe it is possible.
A huge part of the problem is the messiness of the real world and costs.
While there may be a finite set of fundamental problems, the set of possible hardware and software platforms is ever increasing and each has it's own constraints and strengths.
A "universal programming environment" would need something like a "universal hardware interface". That's why things like Java and the Web became so popular despite their poor design.
Also, visual programming is a hard thing to make work and human language skills seem to be far greater than human visual skills.
Perhaps the best that can be done is "starting from zero" and making sure everything, from the ICs in your hardware to the memory models behind your software is thoroughly tested and formally verified.
This is insanely expensive with current technology. Perhaps some fancy math and AI advancements can make formal verification powerful enough for ubiquitous use. Until then, I see little hope for a "universal programming environment".
> Nobody can subtract from the system; everyone just adds.
To tie it to something concrete, just about everyone thinks C++ template metaprogramming is unreasonably difficult to reason about even aside from the syntax, but it exists to be extremely powerful at precisely expressing behavioral contracts of a software component that are difficult to do in other languages. Even then, it barely scratches the surface of what would be useful to express for the interaction of those components to be "simple". The number of design parameters of a component that really matter in various contexts are astronomical. No one can deal with reasoning about that many component traits in practice, so the software engineers simply hide most of them -- the complexity is still there and will manifest in unexpected ways because it is not visible at the interface.
All systems engineering is complex for this reason. Any non-trivial software system has to be reasoned about as a monolith at some level to correctly address complexity, which creates an unavoidable cognitive load. This isn't a software engineering problem, it is a systems-thinking problem. In chemical engineering, for example, there are often complex system design problems that cannot be adequately addressed by decomposing them into a sequence of sub-problems, all that does is hide major complexity at the interface of the sub-problems.
Could you give real world examples of:
* "In reality, components have complex interactions far outside of what is captured in the component interfaces"
* How template metaprogramming "precisely express[es] behavioral contracts... that are difficult to do in other languages"
Traditional engineering disciplines that routinely have this problem of non-decomposability actually treat it as a systems problem (or a supercomputing problem), they don't shy away from it simply because the cognitive complexity is difficult. They convert the entire system into a set of simultaneous equations that needs to be solved for. In software, we would call this designing a monolith, but the reason it is done in traditional engineering disciplines is that you can't wish away the fundamental nature of the problem. In software, because it is not a physical engineering discipline, you can pretend that this issue doesn't exist, for a while at least.
Systems complexity is intrinsic, there is no trivial reduction. If there was engineers of all types would serve little purpose. Not coincidentally, the engineers that command the highest salaries are those that have the cognitive capacity to reason about the most complex systems dynamics, the dynamics that can't be decomposed into independent sub-problems before understanding the behavior of the entire system.
To your first point, in my experience, the best software, including complex platforms that handle massive amounts of traffic and data, can be built and maintained by small teams. I mean less than 20 developers. The larger the number of developers working on an application or platform has an almost inverse correlation to the speed in which new features are built or bugs fixed.
Software architecture has been done visually since perhaps its inception (tools like UML). At most places I've worked, every new project or large feature is diagrammed visually to use as a guide in breaking down the project into component parts.
In my experience complexity arises, not from lack of tools or industry knowledge, but from 3 main causes:
1. Inexperienced developer asked to create project, who just starts building without planning beforehand.
2. The main one - business demands features built that were never expected or planned for, and built as fast as possible. This causes developers to take shortcuts, make inelegant and difficult to maintain design choices. And leads to often inscrutable code that becomes technical debt especially after the original developer leaves the company. This will always happen as long as software is used to make a business money.
3. High team turnover - I've seen places where developers came and went so often that there was a myriad of things half started and never finished.
How I've solved or helped alleviate these issues: Make the business case to company owners or management that technical debt will be an ever increasing impediment to development velocity and the dev team will need a percentage of work in any given sprint to tackle tech debt issues (as opposed to having everybody work 100% on new features and bug fixes all the time).
I have successfully taken a platform that was bug-ridden, difficult to maintain, and where new feature development had slowed to a crawl due to the over-complexity of the software, to a place of stability and ease of development, simply by allowing our team to chip away at tech debt over the course of several years. Tech debt issues were rated by level of complexity, risk to the business in change (regression bugs), and impact on team velocity. We worked on the highest impact, lowest risk items first and kept going until there wasn't much left on that list.
I believe you are wrong, I'd wish you are right ;)
The other reason complexity happens is dependencies and DRY thinking, which is often good, but dependencies, centralization of code, and sharing of code are also a risk. Avoiding repetition is what causes abstractions to grow. Using other people’s libraries means that there are complex interactions you don’t understand. Most of the time it’s all fine, but occasionally sometimes it’s not. This trade is made consciously, for the reason that it’s much faster to develop your app using existing libraries, and not reinvent all wheels, and nearly everyone is doing it. The reason that tooling is unlikely to solve this part of the problem is that failures are so often caused by incorrect expectations - someone using a library hoping that it will do X when it only does Y. Even in theory, better tools could only tell us low-level information about what a library does, but I don’t think it’s possible for tools to tell you at a high level what a library can’t do.
BPMN and all of the UI pallet, drag drop coding applications try to do this very thing, this is super successful at smaller scales but breaks the first real world application.
He said that it used to possible, practically speaking, to treat software like discrete components so that you can reason about how to combine the units and predict the result, etc. By contrast, much programming today is bodging together a bunch of libraries whose behaviors are often poorly defined or otherwise opaque so that one often enough has to treat them like black boxes against which you have to apply scientific reasoning to understand how to use and integrate it. Hence Python and robots.
Presumably the new course teaches skills around managing this complexity, but the shift in model advocated -- from the metaphor of well-behaved discrete components in composition to opaque units whose inputs and outputs (possibly depending on hidden state) must be discovered was interesting to me, and ultimately adds to the accidental complexity of doing software now.
Do you think this sort of problem may have a distinct visual representation that would fundamentally reduce the complexity of library-dominated development? Or are you imagining a whole programming ecosystem built around your idea, so that the fundamental problem Sussmen described is somehow obviated? Or are you really trying to address essential, not accidental complexity?
We already have had, for a long time, a 'universal' model for a specific slice of information processing that consists of a very small set of components: Graphic User Interfaces.
One problem is the diversity of domain semantics and all the associated nuances. In GUI programming, this is the tedious bit of naming the components and mapping them to processing elements.
Another problem is engineering culture (du jour). A reductionist approach that attempts to leverage structural commonalities requires what is poo poo'd as "boiler plate" and all the associated warts of component oriented programming (including factory-factories). A revisionist shift in mindset is required.
“When great thinkers think about problems, they start to see patterns. They look at the problem of people sending each other word-processor files, and then they look at the problem of people sending each other spreadsheets, and they realize that there’s a general pattern: sending files. That’s one level of abstraction already. Then they go up one more level: people send files, but web browsers also “send” requests for web pages. And when you think about it, calling a method on an object is like sending a message to an object! It’s the same thing again! Those are all sending operations, so our clever thinker invents a new, higher, broader abstraction called messaging, but now it’s getting really vague and nobody really knows what they’re talking about any more. Blah.“
https://www.joelonsoftware.com/2001/04/21/dont-let-architect...
If one thinks of software programs as stored knowledge, then it makes sense why they are so buggy, error prone, etc. It's that the knowledge was never complete to begin with. Throw in the difficulties of computation, and you have the mess that is the software landscape.
But FP seems a bit stuck nowadays (to my untrained eye) with the breakthroughs from the greybeards of the last decade.
Do you have any progress thus far? Anything people can contribute to?
It's based on my 20+ years, 50 different teams and many hundreds, perhaps thousands of encounters with software developers.
It's almost never the domain complexity that is high.
Sending a rocket to Mars is complex. And perhaps complex every time.
Dealing with people that has to work together can only be simplified when agreements are made and agreements are kept. Otherwise you cannot trust and have faith in people.
It comes down to far and near organization leadership, mandate and empowerment in small teams with few communication links outwards, clear expectations from whomever is asking for something and process management.
Writing code is rarely hard if you know the expectations and the boundaries of the task.
Empowerment gives the team the possibility to act on what they discover and encounter.
Don't look into the future. Because you cannot. And you are both naive and arrogant if you believe you can. Deal with what is known now!
One thing is complexity in code. Another is complexity in business. Some things are not made for software to solve.
Less experienced people are generally guided by their beliefs around their own ideas. If the mind is locked onto something it's hard not to thing it's a good idea. Usually in software there is simpler approach. Look for it
I can go on because I help companies and teams for a living. But I will stop here
Yes! 100% this. We need less fortune tellers and more pragmatists.
NO!
This mentality is short sighted by definition and leads to stagnation. It may work within large financially healthy companies that aren't trying to rock to the boat.
Making assumptions about the future is incredibly important for us to move forward. But with these caveats:
1) Acknowledge that you are taking a risk and that you may be wrong.
2) Validate your assumptions as quickly and as often as you can.
3) Be prepared to change course whenever your assumptions are invalidated.
> Less experienced people are generally guided by their beliefs around their own ideas. If the mind is locked onto something it's hard not to thing it's a good idea. Usually in software there is simpler approach.
Less experienced people are frequently the driving force behind new ideas precisely because of their belief in it. As experienced people we should be encouraging them to carry forward responsibly instead of projecting our aged and jaded perspectives.
> I can go on because I help companies and teams for a living.
Honestly, that's a weird way to phrase what you do.
But I believe the parent comment was talking about software abstractions. i.e., don't build abstractions because you might need them later. Instead, build the software in the simplest possible way to solve the problems it needs to handle now.
This is because you'll likely be wrong about your assumptions about which abstractions you'll need, and ripping those abstractions out is going to be more expensive than just modifying simple code to do new things.
> don't build abstractions because you might need them later ... [snip] ... This is because you'll likely be wrong about your assumptions about which abstractions you'll need, and ripping those abstractions out is going to be more expensive than just modifying simple code to do new things.
At worst, this is dangerously wrong. At best, it creates a straw man that people can use as a tool to shut down innovative ideas.
What you're describing is a very narrow/specific strategy that one might apply at a company. An example would be when a company is doing well financially and you don't need to risk amassing technical debt for some experimental side projects. Just build the MVPs, see what sticks, and only then decide whether or not to commit serious architecture to the problem.
If you're designing new systems that need to scale quickly (just one example) then you NEED to place bets on abstractions. A well designed abstraction could save you weeks (even months!) of time when you most need it.
I've seen what happens when a successful product is comprised of stitched together MVPs. It's too late for a rewrite and all of that "simple code" is creating a death-by-a-thousand-cuts nightmare.
Bad abstractions will always be bad. Bad system design will always be bad.
It's an aged/jaded mentality to claim that we should just stop designing systems ahead of time because we suck at it. Not everyone does. Some people are actually very good at it. It's possible to design an abstraction defensively so that it isn't more expensive to rip it out later.
Want a concrete example? Payment systems. If I'm writing a billing system then you bet I'm going to abstract away the interface with the payment processor. I would do this irrespective of our plans to expand to other payment processors. It shouldn't even be a debate because there's a way to build that abstraction with minimal overhead to the team.
On the other hand I would never blindly introduce an ORM into a system that doesn't need it. That would be a bad abstraction.
Or it could cost you a lot if your guesses are wrong. It goes both ways.
It's better to write code the simplest way to do what it needs to do now, but in a way that can be pulled out into a new system/abstraction later in an easy way. This is generally done by writing data-oriented code (focus on what the data is and how it needs to be transformed) rather than object-oriented code.
It's not hard to write abstractions defensively. For example you can limit access to your abstraction to a small and strict interface. What you do behind that interface can be the wild west; as long as you can just lop it off later.
> It goes both ways.
It does, but the word "simple" is often used in a deceptive way. A cacophony of "simple" systems are actually very complex in aggregate.
There are many cases (eg: something is your core business) where you'd want to put a lot of thought into an abstraction ahead of time. You'd then want to validate that abstraction's effectiveness and adjust as you go.
Your ORM example is good and exactly what i am referring to. Too much abstraction/gold plating due to unknown requirements in the future is perhaps a better way to put what is a problem.
And you also say that "If you're designing new systems that need to scale quickly..." - if you know that it will scale quickly then of course you know you need to do certain implementations to cater that. But if you do not know whether it will scale quickly then you are looking into an unknown future - and then you should step back on abstractions.
There are innumerable successful examples of committing blindly toward a future few people believe in. These are often cases where the objective evidence of success is very limited. If the objective evidence existed then "everyone would be doing it".
This is the crux of my disagreement. Of course we'll both agree that applying the wrong/stupid abstraction is wrong. But that's not what I'm discussing.
I'm talking about this quote specifically:
> Don't look into the future. Because you cannot. And you are both naive and arrogant if you believe you can. Deal with what is known now!
Perhaps you need to reframe this point to make it more clear?
> Don't look into the future. Because you cannot.
This is silly for all the aforementioned reasons. Especially when innovating in tech!
> And you are both naive and arrogant if you believe you can.
Calling someone arrogant for attempting to predict the future in technology. Come on, man.
> Deal with what is known now!
What is known now is what everyone else knows now. Unless you're dealing with the _extremely_ rare case where you have access to information no one else does, innovation requires one to commit to the unknown.
What I am trying to get at is that I've seen many times where a complex abstraction was introduced, and then it was only ever used for a single implementation of something. I've also seen many times where an abstraction was introduced, to make plugging in other backends easy, but then when it came time to plugin the second backend, the API was sufficiently different that it couldn't be shoehorned into the abstraction without a ton of extra work.
Sometimes you know enough about the problem space up front to avoid those problems, but often you don't. I often the better approach is to keep things as small as possible until you need more.
E.g., for payment processors, just ensuring that the surface area between payment processing and the rest of the code is as small as possible is likely sufficient. Maybe that's done through a simple abstraction, maybe not. But the point is, if you only have one payment processor now, the cost of implementing the abstraction doesn't change between now and when you actually need it (unless, of course, you chose a bad design, which we're obviously trying to avoid with either approach). So you can pay the cost of adding the abstraction when you actually need it, and can spend your time on something else that you need now.
I guess maybe bringing up the word "abstraction" was the wrong way to word it. In my experience, it's usually better to keep the code as small as possible early on, regardless of whether or not its using abstractions, because making it do what you want later is much easier when there is less code (in terms of constructs/concepts, not necessarily lines/characters). Also, try to keep things as orthogonal as possible. I think the reason I mentioned abstractions is that complex abstractions trying to predict all of the possible future extensions to the system is where I most often see design go wrong.
> Don't look into the future. Because you cannot. And you are both naive and arrogant if you believe you can. Deal with what is known now!
See my additional response to him elsewhere in the thread where I tried to break it down sentence by sentence.
Everything you're saying is on point for good system design IMO. What he was saying was totally different and IMO is a "dangerous" way of thinking when innovation is important.
A map is not as easy of a concept in some ways (it's more abstract, you are taking N elements of X to N elements of Y). But a map is simpler than a for loop, because there is no state between steps: it's completely symmetrical.
The for-loop "complects" each iteration together, even if it's as little as incrementing an integer. It's a side effect, however trivial.
To kick it up a notch, now imagine a 2-D loop. Now we have 2 counters. If it's a 2-D array, each addressing operation interacts with 2 counters. But map just needs a second map chained on, and each map is completely "unaware" of the other map. You could map over N dimensions and each map is symmetrical, while the for loop needs an ever increasing number of counters. This slight difference in complexity shows how complexity compounds with other complexity.
doStuff()
But "doStuff()" may be a very complicated function, that's very hard to understand, so it's not simple.
A real-world example might be something like the type conversion that JavaScript and PHP do in comparisons; "just do 0 == '0' and it'll work, easy right?", but turns out those conversion rules aren't simple at all.
I like how he states that simpler software doesn't necessarily mean fewer components (roughly quoting). Where the article argues for fewer boxes and arrows. It's hard to argue specifics, because the article isn't getting into them, but sometimes more boxes and arrows makes things simpler. If you don't have multiple boxes, what do you have? One box of spaghetti?
I also disagree with the article that engineers like complexity. I've maybe known a couple, but usually they are really junior, or really unpopular.
Among other things, it suggests that there are some incentives for complexity: That engineers just enjoy complexity, and that making simple designs might not be good for the respect of your peers/bosses/career: "A simple design might make it seem like you’re not really doing your job."
While that's true, I think the essay isn't clear enough about the fact that simplicity is just plain hard. Complexity is sort of the "natural" outcome of "adding things". It takes time, effort, and skill to create software that can do what is asked for it while maintaining an internal simplicity. Even if everyone involved really wants it. I do not think software complexity comes only, or mainlly, from people not wanting it enough (explicitly or implicity), from incentives to complexity. It's just plain hard. It's a mistake to think just wanting it harder (or having the right incentives) is sufficient.
It may be the key challenge in software engineering -- how do we make software that does everything that's asked for it (powerful, complex things), while maintaining an internal simplicity? You can not accomplish it by cargo-culting any "design patterns" or other approaches. It's still a skilled craft. That comes from years of experience with the intent of developing this skill (years of experience alone aren't enough), looking at other people's code who are also trying to do it, and just some aptitude. And domain knowledge, if you understand the nature of the problems you are working on better, you can design simple solutions for them better--contrary to some current practices treating developers as interchangeable.
Yes, you need your organization to allow you to do it (including taking the more time it will take), but that's necessary but not sufficient for the outcome.
But that means simplicity is a cross-cutting concern, like security, and does not particularly fit into any environment that would attempt to separate design and implementation. It needs “redraw all the module boundaries” to be a plausible (if improbable) response to “this one 300-line part of this one module is awkward and three times longer than it really needs to be”. Aside from the obvious administrative problems, suggests that the complete edifice needs to fit into the mind of a couple of people at most or the iteration simply will not converge most of the time.
I don’t see a good solution to this.
[1] https://quoteinvestigator.com/2012/04/28/shorter-letter/
[2] http://www.ultratechnology.com/forth.htm
[3] There was an example about changing a negative-feedback control loop, written into the spec by hardware engineers emulating analog circuitry, into a couple of fixed-point operations that behaved differently but did the actual job just as well; I can’t find the reference right now but this a minor domain-specific change among the kind I have in mind.
> I don’t see a good solution to this.
I think this is probably true.
Some people take that to basically give up, and say, okay, our software is always going to be a mess, but look at all it's done as a mess, so be it.
Otherwise... the commonly understood solution seems to be composing the thing of independent units (very very very decoupled) each of which can fit into the mind of a couple of people at most. And then the whole, built of these independent units considered as black boxes, can perhaps also fit into the mind of a couple of people at most. I know you said "the complete edifice", which is not the same thing, but this is what we've got: and indeed is "abstraction", basically the entire basis of computer science and what makes possible anything we do. (Even the simplest program of 20 years ago -- can probably only fit into the mind of one or two people if you exclude the hardware all the way down, which you can because it is well abstracted and decoupled; it wasn't always, 50 years ago).
And we can come up with all the challenges and barriers of that -- of course! If it were easy, we'd have simple software. :)
But that's the task, I guess. But a lot of software isn't even built realizing that's the task -- to have one or two people "driving the bus" who have a mental model of the thing they're building, at any level of abstraction (I feel like Fred Brooks wrote about this), meaning that continuity of experience matters and domain knowledge matters, treating developers as commodities to be shifted around continually has a cost to complexity too, as does continually switching technologies and platforms so you can't build up a solid repertoire of encapsulated abstracted pieces to compose.
The same goes for societies. If people have more autonomy (like, not have external pressure to be productive), they might be more willing to contribute to the society as a whole, to fix it, and also take a more long-term view of the future. IME, in established open source software, there is less churn and accidental complexity.
Complexity is also often the result of in-fighting, which again, is the symptom of a society where the trust is low.
Though I am not sure, honestly, if there is a reliable way to increase the trust inside society rather than the actual collapse. That paradoxically often leads to increased cooperation.
"Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?" -- Brian Kernighan
- One is that new systems tend to be rewarded, because people and organizations tend to associate "new" with "better". Related, people and organizations tend to associate high complexity systems with "hard work", and the transactional view of work tends to err on the side of rewarding "hard work".
- Next is that complexity digs a moat of job security. I don't think this is a bad thing, job security is good. Again, unfortunately the transactional view of work can easily devolve into "we can pay someone less to do what you do" and building a complex system is a way to counter that.
- Lastly, a story. At some point I worked on a team that quadrupled in size to tackle huge project, only to then fall into a sort of "identity crisis" because the project was done and there wasn't enough meaningful work afterwards for everyone.
This insane growth, in my opinion, was also fueled by egos. The overarching team manager seemed pretty adamant in finding more people to hire, but I don't necessarily remember us feeling understaffed. They clearly expressed they wanted to use all the headcount the company had authorized, and the primary reason they always mentioned was that we'd "lose it if we didn't use it by the end of the year".
This led to an "induced demand"-like situation. By the end of it, we had a huge team, the bigger meta-project was done, and everyone needed to find the next thing to work on and build. I saw plenty of proposals for solutions looking for a problem, and worked on some myself too before leaving.
I don't think it's that easy. Building simple software is very hard, and often takes several iterations to ensure you even understand the problem. Most developers are under time pressure, and stop as soon as there's a working solution, even if it's more complex than it should be.
That's not necessarily all ego talking. If you have a task to accomplish and you're going to need to spend a week looking at it just to get a handle on how to start getting it done the simplest overall way possible, your boss is just going to insist you do it the "quick and dirty" way "this one time" (every time). As Dilbert says, "our boss can't judge the quality of our work, but he knows when it's late". This is an inviolate law of software development: quick always beats good. Get on board, or you'll be replaced by somebody who is.
Easy to implement for me --> but not so good for overall structural simplicity --> worse for everyone in the long run, kind of thing?
Look at software that has been around for decades, continues to improve, never "needs" a re-write, and will never be re-written. It exists and is not uncommon. Things like the linux kernel, all the GNU utils, emacs, vim, apache, nginx, postgres. The list goes on. We know how to write software that lasts for decades, continuously improves, and doesn't need constant re-writes. You give it to skilled people who care about it, and don't put artificial corporate constraints or impose fad architectures on them.
I'm not saying the corporate model is "wrong" - it works. "Works" means it builds companies and generates profit. That is the goal. But it will not generate stable long lived well designed software. That's okay though because that isn't the goal. Sure, the poor quality cuts into the margins a bit, but not enough to matter.
As a software developer I get paid far above most workers and my job is easier than most other jobs. And yet I could still switch jobs frequently to squeeze out even more money.
At least half of the items on that list are still here purely on momentum and reputation from the time there were no alternatives, not because they were really that marvelous.
I use and even enjoy some of them, but they're far from great engineering.
I love Emacs, but it's a slow, crufty mess.
Vim is a huge pile of code with a lot of complexity in its codebase and hilariously bad design decisions like VimScript.
Apache is a mess from a security perspective (1.7k listed CVEs[1]), and still slower than Nginx.
Want to guess at the amount of lost productivity due to Apache vulnerabilities?
[1] https://www.cvedetails.com/vulnerability-list/vendor_id-45/A...
I think it's time to pour VC money on a fully verified stack.
Perhaps not everything needs to be proven, integrity for some components can be ensured more informally with e.g. refinement types.
Lots of governments and organizations need more robust software.
While it's true that VCs spend lots of money, you do actually need some plausible-ish excuse for how you could make money on it at some point in the future. It's not like VCs are out here funding public parks or charity hospitals. They fund for-profit businesses.
People did sell operating systems in the 1990s, but I'm not sure if that's a strong market in the 2020s. How are you going to 10x a VC investment in basic software tools?
My point is that it can be a very profitable niche, not that VCs fund it pro bono.
I'm not claiming that good engineering doesn't exist or that there aren't amazing FOSS tools, I just take issue with that ideas that either (a) all FOSS tools are great or (b) the FOSS model is somehow able to produce quality tools in a way that the proprietary model is not (or substitute "bazaar" and "cathedral" models if you wish).
The vast majority of software sucks, and I'm not sure that FOSS software even sucks less on average.
Individual parts of Linux go through relatively frequent rewrites (eg filesystems), but not the overall framework.
Also, what happens is that rewrites happen progressively, in small steps.
I mean not really. It does a tiny fraction of what Apache does, it just happens to do that specific fraction better.
The Linux kernel for instance gets big changes. The BKL took a very long time to fully remove.
The GNU utils needing a rewrite would be weird since most of them implement a pre-existing standard, so the need for a huge rewrite is very unlikely.
I thought it was caused by every little piece of software being tied to giant product teams full of MBA majors that want to add 50 features a week
The historical record disagrees. See for example RMS squashing better clang+emacs integration: https://lists.gnu.org/archive/html/emacs-devel/2014-01/msg01...
Even if you have plenty of skilled engineers, it doesn't mean that the business won't impose unrealistic timelines (and insist the timelines be met even if it means taking on loads of tech debt which never gets paid down) or change the requirements abruptly or even demand things that are inherently complex (without understanding the complexity).
In other words, there are plenty of ways that management can create complexity, and in my experience management-induced complexity is the most common kind (note that I'm faulting "management" for things that include "communication between engineers and management" because management ultimately owns the software development process).
1. an engineer under management pressure couldn't easily take an expedient dependency on the private internals of another system
2. teams could deploy their changes on their own cadence, so they didn't feel additional pressure to squeeze poorly-conceived things into one less-frequent release (many frequent releases are theoretically possible for monoliths, but much rarer for pretty understandable reasons)
Moreover, reliability improves because of the aforementioned reduced complexity but also because the blast radius for a reliability issue is scoped to the service, and security improves for the same "blast radius" reason (an attacker who compromises a component can only access that component's secrets and can only talk to that component's collaborators).
My suspicion is that microservices work very well when an organization has a strong architecture/SRE capability irrespective of management or product development capabilities, whereas a monolith depends on all capabilities firing on all cylinders.
The net complexity may well be (and often is IME) less for a microservice than for an otherwise similar monolith. In addition to this net complexity discussion, microservices also have security, scalability, and reliability benefits.
I agree. If both team size and code base are small, the scalability is not needed and the domain logic isn't too complicated, microservices add an overhead with very limited benefits.
If you have different teams for each service - great. If you the same person is working on multiple services - probably a premature move to a more complicated architecture.
They don't get re-written because they can't be re-written without tearing down everything built on top. In every case though there is some newer software out there is inspired by but aiming to "fix" all the short comings of the predecessor.
If anything, Linux isn't afraid to rewrite the parts that need to be rewritten, and that's one of its strengths.
Apache, Vim, and Postgres have been "rewritten." Nginx is an httpd written from scratch that sort of has the same features as Apache. Neovim exists. And newer database technologies that care about performance have been written.
So if you diff 2.6.8..5.14 course you'll get a massive diff. But they got there by a long series of relatively small changes, with regular releases in between.
The tech debt amassed between the rewrites and then rewriting not to learn from past mistakes but to repeat them, is the issue.
The teams that were great to work with and ended up having the biggest wins, were always small teams with a great cultural balance. They always strived for the simplest solutions.
You could trace this balance of engineers in all of the cases back to GREAT hiring by the CEO.
The teams that were the least productive and just burnt through VC money were the ones that had a handful of toxic engineers at their core. Finding complex and seemingly clever (over engineered) ‘solutions’ seemed to be their goal in every meeting.
A way to find out if your team belongs to the second group, measure the amount of jargon that’s being used in meetings.
[1] www.fairpixels.pro
Also talks about the risk of losing knowledge, and a way to avoid that (make multiple copies).
In my experience this is the only safe thing to do when you don't fully understand the system and cannot reason about it in a precise way. You create well defined abstractions over the messy parts, verify the abstractions, and write lots of tests. Eventually you can start deleting code. Although nobody ever ends up deleting code because they skip the "well defined" part and end up adding more indirection, afraid to delete code.
The only real way to tame complexity is to reason about it with precise, mathematically-defined abstractions. You can't invent Paxos using boxes and arrows on a white-board and prove the minimum quorum required without showing your work. Fortunately we have languages and systems that can automate a lot of the effort in checking our work as long as we can express it in formal language.
The other problem with this is that learning how to use these tools and techniques requires a lot of effort that isn't highly valued or rewarded in industry. Most businesses are fine with the trade off of errors and mistakes down the line if they're first to market. Ship it now, fix it later. We do a reasonably good job of preventing enough obvious errors with our tooling and processes. We can have software that obviously has no errors or software that has no obvious errors. Time and again we chose the latter because the former is too difficult and doesn't reward us with money now.
Well, the other option (the most sane one, in fact) would be to not mess with it until you understand it and can reason about it - but in every corporate "how long is that going to take"/"what can we do to speed this up"/"why aren't you finished yet" quick-and-dirty-ocracy, that's not an option unless you can do that in about a day.
...and "yes", I read second ~half of the article. My point here is that the author's analogy works at least as well where he did not bother to apply it as where he did.
Where does this yutz work? Because he doesn't speak for me. Generally, I write a complex program because I don't have time to write a simpler one; and when I have to deal with that complexity later I'm like arrrggh. Rube Goldberg machines are fun to watch, but no one wants to maintain, change, or build on one.
Is that really true? If it's descriptive, then I'm not an engineer (and I may well not be). I don't like complexity at all, in any form. I want the least complex solution to any problem that will work. However, I've mostly given up on decomplexifying, because it's mostly a waste of time. Software becomes complex because getting something done fast, right now, "this time", because, "we're in a hurry this sprint" by duplicating code will always win over trying to modify something to be reusable. If you try to spend time fighting complexity, you won't just be fighting the complexity inherent in the existing system, you'll be fighting your coworkers who've given up on fighting complexity and just get their tickets into a done state.
People get invested in software, their code, their favorite dev community... they get comfortable... they become biased and influenced as they learn (not pragmatic)... they enter cycles of doom largely centered around conways law & organizational complexity. Scaling any organization is a people/business/cultural problem.
E.g. look at React... how tf did it develop into the hook mess. Early versions we so amazing and simple. A react app/oss component that is 2y old now is garbage. In contrast, look at how Angular has developed, or look at Golang, or Kubernetes (yes really) as a modern examples. People are struggling with AI today because python sucks at scale - but they suck at engineering real solutions because they aren't culturally adaptive. Im no AI/data scientist, but our models running in production on Golang, parallelized across K8s, and the data processing our "yesterday's devops engineers" are now doing is working out just fine.
"Software Engineering" was a mistake. Industry never adapted from the physical personality of the word... building bridges and building software are not the same thing.
Edit: grammar/phrasing fix, end of P2.
So, it'd be funny to see, but I don't think it's happening anytime soon.
made me literally LOL. That turn of phrase is gold.
> One thing working in complexity’s favor, though, is that engineers like complexity. Admit it: as much as we complain about other people’s complexity, we love our own. We love sitting around and dreaming up new architectural diagrams that can comfortably sit inside our own heads
It's harder to acquire a sense of mastery & possession, of real understanding, when there's been multiple iterations of teams hacking on a piece of software, when it's many layered & has lost cohesion & concerted intentionality.
Trying to describe how relishable, how enjoyable it can be to explore, to search & quest for meaning, to push designs that hopefully make sense, that hopefully grow, that hopefully wrangle: it's an under-sung & hard to tell story. It's interesting to me having such a big part of the world running off of a knowledge-working that is so under-described, so hard to communicate personally about.
> Most software at Google gets rewritten every few years.
> This may seem incredibly costly. Indeed, it does consume a large fraction of Google’s resources. However, it also has some crucial benefits that are key to Google’s agility and long-term success. In a period of a few years, it is typical for the requirements for a product to change significantly, as the software environment and other technology around it change, and as changes in technology or in the marketplace affect user needs, desires, and expectations. Software that is a few years old was designed around an older set of requirements and is typically not designed in a way that is optimal for current requirements. Furthermore, it has typically accumulated a lot of complexity. Rewriting code cuts away all the unnecessary accumulated complexity that was addressing requirements which are no longer so important. In addition, rewriting code is a way of transferring knowledge and a sense of ownership to newer team members. This sense of ownership is crucial for productivity: engineers naturally put more effort into developing features and fixing problems in code that they feel is “theirs”. Frequent rewrites also encourage mobility of engineers between different projects which helps to encourage cross-pollination of ideas. Frequent rewrites also help to ensure that code is written using modern technology and methodology.
Fix the tools, and you start to fix the code. Unnecessary terminology, confusing APIs, superfluous patterns...it's like watching someone blindfolded in a dark room trying to justify the high price tag on their computer science degree.
After years of running my mouth, I proved my rants about the unnecessary complexity by building my own JS framework [1] and I'll never look back. The emperor is bare-ass naked.
So different, but not less complex overall. But maybe there is value in having some of that complexity consistent across a small number of cloud providers.
No free lunches here.
It's true that a lot of organizations had operational teams that have become money pits, or pushed back on all quality of life improvements because they don't have the talent, the budget, or the imagination to pull it off. When we move to the cloud we start aspiring to these things we didn't have because they were expensive. I don't know of anyone who moved to the cloud and didn't move the goalposts. We were just talking in another thread, as we often do here, about how much YAGNI is going on out there. Yes, the price per feature goes down, but the overall price doesn't seem to. And I get stuck taking care of things someone else used to worry about, which is opportunity cost on top of it all.
What is also true is that developers can learn a lot from their operational peers and avoid expensive mistakes. With cloud we have none of those peers. We have to learn everything first hand. By someone who is all too happy to let us wrap rope around our necks and then 'rescue' us from themselves. That's a perverse incentive and quite a setup for a fairly fucked up codependent relationship. At least with interdepartmental drama some of the money stays in the company longer before going to vendors.
In pre-cloud Internet times, you'd have an untold number of extremely brittle bash scripts, cron jobs, rsync ssh key setup, fleets of build + test boxes to manually worry about disk space, pre-provisioned dev/QA database servers with also untold brittle sql startup/teardown scripts, and all of the requisite people whose job it was to solely maintain this infrastructure along with database tuning, build fleet monitoring, the list of menial tasks just goes on and on and on.
Today, you have a yaml file in your .github/workflows directory.
Now I agree that there are "different" requirements. Understanding the complexity of your workflows etc is no small feat-- but you're replacing such a huge amount of what used to be extremely expensive and brittle architecture with, basically, a text file or two. That's a huge cost savings.
In the build and test example, the answer to "how much compute is running?" is based off developer velocity and so "it depends" is a fair answer. When nobody is shipping any builds, your cost should be $0. (I've found that this is kind of hard for enterprise-y finance departments to wrap their heads around and is why all those esoteric billing notifications AWS services even exist)
Pre-cloud services days, finance departments had a much easier time. You had racks of physical boxes that had static costs attached to them, you had a static monthly bandwidth bill that let you run at a certain speed, and you had salary costs which are also pretty static month-to-month. The idea of "scale to 0" was completely unheard of. What do you mean your QA environment doesn't cost anything on the weekends when nobody's doing anything? etc etc etc, you get my point.
In pre-cloud times, you would spend untold piles of money spinning up racks of storage arrays, switches, firewalls, leasing out gigantic pipes for bandwidth, and again all of the requisite people in order to get that running. You'd rent space in multiple distributed global datacenters, so again you'd have an unreal amount of bash-script-file-sync services so that when someone uploads a new image it gets replicated all over the globe. Millions and millions of dollars. You'd probably have to write a custom resizing service with ImageMagick, hooked into your frontend, so that you were serving customers the correct size and not blowing through your bandwidth allocation. Just incredibly complex.
Today, you click a button in your CDN provider's console; most of that functionality from above just comes for free.
Again you have to do a little bit of munging your front-end to take into consideration the vagaries of your CDN provider, but overall it's such a huge savings of mental energy and time. Put a circa-1999 systems architect in front of the Cloudflare console from today and they wouldn't believe it was real.
> It takes a lot of discipline to resist complexity
That can be taken in many ways.
Self-discipline, to resist convenience, the omniscient and omnipotent desire to control everything and always have the right answer immediately. That takes discipline to let go of. Also to discipline others, to say no, push back, become the master and set firm boundaries against crushing 'demands' for complexity. And intellectual discipline, to see clearly, elegantly, eliminating what is unnecessary. Over-complexity is a failure of the ordering principle of an organism, a step on the road to chaos and entropy.
It’s not a panacea, but it really does help slow the growth of your complexity.
Once we've got a sufficiently complex pile that actually has promise, everybody duplicates the pile. And then puts an additional layer on top. These many pile+layer combos then co-evolve (and co-erode ;)
At some point, something so useful has evolved that we start copying that pile... and put new layers on top. Rinse. Repeat.
The complexity rarely collapses because those new layers can add support beams, if necessary. Worst case, a layer gets ablated, and we try again from one step below. We don't ever go back to the beginning.
(This also explains, to me, why Enterprise software is what it is - the complexity piles can rarely be shared or reused, and so there's no co-evolution, little chance to settle on the best pile)
And, to be fair, it's similar in other disciplines, but maybe a bit slower. Mechanical engineering has individual components, larger components built from those, and so on. It just is more expensive to copy the piles^W components we know to work.
Total collapse is exceedingly rare.
(That even holds for human societies - they, too, often "just" collapse down to simpler societal expressions. And we do co-evolve larger and larger forms of society. It's just that the erode-and-try-again process is uniquely painful for the humans involved)
No. Speak for yourself. I am happiest when my solution to a problem is simplest. I am least happy when a solution requires introducing new infrastructure or major new ideas.
I have no idea who first said it, but one of my favorite sayings is: "Any idiot can build a bridge that stands, but it takes an engineer to build a bridge that barely stands."
I strongly believe that this applies to computers as well. I love it when a system neatly solves its requirements in a way that looks effortless. When an algorithm looks like it just barely works for all the potential edge cases without any special handling because those edge cases just miraculously happen to work out as they fall through the happy path just like any other case. That's when a solution "feels" right.
If I can present my solution/code/explanation to a junior engineer, a C.S. student, or even a non-technical person and they get it, then that's when I feel like I've really done my job. It's music to my ears if I can make a problem look so easy that someone asks "Is that it all it takes?" (until they've tried to solve it on their own :-)
I couldn’t disagree more. The vast majority of companies I worked with recognized and rewarded simpler designs, the vast majority of software engineers I worked with would rather delete code than write code, rather throw away a complex design to a simpler one, and with some NIH here and there, YAGNI and KISS seems to be triumphant and what would get you promoted, not adding more arrows and boxes.
What made up fantasy world do you live in, and are they accepting immigrants?
So complexity has a diminishing return for economical reasons.
And when it comes to keeping complexity low, there is zero theory to back up anything, so it's not taught.
Our trade is somewhat similar to reading/writing, and if you were to teach someone to read/write, you would probably insist on good writing, not just "finish your novel at this date, I won't read it anyway". And you would have good material for teaching to read, material that is currently missing.
Ideally you rotate people between the two teams. And you adjust the number of people when the ones ahead create too much to clean up.
The problem with complexity is that it doesn’t come out of nowhere, it mostly comes from changing requirements. So it inevitably is always cheaper to hack a system to do something it was almost meant to do than to figure out where in the original design things need to change for the new stuff to be supported.
These
https://en.wikipedia.org/wiki/Law_of_Demeter https://en.wikipedia.org/wiki/Principle_of_least_privilege
We just need to do this again and again and again.
For example, we all know there should be / and everything should be done via file descriptors by now. Great, but who is making it happen?!?
No, absolutely not for me.
I want simplicity.
The problem is that our expectations for software systems is so vastly more complex than it ever was in the past. It always grows more complex. The needs and wishes only increase.
Yes there are some developers who like building complex structures. Admittedly it can be fun. But eventually we reach a point where it's too much for one person to keep up with. Then the challenge shifts from complex software systems to complex knowledge and management systems.
It is theoretically possible to have an infinitely complex system which is managed by a perfect, infinitely complex organization. In practice it doesn't work, for numerous reasons.
One reason is that business drives software development, and business is seeking profit. So business doesn't want documentation or complexity management; it wants features. That means a lot of important information gets stored in a few heads of architects and senior devs. And inevitably, people leave organizations after a while.
A benefit of being a broad generalist is that I can do a lot of things in a lot of languages on a lot of platforms with a lot of frameworks. But I can't remember details anymore. So I have to make things pretty obvious. Functional programming patterns help a lot, because I can usually isolate things into well-named functions which do what they say and nothing more. Then with composition, systems of moderate complexity can be fairly well understood and navigated by readers.
The rest of the post doesn't offer any solutions other than to say, "oh well, that's how it goes". It doesn't have to be a hands-up situation. We as developers must try to balance the "demands" of sprints, product managers, and executives to push stuff out. We should repeatedly explain how technical debt will slow us down in reaching future goals, and how we should refactor and rethink our code regularly. It can be done, and I've successfully changed a few companies. It takes persistence and passion, but eventually you have a whole team onboard with an approach to managing complexity. It is very much worth the struggle.
It seems like folks who've built some form of common bus (shared data types and communications channels) do better scaling applications. If I can effectively split off a part of the application and plug a new part in its place, I can scale. OOP seems to love dependencies that accumulate until objects cannot be replaced because of all the connections. Clean separation is a hard problem.
Computers are logical machines. Human laws and regulation are not. Modeling physics is an easier problem (programming not computing need wise) than calculating taxes. At one point in the US, every gallon of bio-diesel had to have its own serial number[1], this adds a level of complexity that goes beyond simple programs.
My own thoughts (or my longtime hobby project) is that Agent oriented programming hasn't really been given a workout, and that decoupling data structures might lead to some improvements. I honestly think being able to replace parts of a running program in an orderly manner might be a indicator of a good direction.
1) https://mnbiofuels.org/media-mba/blog/item/1495-rins-101 - as a programming exercise, write a program to add, subtract gallons from a storage unit - it gets fun.
Honestly, I hate complex solutions. Even my own. What I like, and what impresses me, is the simple solution that takes a lot of work or math to prove it works. Like, emergent behavior type shit.
Control theory, for example, runs systems with very simple runtime behavior. A few open loops with some parameters. The analysis to chose those parameters is where the engineering happens.
Microservice practices are not yet that simple, but some day I hope they can be.
The problem with software is that software deprecation cannot be readily modeled in Excel. We’re back to the problem of quantifying Engineering Debt.
The best I can come up with number of WTF moments per unit of cyclomatic complexity. The second best is aggregate cortisol level of developers divided by the business value produced during a release.
We can only keep going back to the drawing board so much before the cost of rebuilding the system becomes prohibitive.
What do I see instead happening?
I think tools like Github Copilot which assist user's writing code will become ever more sophisticated and broad in their application. That they will not only perform simple bug fixes or write blocks of repetitive code, but even automatically build integrations with third party systems with just an API specification, refactor code to make it cleaner, less repetitive and more decoupled and be able to break apart complex monolithic code bases into smaller libraries or services.
Of course building such a system is a complex undertaking in its own right, but much like how a compiler eventually eaches a point where it can compile its own binary so too such a system might reach a point where it can extend and improve itself taking over from its human creators.
That last part is the key: with the strong preference for short term gains in an economic system where finances are considered and reconsidered quarterly, sustainability is an afterthought. "Growth at all costs" aligns with the dominant ideology of imperialism, and it's just as prevalent in corporations and organizations as it is in nation states. Literal millennia of evidence for that approach out-competing others is hard to beat, and I think if we knew how to survive a different way even at the micro level we'd probably live in a much better society.
So nice, working application after adding next 5,000 more lines here and there suddenly becomes too fragile to perform its main function.
Usual solution for such problems is composition/modularization. Software shall be made of simple but composable modules.
Like text editor (relatively simple) shall be composable with syntax analyzers to make IDE possible. The very same syntax analyzers should be usable as web server modules for the Web.
That dream requires design of (again) simple but powerful architectures of composition/modularization.
Seems like each generation of software philosophers approaching this task anew. Linux with its stdin/stdout modularization. MS Windows with OLE. COM and CORBA, and so on.
Probably it's time to evolve something like COM and CORBA into something new?
We have an old app: dotnet webforms, it's clearly hanging on for its life and needs replacing but the mythical rewrite is impossible if you intend to rebuild and switch, it just involves too much time away from your core product.
Instead, we had to setup a proxy which could send all traffic between one of the two apps so we can move things a few at a time to the new system and all new work just gets built in the new system. It still takes a long time so I am now wondering whether as soon as we finish, we should immediately setup another new app alongside the current new one and repeat indefinitely.
It certainly wasn't easy to do because old and new frameworks are rarely able to share cookie formats/sessions etc so some hand-cranking was required.
Even internet itself has a under threat. That is what we should observe.
Otherwise all is fine. Sone cones some goes, systems moved on. Even mainframe based system moved on.
Not you, perhaps, but some do: https://www.lazarus-ide.org/
Just because you can discover a simpler proof for something doesn't mean you can discover a simpler, less complex proof to anything else. And didn't Godel prove that some things can't be proven at all. That would mean some programs can't be written.
Still, mathematicians don't give up, they do their best. And so do programmers.
The same goes for development...
It's not impossible to rethink things in a more simple manner, it's just that we've stopped listening to the right size shoe for each foot that needs one because of shiny marketing. Technology doesn't become obsolete these days, it just becomes out-marketed and under-supported.
1) Cars are transitioning to electric propulsion, not electronic. Electronics have long been present in engine management of internal combustion engines too.
2) This is actually not new, but just a resurgence of something very old -- at the turn of the century (19th/20th, not 20th/21st), electric propulsion was quite common[1], the land speed record was held by an electric car[2], and the first "series-hybrid" system[3] was introduced[4].
3) Steam engines -- discounting steam turbines -- are also just combustion-driven piston engines; the only difference is that the combustion is external, not internal. OK, so it's not the combustion gases themselves that are expanding within the chamber, but steam heated (created) by them. But that's a minor detail, IMO.[5]
What does this mean for software engineering (or "engineering", in scare quotes)? Not sure... Probably something like:
A) Old tech doesn't go away. It just falls out of use and gets forgotten.
B) "New tech" often isn't. It's just old tech being uncovered or independently rediscovered.
C) What we think is a huge innovation is often just a return to older tech. ("The Cloud" ~= the mainframe?)
A + B + C = "COBOL, here we come!"? :-)
Idunno.
.
EDIT -- oops, footnotes:
___
[1]: https://en.wikipedia.org/wiki/Electric_car#Early_development...
[2]: https://en.wikipedia.org/wiki/La_Jamais_Contente
[3]: Where an internal combustion engine drives a generator that drives the wheels via electric motors and charges the main battery. https://en.wikipedia.org/wiki/Hybrid_vehicle_drivetrain#Seri...
[4]: https://en.wikipedia.org/wiki/Lohner%E2%80%93Porsche
[5]: In fact, AFAICS a four-stroke (preferably diesel?) internal combustion engine could fairly easily be converted into another kind of hybrid, an "external combustion - internal evaporisation" engine. I've been thinking about whether the (alleged?) greater efficiency of continuous combustion in comparison to intermittent would make such a conversion worthwhile. (Just one more of the many half-baked inventions bopping around in my head. :-)
Discipline and complexity are best friends. Laziness and procrastinating seem more valuable for avoiding complexity.
https://www.theatlantic.com/technology/archive/2017/09/savin...
Once I was on a research ship, and the data acquisition system crashed on the 256th day of the year. Care to guess the issue? :)
[0] <https://twitter.com/internetofshit/status/986006653605687296...>
“It seems that perfection is attained not when there is nothing more to add, but when there is nothing more to remove.” Antoine de Saint Exupéry
If we refactor a very large function into smaller sub functions we are essentially adding new boxes... but are we creating complexity, or simplicity?
Maybe we could design better software and better teams using patterns from biology instead of the romans of old.
There's also the matter of different types of architecture at different levels of scale. For cellular biology the foundational properties are statistical mechanics and physical chemistry. What are the foundational properties of computation?
Here is the choice, stagnate and disappear through incapacity or adapt and perish in burnout.
> ...Opinions expressed in this blog are mine *and frequently wrong*.
Emphasis added.
Equating the complexities of software with the complexities of societies (the premise of this post) is a fun and provactive blog post. Which is all this is. I was really hoping for a serious treatment that meaningfully dug into this, but that's my problem.
But not because some esoteric systemic property of complex systems, but just because working projects get deprived of maintainers and eventually there's nobody that understands WTF original programmers intended and how to fix the holes.
So incomprehensible portions of the system are circumvented or replaced with something that just works. Actually the same happens in societies, unless some group sabotages the workaround.
https://sudonull.com/post/111636-IBM-System-360-A-Failing-St...
I think we've seen this with the rise and fall of hardware vendors. Most recently with the Mobile OS ecosystem (symbian, windows mobile)
The causal factors behind why societies fail and why software projects fail are completely different. Complexity is a description of the system and a very bad measure to draw an equivalence between the two.
The risk with drawing this equivalence is it grants license to software engineers to think that similar enough models apply to "managing software" as "managing societies". Because when all you have is a hammer... But society is not a nail.
They claim to be building for the next hundreds of years, but you wouldn’t know it.