What factors explain the nature of software?

New clients were requiring audits and attestations we wouldn’t have been able to pass.

I was able to hire a few seasoned engineers to begin working on the next pass over the monolithic core design. We needed to prepare for a container and cloud based future, given where our clients were going.

The founder just complained and complained. He couldn’t accept that we were working on things the client couldn’t see. I spent a lot of time negotiating for a percentage of total velocity for core design changes.. the rest of our time and effort would be focused on things the customers were demanding and which the founder thought we should focus on.

We had increased velocity about 2000% already. He wanted 4000%. We needed sprints for back end redeployments. Founder was visibly angry when he overheard us taking about refactoring back end things like deployment scripts and data architecture.

Senior engineer gave him a copy of The Cathedral and the Bazaar, just to share something that could lead to some kind of common vocabulary. Founder spent weekend marking up his copy and on Monday firmly announced that he rejected the text in whole.

Senior engineer quit on the spot and walked out.

In any high-level systems architecture, whether software or a chemical plant, you have a collection of notionally discrete and independent components that interface with each other. The properties of these components may be constrained in various ways e.g. the alloy type that can be used in a reactor vessel or the RAM available for a data structure. In a chemical system, that reactor vessel is modeled as a set of complex differential equations that govern its properties given many design inputs. In software we have much less sophisticated models of the architectural components; while there is a high-level concept of space- and time-complexity, those models are usually not coupled to a model of the hardware it is running on or temporal dynamics such that you can reliably predict absolute performance across multiple dimensions that are within a few percentage points of ground truth. With the current state of software engineering tools, there hasn’t been much immediate motivation to make the software component models much better than they currently are.

These models are the axioms of your system. In other engineering disciplines, there are sophisticated solvers that will take a large number of these components connected together arbitrarily with their many inputs and solve for the system with predictable and nearly optimal properties every time when you actually implement that design. You don’t have to guess as to the consequences and side-effects of changing requirements, scope, or implementation constraints, you can just re-run the solvers. The specification for what you want it to optimize for is pretty simple — cost, speed of building, throughput, etc. In software, we do almost none of this in a structured, disciplined way, not even manually. We know little about the properties of the design until after we’ve invested inordinate amounts of time building it first.

Ironically, it is much easier to build detailed and accurate component models in software than chemical engineering. We’ve somehow never gotten around to building systems design solvers in the same way they exist in other physical engineering disciplines. The behavior of the system is not the behavior of its isolated components when decoupled from the hardware environment, but in software we design most things as if this was the case.

Despite coming from a chemical engineering background, I didn’t fully recognize the existence of this gap until I started working in HPC. Now I see it everywhere in software. When I worked in HPC, I started hacking together crude models and solvers to address difficult optimization problems that took an excruciating amount of time to iterate on, a primitive version of what you might do as a chemical engineer, and it was a game changer. But it was an enormous amount of effort because there is no tooling to do this in software, even if it occurred to you that it might be a good idea for the same reason it is a good idea in every other engineering discipline.

I think we got away with it for a long time because early software systems were legitimately quite simple. We could live with it. Software is now vastly more complex. While our tooling has evolved to address problems of abstractions, it has not evolved to address the problem of systems behavior and modeling. The ability to see the effects of system design changes without actually implementing them is powerful. Other engineering disciplines recognized this need quite early; one of the earliest applications of computing was solvers for physical engineering systems problems.

The reason is software tasks are often unique. The most important tasks are most often the unique ones. Therefore they are hard to estimate, and that makes your carefully crafted schedule useless very quickly after a project starts.

Agile, mostly in the form of Scrum is supposed to be the answer, but in many cases Scrum people keep banging their heads against the task estimation brick wall just like their Gantt/CPM chart predecessors.

That's why task uniqueness and the inevitable, intractable, why-even-try unreliability of estimates is an important part of the nature of software.

> The only way to know exactly what software we want to build is to fully specify it: without doing so, there will be gaps between our vague ideas and harsh reality. However, a complete, abstract specification is, in general, at least as much work as creating the software itself — in many cases, it is substantially more work.

Well, there is a way to do this: it's called requirements. There's a whole subfield of Software Engineering called Requirements Engineering that deals with exactly this. The fact that we don't stop to write down even a one page description with bullet points of what software should do is damning as a field.

We'd instead rather just jump right in and then later spend hours, days, weeks, months, even years later refactoring the wrong thing we started with, or banging on it with a hammer every time a new (often anticipatable) requirement seems to appear.

Sure, software development is a young field, but it will never mature if we constantly ignore and even forget the basics.

Absolutely, but the author's explanation of why is both incomplete and unnecessary convoluted.

People pursue that which they can reason about, such as bike shedding[1]. Other answers are discarded without consideration. This is called bias. Perhaps the common bias failure that people employ about software are unfounded assertions. Developers make unfounded assertions typically in the quest for least effort without consideration of total cost of ownership[2]. Software managers frequently make the same errors for the purpose of reducing accounting expenses as opposed to personal time.

Senior executives and military leaders solve for this by surrounding themselves with advisors. Advisors occupy a managerial position of domain specific knowledge that eagerly seek to both keep their boss informed and simultaneously disqualify their boss's bad decisions. This works to eliminate bias so that the boss's job is then limited to balancing competing guidance from their advisors against their own managerial experience in consideration of attaining the stated goal.

[1] https://en.wikipedia.org/wiki/Law_of_triviality

[2] https://www.investopedia.com/terms/t/totalcostofownership.as...

Liminal is key here. I believe this arises from the fact that we frequently have the luxury of simplicity in software enabled by the fact that the imaginary part is that things like generic pointers allow us to pretend as if a physics of software is non-existent, at least until computational complexity catches up and re-asserts physics dominance over our control theory from the stoics.

What's missing is extensibility. In particular, infinite extensibility. All hail Alan Kay.

The relationship between those two has puzzled me for a while, at least since I started writing on this topic post-Covid.

The last things I wrote were along the lines of "bringing Lacan to software" - getting at the idea that there was a Lacanian frame there which needed further exploration. Reminds me of reading the history of cybernetics which IIRC shares the greek root with Kubernetes.

Lev Manovich should get some credit for applying McLuhan's idea of hybridization to software more specifically than McLuhan's target: technology beyond media.

https://mastersofmedia.hum.uva.nl/blog/2014/09/10/the-softwa...

We build houses and conduct surgeries, and planning helps the doing, but doing also helps the planning. One difference is those things are not freely replicable and there are physical costs to building copies or building exponentially improved versions of physical things. So we tend to build roughly the same thing over and over. That iterative feedback loop has been running for decades or centuries, informing the process to the point we can now estimate pretty accurately what it will cost to build a house and estimate how long it will take.

But software is freely copyable. One person can build something and share it with everyone. If it’s widely useful, it gets made into an app, library, framework, or language. We are never spending decades in the plan/build loop honing the process of building the same app over and over. We are, in a sense, perpetually on the forefront building something that hasn’t been built in exactly this way before, and the only way to build the new thing is to plan a little, and build a little, and iterate on it. And it often goes about as well as the first time someone tried to do surgery.

This is the ethos of Agile Development.

Get something in front of users as early and often as possible, because this will change their idea of what the software should be and better to find that out earlier than late in the process when the software is more difficult to change.