Software Is Drowning the World
jamesabley.com
jamesabley.com
Things in the physical world break down beacuse they are made of matter, and the entire concept of "working" is a very simplified modeling of their behaviour that doesn't include second-order effects like tear,consumption of materials etc.
Code, on the other hand, is a social constrution but entirely confined in an abstract and mathematical world. The _only_ reason we still don't have a reliable and solid code infrastructure (in the sense explained by the article) is our inability as a community to once and for all define a solid foundation for the principles of computing. Hence all the hacked together languages, tools and frameworks that people have piled on each other over the decades.
If one century of efforts hasn't been enough, fine. But we absolutely can work towards that goal.
Brian Cantrill had a good observation in a 2007 Google TechTalk introducing Dtrace:
> In software the blueprints are the machine. In software once you design the thing you've built it: the design is the machine. That's why the waterfall model is so fundamentally flawed. This idea that you could design the design before you design it, which is what the waterfall model is essentially saying, is flawed. Software is both, it's both information and machine. (@3m40s)
* https://www.youtube.com/watch?v=6chLw2aodYQ
I originally ran across it on Google Video:
* http://video.google.com/videoplay?docid=-8002801113289007228
Hardware standards shift because of an Apple PR camopaign. We stopped using zeppelins only after one disaster of several dozens made very public by live radio coverage. Non-Sufficiently standardized rockets explode because countries can't pick a measurement system. The auto industry in a country changes overnight because a dictator decides that the roadside to travel on shall be switched on a whim.
Just like software, infrastructure/hardware/whatever is an unfathomably complex web made from science, faith, half-knowledge, geopolitics, coincidences, rules, dick moves, physics and random tweets.
I don't think we are winning anything by trying to lift software onto some kind of pedestal of specialness in every regard...
Is this possible? If code is a social construction as you say, then wouldn't it be impossible to arrive at this point unless society remains unchanging?
> Hence all the hacked together languages, tools and frameworks that people have piled on each other over the decades.
This hints at the what is IMO the real problem, software is hard to replace with confidence. Sure, it's easy to swap one version for another, but doing so with the full understanding of what will change...is basically impossible for any non-toy project today. And there are exceptions to this, but they're the "exception that proves the rule" type.
Social constructs are not immutable any more than physical ones. They're more likely to change, from what I've seen.
Making software is about groups of people interacting, side effects of the software, and lossy interfaces more than it is about mathematics.
I can't remember the link but you can probably find something. the way they code is very careful and deliberate. everyone has an office, none of this "rockstar" nonsense that you have at startups, they are all serious people, the deadlines are sensible, no one pulls all nighters, they come in at 9 and get out at 5. absolutely nothing gets written without documenting it. no line of gets changed without the reason for the change written in documentation. everything is carefully panned. needless to say the code got the ship to the moon and back. this is how coding should be done
Just look at the Windows Internals books to understand the complexity of just the systems in the operating system. It isn't something that needs to be "fixed" just because something is complex.
Everything is more complex than we think it is, and that's not really a problem.
Cars are not a good example to point to. The thing that would make cars a good example would be if cars were shown to exhibit a steady increase in essential complexity. In that case, yes, we might point to them and say, "Maybe the situation for software is the same, and we're mistaken to have any unreasonable expectations to the contrary?" But that doesn't match reality. Car manufacturers are as guilty as software developers of producing things that have gotten steadily complex without good engineering reasons to justify the complexification.
(Pre-emptive request: please do not respond to this comment as if I'm arguing that cars have not become safer or more fuel efficient.)
As for the mechanical components of the car - the modern Bug is a diesel (non-PD) so has a common rail fuel system. I can read fault codes with the OBD2 mechanism but not everything is accessible.
Both tremendous fun but the modern one is 100 times more complex and I would struggle to maintain it single-handedly. But I don't feel the need to reduce the modern car or "fix" the issue of its complexity simply because it is more complex.
My wife has a Mini that has developed a fault but who knows what sensor has failed according to the OBD2 codes?! I don't feel the need to "fix" that and reduce it to the old 1991 Mini Cooper I had (which had its own set of baffling problems, like in-line fuses for important things like the injector but a fuse box for non-essential things like the interior light..... but that's another story).
Similarly, "Functions as a Service" makes many problems the hosting provider's problem. All of the low level goop is theirs. You still have OP's problems but the responsibility surface is reduced.
It's more like "you now have to deal with different problems" much more than it is "you have to deal with less problems". Maybe the problem quantity got reduced from 1000+ to 50+ but the latter might swallow much more time to solve so it's really questionable which of both is better.
https://www.strongtowns.org/the-growth-ponzi-scheme
But I think the initial-to-maintenance ratio of software is both different than for physical infrastructure, but also perhaps more importantly, very different than many decision-makers expect.
Hyperbolic titles are indeed a problem of our age though, I agree on that completely. But your prejudice towards programmers who are supposedly thinking their problems are novel and unexpected is not justified.
That sounds a bit broad. In the general aviation world, engine designs from the 60's are still in widespread use.
As is leaded fuel, to the delight off everyone living within a few miles of airports. Old can be good, until it's not anymore. There's often survivorship bias involved as the worst--or unadaptable--designs have naturally fallen out of use or been supplanted.
You can use designs from the 60s today and the result will be a brand new system that in time will rot away and need to be replaced by, possibly the same design. That doesn't invalidate what he is talking about.
Even the pyramids of Giza will eventually rot away given time.
The difference in software, in my opinion, is speed in which change occurs.
Take Scala for example. There is now mature tooling for keeping your dependencies / libraries up-to-date automatically, using automatic migration-scripts (must be provided by the library author of course).
See here: https://github.com/scala-steward-org/scala-steward/blob/mast...
The difficult part here is of course to write the migrations. This works very well in Scala (and can work as well in certain other languages) because the type-system provides enough information to do automatic rewrites and it is easy to _not_ use unsound techniques such as reflection, code generation, macros etc.
The reason why we don't see such tooling in other languages yet is that they are either dynamically typed, which makes it almost impossible to write migration scripts that pretty much always work. Or they are statically typed but the typesystem is so limited that developers have to fall back on mentioned unsound features.
...Or at least I hope that will be the case in the future. But like you, I believe it's much better positioned to do so compared to the dynamically-typed languages.
I am clipping this for use in future internal discussions. I have trouble communicating this point, this sentence does so concisely.
We've got to stop thinking about software in agile sprints once a product exists. I'm not saying don't use agile for the build. But we've got to learn as software professionals to think like professions that maintain other expensive assets, such as aircraft, ships, or public works. There's a recognition, that a 50 year dam, needs a tear down and replacement in 50 years.
Notice that this includes using code written by other people (e.g., dependencies).
But using off-the-shelf software that you simply install is certainly one form of "using code written by other people." If it is popular software offered by a stable company, I would always estimate it's likely future cost as less than anything written in-house. (Which doesn't mean it's risk free, the software can become unsupported or change in backwards incompat ways incurring costs. You can get unlucky. But I'm almost always going to advise that it's the safer choice with regard to future costs than in-house development).
If we're talking about in-house software projects to begin with, as you probably intended, but evaluating using a third-party dependency vs writing the functionality in-house, figuring out which of those paths will be likely to end up lower-cost is not at all easy to do, and dependent on a lot of context.
But certainly in all cases, not having any software will be lower-maintenance-cost than having any software. Well, at least lower software maintenance cost. :)
Pick technologies that are stable over time frames measured in years, not months. You will never be able to construct any great works on top of fields of quicksand.
For example - Instead of using a fun & shiny new language you read about on HN to construct your company's new loan processing backend, perhaps consider technologies such as Java or .NET. Instead of using a database engine that was developed this decade, look at something that has already stood the test of 1-2 prior decades.
Now, 100% of everything I have just said is totally irrelevant if we are talking about software engineering just for the sake of it (fun, hobby, exploration, etc). This is the important distinction. If the livelihoods of your coworkers depend on you not making a bad technology choice, try to go with the safe bet. Separation of hobby & work mindset can solve a lot of this mess.
We should, however, have these kinds of discussion because I don't think we should outsource this debate to corporations. But this shouldn't take the tone of contempt and resentment towards what we already have.
And I don't want to only be paid an average salary for it.
But... an excellent salary (above-average, maybe even above-high) would motivate me and no, I won't try to copyright that work and mooch off it for billions until I hit 80.
As I said: incentives. Many companies expect programming to be like installing a kitchen sink yet are unwilling to pay to commoditize the activity.
Even if you had a lot of financial support, how would you go about implementing the standards and transformations described in the article? Who would adopt it?
Maybe in the context of a company this could reap benefits and become a case study. Or in a partnership between many companies there could arise such applications. But they would still be locked to a niche. That is why I think even with financial support, the problem is maturity, or lack thereof.
You could argue that once a large enough group adopted practices that proved to be effective in reducing maintenance time and complexity, that others would follow, but that stops being a matter of pure financial support and starts being a matter of adoption and convergence, pointing once again to maturity of the ecosystem.
And another aspect of maturity, standardization and sanitization is that it comes at a cost to diversity, which in turn has benefits of its own, like new ideas and technology. This is kind of the balance between exploration and exploitation we see in training neural networks. I think software is in the transitioning phase from exploration to exploitation and we are living it.
IMO if you make such a tool for something extremely popular like Python or Javascript and then start pivoting upwards from there, people will be coming to you. There are a lot of deficiencies in the everyday work of the programmer to this day.
Like python’s “2to3”.
It’s not a magic bullet; the developer still benefits from thorough testing, but I think changelogs could stand to improve in a lot of projects.
And unless the old way was genuinely bad/hard, why deprecate?
I have a friend that contracted for a company doing y2k work. The main guy have written a system to parse cobol, do pattern matching to find y2k issues, apply transforms to the AST to fix the problems and the serialise back to cobol.
It supported a dozen of so cobol variants. I think it was very successful.
Automating fixes is only possible if you understand fully where the problem is. Then, it still needs testing does it break something else. For testing, automatic tests most likely do not test for everything. And someone would need to create those tests.
There is also a lot of existing code for existing software. Would it be possible to analyze or convert that amount of code to be somehow updateable?
Software is usually made with some programming language and related dependencies. Those dependencies don't necessarily exist for other programming languages and web frameworks.
Problem is, most people are under the illusion that software is like a woodwork or metalwork gadget: you make it once and it can work happily for 10 years without changes. Which is an extra funny thing to believe in because it doesn't apply to those physical gadgets either.
It seems people just want things to be created once and used infinite amount of times and even if I can relate to that desire, it's simply not the universe we are living in.
And in spite of countless evidence to the contrary, it's still an extremely persistent illusion that creates untold amounts of friction between shareholders and programmers every day.
I don't really have a solution for this, I think tech consumers/clients are just always going to want what they want and not care so much about what it takes to make it happen. Thus they will always be surprised and grumble about the price. That just seems like human nature to me.
Absolutely. I see it the same way. After commercially programming for 19 years I've seen exactly zero progress in the human race with regards to that particular problem. It is apparently a built-in bug of our brains and everybody discovers it for themselves down the line (and 99% proceed to never fix it).
I think I recall one quote from a seemingly famous building architect (I think the guy who built the Fallingwater house, so Frank Lloyd Wright?) that was along the lines that you never put all expenses in plain view of the customer at first; you put a reasonable cost plan in front of them and then, after they are heavily invested in you, start giving them the other expenses (which are inevitable and not invented by you, let's accentuate that!). Many people call that a scummy practice but the truth is, EVERYONE is expensive, some are just better at selling themselves. So you have to give the customer the truth spoon by spoon because they prefer lies and truth is too hard to swallow at one bite.
And apparently that's how people tick. I don't like it either. But... blame the game, not the player -- as the saying goes.
Imagine you brought your car for an oil change and the mechanic says "uhh yeah maybe you can get new oil next year but for the next few months we need to... Uh... Address technical debt in your car... So umm... It's gonna cost you thousands of dollars and you aren't going to get anything you perceive as a change"
You'd think they are ripping you off.
However, as a counterpoint to yours, even those of us who can communicate quite clearly get met with very ugly accusations of ripping customers off. People simply come to you with a preconceived notion about how much something must cost (in terms of money, effort and time) and they are absolutely not willing to change that notion -- even if they are on the side of the pure imagination and you are on the side of the facts.
It's basic psychology that many of us had to learn even if we never wanted to.
But what the author is proposing would automate and reduce the cost of the support. A company that could achieve this cost saving and market or successful would have an advantage. Unfortunately it is a very hard problem abs many upgrade transformations can’t be proved to be correct with good automated tests. And then you have the problem of how do you automated the maintenance of the tests.
So not sure how such a thing could be monetised properly. Likely by only offering enterprise-grade services: the tool itself is free, the customer's programmers could employ it to achieve migration but the customer prefers to outsource the cost. Sort of how several Linuxes are free but come with paid support contracts.
(Actually, n log(n) + a constant offset; i.e. partitioning the code has its limits where it becomes detrimental.)
Testing / Certification: A table lamp will be UL/CE approved. This means it will be tested in such a manner that normal use and abuse will not cause unexpected side effects. Knocking a UL approved lamp off a table will not cause your house to get burned down. A light bulb of rated capacity will not cause a short circuit. Normal wear and tear will not cause it to become unsafe.
Outlets: A table lamp plugs into an outlet, of a standard size and characteristics. The outlets are a standard interface that is often nation or region wide. The voltage and current that can be delivered through an outlet are standardized, as is the nature of the loads which may be plugged into it. The outlet itself, and the housing, and wiring are well regulated.
The outlet is connected to a circuit, and a circuit breaker (or fuse). These devices completely and permanently interrupt the delivery of voltage for a number of standard conditions, including over-current, and also ground faults.
The circuit breaker, or fuse socket, is part of a standard and well understood panel, which is designed to allow a variety of options in deployment, with electricians able to configure them in a wide variety of applications. Yet, the user of a breaker of fuse panel is protected from the voltages and power inside, and given a standard and easy to understand model of how power is distributed, and controlled. In fact, some advanced users can use the panel to disable part of their local power network to allow maintenance and modification in a safe manner.
Breakers and fuse panels allow for lock out, which will physically prevent the reapplication of power, while maintenance is ongoing.
Power feeds include metering, and themselves are driven from a circuit, and the system is designed in much the same way as the residential and business circuits, but on a larger scale.
Not only all of those are true, but there is more
Circuit breakers and fuses on devices, etc.. are all designed to be coordinated. The smallest fuse/breaker in the chain from source to load should always trip first. This prevents a scenario such as the one where a person plugging a defective toaster into an outlet brings down the entire power grid.
Circuits and power routing have well separated areas of concern. There is no way that a power system problem can directly cause problems with the sewers or other infrastructure, other than failing to deliver power.
Our software infrastructure lacks all of this sophistication and standardization. You can plug in a fan made 100 years ago into a modern outlet, and it will work. You can't even run an MS-DOS program from the 1980s without resorting to an emulation layer.
You can know that same fan will not cause Russian bots to be able to control the power grid. Nothing plugged into an outlet can ever give you control over the grid. The same is not true of software, no matter how carefully constructed. We're still at the stage where a bad program can take out its host operating system, and then the network its on.
To say we need more skilled programmers, better aware of the security implications of their work, is to insist that we could run a power grid without circuit breakers in every home.
The operating system has the job of protecting the network and its users and applications from each other. There is no equivalent to circuits and breakers. You can't run a program, and know that it will only consume X amount of CPU, network I/O, or RAM, unlike the outlet limited to 15 amps.
You can't be sure that a program won't have side effects. Plugging in a bad toaster in the kitchen shouldn't effect a circuit in the garage. The damage should be limited by default. None of our Operating Systems do that.
Computation lacks maturity, especially our operating systems. We have no stable standards for anything important.
That, in my opinion.. is the real root issue.