Software Rot (2017)
geoff.greer.fm
geoff.greer.fm
Software that is merely rotten can often be saved, with varying levels of effort and skill. Software that is ossified often needs to be replaced.
Ossified code is code that isn't isolated, but tightly bound to several other components. This means that you can't swap it out without making the exact same architectural mistakes. This means you basically need to start over from scratch, which is a much harder and broader problem.
if there aren't system/integration tests - write system tests. you have to be pretty thorough.
have a long and involved discussion about what the new thing is going to be. go through all the frustrations with the current code base. convince yourself at the end that its going to be worth it, because the cost is going to be high. ideally the new version will open up new capabilities that just weren't possible before.
find a cut in the dependency graph. you're going to rewrite the code on one side and leave the other side untouched. ideally that cut will be small-ish and contain some particularly broken stuff that you'd like to get rid of as soon as possible. unfortunately there may not be a small cut that makes sense :-(
build a shim between the old model and the new model across the cut. this shim is only going to last as long as the old code on the other side. this shim might be involved and a total waste of effort. suck it up or look for a different cut.
replace the code on the new side and test against your suite. if its not really that exhaustive, expect a rash of bug reports. run through some kind of soft deployment. if its a request/response kind of thing consider forking your production traffic and comparing the results against the old code base.
repeat until golden brown.
this overall process also can fail. often because of poor test coverage, and more likely because you haven't adequately communicated the scope of the undertaking and its absolute necessity to the rest of the engineering organization and the business as a whole.
however, if you make it through, you've avoided the giant speed bump that comes at the end of the rewrite, and you've been able to fold in new feature and bugfix work along the way.
I was about to come here to say that there's really no such thing as software rot. If all of the environmental variables remain the same, software doesn't really rot. It's actually the environmental variables which change.
Over time, each example became more rigid and hard to change because things that should have been modularized and isolated in one place instead became idioms pervading the entire codebase.
I can deal with idioms. If 1) the idioms are very consistent and 2) you have a reliable syntax driven code rewrite tool, you can drive idioms into a library and/or otherwise translate them. I've been paid to do both.
Rot would be when a program works less and less over time because its external dependencies vanish or change and the program itself stops adapting.
But what stops a program from adapting? It becomes "ossified." Because the code organization isn't up to the challenge, it becomes too hard to add and change features without introducing bugs.
There is no software "rot." It's all different kinds of ossification.
Then it sounds like "rot" is a very aptly used word. I'll put on the jerk hat and quote the Oxford dictionary:
rot (verb)
1 (chiefly of animal or vegetable matter) decay or cause to decay by the action of bacteria and fungi; decompose.
Rot is when something breaks down while environment remains the same. What happens with software is the opposite: it forever remains in pristine condition; it's the environment that changes.
(Compare also with description of evolution. A species does not "rot". It dies off when its niche changes so much its no longer adapted to survive in it.)
Exactly. If you replicate the old environment in an emulator, the old code runs just fine.
Usually it has nothing whatsoever to do with the code organization. The most common cause is lack of active developers to make the adaptation - company dies, developers die or change jobs or lose interest. That's why "rot" is an apt metaphor, like a house slowly falling apart due to lack of maintenance.
> If all of the environmental variables remain the same > I can deal with idioms. If 1) the idioms are very consistent and 2) you have a reliable syntax driven code rewrite tool,
Would you like a pony too? The environmental variables don't remain the same. The tools aren't smart enough to catch every subtle variation of an idiom repeated hundreds of times across a large codebase. Yes, you can automate away the easy cases, but that's not even half the battle. That's why people hire maintenance programmers like you.
It varies from shop to shop, but what I've seen is that code organization has a lot to do with it. Basically, if your code organization/factoring is really bad, then it's more likely that porting becomes untenable. Ironically, these are the projects that can have the longest lifespans, sometimes developing a history of multiple failed attempts to "port" them. (Really, re-implement while incurring huge opportunity costs.)
> If all of the environmental variables remain the same
Would you like a pony too?
You're seriously missing the point. My point is that the environmental variables never remain the same. (I was making two points, and I think you missed where I switched.)
> 2) you have a reliable syntax driven code rewrite tool
Would you like a pony too?
I repeat. I've been paid good money to use such a tool. I've used such a tool to enable the 1-step-removed porting of a project to a different language variant. Instead of doing the port myself, I was refining a tool which automated the port and enabled the maintainers to easily port all the changed code they were checking in. This lets you do a port without incurring the opportunity cost. I know of a consulting company that once made most of their income with this.
Yes, you can automate away the easy cases, but that's not even half the battle.
You can't simply hit a switch with this, of course. Someone has to put real thought into how the translations are made, especially if you want idiomatic, readable code going out the other end. The good thing about consistent idioms, is that they can be leveraged to multiply that work. The manual part can be re-leveraged through automation.
That's why people hire maintenance programmers like you.
That could be read in a way that sounds less than friendly and maybe a bit elitist.
I suggest that your sample is non-representative. Companies don't hire specialists to refactor code that they're abandoning. Dead companies don't hire anyone at all. You don't work on rotten code because nobody does, but that doesn't mean it's not out there.
> That could be read in a way that sounds less than friendly and maybe a bit elitist.
It could, but that would be on the reader. I didn't say anything to denigrate maintenance programmers.
Companies do hire specialists to port code that they're not abandoning.
Dead companies don't hire anyone at all.
"Zombie" companies can still have big revenues and they can and do pay consultants to do things like port old applications. Again, there have been entire companies based on such activity. I've made money from such companies.
You don't work on rotten code because nobody does
I was a consultant for a language vendor for 5 years, then continued working in that niche for another 5. My sample size in this context is in the many dozens at least. There's a lot of code I'd call "rotten" out there that people still work on.
I didn't say anything to denigrate maintenance programmers.
Good. I'm sure I've met a number of them who are probably better coders then either of us when they're half asleep.
This doesn't line up with my understanding of the history of Chrome. I thought Chrome was based on WebKit, therefor definitely not from scratch.
Edit: this post references lifting components from WebKit and Firefox in the initial version, https://googleblog.blogspot.com/2008/09/fresh-take-on-browse...
Point being, even Chrome would have needed to make WebKit compatible with that model. I don’t know if there was a lot of work to do that or not. But “from scratch” implies that there was no existing code to refactor, which would appear be a false statement.
I had a thumbnailing micro service I made years ago that worked happily until 3 months ago. A combination of PIL updates and various Ubuntu security updates made it no longer work properly in various ways from keeping the alpha layer on PNG files to cropping from the wrong position. I consider this software rot since I hadn't updated anything but the surrounding system.
"""So it seems that besides GHC, only ever HBC was used to compiler GHC. HBC is a Haskell compiler where we find the sources of one random version only thanks to archive.org. Parts of it are written in C, so I looked into this: Compile HBC, use it to compile GHC-0.29, and then step for step build every (major) version of GHC until today.
....
[M]ost of [HBC] is written in LML, and the LML compiler is written in LML."""
This is one reason "do one thing and do it well" is useful. The interface is kept simple and should be easy enough to update to a new environment. At the same time, that means for bigger systems a lot of small things will need updating over time, but at least the changes should be straight forward.
This largely works, especially on the C/C++ side.
C++ in particular has also changed a lot since the 90s including how permissive the compilers used to be, even for stuff covered by c++98.
$ make
cc -n -O -c -o sed0.o sed0.c
In file included from sed0.c:2:
sed.h:116: warning: declaration does not declare anything
sed.h:128: warning: declaration does not declare anything
sed0.c: In function ‘main’:
sed0.c:32: error: ‘union reptr’ has no member named ‘ad1’
sed0.c:48: warning: incompatible implicit declaration of built-in function ‘exit’
<snip many more lines>
The problem here is that although sed was written in "C" and we have a "C compiler" on our system, the definition of what exactly constitutes "C" has changed. The modern compiler no longer accepts the same language as the V7 compiler.We can fix the source to deal with this and produce a working binary. But then, what would we do with it? We can't really install it as /bin/sed. The world now expects /bin/sed to support switches and syntax that this sed does not. Trying to use this as /bin/sed would cause lots of programs to fail. The definition of what exactly constitutes a working "sed" program has also changed.
So it seems to me like sed already has rotted in some sense. Looking forward, the sed from a current system is likely to be similarly rotted when you try to use it on a machine 40 years from now.
Standards provide solid points of reference in this mess: 1989 ANSI/ISO C is not going to change. Specific compilers come and go, but the language defined by those documents (one ANSI, one ISO) is unchanging and, more importantly, well-understood such that C compiler implementers both feel the need to implement it correctly and understand how.
> Looking forward, the sed from a current system is likely to be similarly rotted when you try to use it on a machine 40 years from now.
I think it's likely the language won't rot the same way, due to the standardization I mentioned, but the OS interfaces beyond POSIX or similar might rot.
A swift‐flowing stream does not grow stagnant.
Neither sound nor thoughts can travel through a vacuum.
Software rots if not used.
These are great mysteries.
— The Tao of Programming, Geoffrey James, 1987
This largely works if it was written in Delphi.
Well...
The code might very well work the same, but the dependencies were a nightmare: They were scattered across 14 different CDs and web sites. One provider of often used components had gone out of business, luckily those components were released on sourceforge.
Since there wasn't pom.xml (forgive me for mentioning a Java-thing here, but this was long before package.json or ProjectName.csconf was invented) in Delphi projects one also had to go through a bit of research, trial and error to figure out which versions was needed and the exact ones that were compatible with each other.
Luckily we had a seasoned, well paid Delphi expert at hand and we got it working in just under 3 days.
If anything I'd say this is a great example of (one form of) bit rot.
Now that I think of it it might very well have been the same week that I left behind my last bit of prejudice against Java and started openly preferring it.
As time passes, code gets stale. And finally, when people can't remember what the code does, at that point you could consider it rotten. At some point the tooling becomes unavailable and you wont be able to run it!
Ways to prevent code rot:
* Clean (refactor, remove tech debt and legacy code) often.
* Write good comments.
* Keep tooling & dependencies up to date.
Then someone else comes along and is faced with real confusion and before you know it you find yourself with a system which, if you update data one way, the changes are sometimes picked up by another part of the system, sometimes not and... its rotten.
Counterpoint: rewriting your code from scratch is the worst mistake you can make. https://www.joelonsoftware.com/2000/04/06/things-you-should-...
https://medium.com/@herbcaudill/lessons-from-6-software-rewr...
Joel Spolsky's essay is more about rewrites of "ugly" code. E.g. Fix the disorganized code by gradually refactoring into cleaner modules over time instead of a total blank slate rewrite.
In contrast, this Geoff Greer essay is about paradigm-shifting software architectural changes which by their nature, are very difficult to retrofit into an old existing code base. For these, it's often easier to start coding with a "blank slate" rewrite.
As examples Geoff's category of rewrites (new architecture paradigm) vs Joel's category (unhygenic code cleanup), we can look at Joel's ex-employer Microsoft:
- SQL Server database diverged from original Sybase code and rewrote the engine. One of the architecture changes was switching from page locks to row-level locks
- C# compiler completely rewritten from C++ code base to C# codebase. One architecture change was "compiler exposed as a libary service" instead of being the closed "black box" that was assumed by the C++ code.
- operating system: MS Windows NT as a blank slate operating system instead of gradually extending old 16-bit DOS code or 16-bit Windows 1.0 code. One motivating architecture change was the new 32-bit protected mode on the newer Intel 286 & 386 chips. Another factor was the switch from "cooperative multitasking" to "preemptive multitasking".
Somehow, I always felt uncomfortable reading Joel's article because I had never seen spaghetti code on large code bases.
I now realize that it is because I've been using open-source code bases for quite some time and that is probably the reason why.
Recently however I saw some enterprise in-house code and that's when everything fell into place!!
Yes, rewriting your code is usually a bad idea. Rewriting others' is a different story.
Python's rising years have been the same years that we discovered that (a) generalized parallelism is really hard, and (b) many many many applications of muticore are "embarrassingly parallel" and benefit little from shared resources. Stasis around the GIL is a usecase-driven decision to make embarrassing parallelism easy enough (via multiprocessing) and delegate more involved parallelism to linkable libraries or separate services. It helps the language focus on what it's good at.
With async io, you can have a lot going on at once, interleaving nicely while different tasks wait for io responses.
Beyond that, normally you're just as well off to kick off a second process.
You have to get pretty fancy with algorithms to make multi-threaded computation a net-benefit. Most of the time you don't need that. If you do, you're probably not reaching for python.
I've always understood software rot to refer only to cases where the dependencies, including libraries and platform APIs, are changed so that an already-delivered copy of the software can no longer function, and the maintainers have since abandoned it, or can't do a quick fix to get it working again.
A recent example is the removal of support for NPAPI plugins in Firefox [1] in 2017. Firefox is in the role of the 'platform API' here, and this change of theirs broke any extension that didn't update. And, the replacement API differs in features, so the effort of updating an extension approaches that of a rewrite.
Another example is the game 'Star Wars Episode I: Racer', the podracing game from 1999. Newer versions of Windows and DirectX have changed things so that the game's original executable doesn't run [2].
Even open source isn't a remedy. Random projects one can find on a source repo, last updated x years ago, are code that are susceptible to this. Sometimes they're shipped without dependency management, pinned to old versions that themselves no longer work, or track the latest when they should be pinned instead. If more work is required to get it running than just the instructions provided by the author (assuming those were sufficient at some point), then it has suffered software rot.
[1] https://support.mozilla.org/en-US/kb/npapi-plugins [2] https://www.play-old-pc-games.com/2013/12/02/star-wars-episo...
Also, as the poster said, Firefox had a rich Add-on API that they wanted to keep support for but which they eventually decided to drop. The new API isn't close to the versatility of the old API, sadly some add ons work worse now.
A better example would be I think iOS parts of which can't adopt swift and have to stay implemented in ObjectiveC because of binary compatibility concerns.
I have code I wrote 20 years ago that still compiles and runs. I could check again but I don’t think any of it rotted.
That said, I don't buy "software rot" as a proper name. Software does not "rot". The concept of rot implies an internal change that makes something fail whether or not the environment around it has changed. This does not happen to digitizes data. Programs only stop keeping up with the Red Queen's race of computing environments.
Software people love rewriting things. In most cases where I have to maintain an old project I would much rather rewrite it, but it's usually not in the budget.
From another HN thread today: https://news.ycombinator.com/item?id=19245485
> The first 90% of the code takes the first 90% of the time. The remaining 10% takes the other 90% of the time.
And
> With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody
In the end, it went pretty well and it's impossible to know what would have happened if they would have decided to continue on the Navigator track, but Netscape Navigator reached basically 0% market share until Firefox was released.
So the claim is that it would have been less expensive for Firefox to start over from scratch and make a new multi-process browser while at the same time maintaining a single-process browser that stayed current with web standards in the interim?
Microsoft ended up doing it with Edge (I believe), doesn't seem to have hurt them much.
Because they had already lost a large chunk of their share.
[0]https://www.joelonsoftware.com/2000/04/06/things-you-should-...