The joys of maintenance programming (2011)
typicalprogrammer.com
typicalprogrammer.com
My experience is that this is almost never the case.
Generally, management does not understand or appreciate why it's a slog, why you might want to refactor this or that, etc.
Obviously a lot of this falls upon the engineer. It's your job to communicate and demonstrate. Helps if you have other engineers backing you up and presenting a unified message. Also helps if management is technical enough to understand what you're saying instead of simply believing -- or not believing -- you.
But maaaan. Uphill battle! I'm burnt out on it. I've decided I need to seek greenfield stuff. Maybe forever. But definitely for now.
Very few companies understand that the most valuable asset of a programmer is his mental model of your system. You need people who can do the hard work because they understand the system at an intuitive level.
Which is actually something a lot of management would prefer to not have. They often want people to deeply document a system, on the mistaken belief that this knowledge could be transferred out easily.
Unfortunately, it's not. Even a well documented system has inherent complexity that only working on the system would reveal. Otherwise, merely reading the source code would allow for a complete understanding and deskchecking of the whole system!
- Well documented, well maintained code that everybody could work on
- Quickly finished projects that nobody ever touches again, but only the programmer(s) have a mental model of it
The first option is more expensive at first, but way less expensive if you happen to have to rewrite it from scratch at some point (and if you are not maintaining and document it, that moment will come and it might not come at a point in time that fits the schedule).
The second option is less expensive first, but it will bite you in the arse if the programmer(s) are gone or forgot how it is done.
Everywhere else managers try to minimize risks, here they often just don't aeem to get the trade-offs involved.
https://www.theverge.com/tldr/2019/10/4/20899198/google-read...
I've got mixed feelings about this. Google's frequent "killings" have caused me to never trust them or want to buy in to their products.
But at the same time I think that it is good to "honor" "dead" products. Those products encompassed millions of hours of blood and sweat and toil and so on. I don't know the best way to memorialize that but sanitizing them from history entirely ain't it.
Also as you point out, getting funding for preventative maintenance or continuous improvement is extremely difficult in a corporate IT environment, especially if there isn't a product owner to advocate for the system and to secure funding. Although I do believe you need a product owner to drive the work to ensure it delivers business value, and not have it driven solely by developers.
May I ask what sort of company this was? Some "unicorn" startup? Stable oldcorp?
in mature industries, there aren't new revenue generating opportunities, there is only getting better at what you ordinarily do; all marketshare growth comes at the expense of competitors.
Refactoring it is just one of possible parts of the process, much opposed to 'yeah this legacy code is broken dogshit and needs to be rewritten even though the current software is already written, proven and works for 99% of the stuff we need' approach.
[Refactoring] is not what the post is about.
Clients and management generally understand that old code needs to be maintained. They don't love that reality, but that's not generally a tough sell unless they're completely clueless. And they're usually not completely clueless.
The tension in my experience comes when an engineer wants to perform something other than the fastest, sloppiest, filthiest duct-tape maintenance imaginable.
Management accepts the need for fixes, but often does not have the technical/domain knowledge to understand the short- and long-term merits of Proposed Fix A versus Proposed Fix B. And naturally, incentives heavily push management toward the quick fixes by default. Time is money, after all. And managers are expected to meet quarterly and annual OKRs. We don't have a good way to quantify the long-term value of a "proper" fix versus a "duct tape" fix. And so on.
So for me maintenance work is difficult to separate from notions of longer-term software maintenance such as refactors, dependency upgrades, etc.
I personally don't believe refactoring should ever be part of "maintenance".
Maintenance needs to avoid YAGNI and avoid the engineers natural desire for over-engineering and unnecessary perfect polishing and abstruse abstraction.
Refactoring is something that should be driven by new features (or worst case towards rewrite), and few have the skill to do it well.
I'm not arguing the sibling strawman that maintenance be "the fastest, sloppiest, filthiest duct-tape maintenance imaginable."
Excelling at maintenance is highly skillful work and often poorly rewarded.
Refactoring is something that should be driven by
new features (or worst case towards rewrite),
Assuming we're talking about a program that does something useful and isn't just a screensaver, it's probably ingesting data from the outside world. That data can change in a variety of ways. API partners and users can introduce changes on their end and will eventually find new and annoying edge cases. Or maybe the volume of data increases by 100x and changes must be made on your end.
Or maybe your software sort of rots as the security vulnerabilities pile up. Eventually you've got to make major version or point upgrades to the framework and/or various dependencies.
We want to handle these demands judiciously and as a rule, yeah, refactoring isn't generally going to be the answer but sometimes it is.
Of course it shouldn’t, because software maintenance does not exist. Programs don’t degrade with use. All “maintenance” on software is, like new development, the implementation of new context->behavior combinations, and calling it something different is a result of applying an inappropriate physical-product metaphor.
Of course it shouldn’t, because software maintenance
does not exist. Programs don’t degrade with use
The word "maintenance" doesn't imply some kind of physical product metaphor. Sorry. That's not what the word means.
There are several activities in software that are similar to maintenance of physical products. e.g.
* that parsing library that you used 2 years ago and was in good working order then, is now outdated and has known vulnerabilities that are not present in the latest version, apply update.
* Those cloud-hosted databases that you use are an older version that the cloud vendor has deprecated and will some time withdraw support for, apply migration to a later one.
CEO will start looking at this topic if app performance/stability metrics will start making impact on revenue stream, aka app is slow and stop working every day and customers go away because of that.
It's just a lack of understanding. They imagine they understand, because software doesn't get worn out, and doesnt degrade obviously!
Every larger company I've worked at had enough engineering overhead that they could afford to have small teams working on both reworking old stuff and creating new stuff, while the main teams kept the working stuff generating revenue.
As an embedded dev I have thousands of devices that were commissioned and are on 24/7 duty. The oldest still running fleets I believe date back to 2007. Some of them had firmware updates, but many of them (despite numerous bugfixes and improvements since) run on original software. Customers are reluctant to update if there's no specific need arises in their static operational environment.
Naturally in business logic world the requirements are never as static. But when they do change you can likely sell the management on it as necessary development.
I've found this to be true... to the point where some places even prefer manual maintenance to spending (potentially) a little more time on automation, even if that time would be paid back over the next couple of years.
I'm not adept at planning, selling my work, doing the manual maintenance, and doing the extra automation, as eventually that work becomes homework unless you get a boss that's willing to put in some effort here, which I haven't found yet.
But if the whole team is on board and has reviewed your proposal it's going to be more credible. And that's just good engineering anyway.
As the months and years wear on though, things change. Each bug has an obvious solution, and you’ll take the path of least disruption. Implementing new features is a matter of figuring out which trade offs you’re most ok with. Maybe you need to refactor something first, but that’s an obvious path itself.
You’re painted in a corner. A beautiful corner where the only safe thing to do is to tread carefully. But you don’t mind, because you know where you are and what you need to do to get to where you want to go.
Having done lots of both greenfield and maintenance work over four decades, I can unquestionably say that software maintenance is way, way more difficult.
I heard this first from Joel Spolsky, "it's easier to write a program than read it."
Which led me to write an article on the subject, "How To Read A Program," about the cognitive aspects of trying to learn unfamiliar code. https://www.itjungle.com/2009/01/28/fhg012809-story01/
The article uses AS400 applications as an example, but is general enough that it applies to applications in most languages.
This is a field begging for help from AI.
Most of the problems of API design could be remedied by people stepping through their own code and looking askance at the whole process.
If the latter is the case, then you get better at reading code by writing code. Writing lots of code puts those code-patterns into your long term memory, and then when you see those code-patterns again you'll recognize them.
For system design too - if you've designed lots of systems yourself, then when you see a new system, you'll be able to relate it to the systems that you've designed before.
So maybe building greenfield projects also makes you better at maintaining existing projects?
It'd be great if someone could point me to some existing literature on this topic. I've looked around and can't find any.
I think it does, because it builds a higher-level sense of how something "could" or "should" be and familiarity with thinking at the system level.
I've had a lot of problems with people who (seemingly) only have experience maintaining projects. They seem to have a tendency to focus narrowly on "fixing" a bug close to its source, and often lack understanding of a lot of fundamental technologies (because they only attend to things that have broken), and get stuck by a kind streetlight fallacy where they fix things in more familiar areas. The end result is ugly kludges and whack-a-mole bugs where a superficial "fix" in one area causes a new bug to pop up in another.
Only to an extent, i think.
> I've had a lot of problems with people who (seemingly) only have experience maintaining projects.
And similarly I've had problems with people who (seemingly) only have experience with starting a new project and then simultaneously over-engineering and piling on tech debt.
I think "i've never bootstrapped a project before" is easier to cure than "I don't have a good sense of what's expensive vs what's cheap tech debt."
Some tech debt is basically 0% interest. Was this small hack bad? Yeah. Will it need to be fixed, eventually? Yes, for sure. But does it compound every day, or does it just sit there being a little yucky? Very easy to determine in hindsight. Very hard to predict as you write the hack. The end result being the simultaneity in my example. People will over-engineer things that won't end up being problems and under-engineer things that will turn out to require compounding hacks and it would've been cheaper to just get it right the first time.
> And similarly I've had problems with people who (seemingly) only have experience with starting a new project and then simultaneously over-engineering and piling on tech debt.
The best thing is to build a greenfield project and then maintain it for several years, fix your mistakes, then do the same thing on a new project again (including the maintenance).
You really need the full spectrum of experience, and put yourself in the shoes of the future maintainer when you're building something new.
They never have to maintain anything, some poor soul has to fix the broken software.
They also dont even know what mistakes they made, since they arent there to solve it.
I never felt like I had 1 year of experience 5 times because I moved between several kinds of projects at different lifecycle phases and with different constraints, and drew lots of parallels between them. At the end of five years I had project histories in my brain that spanned more than ten years. And got jobs that should have gone to someone with 8 years’ experience. I do not think this was a coincidence.
I maintain two major maps, which are what is the code logically trying to do, but also what exactly it's doing. (i.e. Sanitize inputs, check error conditions, do work, return result as a logical model and then "ensure foo is < 0 and > -10" etc. as the specific checks).
Let's hope what you write the first time is a good practice.
Some projects though just keep on trucking along and doing the next part with the newest shiny thing. You end up with five ways to access data, three ways to send emails and fifteen different methods of report generation. Frankenstein architecture! This architecture also emerges when you have leaders that don't make decisions and peeps debating all day on slack and then doing their own thing anyway because they didn't really want to offend anyone and everyone's ideas are great.
When I wrote that, in the wake of the 2008/9 meltdown, I intended to give programmers some career options to think about. More interesting and stable work out there than FAANGs and startups.
You're still somewhat biased, so you can get put on projects where the style is different from yours. If you happen to think that style is bad, it won't be fun to work on.
You shouldn't want to change the style. Chances are you'll do it incompletely (job change) or the time spent won't be worth it. If someone else comes along and doesn't like your style, we're just going to repeat this process.
There's been way too many different approaches that work for developers to be unified on what styles are best.
At my last job I had to work with a PowerShell codebase that `cloc`ed in at over 100,000 lines, and was absolutely ancient to boot, having been added to in bits and pieces by sysadmin after sysadmin over the course of 12 years total, in 3 different human variables-and-comments languages, all of whom were Windows power users with very clear ideas of what needed to be done, but also all of whom were very clearly learning to program for the first time, at least in the wonky PoSH paradigm. AKA: Classic big ball of mud situation. Every new customer had to have their details run through this codebase, however, and I saw the potential to eventually boil this down to a simple 6- or 7-parameter cmdlet (PoSH for "shell script"). I got my undersized chisel and toy hammer and got to work.
In some ways starting work on the proj felt more like beginning to dig into an ancient COBOL database in a System/360 mainframe than anything resembling the state of the art in 2023, or even 2011. There was a single VM that everything had to run on, and the script couldn't even be called remotely via SSH because the only way in was via a web browser based remoting session. The connection was spotty and would often drop halfway through coding in there in my dinky little Notepad++ session, requiring minutes of reauthentication just to get back to work. Unit testing (which PowerShell had a surprisingly good third party library for, called Pester) was out of the question. And: There was no version control! No wonder it got to 100k lines - most of the code was being entirely unused! But what?
The first few Saturdays I spent didn't go very far. I was mostly trying to figure out where the logic even began and ended in this mess of spaghetti. But eventually I started to piece together an overall structure for the thing. I made a pact to myself not to delete any of the many, many files that had accrued over the years until I knew every file that the code would touch first. I started doing some lightweight encapsulation with the -WhatIf flag, and with firebreaks (sticking `return` after each tiny function) so I could work on one linear piece at a time, and gradually tiptoe my way forward to the finish line. Eventually I got there, and managed to strip the whole thing down to about 1500 lines.
The joy of this kind of programming is that, even if you don't start with regression tests or modern practices, you do know from having ran through the original process a million times exactly what the end product is supposed to look like. Since no one is paying attention to your refactoring work in the first place, you're allowed to take your sweet time. To me it was obvious this mess of code was going to become a blocker for the business at the pace we were going: We were a SaaS, and yet we had to manually provision customers behind the scenes like this, in a process which took days to weeks? And we didn't even have a proper CLI we could call for this? Jackpot!
I left the company about 2 months after wrapping this project up and demoing it to my coworkers in search of better pay, but I'm really glad I got the chance to take such a fearsome codebase and tidy it up like that. There is just something satisfying about that kind of work, it's like gardening. I also got really, really good with Notepad++'s find and replace across all files functionality. There is in fact a skill to it! (Mostly that you should always, always find and replace the longest strings first!) Although I think these days I'd probably reach for a similar function in vanilla vim first, or even in bare `sed` if I was feeling gutsy.
(I'd end with "If you, dear reader, are in a similar PowerShell quagmire you need someone to pull you out of, I'm available for consulting at $X00/weekend", but alas. My weekends for the next few months are already sworn to another equally wonky project for my current employer. At least I'm getting overtime this time. ;) )
In Notepad++ you can mark "Match whole word only" or use a regular expression with word boundaries, e.g.
\bsome string\bThe joys of maintenance programming - https://news.ycombinator.com/item?id=2267261 - Feb 2011 (29 comments)
There are a bunch of reasons for that, but two interesting ones are (1) historical material is particularly welcome here, and (2) we want cohorts of new users to get an 'education' in the 'classics'. It's too easy for grizzled veterans to assume that everyone has already seen the perennials, but even they once read The Story of Mel, or whatever, for the first time!
But the existing project could've used some improvements; there was an inefficient SQL result to XML string to JSON thing happening with every request, so doubling response times could probably have been done within a week of work or so, and quadrupling it with a few weeks of work (there was thousands of LOC of SQL to XML code involved, but in most cases it could've been SQL to JSON directly).
Let's say I make a website that showcases my qualifications and lists out the services I offer.
How do I actually get a client?
I’ve only skimmed it and have started reading but the parts about highlighting/phrasing of one’s marketable skills seems good.