The reality of long-term software maintenance from the maintainer's perspective | Hacker News Reader

The reality of long-term software maintenance from the maintainer's perspective

construct.net

128 pointsAshleysBrain30 comments

As an old software-dev who has both started and 'finished' projects (stayed on to make them actually useful in production for non-trivial data flows), Amen.

In today's world it is in some ways getting worse: You have older applications that are still bringing in money, built on a patchwork of outdated platforms, that e.g. Microsoft no longer supports or offer migration paths away from, apart from a full rewrite.. And unpatched security holes in the older software removes your option of 'just stay'.

I find it painful to explain to business that yes, their 5-6 year old 'greenfield' product now consists almost entirely of parts that no longer offer migration paths or maintenance :-/.

The author makes this point only somewhat implicitly, but I think for a lot of people, new feature development is the fun part of software engineering. Not for everybody (do not let me tell you how to have fun), but a substantial majority. A project's maintainers might also have lots of ideas for new features they would love to work on, but if 9 out of 10 people want to spend their time adding features, and adding the feature is only 10...25% of the work, you can see there is a problem.

There is also little motivation to do the hard part of maintaining things as those positions are often paid similarly but require much more work. If you can hop to jobs that only want new features while getting comp increases, where's the motivation to actually sit still and maintain something?

However, I think the industry might be slowly moving against people who've never stayed at positions for a while and don't have this maintenance experience.

It's not only that it's more fun, it's also usually the better path for self-promotion. Inside a company promotions often go towards those who have most visibly contributed to new projects. In open source projects that is of course not as prevalent, but even then many people will put open source contributions on their resume and "made big feature XYZ" is typically more impressive to recruiters than "fixed lots of small documentation and localisation problems".

I don't think this is justified, as (since the article correctly describes) over time maintenance becomes much more important than the initial feature creation. But at the moment this is the world we live in and people adjust their behavior accordingly.

Management often gets rewarded for new "launches", and rarely for things that already worked continuing to work. To some degree, so does the business (although a track record of things continuing to work over a long period has value, if anyone has a time horizon beyond the next quarterly report). Meanwhile, it is more difficult to measure productivity for maintenance work, because the primary work product is understanding. Managers with the skill and experience to recognize and compensate for this are rare. Developers are not blind to these incentives.

> To some degree, so does the business

Sorry, but that part is wrong. The business gets rewarded for doing the same thing that was already working again and again and again.

Nobody goes out of their way to buy innovative things. It's even worse, customers tend to avoid new things until they had enough time to show themselves as "not new anymore" or "more of the same". There are incredibly few exceptions, and most of them are created artificially at the cost of a large publicity budget.

I actually get a surprising amount of statisfacation on re-writing my old code in a better way. Not even changing languages or using new fancy tools, its just that old code I still use means I know by know exactly what the real use cases are and many of the edge cases that I didn't know when I wrote it originally. So when I do the re-write, I can do it so much better because I really do know what its supposed to do without having to guess at future possible use-cases. Writing good code is less about knowing how to get it to do what its supposed to do and more about knowing what its supposed to do in the first place. In the end the new code is usually very much more compact, faster, and easier to understand. A job well done like that feels good to do.

Of course. We all feel like that and that kind of thing doesn't cause any problems. Issues arise is when somebody comes in asking for new features or bugfixes in code that you haven't used for years and which you are no longer interested in. Worse is when people ask for bugfixes in software that you haven't written yourself, have never used, but are now responsible for maintaining.

Bugfix =! rewriting code, though. Reworking an existing solution to be better is fun, but a bug fix often isn’t that.

(When talking about ”rewriting code” I think of a larger scale than for example three lines of code)

I tend to be picky about code reviews. I'm aware that it causes friction and can come across as unreasonable. It works, we're done, right? Well it's likely to be my problem to maintain this code at some point in the future. Overly complicated? Unstated assumptions? Lack of test coverage? Fits well with nearby things? Solving a problem is fun but software engineering is more than just the fun parts.

+1. Back when I was interning at various places, the conclusion that I came to was that it almost never made sense to push back on changes, because the reality was that I would be gone in 3 months and the reviewer would be stuck with that code for years. Even if I disagreed on something, the right thing to do was to clearly state my reasoning and then change it to whatever they wanted.

Ugh, had this argument recently, to my detriment.

Tests are great in theory said my manager, but if the new doesn't want to write them, why are you making such a fuss. Fast forward three weeks and new guy is gone.

Even the simplest of projects have these engineering challenges, but people cannot see the forest for the trees when it comes to maintenance.

Since I did not enter the industry with a computer science degree, I was relegated to less desirable maintenance work for the majority of my career. I have twenty five years of experience reading other people's sometimes great, frequently benign, and occasionally really bad code.

Unfortunately, most of my experience is wasted and not well respected. Most people do not care to look far ahead, they are far too consumed with 'just getting it up and running'. It's all engineering in the end. The house building analogy holds. Make smart tradeoffs. Don't paint yourself into a corner. Build exits for yourself: from platforms, APIs, libraries, and seldom trodden code paths. If you can write it yourself and it's not that complicated, write it yourself. Dependencies are a nightmare from a security and long term use perspective. They might be good for starting out, but you really have no idea where your project and organization will take you.

If I were a hiring manager I would value this experience more than anything else. It's nice if you can bang out some algorithms, but I think it's much more useful for most organizations if you can make forward-thinking engineering decisions.

But always protecting the maintainers' best interests over those of users or new contributors ultimately leads to the death of the project.

There's a fairly good chance the reason someone wrote that 10k line PR is because they actually understand the problem users are having, and you don't as a maintainer.

Then fork and maintain the project yourself, or maintain it as an extension.

I'd argue the maintainers interests are the only ones that matter and the freedom of open source is that if you as a user don't like what's going on just fork.

“… found it extremely difficult dealing with bug fixes and feature requests relating to the other developer's code, who used a substantially different coding style. In the end we had to rewrite all our plugins for Construct 3, and so we ended up rewriting all the code anyway, which was particularly challenging given we hadn't written it ourselves in the first place and didn't have a deep understanding of how it worked. It's far easier now it's all our own code.”

I love this article and agree with almost all of it. However, this portion I find to be guilty of elitist programming mantra.

The bottom line is when you’re dealing with large code bases, there will be different coding styles present and the likelihood of everyone agreeing on a single coding style is virtually nil. Insistence on coding purity is an issue that can erode camaraderie in a team and develop into petty political in-fighting. Handling this problem in a tactful way is one of the keys to success in large code bases, as developers have their own styles and draconian insistence upon a single style can limit the talent that’s willing to work with you and your team.

There are coding styles that are mainstream, then there are coding styles that are not.

I worked with a guy who insisted in writing his C code in expect style because it was fun. Probably not the greatest choice for long-term maintenance.

What is "expect style"? Like Expect scripts in the Tcl scripting language?

yes

I read this part as the author saying very diplomatically that this code was flat out bad and obscure, and not simply a different style or paradigm.

I think your point holds if it was in fact the latter, but with the lengths the author goes to to stress the important of diplomatic language elsewhere in the article, I think they were just being polite to their former contractors here.

Yet, if I recall, they accepted the code into their source tree, but later judged that the style was so foreign to them that they were unable or at least unwilling to refactor or use the code in anyway. Seems if the code was flat out bad it would’ve been rejected initially.

As a 19-year Chromium dev, this rings true and highlights a lot of reasons "why don't you just..." is rarely the start of a good suggestion.

Speaking as someone who has been working on a system for almost 10 years now (and it's a programming language and UI framework) the amount of times I have seen "this was so easy, we did it two days with three people" proof-of-concepts that legitimately ballooned into "a team of eight devs, two quarters of work" when actually funded makes me say that even the 10K line pull request probably isn't a valid measure of "the work". There's so much more to shipping a useful software facility than just the actual function "A -> B". How does this system interact with other systems (since the world is not actually a pure function side effects from pure functions are in fact observable), how does anyone know to use the new thing, what happens when the sub-systems this system depends on are in a degraded state, and so on and so on and so forth. If you don't solve these problems (or at least guard-rail them) before shipping then whoever owns them after shipping will have to solve them.

> (since the world is not actually a pure function side effects from pure functions are in fact observable)

This one is so obvious and yet boy do a lot of people suffer from not having realized it yet!

One aspect not covered is how varied and wide ranging across the code and deployment platforms maintenance is done. One day you are tracking down a problem that's found to be caused by picking characters from UFT strings using indexes, and the next day the problem was an asynchronous job assuming all exception sub-classes of a common super-class can be ignored. These context switches physically hurt.

Just wait until it's a million lines of AI slop nobody actually understood in the first place!

That will surely make things better right?

> writing the initial code for a feature is about 25% of the total work involved for that feature.

Over a decade ago, I had a job where we were juggling several projects and needed to track our work in 15 minute intervals. The timesheets were mildly annoying but they did allow us to determine that number empirically.

We came up with an average ~20%. Remarkably little variation across projects too.

The implications to estimation should be obvious - any software planning process that resembles a feature factory and only estimates 20% of the work is systematically neglecting the remaining 80% - things like fixing bugs, performance optimization, backup drills, dealing with dependencies, code review, CI/CD/Release process, QA testing, docs, UX testing, training, technical communication, etc. Multiplying the initial estimate by 5x is not just some cheeky heuristic to pad estimates, it's damn near proven by empirical evidence IMO.

Constantly shoving more and more features into software systems without doing any of the engineering work to support it leads to bad outcomes. Who could have guessed /s

The only factor that explained the variation was the success of the project. More successful projects spent less time writing initial code relative to time in maintenance. Failed projects spent a great portion on writing code.

Again, the obvious implication is that if your project is a failure, you sink less time into maintaining it. The only way to reduce the maintenance burden is to not have users. There is some not-so-subtle self-sabotage going on here.

It's difficult to figure out causality in these cases. Did the project fail because of inadequate investment in engineering foundations? Or did we deprioritize work on it because we knew it wasn't going to take off? Likely both, working in a negative feedback loop.

> Failed projects spent a great portion on writing code.

Perhaps because the project requirements were not known or kept changing?

I'm reminded of the quip about "Why spend an hour in a meeting when you can spend weeks writing code to avoid it?"

In this case it was simply that successful projects got used, thus more bugs and maintenance issues were uncovered.

It's super easy to run a web service if it doesn't have any users :-)

Reply on news.ycombinator.com