Hacker News Reader

Comment by JohnBooty | Hacker News Reader

Maintenance programming for pay can be rewarding iffffffffffffffff management has a deep understanding of and appreciation for the task. There can be a lot of joy and satisfaction in adding tests, fixing edge cases, fulfilling user needs, "quality of life" improvements, optimizing. Honing.

My experience is that this is almost never the case.

Generally, management does not understand or appreciate why it's a slog, why you might want to refactor this or that, etc.

Obviously a lot of this falls upon the engineer. It's your job to communicate and demonstrate. Helps if you have other engineers backing you up and presenting a unified message. Also helps if management is technical enough to understand what you're saying instead of simply believing -- or not believing -- you.

But maaaan. Uphill battle! I'm burnt out on it. I've decided I need to seek greenfield stuff. Maybe forever. But definitely for now.

View on HN

It's worse that that. Most companies give career incentives to new projects. Google is probably the most well known offender, but they aren't the only one.

Very few companies understand that the most valuable asset of a programmer is his mental model of your system. You need people who can do the hard work because they understand the system at an intuitive level.

> the most valuable asset of a programmer is his mental model of your system.

Which is actually something a lot of management would prefer to not have. They often want people to deeply document a system, on the mistaken belief that this knowledge could be transferred out easily.

Unfortunately, it's not. Even a well documented system has inherent complexity that only working on the system would reveal. Otherwise, merely reading the source code would allow for a complete understanding and deskchecking of the whole system!

We can also break it down even more, managers are only allowed to pick one of the mutually exclusive options:

- Well documented, well maintained code that everybody could work on

- Quickly finished projects that nobody ever touches again, but only the programmer(s) have a mental model of it

The first option is more expensive at first, but way less expensive if you happen to have to rewrite it from scratch at some point (and if you are not maintaining and document it, that moment will come and it might not come at a point in time that fits the schedule).

The second option is less expensive first, but it will bite you in the arse if the programmer(s) are gone or forgot how it is done.

Everywhere else managers try to minimize risks, here they often just don't aeem to get the trade-offs involved.

The missing bit you forgot is that even if the first option was chosen, it very quickly devolves into the second option, whether the managers like it or not.

The problems with documentation is one, organizing and keeping them up to date is another, and... actually reading it is the other thing. I'm guilty of that myself, because I already assume the documentation will be out of date. But so does everyone else, thinking it's pointless to write something down because it's easier to just ask someone else.

If management cared, not updating documentation with every merge would be a strike, with multiple offenses leading to firing. They don't care. All large development is just slapped together and shipped. By the time the reaper of long term problems comes, the problems are too big to fix.

Google has turned it into a sport though. That deserves special mention.

They’ve even got their own mascot. It’s a gravestone.

Are you being serious right now? Pics. I need pics.

Here's a Halloween thing they did:

https://www.theverge.com/tldr/2019/10/4/20899198/google-read...

I've got mixed feelings about this. Google's frequent "killings" have caused me to never trust them or want to buy in to their products.

But at the same time I think that it is good to "honor" "dead" products. Those products encompassed millions of hours of blood and sweat and toil and so on. I don't know the best way to memorialize that but sanitizing them from history entirely ain't it.

A lot of us got there a few years ago. So the perverse incentive is that many of the people who would shun them have already, so why not keep misbehaving?

I was almost bounced by HR during a screening interview a couple of years ago when they found out I had been doing maintenance work for the previous six months. I had to explain that it was a tiny part of my overall work experience. The hiring hypocrisy is strong.

Also as you point out, getting funding for preventative maintenance or continuous improvement is extremely difficult in a corporate IT environment, especially if there isn't a product owner to advocate for the system and to secure funding. Although I do believe you need a product owner to drive the work to ensure it delivers business value, and not have it driven solely by developers.

That's absolutely mental. I know maintenance work is uncool and kind of a career... well, not a stain, but it's not glamorous.

May I ask what sort of company this was? Some "unicorn" startup? Stable oldcorp?

Agreed. Imagine you’re a CEO: Why on earth would you want your expensive engineers reworking something that already makes money? It seems much more sensible to leave the working code alone and focus on new revenue-generating opportunities.

That's not what the post is about. They describe the process as mostly fixing issues with existing software, so the motivation is that something is broken or not working as required anymore.

Refactoring it is just one of possible parts of the process, much opposed to 'yeah this legacy code is broken dogshit and needs to be rewritten even though the current software is already written, proven and works for 99% of the stuff we need' approach.

    [Refactoring] is not what the post is about.

Not directly, but kinda, at least in my experience.

Clients and management generally understand that old code needs to be maintained. They don't love that reality, but that's not generally a tough sell unless they're completely clueless. And they're usually not completely clueless.

The tension in my experience comes when an engineer wants to perform something other than the fastest, sloppiest, filthiest duct-tape maintenance imaginable.

Management accepts the need for fixes, but often does not have the technical/domain knowledge to understand the short- and long-term merits of Proposed Fix A versus Proposed Fix B. And naturally, incentives heavily push management toward the quick fixes by default. Time is money, after all. And managers are expected to meet quarterly and annual OKRs. We don't have a good way to quantify the long-term value of a "proper" fix versus a "duct tape" fix. And so on.

So for me maintenance work is difficult to separate from notions of longer-term software maintenance such as refactors, dependency upgrades, etc.

> Refactoring

I personally don't believe refactoring should ever be part of "maintenance".

Maintenance needs to avoid YAGNI and avoid the engineers natural desire for over-engineering and unnecessary perfect polishing and abstruse abstraction.

Refactoring is something that should be driven by new features (or worst case towards rewrite), and few have the skill to do it well.

I'm not arguing the sibling strawman that maintenance be "the fastest, sloppiest, filthiest duct-tape maintenance imaginable."

Excelling at maintenance is highly skillful work and often poorly rewarded.

> I personally don't believe refactoring should ever be part of "maintenance".

Of course it shouldn’t, because software maintenance does not exist. Programs don’t degrade with use. All “maintenance” on software is, like new development, the implementation of new context->behavior combinations, and calling it something different is a result of applying an inappropriate physical-product metaphor.

    Of course it shouldn’t, because software maintenance 
    does not exist. Programs don’t degrade with use

This is some real pedantic gymnastics and you're not even correct.

The word "maintenance" doesn't imply some kind of physical product metaphor. Sorry. That's not what the word means.

> software maintenance does not exist

There are several activities in software that are similar to maintenance of physical products. e.g.

* that parsing library that you used 2 years ago and was in good working order then, is now outdated and has known vulnerabilities that are not present in the latest version, apply update.

* Those cloud-hosted databases that you use are an older version that the cloud vendor has deprecated and will some time withdraw support for, apply migration to a later one.

Well, I certainly agree that engineers tend towards YAGNI and this must be avoided.

     Refactoring is something that should be driven by 
     new features (or worst case towards rewrite),

This overlooks a whole other class of factors that can drive maintenance: changes in the environment.

Assuming we're talking about a program that does something useful and isn't just a screensaver, it's probably ingesting data from the outside world. That data can change in a variety of ways. API partners and users can introduce changes on their end and will eventually find new and annoying edge cases. Or maybe the volume of data increases by 100x and changes must be made on your end.

Or maybe your software sort of rots as the security vulnerabilities pile up. Eventually you've got to make major version or point upgrades to the framework and/or various dependencies.

We want to handle these demands judiciously and as a rule, yeah, refactoring isn't generally going to be the answer but sometimes it is.

> Imagine you’re a CEO: Why on earth would you want your expensive engineers reworking something that already makes money?

CEO will start looking at this topic if app performance/stability metrics will start making impact on revenue stream, aka app is slow and stop working every day and customers go away because of that.

Which happens if the CEO is conscientious enough to insist on collecting and looking at those metrics.

Heh. Going to be a lot less maintenance work if they are.

True, but app performance/stability metrics are (often extremely laggy) trailing indicators of code quality. So by the time the app starts falling over, things are extremely rotten, and it's not necessarily correlateable to any specific maintenance decision(s).

By the time the app starts failing, usually the people who wrote it got burned out and left :/

It is not at all obvious that trying to develop new revenue-generating opportunities will actually bring in more revenue long term compared to optimising and improving existing revenue generating sources. And it's not like the old code stops working once you start reworking it. Give your engineers N months to rework the old code, and if it's not better you don't have to deploy it.

Every larger company I've worked at had enough engineering overhead that they could afford to have small teams working on both reworking old stuff and creating new stuff, while the main teams kept the working stuff generating revenue.

Security and risk. You can run several different code/pom/package.json scanner tools that will articulate a risk that will send your IT security department into a panic attack, which you can sell as time you need to spend fixing things.

CEOs tend to be short-sighted when it comes to engineering management, that's true.

and yet the CEO would not find it ridiculous to spend money maintaining plant and equipment in the factory.

It's just a lack of understanding. They imagine they understand, because software doesn't get worn out, and doesnt degrade obviously!

> focus on new revenue-generating opportunities

in mature industries, there aren't new revenue generating opportunities, there is only getting better at what you ordinarily do; all marketshare growth comes at the expense of competitors.

I think that’s overstating things a bit. Even in mature industries, new technology can change the field. But it certainly isn’t all about a hyperfixation on growth.

Why would you turn off manufacturing equipment to do routine maintenance? It’s making you money when it’s turned on, and not when it’s turned off, after all.

Did ball bearings in application's garbage collector wear out?

As an embedded dev I have thousands of devices that were commissioned and are on 24/7 duty. The oldest still running fleets I believe date back to 2007. Some of them had firmware updates, but many of them (despite numerous bugfixes and improvements since) run on original software. Customers are reluctant to update if there's no specific need arises in their static operational environment.

Naturally in business logic world the requirements are never as static. But when they do change you can likely sell the management on it as necessary development.

The “seals” wear out all the damned time, which is basically the same thing.

Does anyone have any tips on this beyond the additional "sales" type stuff where you have to put in a ton of extra effort to measure and sell your work?

I've found this to be true... to the point where some places even prefer manual maintenance to spending (potentially) a little more time on automation, even if that time would be paid back over the next couple of years.

I'm not adept at planning, selling my work, doing the manual maintenance, and doing the extra automation, as eventually that work becomes homework unless you get a boss that's willing to put in some effort here, which I haven't found yet.

I think one of the keys is consensus-building. If one engineer tells management that it would be a great idea to do XYZ, management doesn't necessarily know whether to believe them.

But if the whole team is on board and has reviewed your proposal it's going to be more credible. And that's just good engineering anyway.

Nobody cares about "eliminated a whole class of tickets" aside from the support staff, and really who cares about the support staff at most companies. What would the phone monkeys know about our software. /s

Reply on news.ycombinator.com