Comment by jcadam | Hacker News Reader

Comment by jcadam | Hacker News Reader

Early in my career, I got stuck doing a lot of software maintenance on legacy systems (I suspect this permanently stunted my professional growth, but this was during the great recession and I took whatever work I could find). It's a hard rut to get out of - I spent a lot of time developing relevant skills in my spare time.

Anyway, one place I worked had a 10 page procedure for building the system. It involved running several scripts, logged into 2 or 3 different servers (we were running Solaris 8 - which was already quite old by then) as different users. About half the procedure was written down, the rest was unspoken and passed down via oral tradition.

A particularly intelligent and motivated guy once took it upon himself to unravel the whole mess and come up with a sane build solution. Unfortunately, his quest was not approved by management, the time taken away from bug fixing caused him to fall behind on "points" (i.e., number of tickets closed in the bug tracker), and he was laid off. So we stuck to the old system that sort of worked, most of the time.

Now, one of the user accounts that was required to build a certain module belonged to a person that no longer worked at the company (let's call him Steve - I don't remember his real name). We were certain there was a permissions or environment issue somewhere, but were unable to track down the specific reason why only Steve's user account could successfully build said module. We were successfully ignoring this obviously absurd situation until we got a new sysadmin....

The new sysadmin was unfortunately a very motivated guy who wanted to set things in order. He reviewed all of the user accounts, noticed "Steve" no longer worked for the company, and deleted his account. We were unable to build until the backup containing Steve's account was restored.

We never fixed the FUBARed build procedure, mostly because our customer (this was a defense contract, so the govt was the customer) didn't want to pay for such an effort - every thing we did had to be tied back to a specific req number in our system requirements specification (SRS). Fixing technical debt was not covered in the SRS, so it never got fixed.

Eventually I was caught up in a mass layoff and went on to better things :)

What I got from your story was:

1) Long term software maintenance should not be treated like help desk ticket work nor should it be dumped off on junior programmers. Every company should have at least one guy who enjoys maintaining legacy systems and that should be his sole job.

2) Make sure fixing technical debt is covered in the SRS.

> Every company should have at least one guy who enjoys maintaining legacy systems and that should be his sole job.

That's a career death sentence. I suppose if I was retired and wanted a part-time gig for extra money/amusement, I'd do it. But one thing I learned from working maintenance early in my career was... don't do maintenance work.

It is. Also, people don't want to do it for any amount of money.

I quit a previous job and took less money because I got moved to maintain the crumbling Forte4GL core system - I did get a handsome raise for that, but the writing was also on the wall as they were also working to replace it. Migration project has been ongoing for 8 years and is nowhere near done, I might have been able to retire before it's completed though. It was also soul-sucking.

People don’t want to do it for any amount of money that is offered for that work.

There is an amount of money that will clear that market, though.

Only if you optimize for your current salary instead of optimizing for your future career prospects.

There isn't a reasonable amount of money (it would have to be twice my current salary) for me to accept a job doing VB6 or any other job that would endanger my future employment prospects.

If I can retire one my savings after the job is done I don't care about future prospects. I'd rather work on [insert any hobby with no commercial value] at home than go to work.

$10MM/yr with a one-year notice and severance period gets you as many maintenance programmers as you’ll ever need...

Exactly, that’s why I said any “reasonable” amount. It usually takes me less than a month to find a job as a bog standard “enterprise developer” when I need the “right now” job or contract. It would take me a lot longer if my resume showed that I’ve been maintaining legacy software for two years and wasn’t doing Resume Driven Development.

I would have to spend a few months doing some work I could post on Github using new technology (something I never do).

But there are people that love doing it. My wife is an example of one. To her is it a great mystery novel where she gets to tease out all of the details and try to understand why things were built the way they were. Then try to make them 'better'.

Of course, she was a government employee (mil) and her career path was set regardless of what she actually did and the pay level was set in stone regardless of what she did as well. So there is that.

> Of course, she was a government employee (mil) and her career path was set regardless of what she actually did and the pay level was set in stone regardless of what she did as well. So there is that.

I mean, this is only relevant insofar as that this position allowed her to pursue her interest in what kind of programming she wanted to do. In any private company, the incentives are so perverse that she wouldn't have been able to do any of it, but that wouldn't change her preferences. It's just further indication that, given the right environment, there are people willing to do this kind of thing.

> Then try to make them 'better'.

this. I like this too. but then, in practice, I very rarely get a green flag to actully make it better. Not until is do or die. And by that point necessity forces a rush job, so it's not actually made better, just patched up to live another 'day'

> It is. Also, people don't want to do it for any amount of money.

Like many things, there are people who actually are interested in doing this kind of thing. But since it isn't rewarded in any way, hell, because it's actively punished by many organizations (often by the time you need to look for a new job and you lack the right bullet points to hire you), you're not going to find anybody willing to do it, who actually would like to do it.

I wouldnt entirely agree with that.

I think it depends on the KIND of maintenance work. In my case, I have a few languages under my belt (both OOP and FP). But this means I have a reputation in the area as a "Mikey for Maintenance: He'll fix anything"

Which actually wound up giving me the opportunity to study a lot of both good and bad designs.

I still maintain things, but now people ask me for design advice and help with POCs too.

Which, trust me, is less stressful and more socially rewarding than being a technical lead on a project (done that too).

But that's not a common thing. I do agree with GPs assessment that there are good maintenance devs. But they aren't common, and it can be tough to tell if someone with 20 years of maintenance actually enjoys it... or just could never be good enough at it to get a chance at a different role.

if by maintenance work, you mean "patch up the system so it keep trudging a long" then sure. it sucks.

And I feel that when people refer to maintenance work that is what they end up doing.

I like improving systems (I think it is a much a bigger challenge than making a new system from scratch). I suppose this is a kind of maintenance work.

But I am still trying to figure out how to get the green flag to do this. I often want to re-architecture big parts of the system; but at the same time, I don't want get overworked (neither does anybody really).

I find that I don't really have an incentive to refactor a semi-large part of the code into a better design (how do I even know if it's better? seems subjective). My incentives (deliver the story, complete the sprint's points) push me to just change as little as I can (patch it up) and then forget about it instead of refactoring a lot of it.

And every time I have done that I keep thinking as I do it: "this is exactly how this code got to this point. A few generations of developers with no incentive to refactor this from a slightly bigger picture, just carefully hack in what the story requires and move on". aah the joy of software engineering in real life :/

The other take away is this; for many situations, investments made toward cultivating the existing system can save the company more money in personnel and run time costs than can be gained from developing new functionality for the same system.

There's probably a bit more nuance to this.

I've worked for Gov, Consultancies and Private sector clients. They all have shared part of the blame at some point.

But, I've specifically seen the companies charge £75k for firewall changes or sever upgrades, when the reality was they're already paid £10,000,000 a year manage servers that a couple of good puppet admins could have done.

Point being, it's sometimes not about not wanting to pay, but simply about not wanting to pay for something that should have already been done.

Of course, this all comes back to the fact they gleefully oursorced their tech competency, based on the recommendation of consultancies, based on their lack of long term strategic management, so they apparently get what they deserve.

It's a shame that Gov is spending tax money though.

Outsourcing of technical capability (especially prevalent in government) bites you hard when you lose the ability to work directly with your contract teams and have to rely on their direction. (Especially if the management is non-technical)

There’s a clear conflict of interest on the part of the contractor, and you need to be able to set their direction competently. I would recommend always having an in-house capability, even if it’s just an in-house consultancy, so that you don’t run into these kinds of issues and can provide adequate direction / oversight.

The GS payscale is not set up to be able to hire and retain highly skilled technical experts. Even if I were to be hired at the GS-15 level, I'd take a pay cut from what I earn as a contractor.

I once worked on a contract in a Systems Engineering and Technical Assistance (SETA) capacity, which was both an odd and eye-opening experience - basically advising the govt customer, as a technical expert, on contractors' proposals and sitting in on reviews and such.

That's a good fallback option, though it's still difficult to control turnover and feel like you are aligned in terms of value using that approach. You'd want to find somebody that you trust and bring them on in that special capacity, rather than just putting an RFP on the street for a SETA advisor.

Agreed on the GS payscale and its ability to keep up with high-end private sector pay though I also think that the entire hiring process may be more of a barrier than the pay itself.

All that said, a non-supervisory GS-15 in the DC area (18% locality pay bump - more if you're SF or NY) is maxed out in the 170s which isn't shabby!!

When you combine that wage with generous retirement matching options, job security, good work/life balance, and excellent health insurance at well negotiated rates it ends up being a pretty attractive offer.

Furthermore, there are other government agencies and roles that have their own specialized payscale outside of the GS scale. This can include financial regulators (higher pay to avoid revolving door problems) as well as "cybersecurity" billets across different places.

I find the trick is to do maintenance while doing meaningful (as described by the powers that be) work. So when a new feature or bug fix is asked for, add another 15% to the time required, and clean up the code base a bit.

The knee-jerk, which rarely works, is to demand time to refactor everything or ignore everything until it breaks irreparably.

I think in your case, simplifying a script by a dozen lines a week or so could have been done without taking away "points". Abandoning the status quo and committing too much time will inevitably be looked down on in most industries, however.

The poster was doing a military contract, not working for his own company. His companies money - and thus his bonus - is in putting in hours to do what is requested. Saving money means they get paid less hours now the project is smaller and so the bonus is less because they are paid less.

In private industry it can (but often does not) mean that you can get more profit and thus a larger bonus by saving your company money long term. However in the world of contractors saving the client money this is not rewarded.

this story is so familiar that it’s not even funny. I also have seen this while the project is developed. No incentives for fixing stuff that would exponentially increase the productivity of everyone - just keep adding shit until you move on. Pretend like learning the obscurities of a home-brewed/hastily hacked mess matter.

a few years of that until one day, for example Apple realeases Catalina?

> I suspect this permanently stunted my professional growth, but this was during the great recession and I took whatever work I could find

From reading your post, I think this was excellent experience in understanding office politics; and in software development.

One of the ways to learn how to write maintainable software is to maintain someone else's software!

I’m also curious why OP should think that this might permanently stunt a career.

> permanently stunted my professional growth

This is such a counter-productive attitude.

Reply on news.ycombinator.com