Legacy Software Systems: How to Live with Aging Software Architecture? | Hacker News Reader

Legacy Software Systems: How to Live with Aging Software Architecture?

stratoflow.com

104 pointsmgl71 comments

Having been involved in 3+ $8-figure legacy replacement projects, this article seems to be missing business, management, and cost perspectives. In particular, "non-compliant with recent standards" is the least important reason to consider rewriting a working system. Often, chasing whatever is popular at the moment is the root cause of maintenance problems in the first place.

Wish I could upvote more than once. What I call "Resume Driven Development" creates fractured architectures and reduces quality in stable systems for the sake of "the latest standard". Not the best spend for the business or users/customers.

On the flipside, when your website only works with Internet Explorer 11, forcing your customers to keep an unsupported browser around, it's time to reconsider the aversion to keeping up with changes in the technology.

It's doubly bad when your website is designed for software embedded in kiosks and the like.

Keeping your software up to date as dependencies age and go out of support or the platform you build on evolves is not resume driven development. There’s a business and engineering case that can easily be made for that.

Of course there’s no real dictionary we can go to for this, but I’d define RDD as “chasing technology trends without business or engineering justification”.

Why are we switching to JS-Framework-De-Jour? Why are we using this graphdocumentjsondistibutedimmutablelog database all of a sudden? How does this improve the business? How do see a path to a positive ROI from this decision?

> Keeping your software up to date as dependencies age and go out of support or the platform you build on evolves is not resume driven development. There’s a business and engineering case that can easily be made for that.

I'm not sure your false dichotomy holds. The definition of resume driven development is not whether there's a business case of not for a change, and the definition of legacy system is not the cost of upgrading it.

> Why are we switching to JS-Framework-De-Jour? Why are we using this graphdocumentjsondistibutedimmutablelog database all of a sudden? How does this improve the business? How do see a path to a positive ROI from this decision?

False dichotomies. You're assuming that not maintaining your software is free from cost or business impact, and you're assuming that switching to the latest and greatest does not bring any operational advantage.

I think you just said "stay up-to-date with the latest standards", but in more words.

If your legacy app is incompatible with modern browsers, yeah, that's got to change. No debating that, really.

But,

1. That may or may not mean a full rewrite of the app. Even in terrible old legacy apps, there usually is at least somewhat of a separation between the backend and frontend logic.

2. While not exactly uncommon, the scenario you describe is a subset of legacy apps. A lot of them work just fine, other than the fact that they're not in the latest version of the latest cool framework.

I highly doubt they work as well as an engineer believes. If there are no absolutely no user requested changes - even something as simple as support a new browser type or device type - then maybe. Otherwise, I've seen this viewpoint kill multiple companies because it lead to uncontrollable technical debt.

I doubt all the code behind, dataset ridden ASP.NET got so easily replaced.... Or most JSP etc. There is an absurd amount of applications out there, hacked together over the last 20 years.

Oof, I've seen this at a previous job - lots of microservices written in the newest/most dominant language of the time: Scala, Java, .Net, Python, Javascript, Coffeescript (!), TypeScript and Go services all powering one system.

Clearly the answer is to rewrite them all in Rust. /s

Surprisingly I don't think there was any Rust - they seemed to have settled on TypeScript.

They are still trying to compile the rustc because the new rust project won't accept the old compiler. /s

(i'm only half joking after seeing an error message like: you have rustc version 1.58.0, you need at least 1.58.1)

Yeah, but on the other hand the "we ignore the resume thing" makes it increasingly hard to hire and keep people. Because when tech becomes too much legacy, developers can not afford to stay there. Your employees will be selected for more passive bunch. Or you have to pay more to keep them.

And I do work on project exactly like that. I am paid more then would be market rate for same position and everyone is scared we (as in seniors) are going to leave. It has advantages, not just in pay, definitely. But some of these advantages are disadvantages for the company to be frank ... like that the whole environment gets biased toward more passive and sort of sleepy. (If you are non sleepy you get pay raise, they are trying to reward it, but the force of the system is strong.)

It does not result in the best spend for customer or a company.

The sad fact of software projects is that some part of the work is always a bit of bribery to keep skilled labor engaged. You’re always throwing them bones, letting them work on a few things that don’t really need to be done but make them happy. That’s lost sometimes when I’m lamenting NIH or wheel reinvention, but it’s a matter of degrees. Let them have one big thing and a bunch of small things.

Companies go off the rails and get trashed talked when they clamp down and go for zero fun, 100% financially justifiable task lists.

As a tech ages out you can’t find anyone good to work on it. And when you make them, morale craters.

From my perspective, there's loads of this legacy/boring crap (lots of PHP&Rails). As a "old" coder, I love those legacy apps that folk can't/don't give to their internal team - it becomes consulting work for me :)

any tips on how to get started here? I have plenty of past Rails experience and although I'd prefer rewriting the stack endpoint by endpoint in something like Phoenix, I'd probably be amenable to other kinds of work yet staying in the Ruby ecosystem

I just kept telling my network to hand over their legacy projects. I'm not in the resume builder phase of the career. And folk (managers and devs) on the legacy project are happy to hand it over (provided they trust you). Helps all of their OKRs.

And sometimes you get landed a fun legacy one where you have to rebuild some binary by de-compiling first (thanks for not stripping the binary previous long-gone dev) and then spend days re-factoring it to work with updated AT commands on the new hardware. And nobody's rushing you, cause you're like the only one doing it; and finding another person to even WANT to do this is $HARDWORK.

I'm still out here doing VB6/VBA, Perl as well.

You can get people to stay longer as long as they are getting material to pad their resume. This seems counterintuitive at first, but one thing devs are good (bad?) at is avoiding pressure sales or getting “trapped”. An employee who feels stuck has lowered expectations from another job, and so it’s easier for them to leave. They don’t wait for “better”, they wait for “not here”.

Amongst a group of people I talked about this with, we estimated that places that allowed resume farming saw between 6 and 9 months longer average employe retention times. For a place aiming at 3-4 year turnover that’s a substantial fraction of your goal.

That's the trouble though isn't it—leaving it to mature gracefully in the cellar gets you to a point where only an expert somilier in that particular varietal knows really what to do with it, and you pay through the nose for their knowledge.

What do you do with a 20 year old embedded Linux system which still "works" and where management has been saying they'll replace it for more than ten years but meanwhile don't touch it, and, oh yeah, "make it secure"? I can actually imagine somewhat exotic network firewalls that would be sufficiently secure (might even be a fun project) but there's zero provision for the required effort.

>meanwhile don't touch it, and, oh yeah, "make it secure"?

Yeah never take such a job, this is the manager trying to screw you.

Security is a big word, from stability to observe-ability to hack-ability, there is no way to secure a system without touching it.

I characterize refactoring as an exercise in staying familiar with the code. With security we talk about data at rest versus data in motion, which is a hard problem. You can stare at code a lot and think you understand it, but when you see it actually run it's a different experience. Poking at things teaches you a lot faster and more deeply than treating it like a religious artifact.

Another problem is that too much software these days is built like the rest of our disposable society. We don't build things to last with a solid architecture that can withstand major modifications. Instead we throw together something that just meets an immediate need with minimal commitment of time and money. Little thought is given to how the project will react when business needs change down the road.

The saying 'penny-wise but pound-foolish' applies in many cases.

The comment just above you says that if we build things to last, we don’t find anyone to work on that old tech after the initial 5 years.

The software industry is stuck in a rolling thunder: We engineer things to last for 5 years, precisely because programmers expect to have to prepare their resume for the 5-year-newer-tech.

Only Linux and Postgres beat this doom.

Maybe that's an overreaction to the enterprise Java applications, which often introduce lots of needless complexity in the accept to incorporate flexibility for future changes

I tried to compile seamonkey and firefox from source. They both failed at the rustc compiler checking.

Apparently the trend in SW development is to compile one moving target with another moving target.

Kubernetes with WASM containers say hello.

I feel like I’m reading a rough draft, not a deeply thought out piece of writing. There are some nuggets of good sounding information in here, but the author doesn’t really go very deep.

Chad Fowler had an interesting slide in one of his talks (circa 2009 I think) where he had a graph of something like frequency of errors produced by working code versus number of changes to that code. Code could end up in one of four quadrants:

- frequent changes, lots of errors: suggesting that the domain was still being discovered or there were quality issues

- frequent changes, few errors: suggesting the domain was changing or growing but quality was good

- few changes, lots of errors: suggesting that the code was not adequately maintained or even understood, perhaps poor quality as well

- few changes, few errors: this is the “legacy” code that works hard and costs less to keep running

Clearly it’s a good thing to have that “legacy” code, by this definition, but I remember that he listed some downsides as well, but not the downsides that the junior dev who wants to rewrite then whole system comes up with.

Edit: I think it might have been "Measuring & Analyzing Things That Matter When You Have Too Many Things To Keep Track Of"

At 36:00, he talks about a ruby gem called for Rails "Turbulence" that measures code complexity against number of modifications https://vimeo.com/38252887#t=2160s

He also references this article by Michael Feathers: "Getting Empirical about Refactoring" https://www.stickyminds.com/article/getting-empirical-about-...

The best source for this kind of emprical measurement of code complexity is Your Code as a Crime Scene by Adam Tornhill. See also http://www.adamtornhill.com/code/codemaat.htm

Very cool! It looks like that's evolved into a company called CodeScene. Interesting demo by Adam: https://youtu.be/SWFwPkgLcpo?t=10

If you start a rewrite or a large refactor, for Gods sake, follow through and _finish_. Otherwise you have made things even worse. I can't tell you how often I've come across the remnants of such projects, abandoned with some sliver left. I make a point to never leave it like that.

Few places I've worked ever put improving the system in the plan. It's almost always said that "we'll work on it later, when we have time". Later almost never comes, of course. On the rare cases when there is a little time, the team will start something but have to abandon it when inevitably the business needs change and they get drawn to other work. Remnants of incomplete attempts speckle the codebase like bird droppings on a car parked under a tree for 6 months.

I think that's part of our responsibility as engineers to ensure that technical debt doesn't pile up. Business will want to focus on pushing new stuff and engineers have to be a counterbalancing force. It's literally your job.

This hasn't been a problem in the places I've worked. Perhaps I'm lucky or it's the fact that I don't live in a very hierarchical society (Sweden).

You're correct that it's the engineers need to be responsible, starting with not offering lowball estimates for development and then taking shortcuts and ignoring problems just to get the features out the door. There are plenty of businesses perfectly happy with hiring developers that will ship things as fast as possible over developers that will take responsibility for the code.

I've worked both places. Unsurprisingly, I don't stay too long at places where I find myself working with a team full of shortcutters.

Yes I remember working on a legacy system (PHP application started in 2003) - there were two or three different attempts to make it MVC sprinkled throughout the codebase - obviously a few people had come and decided to improve things (and decided the last guy's work was no good) but left before making any meaningful progress.

80% of the time the answer is "find out if the owners of the code are supportive of improving the code", and when the answer is "no", just put in your hours and collect a paycheck.

Dunno, I successfully refactored some legacy services. Some of them even were a big index.js spaghetti files.

You have to start with the smallest and most conservative to the most "refactory" ones:

- figure out what enters and what exits from the program, and create a test suite that reflects every known combination

- move parts of main() to funcions

- figure out what variables can be converted from mutations to returned values

- move thr ifs(){} to maps, early returns and such

- figure out the database schemas from newly written data

- infer model relationships fro further writes

- attempt to implement some pattern design, even if its just dummy variables

Sometimes you even have to do it at the last stretch of time of your workday, and slowly the mess will become clearer, your test suit more exhaustive and comprehensive.

And at some point it will achieve feature parity with the legacy service

The GP is referring to the political costs of changing bad code, not the technical challenges which are almost always manageable by any experienced programmer. Egos, however.

Yeah. But there were times where I did some hidden refactoring even if people said "no" if I asked for permission.

In part to maintain sanity in a toxic environment, and in part because I knew they just accepted it if it was handed to them.

You’re the guy in corollary 1 in this article.

https://yosefk.com/blog/people-can-read-their-managers-mind....

Either you’re successful AND your managers have an epiphany that you were right and they were wrong, or you fail and your managers will punish you for ignoring their instructions. It’s a risky strategy for an employee.

Until the friction of working in the terrible code becomes bad enough to slow down development to the point that the code owners complain about why it's taking so long to get work done. Of course they still won't support improving the code, but at least you have empirical evidence that never taking out the garbage turns makes the system look like a house in an episode of Hoarders

Well, the car analogy is very good I think. A 35-year old Mercedes-Benz W124 that has been taken care of during its entire lifetime is actually usable and could be reliably maintained. A 10-year old Hyundai with 50k km old oil is meant to be scrapped though.

In the same way, there is legacy software and legacy software. In a decade of contracting I have seen both pieces of crap with 50%+ of dead code which are inmodifiable beyond adding/removing fields from some CRUD entity, and 15+ year old J2EE competently designed, maintained and documented systems, where the only difference to a fancy new project is that there is xdoclet instead of annotations and a ton of plumbing code around.

The big difference is in telling them apart in advance - with cars you know that some brands are more maintainable than others as you see them on the road and as all of them share the same components/weak spots it doesn't take long to figure out if it is an oldtimer or a piece of junk. With software it is much more complicated.

10 years ago I’ve attended a talk from one of German car producer - they talked about 25+ years of software maintenance. You invest into tools 5 years before release to market, and they should work 20 years after release. So companies need to plan the whole cycle not only code maintenance

It goes all the way down the stack, too. I talked to someone working on CI at Nvidia and they were testing up build toolchain to ensure some of their hardware accelerated libraries could be rebuilt in 10+ years (for automotive industry customers). The work mostly involved dynamic systems for ensuring the builds worked without an internet connection (monitoring network activity to make sure everything was cached/saved correctly).

> The work mostly involved dynamic systems for ensuring the builds worked without an internet connection (monitoring network activity to make sure everything was cached/saved correctly).

Documenting and archiving your build tools is also important. I knew a team that checked everything into source control, even Visual Studio. This ensures the tools were all preserved, everyone is using the same versions, and the code is synchronized with the tool versions.

Telling non-technical people about UNIX is often eye-opening.

Development for UNIX started in 1969. It was done in a documented way, using well defined APIs (that later became standard) by world class engineers.

The original code (in C) is still readable. Running it on modern hardware would be a challenge, since computer architectures changed quite a bit, and modern CPU architectures were simply not invented yet! But it's not legacy code; the interfaces, ideas and libraries from 1969 have been improved on and are the foundation of the modern computing landscape.

Yet spaghetti code from two years ago, written by "best value" programmers in a language that's 100% still supported is very much legacy code. Unintelligible comments, no tests, copy-pasted code, inline queries, the list goes on. Often simpler to just re-write.

The thing about never doing that big refactor, is your team loses the ability to do it. Only maintaining legacy codebases results in people who can’t build things.

This is a big, big company problem in my opinion. If you have a dev force mostly just adding to an existing system, they never learn design skills.

Those that built the system inevitably leave over time, and now you’re stuck unable to attract talent because “old tech stack”, and without staff that could build something new.

Don’t stay at a company that’s gotten into that state. If your young, try and gain a mix of experience so you can actually gain those design skills.

What if you're on a team like this right now but the job market is bad...

Shop around anyway. It may turn out less bad than you think.

The legacy software works very well in a static world: when nothing ever changes there is no need to update the software.

Of course, this is not the case. The problems arise when the software updates are not budgeted. Especially the finance sector is notorious for underinvesting in R&D and instead of spending money on stock buybacks - stockholders appreciate.

However this cycle is broken when the world actually changes. If there is no proper maintenance, usually in the form of software R&D, the legacy software change cost has gone prohibitive high and the parent company is unable to do anything. This change is often Schumpeter's creative destruction. There is a new technology. A new startup raises as a competitor, using more efficient software where they can do changes, and execute their core business, with much lower cost, outpacing the established competitors.

Some industrial, like early mentioned finance, are however very-well protected due to barries of entry like the cost of initial capital and regulation. Thus, you are going to see more legacy software in industries where there is never need for a change.

When the b-52 aircraft retires it will have been flying for about 100 years. I can't think of software system that has the same longevity. Maybe a credit card mainframe written in cobol. But what JS developer is going to learn that these days.

I think a problem for software as an industry in it's infancy is that we've built a lot of stuff, and it's really easy to build new stuff, but it's hard to just keep the trains running on time. No one wants to pay for that. So you take the team that delivered the system and move them on. Or you stop them hiring and they age out of the product. The ghost plane fly's on for a long time untill you start to see mountains and need to change course. Then it's panic stations.

Java cracks out a new release every 6? months now. There's a lot of java 8 out there. And it's not going to get upgraded without costly effort.

btw, I have no solutions. I just think we're still at the early stage of this problem. And in truth in the built world. people rip down good buildings to rebuild shiny new ones all the time.

Is this what it's written in: https://en.wikipedia.org/wiki/JOVIAL? Maybe not all of it but some of it at least I'm thinking.

I haven't been programming for long enough to see it or maybe it changes too slowly to notice, but I feel like the new programming languages of the last 15 years have been similar enough (at a high level) while the programming languages of 50 or 60 years ago are vastly different. If that's the case, maybe some of today's software has a better chance of lasting longer.

> There's a lot of java 8 out there. And it's not going to get upgraded without costly effort.

There's a Java LTS version for people that don't want to upgrade every 6 months. Almost all Java 8 code will run on Java 17 LTS so as a developer I don't see that 'costly effort' you're talking about. It's just build, fix and test as usual.

In this space I highly recommend the book "Kill It With Fire".

Just bought the audiobook. Looks okay so far.

"Kill It With Fire" is great. Here's a 2021 thread about a review of the book:

https://news.ycombinator.com/item?id=28571931

And two presentations from the author, Marianne Bellotti

"We Killed These Things With Fire | Øredev 2019" https://www.youtube.com/watch?v=XoEfV0kXXDY

"Future Proofing Systems with Marianne Bellotti" (2020) https://www.youtube.com/watch?v=_ICzo5r-7vY

> Software developers are often regarded as architects, as their work somewhat resembles that of a civil engineer.

I like this analogy.

#1 Keep track of the most important updates

#2 Don’t rewrite what you don’t have to

#3 Maintain good documentation, gather the necessary knowledge

#2 is not universal. There’s a lot of legacy software with outrageous workarounds and sometimes it’s simpler to rewrite part of the system than to even maintain it for a month. It’s all about good judgement.

That’s often the first impression but in my career I’ve never seen a rewrite work substantially better or save any money. Sometimes they are necessary for other reasons but those two are the arguments usually made and they don’t really hold up.

I've got experiences in both camps. Well managed rewrites (typically portions of a system) can fix a lot of problems in one fell swoop, but you have to really know what you're doing and actually have concrete plans for improvements.

If not, all you've done is just replaced a bad system with another bad system. I've also seen that lately. You're right that often it's just "different" and not better, and that makes it hard to sell the case when a rewrite is truly needed.

Agreed - I've worked with one rewrite from scratch and it convinced me that rewriting from scratch is never a good idea. Incidentally, I can see why the company decided on a rewrite from scratch (no separation of MVC logic, no framework, almost no tests, only ran on special environments, production had an extra set of patches that for some reason would never get merged back to master) - honestly, it really was too far gone.

> it’s simpler to rewrite part of the system than to even maintain it for a month.

This is often a career limiting decision. Rewrites are often much more difficult, particularly when you are dealing with a system that is not fully understood. This often leads to what should be a quick re-write taking > 3x more time and money than projected. In the end no one will remember why the technical reason for the rewrite, but they will remember who made the decision.

I've made this mistake a few times in my career, and I always thought, this project, it is the exception to the no-rewrites rule.

The trick is to make someone else make the decision.

Until you find out there is some essential complexity you missed that’s encoded in the convoluted logic… and then your rewrite doesn’t function and you look foolish.

Depending on the case, it at least has the benefit that the essential complexity is now known and can be documented.

Yes but before you can make that call you need to have a reliable estimate of how costly it will be to rewrite the system. This in turn requires a through understanding of the current behavior of the system, which means you will have to study and understand the code anyway.

Rule 0 in Engineering: If it ain't broke, don't fix it.

Fun question to think about what you have to rewrite vs not. This reminded me of my last firm where we had a C# system with tons of Task Parallel Library code, nearly all of it written poorly or not written for the correct use-cases. My colleague and I weren’t part of the development team responsible for putting it together but it was clear it has been haphazardly built that way over time.

We were quite convinced the team needed to rewrite but then things like cost, time, priorities stepped in. We needed other areas addressed. So, the code stayer because it worked, enough, unfortunately.

Reply on news.ycombinator.com