The World of Legacy IT Systems
spectrum.ieee.org
spectrum.ieee.org
The legacy system trap is built on three pillars:
1. Maintenance costs are always lower than upgrade costs. It's even cheaper to pay for neither. Even if you do spend money aggressively to maintain it; every feature added, operator hired, workaround published, becomes a growing suit of armor protecting the system from replacement.
2. Management incentives are rarely aligned with the planning and sacrifices it takes to unwind technical debt. Everyone wants to cut the ribbon on the shiny new big-data/AI/ML/buzzword project, but no one wants to take responsibility for integrating it with the rusting hulks creaking along actually running day to day operations.
3. The software industry can shift paradigms enough that you end up looking to buy a new/faster horse for your buggy. And nothing fits your use case, or even offers a path out.
All three in combination mean you get businesses spending obscene amounts of money on technology they can't use.
It is quite satisfying to successfully maneuver an organization out of the trap. It's like solving a complex puzzle over months. Where hundreds or thousands of real people will be better off because of your team's effort.
This is because people think rewriting is a solution. It very never is. It can help, rarely, but it's never the solution, it just does a good job of hiding what the solution was.
If you can't map out what is really broken in your legacy system and map out a way to refactor it to what you want then a rewrite will always fail.
I often see this idea formulated, in this forum or in articles.
>what is really broken in your legacy system.
Software can be arbitrarily broken (think of only copy-pasting code from thedailywtf): http://www.cs.utexas.edu/users/dsb/CS373S/Images/softwaredes...
I knew a legacy system where everything was broken. There were no proper data model and structures, so every computation involved multiple nested loops, making everything much slower than needed. Everything was modifiable as if it was public, so it was hard to keep things consistent. Allocation/GC and latency were through the roof, they tried to add pooling but noone could figure out where objects could be released. One of the core classes was called Banana (changed name) but actually referred to bananas or to banana trees. And the list goes on (threading, database usage, still plenty to tell).
Then someone, new to both the domain and the language, stepped up to rewrite everything, which took a bit more than a month.
Ten years later the core of it didn't change much, they are still building on it and it's still smooth.
The legacy code was too bad to make refactoring worth it, but surely not every legacy system is in that bad of a shape.
New web-based UI came in (~2004) and it was super customizable and was supposed to be super fast. But nobody new the new hot keys, or there were none.
But it was customizable! So I spent the next few months learning this new architecture, APIs, SDKs for the crap-ware and deployed for our LOs. I got so free beers.
Then I helped another bank with the same issue in their migration - that second go was like free money.
A year of complacency and you have a year of catching up to do and you'll never get that year back unless you want to freeze development.
Before declaring something legacy, a fair evaluation of the future business requirement and fitment of current solution is a must.
12 Years back I worked with an org who had an AS400 which they considered legacy. System is still running and serves the business need perfectly well. All plans to replace just wasted money.
Another company decided to implement a new centralized ERP without realizing that company's highly federated businesses which can also be considered a conglomerate of very small independent businesses doesn't need it and won't use it. Million went down the gutter before higher ups realized the sad fact and fired the CIO.
IT world is literred up with sorry tales like these.
What happens when the hardware fails ?
It's the sort of thing that people who operate those systems know but that knowledge doesn't seem to spread beyond that.
Needed to increase the NodeJS version from 0.16 to 4.0 because a new shiny thing needed it. I knew it was a major version change but the dependencies didn't exist for the newer node versions... so rewrite something that isn't yours and maintain it forever... or do nothing.
Typically a replacement system has a nearer-term spike in costs for development, but is cheaper in the longer run than keeping the old one, because the old one will need gradually more expensive duct-tape, chicken-wire, and harder-to-find technicians willing to support something obsolete.
One advantage of the older system is that the costs are more predictable, because it probably works "good enough" at the time and will likely just gradually "degenerate". An overhauled system could be great or a grand failure.
An exception is if a vendor stops supporting a key legacy component and there is not enough demand for 3rd-party tweakers to take up the slack. Similar can happen if a security hole is found in a key legacy component.
That's the problem, most organizations can't track this.
At a workplace, we replaced a legacy item with something new without doing this analysis... but after that, the move accelerated when it became aware making new features on "New vs Old" was a 2 hour vs 2 week endeavor. The new system covers 70% of the traffic and will slowly be going all the way to 100% of the traffic over the next 1-2 years.
Say your programmers still use CVS for version control and you want to convince management to switch to Git. The CVS server doesn't cost much, a git server wouldn't cost much either. There would be a large cost for changing all your tooling and processes that rely on the CVS model and in training your programmers to use the new system. Why would you do that? How do you quantify the benefits of atomic commits or simple branching?
Really no one is interested in upgrading 'just' git, make a package out of it (new compiler, new CI, new testing, new version controll..fuzz etc) call the project "sustainable nextgen tool-chain...short SuNeTuCha" and sell it to the management like that.
When git seems to be by far the most used system on the market these days, isn't training any new hires in CVS going to be at least as much of a problem? I used it in my first job in 2003 / 2004, but I can't remember how to use it now.
Do companies with legacy code tend to keep staff around for longer?
Through particular requirements that some of these systems have they might shape policy too!
I primarily deal in security assessments of networks. Guess what happens to your password strength requirements if somewhere deep down in a dark and old server room there is a mainframe that can only handle 8 character lowercase passwords? Assume that this system has been the backbone of your, then hyper-modern, completely integrated IAM system for decades?
...
That's right, security policy mandates 'simple' 8 character passwords... Not ideal.
Every time I interacted with them there were hints that things weren't quite right and sure enough they ended up in the news after a huge IT failure.
The smart way to migrate a legacy system is to keep your maintenance crew to retirement age. Once they're approaching retirement bring in some kids to migrate to the shiny new tech and use the old crew to make sure they don't drop stuff.
The Obamacare system is a great example of this. Almost half a billion dollars spent on corporate contractors, then a team of like 10 experienced engineers came in and fixed the damn thing.
It is a people problem. Every department, every group, entry business unit is its own little kingdom that is fiercely defended by its workers. They cling on to it because that’s all they have done for the last 20 years. They know that if this system gets modernized, then their job is toast. Their job is vapor ware and they know it.
Having worked with many state and local governments, I see that these entities are never in favor of a business process redesign that simplified things, unless the consultants carve out a special exception for their little Access database or somehow save their pre historic C# ASP.Net application.
It is a special kind of blindness: We think that just because our tools and machines are so much more powerful that rebuilding a complex legacy application is now childsplay.
From a UI perspective you are most likely right. But it's the couple of million lines of business logic that will be your Waterloo. In a nutshell: forget about it. A bit longer: it is possible but only by a very experienced crew who has nothing else to do under competent leadership and who can chart an incremental path between where they are now and where they want to be. In 5 years or so. A big bang redevelopment is DOA. Every time.
If you're English, that's actually a good thing ! :-)
It seems to me like simplicity (and Occam's Razor) are approaches somehow lost or considered defects intead of values.
Personal example, I have a rather simple DOS (yes DOS) program that alway worked (and still works) just fine, it is Clipper/Dbase based, it does what it is supposed to do, with a single terminal, since 1990 or so.
Since it has financial and tax related connections (invoicing) it cannot (shouldn't) fail.
The business has not basically changed since the '70's, all the programs I could find to replace it now are either:
1) "jokes" not even worth mentioning
2) "worse" semi-legacy (think of MS-Access) crappy programs with terrible UI and layers over layers of "added because a customer asked for it" unneeded "features"
3) complex multifunctional multiuser multiwindow web-enabled real-time dashboard cloud connected <insert here any other buzz-word> based on SQL Server montruosities that actually do the 5 or 6 needed things (and 127 other unneeded ones) through a (slightly) different workflow (which is slower) and with data input masks that take longer to fill (compared to the good ol'DOS one)
I know that the #3 programs can scale with multiple terminals, multiple offices, unlimited number of customers, concurrent access and what not.
Still I don't need anything like that, it is - again since the 70's - a 5,000-10,000 customers per year, 20,000 to 40,000 transactions, 3,000 to 5,000 invoices, it won't become 100,000 or 1,000,000 transactions, ever.
Now I have to replace the good ol' Clipper little thingy, due to some changes in tax laws, and I will need to slow down (slightly) the operations, this is progress.
Banks often have incredibly old systems because their products don't change. A savings account in 1940 is pretty much the same as a savings account today. They basically have no need for DevOps, because they have no need to provide fast-moving, competitive products, and silos are a natural fit in a world of sensitive financial information. So some really old systems are pretty much fine as-is, as long as the hardware can keep running, and a few over the hill programmers are on call.
Another example of useful legacy is old physical infrastructure. The Frankford Avenue Bridge in Philadelphia is a 73-foot long stone bridge that's 323 years old, and carries a substantial amount of car traffic. But it's also not being maintained like it should be, and the more-than-expected wear over time is slowly eroding it. That's the bad part of legacy systems: when you know they need maintenance and they don't get it, so they rot.
Modern infrastructure can suffer the same fate. Concrete bridges, wooden houses, cars... they all need maintenance. So do modern computer systems, where they need to get regular security patches. So the problem is really just a maintenance problem, and we happen to let older systems go to hell more often than we replace newer ones.
The way customers interact with their bank has changed massively over that period. In the UK traditional banks are losing customers to new upstarts that don't have to bear the cost of all that legacy. They are nimble and much more responsive to their customers.
Things don't seem much different today.
On the one hand, HTTP, HTML, and producing HTML server-side in a language like Java or Python and with a SQL database isn't all that different now than it was 20 years ago. There are Java web apps developed in 1999 that are kept up-to-date and work just fine in 2020. On the other hand, there are applications developed with popular technologies a just a few years ago that are hopelessly out-of-date and in need of a rewrite today.
I see it as an end user too. I've been an ING Direct (now Capital One 360) customer for 18 years. The web app that I use in 2020 seems to do the same things as the one in 2002 (though it's slower today), but it seems like it's been completely rewritten at least three times.
[1] https://scholar.google.com/scholar?cluster=23826996884004375...
EDIT: And HN! 13 years old and counting...
In this case the replacement is all about re-gaining the knowledge.