The Problem of Legacy IT Systems | Hacker News Reader

The Problem of Legacy IT Systems

spectrum.ieee.org

88 pointsrbanffy73 comments

I recently (mostly) finished a legacy data migration and here's my advice: Treat it like you would treat a climbing expedition on one of the world's dangerous peaks. It's a given that failure is more likely than success; in fact it's potentially worst-case-scenario catastrophic. You will encounter many optimists along the way, and these are your most dangerous enemies - get them off your team ASAP. The people you can trust are the pessimists who insist on double-checking every assumption (how old is that rope?) and worrying about the what-if's. If folks can't accept your "attitude problem", walk away.

I wish I could up vote this twice. Assume every step in the process will fail and have contingencies. Don't set yourself unrealistic timelines, this will create pressure on you which will lead to more mistakes.

We worked on a Lean Process system within a larger Waterfall organization, and for the first year we constantly got called out for announcing that some milestone was going to be two weeks late. See? You're doing it wrong, you're doing it wrong.

After a while we noticed a pattern. Whoever 'blinked' first got a bunch of theatrics, including claiming a week for week slip every time (even if the problem was only one feature they barely used). Eventually half the teams would also claim a week slip. Anyone who missed that second deadline, the same process would repeat. Some teams who weren't 'the bottleneck' were still writing code week three.

Eventually we figured out that when we said a week, we meant 5-6 business days. When everyone else said a week, they meant 8-15 business days, and we suspected corner cutting even in those numbers. Over time we started 'helping out' people at the interfaces, slowly taking over ownership of more and more surface area. And yet these other teams still struggled to ship on time.

By the end I realized that this was the organizational principle the company was built on. Take a 2d plane of responsibilities, distribute people randomly on the plane, and where one fails to thrive, its neighbors grow to fill the space. You ship when most of the map is covered, and the gaps represent emergencies and/or lawsuits.

Are you hiring? Asking for a friend.

Nope, sorry

Basically when you migrate a project from A to B, your best result would be to have B that works as good as A (people don't want change, people don't want new bugs).

Since you most of the time didn't code A, were not responsible for it, etc. and, at the same time, are expected to understand it thoroughly (you got the fg code after all), you're expected to do at least as good as A.

So you're B will be compared to A and will always be seen as inferior.

How treacherous.

Data migrations aren't that hard if you know every system with access to the database. You just need to run the old and the new system in parallel during the migration, ensure you write to both and read from both. Ensure that all tests still pass if you switch source of truth to the new database. At first you treat the old one as the source of truth in production, keep track of how often they disagree, migrate data slowly so you don't overload your database and crash them, after migration disagreements should be 0, then switch source of truth to the new database and remove the migration code, then turn down the old database. Congratulations, you now have migrated data without downtime!

There is no special tooling for this since there can't be, you just have to roll up your sleeves and do the dirty work needed. It isn't hard, just a bit tedious if you want to do it properly. A junior developer can do it.

The big issue with data migration are inconsistent data in source, general news in structure and incompatibile approaches in original and destination db structure.

The last one may mean you are solving np-complete problems to satisfy new bd.

"It's not hard, it's just a lot of work."

The trick is that the people asking for it almost always assume it's not going to be that much work, and won't be as slow.

So politically it's gonna be hard for you if you don't properly set those expectations on day 1.

In the tricky data migration cases it's not plausible to know every system and be able to modify them (e.g. to ensure that you write and read from both), many of them will be outside of your control, likely in different organizations. That other organization will not run the same operation on two sets of data to verify if they match - they'll just import it in their workflow, and what happens there will have consequences immediately as soon as some item e.g. a person is switched to a new system. And it's plausible that you won't have any direct visibility in any flaws in that migration until that person complains to that organization and perhaps maybe eventually they will identify that your data migration is involved instead of simply screwing that person for months.

My organization narrowly avoided a business-crushing event a few months ago. Years of optimistic promises were finally put to rest and the sobering reality came to light: our legacy migration is, at best, going to be done in 3-5 years.

We were just barely within the window to order a pair of new servers before the Itanium order book closed forever. Naturally, our old servers are too old for VSI to support, so HP was willing to sell us a license to give us time to migrate from the old legacy servers to the new legacy servers. It came with a warning that THIS IS THE LAST OPENVMS LICENSE YOU WILL EVER GET FROM HP AND YOUR CEO MUST SIGN THIS BEFORE WE GIVE IT TO YOU.

Its a shame FreeVMS never took off:

http://www.pvv.org/~roart/freevms.html https://github.com/rroart/freevms

My fear is that the “skeptics” who really know what’s going on are rapidly retiring. Nobody can even patch these systems.

Generally speaking in the IT industry, "Legacy" is something thats no cool any more.

I have a gut feeling that there are numerous legacy systems out there that run circles around "Modern" (which really means k8s) systems in terms of TCO because they were engineered thoughtfully and ignored hype train. So many lies & deceit & click bait.

On another side of the coin though: "Legacy" systems are ones no new comer is training for.

That's one definition of "legacy". The other is "oh shit, this stuff isn't supported any more and is now a liability".

Which is a problem, because the longer you're unaware, the more the liability and unknown unknowns grow. Eventually, the liability and cost to replace are huge/bigger than it being an asset. The problem just gets harder; especially if you ignore it until it goes horribly wrong. See a comment above [0] for how this could happen.

Responsible, senior devs should be able to provide some guidance to the business at what point the system or sub-system is on that liability vs asset curve. "Not cool any more" isn't that kind of advice.

[0] https://news.ycombinator.com/item?id=24769695

I think part of the value that senior people (those of us who have been in the industry for decades) bring is the wisdom of when a system has gone from uncool to dangerously unmaintainable.

Part of me wonders if it's because we old-timers ourselves know what it's like to be on that specific journey personally.

It's not lost on me just how intellectually interesting dangerously unmaintainable becomes. I love working on new things, they're fun - but things that are legitimately mind breaking are a fascinating challenge.

The lack of support is critical. They are time bombs.

The moment something becomes uncool, it starts the process of becoming harder to find developers. Something that has fewer available developers is more of a liability.

"Does everybody agree on what legacy IT is? What counts as legacy?"

The salesperson's standard definition is, "That which I didn't sell you."

The coder's definition too often is, "Not in a language I think cool."

Or something that many can’t service. Ive seen legacy apps in PROD for which only the binary existed and nobody was exactly sure what it was doing, they had an idea and workarounds for when it breaks.

I think you could make a good definition based on the institutional knowledge of a system, and how much of the system is not included in that knowledge.

Flashback to a utility that only existed as a binary, but was key to some prod processes. Even better - by the time I met it, said utility crashed on a SEGV after it accomplished it's task. So SOP became - ok, xxx crashed, we're good to proceed.

Coders definition is: "code I wrote last week"

Or code someone else wrote.

Code I wrote before lunch today ;)

How about: Using no longer supported libraries

Slow development speed due to old standard

Old and slow source code repo tool

Outdated (os, vms, etc.) development landscape

Bad code like missing Tests

Isn't legacy a system which was not maintained for a while? I have never seen a well working, up to date, modern system being called legacy.

You become legacy when something else superseeded you and now the legacy system needs to be kept alive for whatever migration issue.

"Denoting or relating to software or hardware that has been superseded but is difficult to replace because of its wide use." [1]

[1] https://www.lexico.com/en/definition/legacy

The real definition of legacy for most developers is “whatever is running in production right now”.

Precisely, and often "the thing that's making the company money".

Maybe it’s code a bit like the Iliad.

A cornerstone of your organisation but few know how it got there or how it works. And even those who do can’t verify it historically. And monsters. Those too.

I generally try to phrase "legacy" as "something that is surprisingly expensive to maintain and improve".

A Mercedes limousine from the 1930s might still drive, and be reliable and stable. Finding parts for it probably requires custom-making them, and that's why I would consider it a legacy car.

You can consider "legacy" as something that's in an ascending curve in TCO.

Literally every system that is scaling up is legacy by that metric.

TCO per customer? If serving your customers, on average, becomes more expensive with time, your system becomes less and less adequate for business.

I would define legacy as when the IT system drives the business.

When working in government it was really obvious when you’d find these systems, as they were usually modeled after a paper process. The organization usually outgrows the process, then they start building shims to adapt.

Newer legacy is harder, as the paper processes were usually better than whatever nonsense was cooked up circa 1997-2007. Paper process people almost always understood the business better than their successors.

My definition is pretty straightforward: legacy is every system of which maintainers don't have complete knowledge.

Also legacy is a spectrum: there is a difference between that side project you coded 2 months ago and forgot about, and that mysterious cobol mainframe the bank found when reworking the office to open floor.

I’d argue that “legacy” is only really used to try to sell you something. It’s a marketing term and doesn’t really say anything useful about the nature of a system.

The way it is usually used, it tends to mean "something we gave-up on paying most of the technical debt".

But I've seen some very different usages too.

coder here: "The thing I'll get yelled at for fixing when someone reports it's broken".

Similarly: "Thing I'm not allowed make any enhancements to"

I worked at a financial institution during a migration from one core system to another. I was their "Administrator of Legacy Systems." At the time I thought that the "legacy" was the data we kept in the system - all of the transaction records, account information, etc. At the time I didn't consider it a pejorative, but seeing how the term draws derision these days I'm not really so sure!

One definition I've heard before (and like) is "code without tests". I don't think that's the only factor, but it's a pretty common indicator.

The manager’s definition is “That which is no longer supported”

The retiree’s definition is “That which pays for my grandkid’s college”

Legacy is contextual. It means different things in different context so it might be more helpful to understand what legacy means in this context.

Generally, systems are almost never explicitly designed with an exit plan in place. A way to get the data out, inventory the integration points, enumerate the site-specific user procedures oriented towards business teams that need to be re-oriented towards a new system, a central location of customizations (containers are slightly helping here, with their notion of a centralized location of persistent configuration data across builds), how to test, and so on, is nearly always left to the team tasked years and sometimes decades down the road to figure out, instead of the team that originally stood up the system and has the information fresher in their minds.

This generally isn't done because no one wants to budget for those activities; they don't show up as revenue-positive. It will take a cultural change to adopt across the board, like leadership who controls the budgets now more commonly accept the overhead of version control.

Re: systems are almost never explicitly designed with an exit plan in place.

How do you plan for the future when nobody can accurately predict the future? I remember where it was claimed that OOP UI api's could "last forever" because they were an abstract interface rather than implementation. However, they couldn't handle the stateless-ness of the web because OOP is inherently stateful. It's almost like having a printer API that assumes a color printer, and suddenly it's being asked to translate into black and white. You'll lose output meaning without reworking everything. Abstraction usually has to make assumptions, and these assumptions may turn out wrong down the road.

Re: no one wants to budget for those activities

True. That's one thing an org can and should plan for: money to pay for upgrading. In other words, you cannot predict future technology, but you can safely predict that adjusting to the future requires resources the vast majority of the time.

Next question: how many resources?

Re: How do you plan for the future when nobody can accurately predict the future?

It isn't so much 'planning for the future' as it is documenting the present. What are the things that you are doing _today_ which haven't been documented. Processes and procedures are often left undocumented because, well, ask Bob he knows how to do it.

... when Bob goes to retire it becomes a whole different discussion.

Ye I have noticed this bullshit antidocumentation attitude everywhere. It is like something people make up as comfort since secretaries and technical writers were sacked in the distant past and people are not even trying to keep things in order anymore.

It has got way worse becouse of Scrum and constant churn living in the now and not planning two sprints ahead.

Isn't this largely confirmation bias though?

I know a lot of systems which used standardised structures as a migration pattern, but those are the things that were basically painless to replace; and those that weren't painless stuck around.

Ironically those systems which had easy exit paths are often replaced by things which don't.

>Ironically those systems which had easy exit paths are often replaced by things which don't.

Sad but logical - stuff get's upgraded until it's no longer easy to do so. Often the opposite of what is needed!

Sustrik's law:

"Well-designed components are easy to replace. Eventually, they will be replaced by ones that are not so easy to replace."

> In a long-lived project, components are being replaced. Nice reusable components are easy to replace and so they are. Ugly non-reusable components are pain to replace and each replacement means both a considerable risk and considerable cost. Thus, more often then not, they are not replaced. As the years go by, reusable components pass away and only the hairy ones remain. In the end the project turns into a monolithic cluster of ugly components melted one into another.

http://250bpm.com/blog:49

Moving to the cloud has changed this mindset somewhat - now there's no longer perpetual licenses, and we can't expect that the application will last forever, there's some consideration given to how application data can be repatriated.

Having cloud apps come with APIs as the only customisation points helps with our business users - we tell them that they need to chose a product that does what they want, as we won't be able to bend things around for them like we used to.

Yes, legacy systems cost a lot to replace, but continuous development over 20 years isn’t exactly cheap either. Hardly anyone talks about that though.

And let’s be real, if a system is non-legacy it’s likely being completely rewritten every 5-10 years anyway. Not many active codebases have 10 year old code still running.

10 year old code would be considered very new where I work. I know for a fact code I wrote 10 years ago is still the heart of our system (with very few changes). I know of code that someone else wrote 30 years ago that is still in use, getting that code tested is very important so we can confidently make changes for the next product that will use it.

A few years back we started an effort to rewrite some old C code into a modern system with formal proofs of correctness. That was given up because the new improved process turned out to be 3x more expensive to write new code vs new code in the old process. (formal proofs were the reason to use the new process, but not the reason it was 3x more expensive. I'm not allowed to comment beyond this so don't ask...)

If you book an airline ticket, odds are under the hood you're hitting an IBM TPF application (prolly PARS/IPARS). When I was involved in that world 20 some-odd years ago the programmers would regularly show me files with 'last modified' in the 80's and 90's. These aren't things that someone casually modifies, and I would expect you'd find an awful lot of the same even now.

I'm typing this from a legacy browser (first released 18 years ago) running on a legacy OS (first released 29 years ago, a rehash of a design from 50+ years), and will switch to my legacy editor (first released 44 years ago) to continue work.

Firefox, Linux, and vi aren't "legacy" software by any reasonable definition of the word, so no you're not.

Emacs; vi is older.

But all of them still contain crucial central parts that have been created decades ago, and famously have not been redone from scratch in their history, unlike, say, Windows or vim.

I don't see what is the principal difference between these pieces of software, and some COBOL-based accounting packages still doing heavy lifting in s bank, while being actively maintained and even receiving new features.

Legacy systems aren't just old. Old systems that are maintained aren't the legacy systems that create problems, and really aren't legacy systems at all (just old).

It's the old systems that "just work" until they don't. Where they run on old hardware, and the compiler is possibly on something even older. You can't port them to a new system because you may not have a compiler or even the source code. Or the source code is heavily dependent on the original physical hardware. Maybe it runs in a VM, if it was on an IBM mainframe it probably does. But that doesn't fix the problem of old code that can't be easily altered at this point in time.

I like Michael Feathers' definition of legacy code being code without tests. But it's not just tests, it's comprehensibility and (to a lesser extent) portability. Tests help to make a system comprehensible and portable. Since architecting for testing requires better modularity which makes things both more portable and more comprehensible in most cases. Portability means proper abstraction over the hardware or other things so that the core logic is separable. Imagine if the IRS's tax system were written in this way, where the hard-to-port part would be the interface with databases and networks, but the core part would be easily portable (or more easily portable) to another OS because it'd be properly separated. We could port that core to another system and change how it gets called more easily. But if you tied your software to a specific DB or a specific storage medium (like it has to be a tape drive, or something that behaves like one) you've reduced portability, and created the start of a legacy system.

Vi and Emacs were both originally released in 1976.

> Firefox, [...] aren't "legacy"

It is if you're on the most recent working version (3.6).

My iPhone 12 is legacy because it's based on an iPhone released in 2007 and the software is also legacy because it's based on Unix which is 50+ years old. Glass is thousands of years old so it's legacy technology and the molecules are the same as those from the age of dinosaurs so it's very very legacy.

Those 20 years weren't about current functionality, but tons of changes reflecting how business requirements were changing. New system from the scratch in the same technology would be very cheap (given there are folks around who still know original language/platform).

System being completely rewritten - I can't agree on that one either. Some, in some companies do, most don't. I personally manage quite a few of those and there isn't budget for costly migration experiments if things work. Working in one of the biggest global banks in Switzerland. What works is if ie OS/browser that is absolutely required isn't supported anymore and there is no way around it.

There are very few cases where the approach 'if it isn't broken don't fix it' isn't the best approach for the company. But good luck trying to explain it to devs/teams who want to try new things (basically at the cost of employer, I don't blame them but it is what it is).

The primary money making code where I work is well over 10 years old and still going strong. Still being maintained and improved. Still getting features added.

Lol I was at a company before and they had like 50 people build the original system. Not one was left and I was on a team of like 8 people supporting it. In addition, managers and other people on the team were bad and I was junior. What a nightmare.

I’ve been there. The manager told me that the system wasn’t in maintenance mode, that we were adding new features and finding new customers. Turned out they were just lying to themselves, that there wasn’t enough staff to add anything new to the project, and we should probably focus on migrating the customers to other products.

I remember the moment when I realized that the other people on the team were adding bugs faster than I could fix them. That it would be never-ending.

managment does this explicitly to exert control over engineers; I have seen this at least twice in several decades

From 1997 to 1999 I had a contract with Caterpillar. Dealers used an AS/400 application called Service Advisor via good old-fashioned 5250 terminals [0]. For those not born when reality was rendered in shades of green, a real 5250 terminal weighed close to 85 pounds (including keyboard) and was built to withstand armed invasion (only partially joking). They were rugged. All IBM hardware of that era was durable. Caterpillar dealers fix large earthmoving equipment and the users of this application were mechanics and other shop staff. Grease, oil, diesel fuel, and any other kind of grime you can imagine were slathered over these terminals. They still worked. No mouse. No GUI. As is the case with most "green screen" applications, users had a mental map of every panel and could "type ahead" several screens and would do so routinely.

It was decided that this application would be "modernized". Y2K hysteria was ramping up and the nascent dotcom boom was brewing. I lost count of how many times I heard, "We need to put some lipstick on this pig." The new hotness was Java and applets. Yes, you read that correctly: the new front-end was going to be written in Java and deployed as applets. The AS/400 would remain as the back-end. For a period of time DCE [1] plus C and C++ shims (running on the AS/400) glued the Java front-end to the back-end.

The project did get "completed" but in name only. Dealers hated the new interface: it was slow, it required a mouse, it did not support any kind of type-ahead, and PC equipment wasn't designed for such a brutal environment. My last interface with these systems was in in 2001 for a very short follow-up project. At that time the green screen was still ruling the roost. Perhaps they eventually did kill off the AS/400 and all of the COBOL. Nah, probably not.

The AS/400 wasn't then (and isn't now) sexy. Green screen applications aren't sexy. COBOL, RPG, and other IBM-centric technologies like CICS [2] aren't sexy. That doesn't mean they don't work. Ironically, they tend to work too well.

This is but one example from oh so many over the years. Sometimes, very rarely, an old technology truly requires complete replacement. Often, calling something legacy is used as cover to cargo cult and "keep up with the neighbors".

[0] https://en.wikipedia.org/wiki/IBM_5250

[1] https://en.wikipedia.org/wiki/DCE/RPC

[2] https://en.wikipedia.org/wiki/CICS

EDITED: formatting

The tech industry, but especially and most drastically open source, has been completely taken over by expert marketers called "dev relations" in the past decade. What you say about working too well is true; the business model of things like Kubernetes or new datastore du jour require a high touch to sustain the business model.

>> a real 5250 terminal weighed close to 85 pounds

I came into networking around the time when those were being replaced by 5250 terminal emulation in Windows through an application. So much desk space was freed up, now that you only had to have one 'computer' each desk.

Some customers would run new CAT5, and the others would just slap ethernet adapters on their twinax. Still run into a huge bundle of that stuff in a ceiling from time to time.

That brings back memories (not good ones). Those twinax baluns (and the equally evil token ring type 1 to ethernet baluns) were awful to keep running. I guess in the short run it was less expensive than running new cable, but in the long run my feeling is the TCO probably wasn't worth waiting to bite the bullet.

If you didn't have to work with RPG ("Report Program Generator"), count yourself fortunate.

Reply on news.ycombinator.com