Banks scramble to fix old systems as IT 'cowboys' ride into sunset (2017)

Spending money on upgrades is not the problem. Firing people who know about your systems is.

"absolve relevant parties of their responsibilities" - what relevant parties? What responsibilities? It sounds great and responsible, giving a good impression of dismissal of the point made, but the wording is so general as to be impossible to act upon - just as with "a bank". I'm sure standardised checklists were ticked, committees reviewed the decisions and came to the conclusion that it was best, the board was satisfied with the new direction taken by the CTO's organisation, and the shareholders happy with the cost reduction at the time as described by the board. To paraphrase Thatcher, what colour trousers do the relevant parties wear?

You see this defection pattern everywhere. For example, the never ending stream of useless new features in most Google and some Microsoft products which coincidentally get slower and buggier every year, because you can get promoted for the visible addition of a feature as a PM, but it is much harder to prove that getting the Gmail app load time down by 70% has stopped the feeding of your best power users to the competition (Calendar is too easy a target for criticism).

The classic defector on the tech side is the "ninja rockstar developer" whose prolific output of code and visually attractive but functionally superficial features is matched only by the amount of frustration generated after they are promoted somewhere else or changes jobs if it doesn't happen fast enough, and the rest of the team has to deal with the technical debt and new, unrealistic user expectations.

Security is another good candidate for shortcuts, since there is no visible upside to good security (maybe you were just not attacked) and tons of downside for taking responsibility since you make yourself a visible scapegoat for blame when a crisis does happen.

And note that none of these examples are in "move fast and break things" i.e. "do illegal things and hope they become legal ex-post" territory! I personally take great pains to understand these dynamics so that I can avoid them in my own company, which is a matter of survival.

Whenever you have a systemically pathological situation like big, enterprise software projects often get into... it's probably an incentive problem.

Tldr agreed

Most big banks have a bunch of security people whose sole job is to make people’s life hell by spreading as much FUD as possible[2] — that’s what keeps their jobs secure, after all. And they are the only “engineers” who are heard. They make sure that anything new is not adopted till the rest of the world has already started abandoning it.

The primary focus in the technology departments of big banks is to maintain the status quo. New development is rare. New ideas are crushed before they can see the light of the day. As a result, good engineers leave as son as they can and the only ones left behind are the ones who are content living a life of mediocrity — maintaining the status quo.

[1] I know someone who was told something along those lines when they joined a big financial institution as an engineering manager. And not all financial companies work that way. Those who embraced technology reaped the benefits and continue to do so.

[2] I am not suggesting that security is not important. But security people at these institutions take it to a whole different level — to the point where things that should not take more thane a few days end up taking months — if they allow it, that is. Funny how all their objections vanish when someone from upper management asks for the same thing.

Imagine that you need to replace a live system, downtimes are not allowed, probably provide a new UI while offering 100% of existing features.

There is no documentation available, or even if there is one, you won't find anyone in the building that will put their hand on the fire for how much of it actually maps to the running application.

So a project of reverse engineering existing application into a requirements document or updating the existing documentation usually takes several months.

Now just for a very basic project cost, consider the monthly salary of everyone involved, multipled by the amount of months.

And we are just speaking of phase I, then there is the actual re-write, followed by verification that 100% of the old features are still present on the new system.

Naturally it is cheaper to keep existing systems running.

This applies to any old platform, not only COBOL systems.

You can't just throw it away.

It is not un-replacable in itself, I'd imagine it is just a huge undertaking and many higher ups are afraid to do it and risk something going wrong.

As above, may be these COBOL systems have been battle tested for a long time and they are good. Other reasons could be, "if it ain't broke then why fix it?" or that there aren't enough people to act as a bridge between the old and new systems so it is hard to do such projects (as other comments have pointed out)

So at any specific time, the safer and cheaper option is to stick with the old system, despite the growing risks and costs in the long run.

To use hip modern buzzwords, it’s a DSL. In addition, it runs on hardware specifically designed around its idioms.

MSP and Cloud are separate concepts. The typical MSP has a bunch of technicians who drive around to client sites and tend to the servers in their closets.

Oh, but they can't increase the estimate. The customer signed off on that estimate, so we're committed to deliver by that deadline.

Fortunately I had already resigned at that point.

I've been programming for decades and my current project is frequently praised by other developers who work with it (it's a framework) for the quality of its documentation.

So, in fact, I do have such an idea. It's hard work. We do it anyway because we're professionals. The fact that people frequently don't do it doesn't change this.

One technique I use is to keep documentation in the same repository as the code, and force people to write docs via the code review process. That is, a proposed change doesn't pass review until it also updates the documentation files. People want to get their code merged, so they know they'll have to take docs seriously or else it'll get picked up in review.

Now maybe back in the mainframe days there was no culture of code review, but these are banks we're talking about. They excel at process. So it seems kind of unlikely they'd not want things documented, if management had understood what needed to be documented in the first place. But ultimately it was only the employees who knew that.

This, he could have documented it perfectly well in a time when everyone knew COBOL.

But that documentation would all be useless to me today with my knowledge of Java and Kotlin, but understandable to someone else from is generation that works with COBOL.

However, writing useful documentation isn't trivial, and is best done with a technical writer, and takes time away from development. If management isn't willing to invest the time and resources necessary to produce useful documentation, then they're going to have problems in the future as the gurus leave.

I just had to go through an open source project that uses guice, in order to try to trace a particular code path, since the user group was less than helpful.

Trying to figure what was running was an absolute nightmare of nested magic spread out through a score of files. Meaning as soon as I had a clue, I'd lose it as I had to retrace back through all those files.

I'm sure it's clear as crystal to the people who wrote it, but DI can be obtuse as hell if you have to figure what's actually running on your own.