Hacker News Reader

Comment by scrollaway | Hacker News Reader

Realistically, if, say, one of these big corps/banks with deeply-integrated COBOL systems spent a few years revamping their systems to something more modern and usable today, how much would it cost?

Running mainframes isn't cheap. COBOL programmers aren't cheap. Not being able to improve these systems / innovate with them can cost the business a lot of time/money. And if we're talking COBOL, we're talking at least 25 year old corps.

How long's the ROI on moving over? How expensive does the maintenance of antiquated systems get? It's easy to say that they're rock-solid and never need to be touched, but is that really true? For example, lot of banks today have a lot of absurd UX, often because of absurd internals and I know a lot of that usually is "the system is ancient and nobody really touches it anymore, that's just what it is now".

View on HN

> Realistically, if, say, one of these big corps/banks with deeply-integrated COBOL systems spent a few years revamping their systems to something more modern and usable today, how much would it cost?

What language would you suggest? Remember that decimal reliability and predictability is important. Plus, let's pick a language likely to have a very long service life because we need to have our investment rewarded.

Frankly, the iSeries machine we have is pretty cheap to run. We bought it 10 years ago for about $15k and have had a maintenance contract that doesn't cost that much (about the same as our Red Hat service contract). We probably will buy another one since they just released a sub $10k one that will more than meet our needs. Never lost any data, easy enough backup that the biz office does it themselves[1], and IBM is pretty quick about any fixes. Much cheaper than any Linux, BSD, or Windows system we have ever had. Plus, I don't mess with it like the others. It just works.

1) we've tested it multiple times, after all, an unrestored backup is not a true backup

I don't know enough about these systems to suggest a language, but I imagine within the "a few years of development" is budgeted some time to figure out what tool to use. Surely you're not telling me that mainframes running COBOL is the best tool for the job. What languages do younger banks use?

"I don't know enough about these systems to suggest a language", so how can you say "Surely you're not telling me that mainframes running COBOL is the best tool for the job."?

Are you telling me I'm not allowed to ask questions?

You weren't asking a question, you were making a statement about how foolish it was that banks are running COBOL. Question would be about the why and what value do people see in the situation that stays their hands at going to a new technology. Why did banks spend the money and time (which was not insignificant) to modify code for year 2000 instead of making the switch then at a natural change point? It would be the most logical point, yet they didn't do it. What does the mainframe and even the mini-mainframes bring that Docker does not? What made COBOL so hard to replace given later things like PL/1 which were directly supposed to replace it? Where do all the RPG programs fit into this?

Maybe someone should be thinking about a COBOL replacement language instead of the plethora of C / C++ replacement languages.

I imagine within the "a few years of development" is budgeted some time to figure out what tool to use

It will be a lot more than a few. Things absolutely cannot go wrong or even change in production. One slip-up can result in real money being lost. This whole "go fast and break things" doesn't apply to bank accounts.

These systems can be at the very core of the company's operations, so you're talking not just a revamp of a bunch of COBOL programs and mainframe servers, but a significant portion of the company's workflow would potentially need to be refactored or replaced. The mainframes themselves sit in a box, but many other systems rely on their data and their guaranteed processing. And all of those connections and data structures are custom. Even if you were to figure out some way to completely emulate the system (and that itself would probably take several years), you would then need to integrate your emulated system into other real-time systems. A bank just can't shut down for a day. And if they screw up a few minutes worth of transactions, then things get very bad. So your testing and integration will be slow and methodical and expensive. And even then, you're still emulating a crappy, old system. If you wanted to actually replace the entire system, how would you do it? These things aren't modular. They were custom-built with no intention of ever being parted-out. You would have to completely re-architect the company's infrastructure and workflow.

Or you could not do any of that. You could just accept your situation. Mainframes are essentially a subscription service. For a yearly cost, IBM has offered you guaranteed performance and guaranteed hardware (they fix it immediately if there is a problem). So, all you have to do is pay your subscription and pay some people who like to have a steady, unexciting job and your business will continue to run like clockwork.

I understand what you're saying, but does this logic hold over decades?

I'm obviously not saying banks should just switch today cause it's easy and all, but if the ROI on switching is 10, even 20 years, it still makes sense to do if you're a bank.

It's not 10 years, it's probably closer to 100. Reasons being a) the ridiculously large cost of rewriting and testing the new system on a real-time system that can't be taken down, and b) the ridiculously low cost of maintaining the current systems.

How do newer banks do it? It certainly doesn't take 100 years to bootstrap a bank. I also mathematically can't get behind the concept of reproducing a known state taking longer than the time it's taken to get to it.

I can't find the post now, but some folks from the US govt who frequent HN talked at length about the ongoing replacement of mainframe-type services with newer tech. What I gathered from that is that the cost isn't ridiculously low at all when you take into account the impact it has on the organization's ability to get things done.

I had the fortune to work on COBOL software for brief period of time at my last job, which was at a financial clearing company. This group of 50+ COBOL applications that was the lifeblood of the company and a bad run could cost the company millions. The software was started in the early 90s, so it was easily 20+ years old when I was working on it. There was very little documentation regarding the process and the people who worked on the system were gone. I bring this up because when you have software that old you typically aren't going to have the original people who wrote it at the company nor will you have the people who gathered all of the requirements. All of the knowledge that went into that product was virtually gone.

You are at a known state but you now have to re-create the steps to get to that known state as I'm sure a lot of the domain knowledge around the product is not there anymore. Not only that, you now most likely have other sources consuming the output of the original COBOL application so you have to ensure that the other applications are able to consume the new data and that those don't break either.

> I also mathematically can't get behind the concept of reproducing a known state taking longer than the time it's taken to get to it.

What you have to remember is that when the COBOL code was written, it replaced hundreds, maybe thousands of people doing manual data entry and manipulation, maybe even pen-on-paper. That gives you a fantastic ROI. After that's been done, replacing one computer system with a newer one is completely different, a spectacular case of diminishing returns.

While I'd agree that its easier to start a new bank, thats not the case here. What you are doing is trying to recreate something that currently exists, and if it goes like almost every job I've had then there is no budging on any of features, they want stuff duplicated down to the same bugs, and none of the features are documented. You end up spending half your time just reverse engineering what currently exists before you can even begin work

I'm not sure about most banks, but some use other industry-specific software, there's one being developed in Uruguay for small banks and financial companies called Bantotal, which is written in Genexus, a 4GL language or metalanguage that generates whichever language you want (Java, Ruby, .NET, even Cobol and RPG)

http://www.bantotal.com/?lang=en

https://en.wikipedia.org/wiki/GeneXus

most moves from mainframe to *nix, windows or others are a port of existing code to a new OS. NOT a complete rewrite.

You're absolutely right. But to start a decades-long project that may or may not work and has no near-term ROI is a hard thing to rally for. The cost of the project would eclipse what is being paid to the mainframe provider and to company personnel. And these companies have a difficult time bringing in top talent, so they don't really have many tech-savvy people to preside over it.

I've seen it attempted first-hand and it was a dismal failure. And it was their third or fourth attempt. And it wasn't even a full replacement! Just a simple data lake to dump things into. Google or Facebook or Amazon could do it, but few companies are like them.

Success isn't guaranteed, maybe as low as 50%. When you say 'few years' it intimates you don't appreciate the scale of some these systems. In the end, retiring legacy systems isn't a technical problem, it's a product owner, product management, and business problem. The business owners have to make concerted, multi year effort to document their process as implemented, define the replacement, and manage it's (re)implementation. This never happens.

> The business owners have to make concerted, multi year effort to document their process as implemented, define the replacement, and manage it's (re)implementation. This never happens.

I wouldn't say it never happens, but it certainly takes a couple tries and a lot of time and money.

My employer is close to being off our mainframe(s), but it will probably take at least another 10 years to get rid of the last little dependencies. We've already spent about 15 years (with hundreds of developers) getting to the point where all the major online and batch processes have been reengineered and have very small mainframe dependencies. The mainframe development teams are very small and mostly focused on decommissioning work.

But yeah, it's a massive undertaking. In our case, the business was motivated to pay for it because mainframes just couldn't scale for our main use cases.

Any chances we could read more about this somewhere? Sounds very interesting.

> Any chances we could read more about this somewhere? Sounds very interesting.

Very, very unlikely. The company doesn't even have technical writers anymore for internal projects, let alone technical historians. And with the time it's taken and normal staff turnover, I doubt that there's anyone who really could write about the whole effort based on their own experiences.

Basically, it consisted of building of building a new central datastore for our content (that was pretty mature by 2005, when I started). Then hundreds of smaller projects to migrate this or that process to read from or update the new datastore or keep it in-sync with the old one. There was also a lot of opportunistic piggybacking of new feature work onto migration projects or migration work into new feature work. However, I only have the view of a regular developer.

When I say "few years" I mean 10, 15 years? 20? 30? In the context of a bank, which may be around hundreds of years, that's still not a lot. But maybe I'm still way off, hence my question: How long would it take?

I see a lot of people here telling me I don't know what I'm talking about without actually giving an answer as to how long it would take and how much it would cost, which was my question in the first place and I'm being downvoted for asking it. Seriously, WTF HN?

Well, those questions depend on the system in question, and for any particular system, it's probably impossible to say what the cost of replacing would be, because no one understands everything the system does, and how it interacts with external systems. This is what others are alluding too, your question is unanswerable.

What you can do is estimate the cost of implementation in today dollars. For example suppose Bank System X has been under development for 40 year with an averge of 10 developers per year. That's 400 person years. Suppose you find cheap labor at $100k per year, the sunk cost in personnel alone is $40M in todays dollars. So 40 people maybe could do it 10 years.

The above analysis is not satisfying.

> Running mainframes isn't cheap.

Price for performance for certain workloads, as I understand, it can be.

> Not being able to improve these systems / innovate with them can cost the business a lot of time/money.

The banks use other technology for peripheral systems that interact with the core functions all the time (look at the drivers behind AMQP, sometime.) The core functions, OTOH, are about impelmenting requirements that haven't changed in centuries; until there is a desire to implement a fundamental new way of doing things (like blockchain, which financial institutions are exploring and probably aren't using COBOL on mainframes for their experiments) theres nothing to innovate on.

Most banks looking to move off mainframe/COBOL/CICS/etc don't look to rewrite everything themselves. They look to buy packaged software which they will then customise/enhance. I think most of those packaged software offerings are written in Java. (Disclaimer: My employer sells packaged banking software written in Java.)

In fact, if you look at the existing COBOL banking applications, many of them aren't pure inhouse development, but packaged applications like CSC Hogan. Hogan runs on top of z/OS and CICS and is written in COBOL. Hogan shops will have a lot of their own COBOL they add to the Hogan system to implement customer-specific requirements. So for many banks moving off mainframe/COBOL, it is actually a migration from one packaged app to another, but of course they need to rewrite from scratch all their customisations.

In my personal experience, most banks doing this don't try to do it as a big bang. They identify subsets of functionality where they feel the current mainframe system is most limited – e.g. mortgages, mobile banking, whatever – and start by replacing that function only with the new non-mainframe software, and have it integrate with the pre-existing mainframe app for other functions. Then, the idea is, progressively they will migrate more and more functionality to the new system, and eventually the mainframe/COBOL systems can be retired. A piecemeal approach often involves less risk, costs less (in the short term) and is easier to justify to the business since they can see tangible results much easier.

It cost one large bank in Australia $1 Billion over 5 years to convert their system. That's 2-3% of their annual profit over 5 years to fix the problem, so we can assume the costs of running the current system are probably some small multiple of that. And they would still need to pay to run the new system, and most likely both systems at the same time towards the end of the transition.

Reply on news.ycombinator.com