But how complex can these programs be? Surely anything written before 1992-ish can’t involve that much code just because of the limitation of the machines they were designed to run on. What makes it that expensive that no one wants to rewrite them in a modern language?
You would be utterly shocked.
I did some COBOL work for a bank.
Most of the new code they write is in C++.
However, COBOL was still a huge part of their stack, and they still had several heavy duty mainframes that it ran on, and maintaining that was part of my job.
For an example of how much COBOL was written back then, and how huge the technical debt to replace it would be:
* The interbank transfer system alone was ~750,000 lines of COBOL.
* It handled around 1.2million transactions every three seconds. That's not capacity, that's real world data.
* They had less than five minutes of downtime in the last twelve months, and that was considered poor.
It's a mammoth task to replace the system, and getting transactions from the old to the new without incurring a huge overhead is nigh impossible. And yes, dropped transactions can mean dropped customers, and big customers.
It's far cheaper to hire and train programmers in COBOL, than to rewrite everything. COBOL and Fortran have millions of lines of code in use, and such frequent use that replacing them may cost more than the lifetime of returns would give.
Sounds like jumping from the frying pan into the fire.
Yep, that. I did a two-year stint in a major financial network. I guess it's not a lot of time, but while I was there, there was a big outage and it was not on the mainframe side of things- some all modern server farm bought the, er, farm.
From what I can tell, at least in that company, mainframes are still in use because their stability is simply unmatched by anything in the "distributed" world; Cobol is still in use because mainframes are still in use; and nobody dares replace the Cobol code with something else, because the code itself is damn stable, as a result of hundreds of people having worked on it for several decades now- and counting.
In general, Cobol on mainframes is a rock-solid system that doesn't fall down and that nobody wants to disturb.
Also, I've yet to see such businesses opt for the safest porting approach. Build a transaction replication system, then a component of the ported system. Inbound transactions are replicated and dispatched to the legacy and the component of the ported system simultaneously. Build scaffolding to monitor key transaction metrics. Compare output and performance results. Let that "bake" for a year or two. Rinse and repeat with the next component (many such ported components can be developed in parallel, etc.), and so on, until the ported system reaches feature parity with the legacy system, then let the completed version "burn in" for 2-3 of the longest periodic process lifecycles. For example, if the the longest periodic process the application supports is once every 7 years, then you burn in for up to 21 years, comparing during the entire duration.
By the time you "throw the switch" to the ported system, you have a very high certainty that it will work just as well if not better than the legacy system. Extremely low-risk, but extremely expensive, and considered extremely impractical. And with how business-critical some of these systems are, pretty much any risk mitigation short of this will get nixed by senior management.
This doesn't even begin to get into other factors, like changing regulatory and business environment as you are porting (so you're porting and modifying for new requirements simultaneously), such a project can easily be targeted for the cutting block for just the cost savings to make someone in Finance look good, internal political issues, developer churn over such a long period, developers on the legacy system withholding key implementation or even business requirements information, that information simply not being available, binary-only parts of legacy, etc.
I would disagree with the assessment that management is too dysfunctional, it depends. But if they all were like you described, i would be out of work. And it just happens that we turned off a mainframe just 2 weeks ago.
From seeing how the risk is assessed in some of my customers, the kind of emulation approach you described (I've seen one for the old HP 3000, and it worked great) sells easily when the legacy vendor is so sclerotic and rapacious that the customer's management is practically pushed out of the relationship, and the economics don't have to completely pencil out to close the sale.
The customer is tied down to a smaller (usually) vendor's emulation, and trust the fidelity of that emulation over time, and furthermore, over new versions. The customer must maintain a skill inventory that isn't "current" (though some people might consider that a plus).
I can think of a couple ways to address those sales objections (lock-in isn't real: can always switch back to legacy vendor if all else fails, no need to revamp staff with crucial business/domain knowledge, etc.). But I'd be really interested in how your sales team deflects them, because I anticipate this legacy software issue will only get worse in our industry over time, and not just on mainframes. I'd like to try to figure out how to bring down the friction of such sales in the future.
Thanks for your feedback, really appreciated.
Aw. I hope you didn't hurt it.
I don't think that's the case. In the place I worked (major financial network) the average time people seemed to have ben with the company was 10 years. Especially in big companies, you can expect to see lots of people who go for a long-term career, instead of hopping from job to job (and promotion).
I think the real reason why management doesn't dare to make big changes is that they understand there's a very real risk to cause a significant loss to the company (and lose their job, as a consequence).
Which is why Cobol shops have absolutely mad processes. Where I worked, to make a change to a Cobol program -even if it was just the JCL that would run it- we had to complete a ten-page form, submit it and wait. They basically just don't let you touch the code. You work around it, sorta.
What do you think about the .NET and JVM Cobol compilers?
They seem to be a way to move that code into new systems with no rewrites as well.
But for the generic case, the mainframe is quite an integrated system, and big clients tend to use a lot of the features at the same time. Even if you can produce an executable or a library, but you ll still need a transaction manager (CICS/IMS-TM), a database (DB2/IMS), MQ, RACF, Datasets, GDG, REXX/JCL, etc. You need the whole ecosystem. And all the binaries will expect a DB2, or a MQ listening on the other side, with near-perfect emulation (return codes, db behaviour such as EBCDIC sorting, etc). Just compiling wont help you much unless you have a very basic use of mostly JCL and batches.
1. Introduction of errors (both overt and subtle)
2. Fixing of what seem to be issues in the original code that are actually undocumented workarounds (or similar things) to potential bugs (in COBOL or who-knows-what-else-in-the-codebase)
Both are big issues in such a move; software engineers don't often appreciate that much of this old stuff is still around because it WORKS. It has been virtually fully debugged, just through the iteration of time and experience.
So porting can easily introduce bugs that may not be apparent immediately; and because software engineers are always wanting to make thing better, if they are working with the COBOL codebase and see something that they think could be done "better" - that change might actually cause problems down the road, perhaps in ways that aren't recognized immediately, or only in edge cases - things that were worked around decades ago in the COBOL implementation.
An ex-colleague used to work for a major US insurance company, and this is what he told me: They had a bunch of products, and due to the state-by-state nature of US insurance regulation, they had a slightly different version of each product for each of the 50 US states. And then, for every one of those different versions of those products, they had a separate suite of COBOL programs to manage them. And, all these COBOL programs basically did the same things, but all with slight variations due to the different rules of each state's insurance regulators. So they basically had hundreds, maybe even thousands, of separate COBOL programs to maintain, a mess of horribly repetitive code. Their problem with moving off COBOL was not that any of the code was particularly complex, just that there was so much of it, and trying to rewrite it into another language (or, maybe more sensibly, translate it into some sort of business rules engine or insurance policy automation product) would take major effort and run a big risk of getting something wrong in the translation. (He worked there more than 15 years ago, so things at this insurance company might be very different by now.)
COBOL will outlive us all.
[Edit: changed period to question mark]
Regarding "how complex can these programs be", these programs run core operations for banks and insurance companies, so you can probably imagine the amount of hidden business logic and technical hacks that have stacked up over decades of operation and changes. This will not surface in requirements gathering meetings without A LOT of blood, sweat and tears. The alternative would be to reenginer the company's processes before the building the software to support them, but that would mean a lot of change and that tends to be harder than most software engineering endeavours.
Not only are these programs complex (projects to re-implement them from scratch in J2EE would certainly be measured in years), but they are mission critical and the organization cannot afford to live without them or something 100% equivalent.
There was a lengthy discussion about this on HN some time ago: https://news.ycombinator.com/item?id=12096250
You'd be surprised. The business rules and exactitude of results can be documented in 1000s of pages.
It’s in a local maximum. Just like all technical debt. And as far as I hear, cobol is used for things that put that risk and liability factor on steroids. I know I wouldn’t touch it without some serious budget to set up massive amounts of redundancy and fail safes.
From a business perspective there may not be much to gain from doing so. If the old software works in a current environment then there often isn't a justification for the cost of rewriting/testing/integrating just to end-up with the same thing in a shiny new language.