Why and how COBOL is still used

True, I started as a COBOL programmer (did a lot of TELON online screens, JCL, CICS, DB2...) as my first job in 2009, for a biggest retailer - personally it taught me a great deal of software engineering - having a background in electronics engineering helped I guess.

The whole stack, and the complex business built around it over many years makes it so difficult for these large institutions to port their solutions on to a new platform in my experience too.

For a lot of these companies, these backend systems on Mainframes just works and I believe there isn't a proper incentive to move on to a new platform anyway.

People tend to think migrating an application is all about rewriting the program itself. But actually rewriting is only a small fragment of efforts needed. For some large organizations the legacy code base is so large that it would be crazy to rewrite them in a different language in the first place. This does not event mention the effort to test or validate if the new application works as expected. Put those thing aside, building a new infrastructure with PC servers and hire/retrain the operations would almost much larger than anticipated by most of the people. After evaluating all the options, scale up/out with MFs would be the cheapest and most reliable solution to provide more capacity for a perfectly working application. IBM has done a decent job on keeping MF backward compatible. There are lots of core banking systems running S360 Assembly applications on latest MFs today, in comparison, COBOL is not that old at all.

You can absolutely make a web interface that is snappier (not quite like a terminal application). You need to spend a lot of time with your layouts and jettison a lot of your web assumptions (like using javascript for a lot of things, spending a lot of time on tab progression, and forget about responsive designs). My current job when I started maintained a web app for a medical research outfit. The person that developed it came from a COBOL/RPG background and applied that knowledge to this application (which required tons of data entry via web forms). It was a python backend, minimal JS on the front end, and our data entry folks could fly when they were entering data. The endpoint that the data entry screens pointed to were implemented with speed in mind so saving a form was almost instant (from the user perspective).

For a long time it was also the largest program that had been formally verified. Not sure if that is still the case.

In what way does the security best practices of the programming language are different?

(I've worked at a security related startup that worked with mainframes which used cobol... The security practices of the runtime do not seem to me less advanced than what we do in other languages.)

Presumably cost is the biggest issue preventing MF adoption for new users - hasn’t the hardware and software been IBM’s cash cow for a number of years?

BLIS/COBOL has entered the chat.

Of course not - that's what IBM Basic Assembly Language is for!

Having the right tool for the job is always a joy.

Yes. Cobol is not C/C++. Nor does it try to be.

>If the systems are so vital surely there is a detailed specification

In my experience those things not always go together.

I've seen "mission critical" software with basically zero formal specification.

Maybe this also happens with COBOL?

I bet there were numerous failed attempt at modernization with these platforms over the years, and after wasting enough money/resources on them, management made a blanket decision of No More.

I think you greatly overestimate how well software is documented. Just one example: When the Y2K problem was discussed in the late 90s, one of the big problems was that almost nobody understood the decades old code base. They hired experts, of course, to make sure there were no difficulties when they switched to the year 2000. They made such hair-raising discoveries in some code trees that a lot of the variables had completely useless names, like "Mickey Mouse".

A lot of knowledge is probably institutional knowledge, especially about how the different systems interact (over the decades everything starts relying on implementation details and bugs of everything else). And the people who hold that institutional knowledge either left at some point in the last 60 years or have no interest in undermining their job security by telling you

> writing in an easier to maintain language would repay the effort

COBOL is very easy to learn, but, as was pointed out in [1], it's not only COBOL, but COBOL and CICS. CICS is an extremely optimized transaction manager that's been continuously improved and has been coevolving with IBM hardware since the 60's.

There is also the workflows around the very non-Unixy nature of zOS. IBM has been doing an excellent job at introducing good operations practices and adding tools and functionality to facilitate that, but it's a long process.

Also, remember these machines also run Linux incredibly well. They are unbelievably fast.

1- https://news.ycombinator.com/item?id=29229525

I have retired some COBOL in my life, I can’t say that my experience is universal, but here’s what happened during my time consulting for a Fortune 500 company sunsetting COBOL into Java (1.4).

> If the systems are so vital surely there is a detailed specification

This is hilarious, and wrong. There probably is a specification somewhere, but there is drift between the implementation and the specification. When that 4am bug pops up and stops processing a billion dollars worth of transactions because of a weird corner case, someone fixed that to get the system running again, did they update the master specification, maybe. Half of the job is figuring out how the system really works, all the bugs, all the quirks, all the things that other systems have come to expect.

Sunsetting is often a bug-for-bug type of conversion, because the output of COBOL programs are similar to database tables. Other programs are built on top of this data and that consumer code introduces a bunch of expectations on how the output of your replacement must behave to keep everything working. The number of consumer programs that have some expectation on the data can range from dozens to thousands.

So you can either read and understand and fix all the consumers as you make your replacement behave to spec instead of behaving identically, or you do what most sunset teams do and you make it produce identical output.

This now means that you are reading the target code base and understanding hundreds of thousands of lines of COBOL, you are unearthing decades of extensions, bug fixes, hacks, and mistakes.

Your job then is to make sure the replacement behaves exactly the same way over decades of input. This is the best test of course, learn JCL and beg the mainframe operators to let you run a massive job that will process historical records with the target and replacement programs and compare the results.

Your days begin to look like the following. Get in, get the report from last nights run which tells you all the discrepencies. For each one you can run your code and the cobol locally with some work. Formulate a theory for why some figure is wrong, fix it, move onto the next. Do that for 8 hours. At the end of the day kick off your massive JCL and do it all again tomorrow.

Just do that everyday for a few weeks or months or however long it takes until your replacement program behaves the same as the COBOL program and then you can swap them without breaking 1000 other applications.

Then you get the long tail of bug reports. The zipjack flim-florp report isn’t working. So now you dig into that, the flim-florps are processed by the COBOL you replaced, so maybe the zipjack system had a requirement you didn’t realize. Dig dig dig, days go by, your will to live eroding, but finally you find it, deep in the CVS history bug-1376892 looks like the target program accidentally added an extra hour to the business day in Seoul on leap years because the time zone data is wonky and they have special code expecting this wonkiness and correcting on their side because the mainframe programmers in 1994 decided it was too much work to fix on the mainframe side.

Our program uses a different timezone database and it knows how to properly calculate this, so just add a data transform at the end to detect this scenario and fudge the numbers back to what the zipjack (and who knows how many other systems) expects.

The COBOL systems are rarely used directly, they are batch processors taking in data and pumping out data. Applications are then written on top of their output and bind the behavior of the COBOL system with all their expectations. And this is where the complexity lies, the specification for how the system works is possibly written down somewhere, but the actual specification is in the COBOL code and the hundreds or thousands of consumer programs and all their expectations. To do the job well you have to write something that replaces the actual real-world specification encoded in all that other software, not the one jotted down in wordstar a few decades ago.

I had a brief exposure to a COBOL application and started to understand some of the technical hurdles that make this hard and largely lead to 'big system replacement' type projects which are extremely high-risk.

If you've ever worked a VisualBasic Application, you'll have some inkling of what COBOL is like.

One of the core things I saw with COBOl was strong coupling of application and data persistence. Further, where modern apps take a stateless approach for say a single API call, many COBOL application predate this concept. So not only are application and persistence coupled, the applications are VERY stateful.

This makes replacing a 'piece' of the application extremely difficult as there is a lot of shared state to tease out of the system. Nor can you start building a second system that owns some piece of the data model, as COBOL's persistence models aren't built for shared write activity that a normal DB facilitates. You are forced to still write through the COBOl data model.

Clearly not impossible, just difficult.

In many cases these system may process millions or billions of dollars of transactions a day. They may be a specification or there may not be and there are probably a dozen corner cases that are not specified. Nobody wants to be the one who decided to replace it with a shiny kube cluster running node js or something and then have it all go up in smoke and losing money, customer, reputation.

Realistically If there were detailed specifications, they were hard copy print-outs from the 80s that are probably no longer readable, and undoubtedly were thrown away in the intervening decades.

Many of these old systems rarely require changes in most modules. (As in, of the hundreds of thousands of lines in the system, many subsections just never get touched).

This means for the the parts that never get touched, there is a negative return on investment for rewriting them, so there has to be a correspondingly larger positive return on investment in the higher churn parts to make a rewrite worthwhile. Most of the times when this is studied, the return on investment from the rewrite is either tiny or net negative from this effect.

Ah you say, then just rewrite only the parts that tend to change a lot, and leave the other stuff as COBOL. That sounds good right? Except that generally the cross language interop code needed becomes an absolute nightmare. I've dealt with this in an near ideal enviroment, and it was still a royal pain.

This near ideal environment was: COBOL code restricted to just a subset used by the company, which was being translated to C code by the companies own compiler, which allowed for a C-compatible ABI to be defined allowing for surprisingly simple and direct C to Cobol interop. Yet even then the code code was still a royal pain, just due to how COBOL data structures differ so much from the idiomatic data structures for C or C++ code. My code was actually C# code interacting via C++/CLI with some C++ code that interacted with the the COBOL. But the interop code was still some of the worst code to touch.

Basically it very difficult, because no one truly no one understands the whole business logic implemented in Cobol. Take life insurance. You have contracts running for decades. You just don’t want to break something. There are no tests and so on. So if you would start from scratch you would be faster. So it’s just very very expensive with a lot of downside risks and little upsides.

> If the systems are so vital surely there is a detailed specification and writing in an easier to maintain language would repay the effort?

As others have said, no, there's probably not a complete specification. But even if there is, it's not nearly complete enough.

That code may have 50 years of bug fixes and corner case fixes in it. How many times was the spec changed to reflect those fixes? Probably not always. If you rewrite it from the spec, how many of those issues are you going to re-introduce?

And, how many new bugs are you going to introduce? The old code has 50 years of "getting the bugs out". It may be hard to maintain, but for what it does, it probably does it pretty solidly. Will the new code be as solid?

Re-writing isn't as easy as it looks...

Here's the original spec in Interleaf v5 format and a few thousand pages of notes and emails. We might have a 25 year old PC we can boot to print that.

That said, all that code has been running for 3 or 4 decades and anything that gets ported will have a shelf life of < 10 years.

IMO One of the reasons is the tight integration with other IBM Products and the operating system. In short you can do distributed, networked transactions seamlessly with COBOL, involving files, database and messaging all at once, without the programmer even knowing what distributed transactions are and how to implement them "manually" - few do.

BanaMex's core runs on VMs that still run COBOL.

(Source: I met one of the debuggers while I was in Patagonia. He is in his 70s and sails around the world with his wife and fixes bugs when he's near WiFi. Either he was telling the truth, or he's a really good storyteller.)

haha that was my thought as well. That's just a different way of punishing yourself!

That's the point isn't it? e.g. you might say "I would rather stub my toe 1000 times than run COBOL once". The comparison only makes sense if it's something you consider to be bad.