Our Government Runs on a 60-Year-Old Coding Language, and Now Its Falling Apart

COBOL isn’t falling apart, the systems are exceeding their design limits. “Oh if it were just written in a modern language everything would be fine”. Not true - we create modern systems every day that would choke and die if they had to scale several orders of magnitude instantly - in fact it happens about weekly when an unsuspecting link hits the HN front page. Also, I would wager 90%+ of software written today is 10-100x more fragile over time than whatever COBOL software is on these machines. What do you think the chances are of those 100 fancifully orchestrated micro-services running 10 abstraction layers up on 3rd party infrastructure are of running at all a decade from now, never mind within their original design constraints.

Old doesn’t necessarily mean worse and newer doesn’t necessarily mean better, but that logical foundation is easy to forget in a tech culture so often myopically focused on innovation(tm) and short-term profit or austerity above robustness.

The relative rarity of COBOL programmers aside, this is a just an garden variety f* up of keeping legacy systems around too long because they aren’t “broken”. And like technical debt, the money and effort to fix them, despite many warnings and pleas, is called a waste because there is no immediate benefit or profit.

Then something like this happens, and everyone forgets the shortsighted management or decision makers that ignored the risk and collected bonuses and praise for saving the Corp/taxpayers money. Those who warned about the debt coming due get nothing except being able to say “I told you so” for a few weeks until the whole thing blows over.

Attributing these problems to COBOL is short sighted. While COBOL is obsolete for good reasons, it does provide high performance and makes it relatively easy to construct new batch-processing tasks quickly, due to its built-in data model (a concept which is largely absent from modern programming languages but does pop up from time to time in forms like MS LINQ - historically it was far more common, see also MUMPS). Because COBOL applications run predominantly in batch-mode they are also fairly robust and easy to troubleshoot compared to many modern systems, isolating a problem being a matter of running the batch jobs individually and finding the bad intermediary state, and recovering from a failure being a matter of just restarting the job. Intermediate state is almost always persisted to storage so it's easy to work back to where the problem occurred, unlike in e.g. a microservice architecture where doing so can require a dedicated tracing system and still be quite difficult.

Consider that we could write a headline like "Amazon runs on a 50-year-old operating system, and now it's falling apart." The sentence is not entirely untrue but rather misses the point.

The problem is this: these systems are old and, more significantly, have received extremely limited maintenance and enhancement over their lifetimes. This means that there are few people familiar with each system (a "bus factor" problem) and, more broadly, that the talent-base of COBOL programmers has been allowed to recede to near-zero. While a common modernization (or "recapitalization," a term preferred in government contracting) route in the tech industry might be to take a legacy system written in e.g. Java and progressively reimplement it in e.g. Python with a simplified architecture, a common recap route in government is to take a legacy system written in COBOL or s/370 assembly and transition it to an s/370 emulator running on HPUX on NonStop. I've been involved in such processes a couple of times. It has the advantage of maintaining the reliability properties of the legacy system at relatively low-cost since very little software work is involved. It has the tremendous disadvantage of allowing the organization to continue to not touch the software, with much of the work often outsourced to a vendor who need not be very familiar with the codebase either

You certainly wouldn't write anything in COBOL in 2020, but COBOL itself does have a robust and even fairly maintainable design. The problem is that any application written 20 years ago or more, regardless of language, is going to be tremendously difficult to modify to meet changing requirements and increased demand. Unemployment insurance applications reaching six million per week, for example, would strain a system designed using the most modern methodology which was specified for a few hundred thousand. The difference is that, with a newer system, there are likely to be in-house engineers or vendors who are very familiar with the system and will be able to expand its capacity more rapidly.

Another underlying problem is that legacy systems designed using a mainframe methodology are intended for vertical scaling - to process more records you buy a bigger machine (or expand the LPAR or what have you). While horizontal scaling has its own disadvantages there's a reason it's so popular today - it's a lot faster to buy more capacity from a cloud provider than to requisition a larger mainframe^w midcomputer^w emulation host. There are few vendors and government acquisitions are slow at the best of times, even under emergency sole-source rules.

In a way the whole thing is ironic, because a lot of COBOL application interfaces are faster than any software we use today. From the perspective of a human operator they can be absolutely outstanding once you get over a learning curve. However, the lack of any enhancements made to these systems mean that requirements changes are often "patched in peopleware" if you will, and you end up with clerks that can enter data an order of magnitude more efficiently than with a likely modern solution, but they then have to send the records to a virtual printer to be ingested into a Java EE solution... this kind of "enterprise patching" rapidly adds up to an extremely inefficient system.

What is far more scary than the old stuff our government is running in COBOL, is the new stuff running in Excel. Excel became a de facto Common Business Oriented Language (COBOL), accessibly to anybody. Old COBOL programs were still developed with some software engineering discipline. Current Excel spreadsheets have none of that. And, this doesn't apply to the government only - private enterprise runs bazillions(?) untested, unverified spreadsheets. Wish I could share what I've seen.

My first computing job, many years ago, was maintaining a large, COBOL program for an insurance company. I doubt the problem the American government is now facing has much to do with COBOL in itself.

It probably has much more to do with the fact that these COBOL programs were designed to run on IBM mainframe computers and to interact with various proprietary IBM subsystems (with names like CICS, VSAM, VTAM, DB2, etc.) These subsystems are non-trivial to learn and there’s been little/no incentive for anyone new to learn them for the last (at least) twenty years.

Finally, back in the 80’s and 90’s, some programs were written in IBM assembler language instead of COBOL, for example when there was a need for greater speed. Understanding and maintaining (or porting these programs to a non-mainframe environment) is also non-trivial.

I left the IBM mainframe world completely 20 years ago, switched over to working only on Unix/Linux systems, did that for 20 years, and am now happily retired. However I’m sure there are still lots of older IBM mainframe people around to handle these problems, they just cost money.

Not a great headline really, C is nearly 50, but most people wouldn't consider it a disaster for everything to be written in C. The problem isn't that COBOL is old, it's that it's largely obsolete.

The problem with rewriting the COBOL code base in a newer, sexier language, is that this new language will itself become obsolete in ten or twenty years. Then you're back to square one re-writing the app in another newer language.

So businesses faced with this problem have a choice: accept that their code is written in an old language, and train programmers to learn it. Or pay big money to convert their code into... a language that will soon be obsolete, in an endless treadmill.

Isn't it better to be obsolete and not pay conversion costs, rather than be obsolete and pay perpetual conversion costs?

Yes, programmers are more productive in newer languages. However, you are never going to get rid of the old stuff, so you will always need some programmers versed in every language, at every stage of your conversion hamster wheel.

I think that The Onion should run an article: "Coders Shocked to Find Accounting Systems Written in a Language Designed for Accounting Systems."

It seems like the age of something is the argument.

I starting thinking of this and thought:

- Much modern AI software depends on 63-year old fortran numerical methods code.

- the Brooklyn Bridge is 137 years old.

- the White House is 220 years old

- Aristotle was using logical reasoning over 2300 years ago.

Now I know how to reply when someone posts “I have a week with nothing to do. What language should I learn next?” COBOL.

Seriously, anyone with some programming experience can learn COBOL or any other language in a few weeks. Maybe with so many hip startups for concierge cat sitting closing down more programmers with time on their hands and no paycheck will crack the McCracken book (the one I learned COBOL from ages ago).

Informative comments. Thanks!

*60-Year-Old software systems