COBOL – Still standing the test of time
blog.microfocus.com
blog.microfocus.com
Also, COBOL work is kinda crap. Debugging 20-year-old Java code sucks, but a 40-year-old COBOL project, with source code backed up to 5-inch floppy disks - forget about it. (I just watched the trailer for 'The Many Saints of Newark'. LOL)
I guess there just aren't enough takers.
The old -- and likely not so bad -- advice if you'd like to learn how to use a Mainframe is: Get access to a one, get a copy of latest Red Book [2], and start playing around :)
[1]: https://www.ibm.com/it-infrastructure/z/education/zxplore (this isn't the program I went through, but my old bookmark now redirects here)
[2]: https://www.redbooks.ibm.com/abstracts/sg248852.html (IBMs mainframe Technical Guides traditionally have a red cover)
The pays for COBOL do not even look that compelling and you are walking down a dead end when these systems finally modernize. I wouldn't touch cobol for anything short of $300k and the promise of enough work that I could retire within a few years.
They don't have enough work to train them on and whatever they have is actually critical enough to need someone experienced.
My partner was hired to be a COBOL programmer right out of college in 2004 & trained to work on AS/400s.
The problem was that everything was a small part of a big project, but you got no experience building a big project over its lifetime to see how it should be done. And on top of it, a lot of things which were "the right way" in 1991 wasn't so great in 2005. So a lot of work was actually bringing the existing work to code before you could do the tiny bit of work which is actually needed.
Sigh , those single liner that both illustrate the ignorance and the status of the author.
I’m an enterprise architect in banking , 6 month ago I was hired for IT Transformation.
My mission was very simple « move the bank the out of mainframe »
In 2 weeks or so I presented a Kafka based runtime based with JVM contracts that would enable the bank to perform in a near real-time manner as opposed to « batch » processing while covering and simplifying 90% of banks related scenario ( SEPA , MasterCard , AML etc...)
The project was accepted by directors but devs refused to go into that because much like the authors they are 30 years in the banks and don’t want to learn something else than what they know « cobol ».
90% of our contractors work is spent dealing with mainframe constraint and writing interfaces and top of that piece of crap that can only process data at night or during the weekend.
Mainframe is not there because « it’s superior » , distributed system have largely proven their capability and maturity.
Mainframe are still there because of Corporates Politics and lack of Leadership from top management.
When you are reminded that Citibank lost 0.5 Billions because they spent 0$ on their UI, you may start to understand how much corporates world is rotten to its core and why mainframe is still there.
Has nothing to do with it’s capability , period.
The way to fix that is to use the method IBM used to introduce the IBM-PC back in 1981. They set up a completely independent project group that had no connection with the main-frame boys, and so weren't 'brainwashed' into the IBM 'way of life'. The rest is history.
Incidentally, while I no longer program in COBOL, I still like it. It was always easy to do maintenance on a program that I might not have looked at for decades because of its wordiness. I normally program in C these days, but it's not as 'maintenance-friendly' as COBOL unless there are lots of comments.
> COBOL- still standing the test of time
.. like the plague.
I worked on mainframes and this seems like some deliberate policy not a mainframe limitation.
Also your Kafka+Java architecture is unlikely to still be supportable in 2 decades. Will have the same problems with Java and Kafka in the future as you have with Cobol today.
I doubt that. Java has shown a huge commitment to backward compatibility; you can take code from 20 years ago and run it unmodified today. Kafka is younger but it's also a project that takes compatibility seriously.
Mainframe is still there because crypto isn't yet.
That wasn’t actually on a mainframe or in COBOL, it was an Oracle app (Oracle Forms/Reports, PL/SQL, Java, etc). And, it was a product from an Oracle subsidiary (OFSS), the software itself was not maintained in-house. Although, to complicate the story, that subsidiary was started by Citibank and then sold to Oracle in the mid-2000s.
So Citibank no longer directly controls the decision on whether the UI is updated, now that is up to Oracle. They can encourage Oracle to do that, and decide how quickly to upgrade if/when Oracle delivers it, but Oracle controls the actual UI. Or they could decide to look for a new product to replace it with.
(Disclaimer: former Oracle employee, was peripherally involved with OFSS banking products during my time at Oracle, although I never had anything to do with Citibank, and I never saw this specific banking product either.)
[1] https://www.bloomberg.com/opinion/articles/2021-02-17/citi-c... [2]. https://news.ycombinator.com/item?id=26180785
In the early 1990s, Citibank decided to outsource banking software development to India. Not an unusual decision, but somewhat unusual the approach to it they chose – they set up an Indian subsidiary (iFlex) to develop banking software for them, but also decided the subsidiary would sell the software as a product to other banks. So Citibank owned an Indian subsidiary which developed banking software both for Citibank and also for others. And then Citibank sold that subsidiary to Oracle in the 2000s, and Oracle renamed it from i-Flex to OFSS. Actually Oracle owns the majority of it but a minority of it is publicly listed on the stock exchange in Mumbai.
It isn't just one product, it is a whole suite of banking software applications. I know there are banks who have deployed just certain apps out of the suite and integrated those apps with their legacy core banking. Or, you can buy it all and use it for everything. Given Citibank is the original customer, I imagine they use more of it rather than less of it, but I’m just guessing.
This quote made me pause though, "COBOL remains the preferred choice for systems where application quality and operating cost remain important considerations". How much is COBOL being "chosen" vs. COBOL is basically required b/c of the existing codebase and systems that need to be integrated with?
The latter is my guess, and the choice seems to be really, "what's the best way to extend this existing system?" If that system is in Java, it's probably Java... C it's probably C... COBOL it's probably COBOL.
At the time, the cost of an outage was estimated in the $100K's per minute, and is likely significantly higher now.
When I moved away from FinServ, I found it surprising at first how cavalier some orgs are about ripping/replacing core services in the name of "modernization". Move fast & break things and all of that...
I'm not allowed to talk about it, but I know people in tier-2 support who were on an all night phone call with a big bank customer because if the computer wasn't up by morning the government would shut them down. (somewhat like the government did shut down some big banks in the 2008 crisis, but this was in the 1990s) At the start of the call the rep told them "We are pretty sure we know where the problem is and it isn't you, but we need you on the phone anyway just in case anyone has a question you are ready to answer." That is also why big companies have the million dollar support contracts: someone who can stay up all night and answer obscure questions.
Break things is a bad thing sometimes. I wish more of the move fast and break things companies would realize who annoying it is when they break something.
>the cost of replacement is high enough (and it works fine doing what it's doing) that no one wants to invest the time and money to replace it's usage
That's pretty much what it is. They're not hard to manage as they are and ... doing their thing.
It's a little clunky, but still going strong!
Honestly, I've found ColdFusion is a great solution for a lot of web development problems. It tends to break down with very complex apps, but for a lot of simple-to-mid comexity apps CF is great!
The trade offs are quite apparent, but the value these systems provide is perfectly obvious too.
I’m not sure I buy the extensibility argument either. In my experience the COBOL typically runs on the core processing system, the bit of the bank that runs the central ledger. This is usually a small portion of the banks technology footprint, and the rest of their integrating systems can run anything they like. I’ve worked in quite a few banks, and in my experience .NET is a lot more ubiquitous than COBOL is.
In my experience it's a very narrow kind of reliability, where no-one cares whether the calculations are right as long as they're the same ones you've always run, and no-one's ever validated what or why is being calculated or sanity-checked it. Probably one of the vital constants in the system is 10x what it should be because someone made a typo it in 1982, but that's still considered "reliable".
> How many of the new Kafka systems being stood up today would you realistically expect to still be running in 2061? How reliably do you expect those systems to perform over that 40 year period?
If they were babied as much as the COBOL systems (no changes without months of testing, if data comes in in an incompatible format then we preprocess it...) then I'd expect them to last longer. They probably won't be, but if you decided to run the Kafka systems in a reliability-above-all-else way, I bet they'd do it better than the COBOL ones.
I'm skeptical on a pure hardware basis.
Your classic mainframe was a very expensive piece of kit and outfitted as such. Everything redundant, self-monitoring, and hot swappable. The hardware support and backwards compatibility runs into the decades.
Today we're renting a cloud service, that's made of pig-in-a-poke hardware. Reliable/high availability hardware is simulated with a bunch of disposable instances, at the cost of needing to design systems at the software level to sync and preserve data rather than having it built into the hardware.
I'd be pleasantly surprised if I can snapshot an AWS/Azure/GCP image today, and fire it up in 2041 with no changes. (Hell, we have no assurances they'll be offering native x86-64 hardware in 20 years)
Right, but Kafka's already set up to do that. I've worked on Zookeeper clusters that have been continuously up for 10 years - not the individual hardware, but the overall system was continuously available for all that time.
> I'd be pleasantly surprised if I can snapshot an AWS/Azure/GCP image today, and fire it up in 2041 with no changes. (Hell, we have no assurances they'll be offering native x86-64 hardware in 20 years)
We'll be able to emulate them, and even if not, the Kafka stack is all JVM-based, so you don't need x86 to run it.
This doesn't mean that every function in the code works correctly for arbitrary inputs that appear to be allowed by the function interface (formally a bug!) but it does the right thing for the inputs that have historically been provided (in practice very stable and reliable).
That kind of reliability makes it hard to rewrite the code. There are cases for historically unused input combinations where you have no particular reason to believe the old code is bug free. So when your rewrite produces a different result in an obscure case, it is a long and painful business logic task to decide which version is right.
In my experience that's only true for a minority of them. The likes of e.g. VAR calculations were often straight-up wrong. And even for monetary transactions, you only have to look at the UK post office lawsuits where the code was simply wrong but rather than checking the code they spent years assuming that any discrepancies were the results of employee theft.
> This doesn't mean that every function in the code works correctly for arbitrary inputs that appear to be allowed by the function interface (formally a bug!) but it does the right thing for the inputs that have historically been provided (in practice very stable and reliable).
It's stable and reliable as long as nothing changes. But the world is always changing.
> That kind of reliability makes it hard to rewrite the code. There are cases for historically unused input combinations where you have no particular reason to believe the old code is bug free. So when your rewrite produces a different result in an obscure case, it is a long and painful business logic task to decide which version is right.
Right, but in practice you get an absurd double-standard. The old code is just as likely to be wrong as the new code, but no-one cares about the business logic, they only care about whether the result is the same as the old code.
For example, the entire "mountains of dependencies" mindset feels relatively new. There was a time before .node_modules, Composer and Ruby Gems.
If your codebase is legitimately decades old, your dependencies are either "something we built 100%" or "something we bought/fully licensed and retain 100% control over our copy" Even if you aren't worried about a left-pad incident breaking the entire payroll system, you probably have compliance/audit mandates that are much more easily satisified by saying "here is the master disc that is untouched since the Carter administration".
I'd also expect that the mainframe/batch-processing mindsets have wildly different philosophies on reliability and performance than most consumer services. You can't get away with returning a cutesy animated gorilla and telling people to retry a $7 million B2B payment, or developing elaborate GraphQL-ready chainable APIs when your connected device is a 20-year-old fixed-function terminal with a few dozen kilobytes of working memory.
Now, I'm sure you can build to those same constraints in Python or Node or C# if you want to, but the ecosystem is not really built to support you. There's a lot more move-fast-and-break-stuff.
I've always been conceptually interested in COBOL-- I have that "born too late" mentality; I missed when "you work in the same room as THE COMPUTER" was impressive and you had to wear a labcoat. For me, the problem is how you get started in the sense of "what's the toy problems to even start grasping syntax and build flow." What's the COBOL answer to the "demo to-do list" for web technologies or the "half broken Blackjack game that doesn't actually pay correct odds" everyone wrote in interpreted BASIC in the 1980s? Are there even readily usable toy problems that play well to its strengths?
The systems that managed the forecasting for replenishable and non-replinshable products, markdowns (i.e., discounts) on products in the stores, stock warehouse etc. across the country was completely built on COBOL/JCL+DB2 stack combined with TELON online screens for business to manage the solution/system.
This system practically worked round the clock with hardly any downtime, and in the peak season (like Christmas when the sales/transactions were at it's high) too worked without any major issues. I'm sure IBM still earns a big chunk of revenue from mainframes from these big customers.
I remember an attempt was made to migrate some (if not all) to a new generation data analytics platform and it didn't go very well, so they just stuck with what they had.
One of the main reason I can think of is the the cost of this migration was too high (vs. the realised benefit) as the system just worked for the business and was quite resilient.
I since then jumped ship to the very technology they were trying to migrate too and have never looked back really. I think I can still work my way out of COBOL/JCL + TELON/CICS, perhaps I will forget them very soon :)
world (and a lot of HN readers) needs to change mindset away from this lang versus that lang towards a this DSL and its problem space versus others DSL and its problem space. For me if I had to write a payroll engine I would prefer use COBOL than javascript but would likely turn to java just because of easier to get reference material.
I learned fortran, Pascal, cobol. then worked in COBOL and assembler. then RPG and awk. nowadays mostly python, javascript, some c for projects.
FWIW I aspire to learn and use Ada.
If general OOP-language software engineers had the same army of QA, PR, marketing and sales people that mainframes have you'd probably end up with the same thing. I'd say COBOLs continued existence isn't the main case, it's just a side-effect.
Edit: please add (2012) to the title.
Pleasant, relaxed bunch.
Nowadays a "mainframe" is just a 1U server and it's incredible how some old names in IT can still charge enormous fees for some very old software.
No it isn't. But I grant that there are probably people who incorrectly call their 1U servers "mainframes".
For example, I've got a decent-sized network and I've only ever met one person who works in government tech. And it wasn't through that network. But you can't tell me the government doesn't have legions of people working on its systems.
Even talking with this friend about work is like speaking another language. He builds, tests, and manages huge digital systems, but all of it is done using this product called Microsoft Dynamics, which I'd never even heard of. He doesn't really write any code, but the medium he works in is as complex and powerful as a programming language ecosystem. Yet there's almost no crossover between the jobs each of us might apply for. It's like an uncanny, alternate universe, coexisting with the one I live in. I think there are many such universes and it's entirely possible to go through your career without ever meeting someone from one of the others.
SAP is another ERP parallel universe, with its own language (ABAP), industry certification schemes, conferences, publications, culture...
These vendors specialize in solving the nexus of unfun problems that sit at the heart of every large organization (invoicing & expenses, accounting, payroll, budgeting & forecasting, financial statements…) and I believe them to be quite well insulated from the forces of open source and consumerization that have shaped the wider industry, because there is no consumer/hobbyist path to creating an alternative. Workday has come closest.
I think one of the biggest impediments is that the audience is highly risk-adverse, and the problem domain is highly risk-prone. There's an enormous incentive to go with an industry "known good" even if you know you're getting thrown over a barrel.
ERP conversion are highly, highly likely to fail in terms of on-budget, on-time (more so than the general abysmal record of IT projects). Lack of business ownership/knowledge of processes and requirements is usually at the heart of it, IME. I've been through a couple SAP implementations, a couple SAP upgrades, and conversions to other platforms in my earlier life (Deltek, Dynamics), and they are truly the worst projects on the planet to do.
In the government contracting space, Deltek (which most have never even heard of) is such a winner with accounting departments because they're playing to government auditors - the level of scrutiny from just saying "we run Deltek" compared to some roll-your-own solution is miles of difference and ultimate cost. I don't have any relationship with or love for Deltek, btw, this is just my IRL observation and an example of what the "customer" is thinking about - they could care less about technology stack.
I think most finance departments just "mind the machine" they have - it's really rare (again, IME) to find people in a finance department who have expert knowledge of the key business processes, report audiences/dependencies, and how everything ties together. You can throw a ton of business analysts at it and still find dark corners that went unaddressed after or the night before go-live. And you're impacting things like payroll, benefits calculations, vendor payments, invoicing, taxation, etc. High risk stuff to kludge up.
For example: Hedge fund traders, Game Engine lead devs, scientists, profession esports players, aerospace engineers etc.
I particularly enjoy that D has given a bunch of smart people with no qualifications a venue to do what they love and get paid a small fortune doing it.
That being said, I like languages that came later much more, and absolutely would not choose to create a project today using it. I also wrote a lot of Pascal in the 80's, then moved to C/C++ and Python. Python is the language I've used the most in the 2000's, but now I'm studying Typescript, and think that is a perfectly fine language which I plan to use professionally. I've also studied OCaml and Rust and would absolutely choose the latter now for anything where efficiency was a prime requirement.
I've met quite a few. I spent a year in a mainframe team in Amex and I worked in COBOL, JCL and REXX. I preferred REXX, but COBOL is not that bad. JCL is a horror. I worked in that team in 2016. The median age in the team was around 40 ish but there were a few outliers, including myself and a couple of others. I was also acquainted with many dozens of third-party contractors who worked for Amex programming mainframes.
If you haven't met many COBOL programmers yourself, the reason might be that a) you've never worked in a big financial corp and b) you don't hang out with programmers from big contractor companies a lot.
To clarify, most of my career in the industry before and after that I was working with more modern languages, like C# and Python.
Almost all of them were older than 50 though.
Probably most governments still have them. Social security and health payments from governments probably go through Cobol code.
"That oil in the fryer at your local bowling alley that has gone rancid but they refuse to change it because the guy who knew how to drain the old oil quit - still standing the test of time"
While many also debate the status of new oil in relation to rancid oil for french fries, rancid oil remains the preferred choice for deep fryers where knowing how to change the oil and cost remain important considerations, so often the case when addressing the ever-present issue of stuff only Bill knew how to do. When many bowling alleys are facing mounting problems due to not making sure someone else could drain the oil besides 'their Bill', the average cost per gallon of oil for rancid oil already in the fryer was projected to be £0.00 whereas the cost to go to Costco and get 15 gallons of new oil was £8.73, according to Chelsea who is at Costco like every day.
* It has a data division which lets you describe nested, repeating, and overlapping data at the byte level with a clear, unambiguous notation.
* Numbers are treated as decimal values (generally) and math is likewise done in decimal, not binary.
* It is profoundly cross-platform. You can take a memory-map from one architecture, slam it to disk, transfer it to another machine with a different architecture, load it into memory, and have perfect fidelity.
What other languages have these properties?
Yes, it is clunky as hell. It is not good for many things. However, for its intended use, it is very good.
You can do a lot more with a line of Java and, of course, with a line of Perl or Python you can do very complex things.
That stands to reason, yet much software is decomissioned or replaced every year. All of it deterministic software. Perhaps the software is expensive to maintain or use, doesn't adequately meet the needs of the business, is error-prone, or any one of a million reasons.
COBOL works at the heart of our industry and our society, despite COBOL being an ancient crufty language. In the same way that may of us never see our hearts or become heart surgeons, I may never meet a COBOL program or a COBOL programmer. I am still thankful they are there, and I recognize they serve a function other software likely would not.
Deprecation is a new term in mainframe.
Don't fix what's not broken
------
It's a common saying that mainframers don't try to learn new tech. Well that saying goes two ways, person saying has really tried to learning mainframe and work on it for a month. Actually there isn't much to learn other than couple of languages COBOL, REXX, JCL and SAS. Other than that it's just ISPF interfaces.
Mainframers actually have to learn tech once and then go on to learn business logics and business domain. Isnt that actually good when comparing to always learning new tech and writing migrating programs to support existing interface in new Lang but not too much about business you are working on.
On the other hand mainframers do spend on learning and upskilling but in mainframe itself. There are lots of IBM Usergroups like IDUG for instance which is related to Db2, being a DBA that's a very useful training programme for me.
COBOL codes are highly modularized to do one or couple of functions to my knowledge and there are definitely coding standards each shop maintains.
Having a bunch of nodeJs cloud functions i get an email from Google saying that I need to migrate my NodeJs or it will stop functioning and while trying to migrate i find couple of modules are not longer supported and I need recode. Here I am not adding any new features just trying to make it work as before with new library which I have to find, learn, test and analyse does it suit then recode. All this costs time which you don't have to do in mainframe. You can say why not run in a VM, then what's the point running in cloud.
Mainframe is not a legacy system. IBM introduces a new machine every 3-4 years and send their sales team to visit corps and run presentations :) . Currently IBM has changed their billing style, it's not flat or peak hour, it's custom. It's the smallest footprint in your data center.
What mainframe actually did is, job separation. So in a career a person will sort of become an expert in their line or software. He will know all the ins and outs. For example a RACF security person will not know about Db2 and how it functions only part he knows is what interfaces with RACF. Same with CICS, MQ.... From application/dev team they get to know business like in a requirement meeting when business talks devs will be mapping business with program modules they had to make a change.
In mainframe people get to specialize.
IBM Redbooks are quite good.