Hacker News Reader

Comment by _8j50 | Hacker News Reader

Right... they can't train someone on COBOL? I call b.s. on that. It's an HR issue, you need blah blah degrees and yadi yadi yada experience. Big corps don't adopt to job economy changes over time like this unless their core business is built to adopt to economic changes. Inability to make exemptions and place the right first line managers over teams is what kills big companies, or forces them into an endless loop of unoriginal aquisitions and failures imo.

View on HN

Banks are genetically incapable of doing anything useful in terms of technology. Their mainframe architecture was outdated 40 years ago! They will forever maintain their hacks and buggy software.

Please define "buggy software." Can you point to a single confirmed case in the US of a person's bank account balance disappearing, or even a deposit not appearing, due to a software bug? I doubt it. Banks' core software is rock solid or they would be out of business very, very quickly.

Transaction reversibility, fear of law enforcement and the ability to react to things is the reason why the bank tire fire doesn't collapse on itself.

im sure glad the cryptocurrency folks seem intent on throwing all three of those away at once.

It is not, based on the loopholes one has to jump through for basic support.

Banks have procedures and regulations that protect customers from those issues. In fact there's a lot of of secondary checks and offline reconciliation precisely to catch potential software errors.

Not the US, but in the UK which I think is comparable:

* https://www.theguardian.com/money/2015/aug/28/many-hsbc-cust... (I believe this one was caused by firing greybeards who understood a certain very complicated batch processing system and offshoring to undertrained staff in India who then make a mistake and basically brough the entire system down as a knock-on effect)

* https://www.theguardian.com/business/2019/nov/19/tsb-it-melt... (Insufficiently tested software upgrade although to be fair this was a major upgrade)

* https://scottishfinancialnews.com/article/rbs-it-failure-pre... (Not sure on root cause. Other interesting stats at the bottom of the article regarding failure counts)

Now, in all these cases I believe the outages lasted a few days at least where people had limited or no access to bank accounts, salaries not paid on time, incorrectly charged overdraft fees, automated payments not happening etc.

It can probably be argued what is "core" here and as far as I know no money actually disappeared into the ether but, in the UK at least, I am not sure I would be so confident in my declaration of Rock Solid software.

Thanks, these examples confirm my point. The old, ancient, "outdated" (as the original poster claimed) software worked fine for years/decades. The failures that you cited were introduced during an upgrade, a rewrite/outsourcing, or due to a 3rd party outage.

But requirements change, so upgrades are a normal part of business, due to at the bare minimum legal reporting requirements changing, even if there weren't all sorts of new things like internet banking making their old APIs inadequate.

If the software isn't extendable or changeable then it isn't 'working fine', because if you can't offer internet banking or do FATCA reporting you go out of business or get shut down.

You've cited two comparatively minor failures confined to a very small subset of what banks do. There's a whole lot of critical software systems in banks that don't touch personal bank accounts. If you've ever worked on bank systems before, it was certainly in a very limited capacity.

I did code-level support for a large internal data management system for the federal reserve bank in the mid aughts. I'm not really interested in disclosing the specific bland but critically important data we moved/managed. It involved a giant amalgam of legacy commercial AIX and HP-UX software with some newer Java stuff in a Websphere environment. It was all duct-tape-and-bubble-gummed together with huge ksh88 scripts that were in constant development by devs and support people like me who'd just ssh into production systems to make changes. None of the internal software had dev/test systems, there was definitely no code review, and there was no version control. Yep. The system on a whole required constant monitoring and intervention to keep it plodding along. We did periodically see data loss that was unrecoverable from a technological standpoint but recovered with physical media kept around from their necessarily belt-and-suspenders business practices and tons of available staff for data entry. I reckon tens-of-thousands of folks outside the bank would have been affected per incident.

Money disappearing from a bank account is a "comparatively minor failure"? Sorry, I stopped reading at that point.

There have been major bank tech failures too, if you wanted to see major vs minor side by side...

If you're committed to not knowing something then I can't stop you.

But compared to, say, tens of thousands of investment transactions for pension and retirement funds evaporating, disbursements to a hospital's operations budget being delayed, sending confidential historical balance records in bulk to a large corporate customer's competitor, having an online investment system down during a period of extreme market volatility, or even large-scale customer data breaches, then yes. 'Money missing from a bank account' is in every significant way a comparatively minor problem probably fixable with a phone call and a couple hours of investigation on the bank's part, even with a significant sum of money. The bank would almost certainly catch it themselves during an automated audit. It's exactly the sort of problem that bookkeeping organizations are designed to mitigate and root out if they do happen.

The others could affect many, many more people's lives for much longer— potentially permanently.

Money disappearing is a minor failure in the sense that it is easy to track. Banks do daily audits to check that money sent in one side is deposited in another. If there is any issue, they will track it and rectify within the 3 days grace period created exactly to work around these bugs.

Has that really not happened to you or someone you know? It is (typically) eventually recovered, yes; and you can often understand how/why it happened. But it definitely happens, and requires manual human intervention for fixing. Here's 2 incidents that happened to me or friends in the last month:

- took a trip to the neighboring country, my friend tried to take out money from an ATM; first two transactions failed, third succeeded (at the atm). In the bank though, all "succeeded", and money were gone (first 2 eventually were reversed with manual intervention).

- I sold a house (in a remote mountain village), buyer sent 8100EUR from Germany, 78xy.z EUR made it to my personal bank account. Apparently due to multiple currency exchanges, but this is a SEPA transfer between two accounts both denominated in EUR, this was absolutely not supposed to happen and nobody was able to articulate exactly "why it happened" (or even exactly what happened). For this one the buyer decided to just eat the loss and sent me an additional 300EUR.

What do you mean by 'useful' here? Facebook goes down for half a day and everybody laughs. Take 'banks' down for the same amount of time and see what happens.

It's pretty standard for banks to "go down" outside the hours of 9am-5pm. Nearly everything meaningful that a bank does is processed in batch, often with days-long delays.

Operationally, Facebook is a vastly harder problem.

Multi-hour outages are already relatively common and getting more frequent by the year for banks. About 10 days ago there was a massive Bank of America outage.

You certainly don't use banks often, because their systems being offline for several hours is a relative common event.

Not if they can’t find anyone willing to learn COBOL.

There’s huge demand for developers on modern systems. Seriously, how much are you going to pay a developer to do mainframe COBOL - and is that developer going to produce commensurate value, really able to hold together a crumbling antique infrastructure?

Fwiw, one of the highest paid coders I ever knew worked for a big stuffy bank. He was unfireable because he was the only one that knew the internal systems as well as he did. Learning COBOL may be a huge long term competitive advantage in the workforce, especially if the current dotcom-esque funny money dries up.

Plenty of people working helpdesk that know a bit of python who can learn COBOL if they knew a job offer was on the other end of the investment in learning the language. There are hoards of people learning python and powershell just so they can make more money, trust me when I say they (the people I see learning python for career prospects) have no sentimental attachment to the language.

I don't think it's learning COBOL language that's the hard part. An inexperienced programmer who hasn't worked on financial systems is going to make security mistakes and write buggy code for awhile, both of which can be deadly to a bank. But it would be better to have that person rewrite the infrastructure than try to fix bugs. The problem with these ancient systems is trying to understand how bugs arise from subtle unintended interactions within millions of lines of code. I've been coding many languages for 25 years and I would consider it extremely daunting to try to tackle a bank bug knowing that one wrong assumption about one line of code could crash whole systems or cost millions of dollars.

Eh, someone's got to do it. Letting thing stagnate in perpetuity isn't cheap either. Move slowly and write tests.

Sure, but a veteran coder at least understands that tampering with something may affect some other subsystem you don't even know exists until it breaks... which you wouldn't even think to test. The idea of throwing the phone staff into COBOL instead of them learning Python just obfuscates the real problem which isn't that the language is uncool and dying but that the systems are like the rules of the NFL. They stopped making sense a long time ago to anyone who wasn't immersed and paying attention.

Ah, yeah, fully agree - I just saw your mention of having 25 years experience, which made me think basically "if not you, then who?"

I didn’t intend to imply that the problem could be solved by an army of people fresh out of boot camps with no senior level supervision.

I'm also realizing now that just because you "would consider it extremely daunting" does not necessarily mean that you don't think you could be a net positive contributor.

For enough money they will find someone willing to learn COBOL.

Maybe, or maybe not.

Suppose you accept 1.5x or 2x or even 3x your annual salary to take a COBOL job. Nice... for now.

What happens in 2, 5, or 10 years when this COBOL gig ends?

Now your skills have withered or disappeared and you're 2, 5, or 10 years behind when it comes to "modern" technologies where 99.9% of the job opportunities are.

In a sense, that COBOL job is a gamble. You are gambling that you can ride that sucker until retirement.

Of course there are other possibilities. You could sock most of that extra $$$ away and spend a year retraining yourself, or something, if/when that COBOL job ends. Or whatever.

I'd rather learn COBOL and get paid 2x salary for the next 5 years than learn some BS JS framework that I'll have to forget in the same timeframe.

React has been around for 8 years and isn't going anywhere anytime soon.

Staking a career on something that has existed for only 8 years is a huge gamble. Most people build careers around concepts and tools that have existed anywhere from decades to millennia.

> What happens in 2, 5, or 10 years when this COBOL ends?

What happens in 1 or 2 years when the current hottest new full-stack, mvc, reactive 3.0 whatever is old and tired and you have to learn the new thing? You learn the new thing.

In fairness, the new things tend to build on top of each other. You can get off and on the treadmill but it comes with a cost.

Sorry, I forgot a word there. Post now edited.

I meant, "when this COBOL gig ends", not "when this COBOL ends" which is what I initially typed.

I agree that COBOL will outlive a lot of these modern flash-in-the-pan frameworks.

However, the number of COBOL jobs will dwindle over time, and if I took a theoretical COBOL job today it's not clear to me that I would be particularly employable if/when that job ended. Particularly if I am unable/unwilling to relocate as I suspect COBOL jobs may be much less likely to be remote.

Whereas, in comparison, if one stays current in Ruby/Python/JS/.NET/Java then one's prospects would remain strong for the forseeable future. "Staying current" on the tech treadmill is of course its own special hell, but it is probably the safer road.

    What happens in 1 or 2 years when the current 
    hottest new full-stack, mvc, reactive 3.0 whatever 
    is old and tired and you have to learn the new 
    thing? You learn the new thing.

OK, but how employable are you at that point?

It's going to take some significant effort to catch back up.

Perhaps more crucially, your resume is not going to be appealing to potential employers. If they're looking for hotshots in the latest framework, do you think they're going to look kindly upon a candidate who's spent the last X years working with COBOL?

There are ways to address that, such as getting up to speed in Trendy Framework XYZ and making some portfolio pieces, open source contributions, etc. But, that's not always a smooth road.

>OK, but how employable are you at that point?

>It's going to take some significant effort to catch back up.

You got 2x (or whatever) the salary until then. That should give you plenty of time to catch up. Currently anybody is employed anyways if one knows how to use a keyboard. I don't think that is ending anytime soon.

    You got 2x (or whatever) the salary until then. 
    That should give you plenty of time to catch up.

You can catch up, but your resume can't.

Let's say you take a cushy COBOL job for X years and then spend Y years on your own "catching up."

At that point you're pretty proficient in whatever the trendy thing is at that time.

But what do recruiters see? They see, "this person hasn't worked with anything modern in X+Y years, and hasn't even had a job in Y years"

It can be overcome, but... not ideal.

    Currently anybody is employed anyways if one knows 
    how to use a keyboard. I don't think that is ending
    anytime soon.

It might still be that way in X+Y years. It might not.

Will they learn enough COBOL to be worth the money? Seeing how hard it is to find good mobile developers, and having lived in a COBOL shop back in the heyday, I don’t see a meaningful number of talented “show me the money” types passing up modern tech for antiques.

>> crumbling antique infrastructure

Are you talking about the hardware or software? It's worth considering that Lindy's Law [1] might apply here.

1. https://en.wikipedia.org/wiki/Lindy_effect

Reply on news.ycombinator.com