New Jersey needs COBOL programmers for their unemployment claims system | Hacker News Reader

New Jersey needs COBOL programmers for their unemployment claims system

twitter.com

447 pointsenraged_camel295 comments

I was sort of tempted to pursue this. I did a few years in the salty COBOL mines back when I was a PFY about 20...plus years ago. This niche probably hasn't moved very much since then and I could probably recall most of it after a few hours. I'm freelancing right now anyway and some other projects just vaporized.

But.

A quick skim of the postings is finding that what they're really screaming for is somebody who's been familiar with their particular pile of toxic waste for at least four years at minimum. They're asking for people with 4+ years of COBOL, and JCL (ok, fine), and DB2, and VSAM, and "SDLC" (proto-waterfall), and, oh yeah, a four-year degree.

Heck, there's a posting asking for somebody to draw diagrams and document their existing processes, and for that they expect you to have 10 years of COBOL experience.

Okay then. This is not yet a mess that they're motivated enough to fix.

Or they, like many non-technical organizations, have no idea how to hire programmers. Could be worth a shot; even at regular companies they often over-state resume requirements.

As someone who works with a lot of state IT people, if they don't know how to hire programmers, they probably don't know how to manage programmers. In a bureaucracy, if the administrators are making these kinds of unrealistic hiring demands in a crisis, it probably means that the administrators outrank all programmers and don't trust them to make their own decisions. This is not an environment to work in.

This is a crisis, not a normal time. Fixing this system will make the difference between real people getting unemployment checks or nothing. That's worth enduring a bad boss for.

Did you apply? I think it would be interesting to hear the story of what it looks like inside if you decide to work with them.

Problem is, that's how they get away with everything.

They fix nothing, then wait for somebody to need them enough, or have compassion enough to handle their mess.

Basically here, they are saying: we didn't listen to all those competent people that told us we should have cleaned this system a long time ago because one day it will bite us back. We didn't think technical debt was a thing. It has always worked. And it's virtual. And IT is always exaggerating anyway.

Now it does bite them back, and do they take responsibility ?

No. They ask for competent people to have mercy and save the day because the situation is dire. Because this is special this time, unlike the other times.

This is completely not like with the health care system that never got fixed and now the medical personal is paying for it so that the people don't suffer as much from this mismanagement.

They will not pay more. They will no fix anything. They will not respect you more, change the way things are done, or take any responsibility for this. And in fact they will probably guilt trip you into give up your health to overwork on this with less resources and for less pay because it's an emergency.

A friend of mine works in an hospital right now. She has been at work for 15 days straight. They already told her she won't get paid for the days she was not supposed to work but did.

Then as a final fuck you, if this turns out ok, if the catastrophe is avoided, they will get the reward.

The nurses or the programmers ? They will be forgotten in a week. Nothing will change.

But for the mismanagers, their superior will thank them for managing this situation well. They may even get promoted and have more responsibility.

So you have the choice between saving the money of poor people now and reinforcing the suck of the whole system, of letting people suffer for a chance of a redo and purging the ones that lead to this.

This is a terrible choice to have to make.

Especially since we have the empathy to make us discuss this, while clearly they don't.

Getting Atlas Shrugged vibes. Lots of parallels with this situation actually.

Seems like OP is describing the plight of the average worker who gets left to clean up the mess with no credit at the end of it all and less of the business leaders or executives.

IIRC, the capable workers were the first gone to Galt's Gulch. The story dwells on slightly-less-bad business people running around like decapitated chickens trying to keep things working, but the real heroes just quietly noped out.

Yes, don't forget there will also be some deniers calling it a fraud afterwards: https://twitter.com/TheKouk/status/1211123181051768832

This is honestly a terrible take. NJ did try to modernize the benefits system in 2007, awarding the contract to a company that is now owned by HP. News article: https://www.nj.com/politics/2014/11/nj_ends_118_million_cont...

EDIT: surprised at the downvotes. Parent asserted:

> we didn't listen to all those competent people that told us we should have cleaned this system a long time ago because one day it will bite us back.

And the historical record showed that NJ made that realization at least before 2007 (which is when the contract was awarded)

https://twitter.com/tradel/status/1246526465207869440

Consistently assigning work to big firms who consistently screw up... All they want to do is show that "they're trying", and the charade will continue until someone does get fired for buying IBN.

I bet NJ wouldn’t accept any excuses from a small business that tried to claim “they’re trying” if they had technical issues with payroll.

Surely outsourced and offshored isn't something to praise them for

Given a choice between the government doing it and a company that has been specializing in this type of thing for decades, why wasn’t this the best choice?

It’s not like HP is some unknown foreign company.

> Given a choice between the government doing it and a company that has been specializing in this type of thing for decades, why wasn’t this the best choice?

Because the only thing they are specialized in is sucking tax money from the government.

> It’s not like HP is some unknown foreign company.

That's their only pedigree. In other countries they also took a lot of money and delivered crap. So people should be aware by now. But with all this corr^W lobby ...

Like off-shored protective medical gear, off-shored pharmaceuticals manufacturing, and now a country full of unskilled 20 somethings that can only do phone apps and coffee baristas.

Companies will not like it, consumers will not like it, State budgets will not like it,

But like cobol programmers, we need to learn how to make things and build our own supply chain.

My old employer did not listen last year when I retired. Nobody was assigned to make an annual change to a REXX program. I learned it on my own because it predated me. But the process was so rigorous because we had to be our own RACF Admin to make our own RACF I’d to get to the datasets to change, then run the programs, but they got rid of printers connected to the mainframe and tighter up FTP so much that if took days to refresh passwords and ftp print files to your pc hard drive. Then run programs in C# that scalped off the first column and read the mainframe page skips....

Am I was suppose to teach this to a C# programmer who would not need to run this for 6 more months. I blame management. They loved outside consultants and treated our own programmers like loading dock employees, write ups when 5 minutes late for work.

They wanted me to take a brand new high end pc with me when I retired. I left it on my desk when I left. They had 6 months to learn it.

No, it's not. These state IT systems and this habit that leads to their inevitable downfall needs to die.

They respect nothing. Scope? No, we need to fulfill every need and some. Budget? No, of course we don't have cash, and we have to be very careful, it's the taxpayers' money! Time? It's already late! (Yet they weren't able to get it done in 10+ years.)

They have no real competency, they don't even have competency to delegate this to someone competent. And they lack the competency to manage their own inconsistency regarding these issues.

The linked tweet thread is a perfect example of this. (Rampant project mismanagement; too big to fail; they introduced some Hadoop scoring system to match inconsistent records and whatnot instead of simply throwing out all the bad data and handing the rest separately - eg hiring a bunch of unemployed people to go over them.)

A state that should be at the top of any kind of project management and procurement hierarchies wasn't able to supervise an IT system project. It's so incompetent just thinking about it will lead to spontaneous combustion.

Yup. Exactly right. And they think the 26 year old outside consultant knows how to fix the problem better than the 50 year old government programmer, who management has spent 17 years beating down their self respect.

> So you have the choice between saving the money of poor people now and reinforcing the suck of the whole system, of letting people suffer for a chance of a redo and purging the ones that lead to this.

What a terribly wrong take. If you really believe the system is that corrupted, and that horrible then you know darn well that the count of people who suffered during this emergency won't make a difference. There will be "accountability" in some form, but it won't be driven by how many people did or didn't get money for food at all.

Whether you help them or not won't change the system. So instead you're saying scores of families should go hungry so you can stay on your high horse and lecture.

That's a good reason. You don't need a moral reason to not choose a paid job. The reason scores of families will go hungry because of the mismanagement, not because one human did not step up to keep the mess rolling. In analogy, the reason Africa is starving is because of the past and present extreme exploitation from foreign powers, and said foreign powers sponsored corruption and violence. Not because "a heartless middle class salary man did not send his monthly 10 dollars", turning a blind eye to the real causes of the situation.

The big corporations, will get $18,000 from every taxpayer from their future tax dollar.

But they struggle just to give each worker in this country $1200.

It is hush money. Then next year they will tell us it is taxable income. Watch.

> Fixing this system will make the difference between real people getting unemployment checks or nothing. That's worth enduring a bad boss for.

In absolute terms, perhaps. But if you are moving away from taking jobs out of self interest, and into doing something for the public good; you might as well check out https://80000hours.org/ and pick the job that has the most positive impact.

My guess is that working any kind of 'normal' programmer job and giving 10% of your income to an effective charity would beat out enduring the bad boss in New Jersey.

10% of my salary wouldn’t be nearly as impactful as helping millions of people get unemployment checks quickly.

You'd have to look at the marginal impacts.

You working for them would most likely not push them from failure to success. It would perhaps make success marginally more likely, or perhaps decrease schedule overruns slightly. Or perhaps make not much of a difference at all, depending on how screwed they are.

Of course, the estimate of impact also depends on how you value humans. If you value all humans fairly equally, then Americans who had a job until fairly recently (ie those eligible for unemployment insurance) are already fairly well off compared to the people who benefit from more malaria nets. Especially since their unemployment benefits would merely be delayed, not lost.

Lots of people value those closer to them higher. The most prominent example are family and friends. But valuing compatriots higher is fairly common as well.

Of course, you can give more than 10% of your income as well.

Then why don't _you_ do it ?

I didn’t say I would do either. It is a generic discussion about tradeoffs.

Would do it myself. For free because of the unemployed. But I see no link to who is hiring, not going thru a head hunter who takes a rake, and states cannot send you a firewall link and password without a month or more of red tape. And rightly so if it is a payroll system, the opening it would make for abuse.

They'll change their tune when they won't be able to find anyone. The free market is as honest as it gets. If you underpay and undertreat your employees, they will seek an alternative. It is simple opportunity cost.

It will probably not be all or nothing.

They'll probably find a few people who will give it a go. The result will probably be delays and failure. Business as usual.

> Fixing this system will make the difference between real people getting unemployment checks or nothing.

Let them get nothing. Then they'll call for the heads of the governors, the very people who couldn't care less about the system until it stopped working. They are the ones who could have made a difference but actively chose not to because it just wasn't that important to them.

No. It is worth holding the bad boss accountable and firing the bad boss who put them in this situation in the first place.

There's a difference between a generic "bad boss" and an incompetent one. An incompetent one will kill the project.

If NJ have failed to maintain legacy systems that good COBOL development needs, almost by definition they have incompetent management (and a Governor who can't even pronounce COBOL). They likely don't have the dev or test boxes that they should have. They will either put up barriers to the other systems the developer needs to understand, or will throw open the doors so that my fixes get broken by everyone else's fixes.

This would work if we had only one problem to solve. Real people getting unemployment checks is an important problem for sure, but not the only one.

It's better to direct our limited resources to better porjects with a higher success rate and/or impact.

> That's worth enduring a bad boss for.

A bad boss will absolutely give you burnout, and that lasts for years. You are of course correct, but at the same time I would never ask this of someone and I wouldn't subject myself to it either. Bad bosses will destroy your quality of life in the medium-long term.

I would normally agree, but emergencies can create opportunities for improvements in the process. See healthcare.gov for example.

"Chaos is a ladder" though it didn't turn out so well for that fictional character, his assessment wasn't wrong.

Or maybe that’s their way of aiming high to land lower when that particular assortment of requirements isn’t yielding them any best-case options.

Hell most technical organizations have no idea how to hire programmers.

So what if they don’t know how to hire programmers? All they have to do is hire someone who has had engineering leadership role/experience and let them hire programmers, isn’t it?

Isn’t it easier to verify someone’s leadership credentials than COBOL skills?

Most government hiring does not work like that. "HR Specialists" decide if incoming applicants are qualified, and the hiring manager has little or nothing to do with it.

Lots of people are giving up before even trying.

States have done this to themselves. They can't expect people not to treat them warily just because they've suddenly discovered that their inattention to their own problems has consequences.

Yes, they hired a 24 year old college grad into personnel and she/he copied some old specs used by companies that wanted to bring in H1b programmers from off-shore.

The job specs were 40 lines long and pay was $60,000 a year for 10 years experience.

Then there were the places that brought in consultants that were doing a big rewrite, ignored the local staff and impressed management with their dark blue suits and white shirts, and left 6 years later without the project done. During that time the best of their own programmers left and they were left with the dregs and those a few years from retirement. Thought about doing this but I remember how we were treated. I left a job like that in Texas, worked in a shop in Connecticut with a room full of empty cubicles, and one very old guy and one new hire who knew Access.

Even the Director of MIS was being cut there but allowed to stay a few more months to apply for other jobs without saying he was unemployed.

I left.

Just apply. There is no penalty for applying. If you want work, they definitely will at least talk to you.

Like the other guy said, they're probably just listing their ideal candidate and not what the job may actually require

Do they mean that SDLC? Or do they mean "synchronous data link control"?

Nope - http://www.ambysoft.com/essays/agileLifecycle.html Though I would of prefered your flavour as way more fun and less admin work overheads.

And the very fact that you refer to "admin work overheads" in something claiming to be "agile" indicates that it's Agile(TM), not actual agile. (Though I got that idea very quickly from your link, as well...)

Yes, naming of projects was one of the most fun aspects in projects. Seen that play out in many ways, often case of thinking of what you wanted and working out some acronym or reason to go with that. One team the project manger was a fan of the TV show BILKO, so the new invoiceing system became called that once a passable acronym was worked out. Was something like Billing and Invoicing ...was so long ago but was clever and worked. Then their was the buzzword period and marketing departments, so would get whole rafts of things with Rapid as a prefix and the irony played out.

Still, maintenance is the biggest cost of any code that has lived so long, so with that such systems overall may well be agile, you just smooth out the pain over time in a more manageable way.

I can tick the 10+ years COBOL box nicely, though that was 30+ years ago and with that, somebody with 1-3 years COBOL would be more effective than somebody as myself.

Still, in an era of movie superheroes, who knew that having 10+ years upon your CV would enable you to go, step aside good citizens, for I shall save you. COBOL man to the rescue.

Everytime I see COBOL, I always think of Newman from Jurassic Park and imagine that is a stereotypical COBOL programmer haha!

If you see any bosses with a weird scarf, back away cause they may spit corona-acid in your face!

Newman is some sort of Pascal programmer https://movies.stackexchange.com/questions/28030/source-code...

That film's horrible misrepresentation of UNIX almost ruined it for me.

How exactly? Versus something like CSI, Jurassic Park got a lot of things right. The computers used were real SGI SPARCStations and they had a bunch of Thinking Machines CM-5s for gene sequencing in the background shots. I believe all of the on-screen shots were done with real Irix software (including the scene where they locked the doors, which was done using a real IRIX file manager called "fsn").

Sure it was given the Hollywood treatment, but it'd be pretty boring if they didn't.

Sun made SPARCstations, not SGI, who used MIPS, not SPARC, and just called them workstations. Then there was the SGI Indy, which was an SGI Indigo without the go. ;)

The 3d desktop file system navigator software running on the SGI workstation was actually a thing that SGI shipped, by Joel Sherrill <joel@sgi.com>, called SGI Fusion, or FSN!

Original URL of binary distribution:

ftp://ftp.sgi.com/sgi/fsn/fsn.tar.Z

Now archived at:

ftp://ftp.funet.fi/pub/unix/sgi/bin/fsn.tar.Z

https://github.com/jdm7dv/3d-file-system-navigator/blob/mast...

https://en.wikipedia.org/wiki/Fsn_(file_manager)

>File System Navigator (fsn; pronounced "fusion") is an experimental application to view a file system in 3D, made by SGI for IRIX systems.

>Even though it was never developed to a fully functional file manager, it gained some fame after appearing in the movie Jurassic Park in 1993. In a scene in the film, the character Lex Murphy, played by Ariana Richards, finds a computer displaying the interface. She exclaims "It's a UNIX system! I know this!" and proceeds to restart the building's access control system, locking the control room's doors. After the release of the film, some perceived the visualization as an example of media misrepresentation of computers, citing the computer game-like display as being an unrealistic Hollywood mockup.

FSN – the IRIX 3D file system tool from Jurassic Park

http://www.siliconbunny.com/fsn-the-irix-3d-file-system-tool...

SGI Fusion - FSN 3d File System Navigator

SGI File System Navigator, otherwise known as Fusion. Running on IRIX 6.5 on an Octane. If you have an SGI, a binary of this program is available at ftp.sgi.com

https://www.youtube.com/watch?v=zaRHU1XxMJQ

It's a unix system

https://www.youtube.com/watch?v=dxIPcbmo1_U

"Indigo without the go." -Mark Hughes (?)

https://www.youtube.com/watch?v=lEcw_oTKTa0?t=6m8s

https://news.ycombinator.com/item?id=7341455

Software Usability II

http://www.art.net/~hopkins/Don/unix-haters/tirix/embarrassi...

My vivid memory is of that excruciatingly slow zoom into a bar graph representation of a directory, to which the girl exclaims, "this is UNIX! I know UNIX!". A real world UNIXer would go straight to a shell prompt, especially in a time crunch with Cretaceous predators at the door.

You have to go to the original Crichton novel for the real cringeworthy misrepresentation of tech: the "code" listings will make you weep. He's a great writer, don't get me wrong, but would a little bit of research and/or editorial assistance have hurt ?!

Or Cobol Woman! ;) Cobol was created by Admiral Grace Hopper.

https://en.wikipedia.org/wiki/Grace_Hopper

https://medium.com/@donhopkins/cobol-forever-1a49f7d28a39

I am pretty sure that individual was referring to themselves with that term.

I am pretty sure my intent was not to misgender him (else I wouldn't have said "Or"), but to point out that women can be Super Cobol Programmers too, and in fact the first Super Cobol Programmer was a woman.

If you're touchy about people gently correcting accidental unintended sexism, take a moment to think about how women like Grace Hopper and Ada Lovelace have felt about being on the receiving end of systematic intentional sexism all their lives.

https://geekfeminism.wikia.org/wiki/Ada_Lovelace

>If you're touchy about people gently correcting accidental unintended sexism

The point was that there is no sexism, unintended or otherwise, in the post you are "correcting". Projecting your touchiness onto others isn't very polite or constructive.

You are correct but the poster does raise fair and yes, nothing intended directly nor indirectly - however it would be possible to read that into it and the reply initially was fair and did restore balance for those that may mis-read and all done in the best possible taste - they used a geek-wink ";)".

However, it is easy to view such balance as over-balance and it can then snowball into something of which it was not, and words, like code - heck nobody writes perfect code first time, let alone wording. So you can easily get bugs, be that in the code or in how things are worded that open those words up to tangents or ambiguity that was never the intention.

Anyhow - we are all on the same page of that issue I do feel and with that, let's poor one out for Grace, who without her we would not even be having this conversation that COBOL has given us today.

>but to point out that women can be Super Cobol Programmers too, and in fact the first Super Cobol Programmer was a woman.

Well said and an extremly valid point upon many levels, let us not forget Grace.

I perhaps should of phrased it as COBOL Human, though that may of confused others more than not, more so with the mixed superheroes conversation and that in itself had legacy baggage as do many things that are easily tripped up upon.

I'd upvote you more as detail is as always important and should never be overlooked.

I just vouched this after it just got flagged and as it is in response to myself that sent this path in motion and already been covered in this thread and no offence was meant or intended by myself of Don Hopkins, can we please read thru it all is cool.

We are all human after all and words are like code - bugs happen and we all operate our own unique brain-CPU's so things can and do run differently. No lines need to be read between here. Be well all and sorry, just thought I should clarify in the name of fairness for all and Don did nothing wrong in my eye's and it's all good.

I was indeed, but still a very valid point and after all without Grace - there would be no COBOL.

Unnecessary

Yes, but from my perspective - not unwelcome and I actually support the comment and took no misalignment or judgment from it - I'm all about balance and fairness - not that everytime the word man or women is used we have to then add lots of small print disclaimers. But it does remind us how some can read into things and reminds us to be more mindful of that aspect.

COBOL human to the rescue would of been better perhaps, so again unnecessary, yet not unwelcome and with that - everyone gets some karma.

> JCL

  ERROR IN OR NEAR CARD 1 COLUMN 1
  0C7 ABEND
  ABORT
  CORE DUMP FOLLOWS:

....

About the only other thing I remember is:

  //SYSIN DD *

No. Thank. You.

If you get an S0C7, your JCL is probably fine (else you wouldn't get far enough to get a data exception).

JCL is just another scripting language. I found it fun. I did all the JCL for my group's production jobs and commented the hell out of the esoteric issues to help Operations triage things. A small, well-planned effort prevented a crapload of, well, crap.

> JCL is just another scripting language

One feature of JCL which I think is cool, is that whereas on Unix processes inherit file descriptors which are numeric only (and hence few applications know what to do with anything other than FDs 0-2), in JCL you instead have DDNAMEs which are 1 to 8 characters. I think, if Unix had alphabetic names for inherited streams rather than just numbers, you might see more Unix programs supporting more than three inherited streams.

Another feature I think is cool is catalogs. A database mapping dataset names (file paths basically) to the filesystem volume (disk or tape) on which the file lives. You can move a file to another disk or to tape without changing its name/path.

(Strictly speaking both of these are features of the MVS operating system as a whole, not just JCL per se, but JCL is one of the main interfaces to them.)

JCL is certainly not any worse than bash.

Don’t be surprised if this position then goes to an H1B contractor though. If enough anericans who have this knowledge do not apply because they - Hate the tech - Can’t provide their feedback to improve the system - Don’t like the salary offered even if’s greater than their current unemployment income, then is it for an offshore company to come in and fill that gap.

I don't think the point is that Americans should consider themselves too good for the COBOL jobs. The point is that the organization has stated that it will refuse to hire many people who would be willing and able to do the jobs.

People love to attribute bad faith into a government just trying to get their shit together in the middle of an emergency. Where and when have they refused?

Which popular H1B source countries teach COBOL? I feel like the advent of cheap offshore programming came well after the decline of COBOL. Though my sense of history can very well be off.

They can just lie about experience can they not?

A lot of Indian degree courses teach COBOL for exactly this reason.

I'm sure that Tata/Infosys/Cognizant will be happy to supply you with dozens of CVs claiming COBOL experience and then give crash courses to any candidates selected.

And that's why H1B is a sham. They should just auction off those visa slots to the highest bidders.

(The auction would also give politicians a nice incentive to find ways to increase the revenue received.)

And that's why most American programmers hate H1B. It's not about filling jobs that we don't have the skill for, it's about filling jobs that companies don't want to spend money on. Offer a half mil a year, and watch how many qualified Americans come out of the woodwork.

This isn’t specific to programming. The whole point of hiring foreign is they do more work for less money and don’t complain (lest they lose their visa).

USA has the skill & unemployment to fill farming, construction, housekeeping, etc. jobs no problem. But why would companies pay 2X to hire an American that’s liable to quit at any point when they can get a foreign for X that’s legally bound to the work?

SDLC is often used to emphasize that you need to be familiar with the full "Software Development Lifecycle". When I read SDLC I just assume they mean, "you need to be able to take a project from concept to deployment (including documentation for supporting it long term)". Depending on org structures, a lot of engineers only work on one part of that pipeline throughout their career.

Nah, this is clearly an IBM mainframe shop. SDLC is a layer 2 network protocol in the SNA stack.

https://en.wikipedia.org/wiki/Synchronous_Data_Link_Control

They aren't asking for so much. They want:

1. understanding the language (COBOL)

2. IBM mainframe experience (JCL, DB2, VSAM, SDLC)

3. credentialed (4-year degree)

Note that SDLC is a layer 2 network protocol in the SNA stack:

https://en.wikipedia.org/wiki/Synchronous_Data_Link_Control

3.) is what gets me...

Why do you need to pay a college to get a piece of paper to enable someone to pay you for skills you otherwise posses?

I am not saying I agree with this system, but am assuming one of the reasons is that the hiring company see the credentials as a way to not gamble on whether you "are familiar" with a subject or actually has deep knowledge of core concepts taught along a planned syllabus.

You say you have skills. Everybody says that. Prove it.

Assume the interviewer doesn't have time for a multi-day interview in which your skills are tested. Maybe the employer lost the last person with equivalent skills, so a non-technical interviewer is trying to find a replacement person.

>You say you have skills. Everybody says that. Prove it.

I feel like this is a game of chicken that FAANG can win, but NJ Unemployment Office (or whatever) cannot.

The shortage isn't really cash. You could hire some bright high school students for cheap, and, given enough time, and half-supervision by an experienced dev (who doesn't even know COBOL), they could fix whatever bugs are ailing the system. You could hire a couple (again, COBOL-ignorant!) web devs at or above market rate, and they could also deliver a working system. It would take time, but they could do it.

The actual shortage is management. They don't have anyone who'd be confident fixing things if said high school students screwed up. They don't have backups, or don't know how they work. It's unlikely they know what actually needs to be done.

So what they actually need is a tech lead. But they think they just need "programmers," so they advertise for programmers (with inflated requirements, because they need them to actually be a lead) and the pay is commensurate for "programmers," rather than what they actually need.

It makes no sense to me... JCL, DB2, VSAM -- ok. SDLC?

If it referred to SNA, it should be SNA/SDLC. Not referencing that (or VTAM) means Software Development Life Cycle.

Looks like they want a programmer to do the job of an entire IT department.

> I was sort of tempted to pursue this.

For me the exact opposite. 15 years ago I removed any and all reverence to my history of COBOL because I was suddenly actively being contacted for COBOL coding jobs.

Although I had a great time, as a starting coder, with COBOL jobs (transcoding COBOL to C at the time), I know one thing: ever ever again will I utter one like of COBOL code.

There are still people making very decent money doing it, but the lockin to one particular source of work just does not sound like what I could bare.

> I did a few years in the salty COBOL mines

In the mid-80s, my first year CS professor anticipated most us ending up "in the lead mines of COBOL programming" — it was commonly assumed that we'd never manage to get rid of the dominance of COBOL.

Are references to mining common in the COBOL scene?

I can’t tell if you are joking, but “the salt mines” is a rather common bit of workplace humor, at least in the US.

https://en.wikipedia.org/wiki/Salt_mining#Idiomatic_use

Thanks! I genuinely was not aware of "salt mines" specifically, though I have come across mining references, and even made some myself, now that I think about it.

Or they are fulfilling Rhett requirement to post the job publications when they are rehiring a specific person quickly.

4-year degree? So that rules out Bill Gates, Larry Ellison, Michael Dell, Mark Zuckerberg.

To get serious, they need to move HR aside. Many of the best techs from the '70s and 80s dropped out of college. What was called for back then was real-world experience, not slips of paper.

I seem to spend most of my working life trying not to embarrass people with 4-year degrees who can't even spell things like "COBOL".

dont really know anything about this particular case, but:

Isn't it common for job applications like this to state what they "want", even if they know that it doesn't exist? (such as 3 years of experience with an 1 year old product).

Then they just hire the person who gets the closest to what they want.

Techies: What can we do to help? 3D print ventilators? Process terabytes of data in seconds? AI?!

Govt: We need COBOL programmers.

Techies: I am powerless to help.

But seriously some of the brightest techies I know quit their jobs to do a stint of government work which included slinging VBA and ASP Classic to remove SQL injections from VA websites and other similarly unglamorous tasks with huge impact.

If you want to be patriotic. If you want to make a difference. Government work can do that. You're probably not going to train a neural net to solve coronavirus, but you'll probably be able to stop 1000 hacks from ever happening or get help to someone who really truly needs it.

I could help rewrite in some other language.

Can you also guarantee that the rewritten code does not change any undocumented behavior? I think that is the real challenge.

Maybe we'd be better off writing tests then...

You can't guarantee that for updates to the existing code either.

We still haven’t learned.

https://www.joelonsoftware.com/2000/04/06/things-you-should-...

Surely we can make an exception for COBOL.

It was a bad rule back then, it is even worse now and it didn't even stand the test of time at Fog Creek (or Glitch, looks like they have renamed themselves), where they had to do it for FogBugz to finally leave Wasabi behind. I could go on about how Firefox ended up very well and not as the complete disaster that Joel predicted back then, but that's besides the point here, namely: There can be good reasons to rewrite something and "hey, we do not find remotely enough people to work on the current code base" is pretty far up on that list.

So you don’t consider it a “complete disaster” that Netscape went from 80% marketshare to almost nothing overnight?

Not to mention that currently their existence is almost entirely dependent on their competitor - Google.

Do elaborate how its even remotely Rust's fault that FF has lost its marketshare.

What does Rust have to do with anything? Joel’s article I referenced was about Netscape rewriting their browser during the browser wars over a decade ago instead of improving their existing browser to compete with IE.

> Not to mention that currently their existence is almost entirely dependent on their competitor - Google.

You wrote this sentence, it talks about Firefox and implies something went wrong there?

Joel’s article was about Netscape. Firefox was the ultimate result of Netscape trying to rewrite Netscape Navigator.

The article is about the disastrous complete rewrite of the Netscape browser. More recently Mozilla have rewritten some parts of their browser in Rust, but this time they are doing it incrementally.

Netscape went from the dominant browser to a negligible browser share. They basically handed the browser market to IE on a platter. They only rebounded somewhat because MS abandoned IE development altogether. And the result of the rewrite was not even that good, which can be seen from everybody else building on webkit rather than Mozilla.

I think the unspoken and (imo) wrong assumption here is that if you have a codebase which nobody understands, and the success of your organization relies on iterating rapidly and correctly on that codebase for its continued survival, there is anything you can do that has a high probability of success.

If you are in that situation, and you do not do a rewrite, you are likely to fail.

If you are in that situation, and you do a rewrite, you are likely to fail.

Sometimes there is no winning move.

I don't doubt the existing code was a mess, but I'm pretty sure their time would have been better spent refactoring it into a more modular structure. For example, what actually made people switch to Firefox eventually was tabs and a popup-blocker. There was no need to rewrite the entire network stack from scratch to support this.

Nobody's going to eat the government's lunch. There are still people around today who know COBOL, but how many years will that last?

It’s not like COBOL is some indecipherable language that requires a Rosetta Stone to translate. If there is a market for it and someone is willing to pay enough, people will learn it.

But didn’t IBM have a system to convert COBOL to Java that the federal government was using?

Imagine making even a minor change to the COBOL code. It's not likely to be very well structured or comprehensible. I'm guessing there would be a lot of global variables and strange database / file system interfaces. Very hard to do it without breaking something somewhere.

Edit: per the thread starting with vaidhy's comment.

Just yesterday there was an exciting post about printing off pages of assembly and reverse engineering archaic encryption using a crayon.

If you thought that was exciting I can't imagine hacking on archaic COBOL codebase could be much worse in terms of quality of tooling!

There's also a much better chance you'll have a say in the replacement if you work on and understand the original.

Fun fact: the third-party dump formatting package we had in the 80s could show line by line what assembly code corresponded to each line of COBOL. I actually diagnosed a CAPEX optimizer bug that way once.

I couldn't track it down. Would you be able to relocate and link it?

I'd love to see it, as someone who draws and colors all over printouts for reverse engineering challenges.

Probably this https://news.ycombinator.com/item?id=22774057

Oh! I should have known :)

Thanks!

a person is more likely 'to train a neural net to solve coronavirus' while working for the government than in the private sector

> If you want to be patriotic. If you want to make a difference.

And continue to incentivize politicians (and their voters) to underpay employees and delay infrastructure spending until a crisis, and then beg for more “patriotism”.

you make relieveing the government of its social obligations so it can do more wars / interventions / bailouts sound like it has some kind of humanitarian dimension...

Governments are made by people, of people, and for people. If you disagree with how the US government functions the answer is to get more involved, not less.

Slinging COBOL for New Jersey is not going to drop more bombs in Syria, but it might just save some lives by managing government operations more efficiently in a moment when every hour and dollar counts.

i must concede that bailing out local bureaucrats who haven't been doing their jobs in order to help predominantly able-bodied dependents of the state is a slight improvement.

People are asking "Can I bring my comparative advantage to bear?" because obviously the best way I can contribute if I were making half a million dollars is to take $50k of my post tax income and allocate it to boosting the salary offered.

I used to work on mainframe COBOL during the Y2K times. While the language is easy to pick up and the OS specific things are not too hard, the style of programming can lead to issues. Typically, shared data structures are often stored in separate files called copybooks and they can be hard to track down. Most of the code is not in any source control repositories which means no one knows which is the actual deployed version. It was all fun times then..

COBOL copybooks are just .h files basically.

That's not an issue with "style of programming", it's just complete disregard of software engineering basics.

Easy there. We're standing on the shoulders of those COBOL giants.

What's obvious in 2020 was cutting-edge and forward-thinking in 1960. The rude kids of 2080 may sneer at your best code circa 2020.

Don't mean to brag, but I already sneer at 2020 code!

I've certainly found myself sneering at code I wrote just last week.

In an effort to be more efficient, I've begun sneering at it as I write it.

I just don't write any code to avoid the sneer!

Why not sneer proactively ahead of time?

We used to maintain multi-user machines with useful software preinstalled. It took tremendous post-deployment sneering to get from there to Docker.

I just go m-x auto-sneer-mode and Emacs automatically sneers at my code as I write it!

I like to edit the source code to auto-sneer-mode.el in sneer mode, so it sneers at itself! Nothing more snarky than self-sneering code. I had to manually sneer by myself while writing it, to get it bootstrapped.

A classic JIT sneerer.

We are not standing on the shoulders of COBOL. We are standing on the shoulders of ALGOL and LISP. Those languages influenced everything that came after in every field of computing. COBOL has had little influence outside of its silo.

We should absolutely respect the history of our field, but the mere fact that COBOL is old doesn't make it an important part of history.

That fact that the many industries, most importantly the financial industry heavily rely on it makes it an important part of history.

I agree cobol as a language is not particularly glamorous in terms of what it brought it computer science. I say this as a cobol dev, so I’m laughing, but it’s true: cobol sucks.

Did software engineering exist back then.

COBOL was created in 1959. The term software engineering was coined in the 1960s. So it existed for most of COBOL's lifetime.

The term teleportation was coined awhile back too.

The term "software engineering", at least as used by Margaret H. Hamilton in the 1960s, was used to describe existing practices.

"Teleportation" was not.

Yes, it did. In fact, CASE tools were going to be all the rage: computer-aided software engineering.

One of them was METHOD/1 from Andersen Consulting (now Accenture). If you wanted to create a variable to hold, for example, a part number... you'd create an entry in the CASE tool for it and from then on, everyone was supposed to use that one. It would have validation rules, like 30 characters, alphanumeric, whatever.

The CASE tool would do code generation for screens and forms and whatnot. And.... that's about as much as I remember at the moment.

CASE tools are widely used today, but, you will see them scoped very specifically the databases in systems where everything is built around the massive pillar of a major db implementation.

I worked for a company for little while that made extremely effective use of this approach allowing hundreds of devs to effectively work simultaneously on a single monolithic system. It was shocking. I was so impressed. I've still never seen any software organization so... organized. And they owed it largely to the protocols revolving around their use of a CASE tool.

Had source code control, albeit printed manual audited to the nth degree with any production changes having to be fully audited and signed off and compared with original with light boxes twice by separate people level of SCC. Yeah experienced that back in the 80's, it got easier though.

As for standards you would see from one company to another - early days and lucky to have separate department use same standards. So payroll would have different standards than marketing, but separate ecosystems and less of an issue, beyond team cultural changes and analysts quirks.

But then, standards change over time and become usurped.

So could very well ask the same question in 20 years time as the approach is always changing. May even be a call for C programmers then.

Did you ever wonder how Internet was created before the Internet

That's an incredibly bad argument. If they wrote the programs 40 years ago and still use them they had 40 years to fix the business process. You can't argue that something is broken because it's old and then argue that you didn't have enough time to fix the process.

I wouldn't assume the fault is with the people who built the system. There is a reason they're looking desperately for people who know COBOL. Implied: they don't have any, and the systems have been ticking along for decades with the occasional call to a consultant who asked more than they would pay to modernize them.

Now management will pay even more because they're desperate and end up with a rushed solution that breaks again in the next crisis.

No. Nothing existed before you started using Javascript in your moms basement a couple of years ago. Wirth, Booch, Brooks, Neumann, Bachman, Dijkstra, Belady, Parnas and the rest never did anything worth much.

Why do some people take joy in wallowing in ignorance. Sad.

Site standards would play a huge part, you can write code without using GOTO in COBOL fine, and some programmers loved JSP (Jackson Structured Programming), yet modifying that code would make things open to whoever modified it and so the nuances and quirks of the programmer play out and code that has had many years of evolving standards and programming styles can be a right nasty mess that always begs a rewrite but the budget is for the change and no time for the rewrite. So the legacy snowballs.

Hence not all code equal and the site standards and that history as well as knowing that can make an almighty difference.

Never had copybook issues with ICL or Honeywell BULL platforms ;). That would be an IBM thang, but then, is there anything else still running - probably.

Usually you just compile the code to get the copy books. Ezpz. But yes its common practice in my office to carry a thicc black binder around for each companies’ copy book.

I’m newer to the mainframe industry but not that new and I’ve never seen a place that doesn’t have a source control repository. In fact, it’s required by law in every company I’ve worked for.

Man I remember copy books. I could do... just no where near NJ. And I am not interested in re-writing anything. Just the stuff working. :)

This is all second-hand, etc., but from what I've heard, NJ's IT systems are a mess across the board.

The most shocking story was about their system for tracking Medicaid patients. Their system had a fixed-width field for the patient's ID, and they ran out of identifiers - so they just started to recycle them again from the beginning. This problem was multiplied by the fact that, to say money, they didn't have backups of the original, clean data before the recycling. Thus it became pretty much impossible to understand what benefits had been afforded (or not) to whom.

Governmental budgeting seems biased strongly in favor of labor expenses, especially those with seniority, to the detriment of the maintenance of capital assets. This is pretty much a recipe for technological debt.

Any state government it department is going to be a royal mess. I worked for the Washington state government for 6 months a few years ago and it was the worst job I’d ever had.

I was at the top of my pay grade just under architect and only two years out of college making around 68k a year in Olympia. Olympia is not a cheap place to live. On top of that you’re basically legislates NOT to do work.

hope they print those recycled record out!

This issue highlights one of my main fears about a pandemic such as COVID-19: if enough people with the necessary amount of knowledge to maintain necessary infrastructure die without sufficiently and timely trained replacements, then civilization as we know it becomes one more step closer towards total irrecoverable collapse.

The prevalence of COBOL and other older programming languages in many parts of the world's critical infrastructure (unemployment claims here, finance, government, defense) means that the average age of someone maintaining these systems tends older. Older people tend to have more health issues. From the summaries of reports I read about COVID-19, the majority of the deaths happen among the elderly and those with other health conditions.

The idea of civilization collapsing might seem fanciful and farfetched, but the idea struck me after watching a video that was submitted here on HN and the videos it references [0][1][2][3].

[0] Jonathan Blow - Preventing the Collapse of Civilization (English only) - https://www.youtube.com/watch?v=pW-SOdj4Kkk

[1] Preventing the Collapse of Civilization [video] - https://news.ycombinator.com/item?id=19945452

[2] https://www.youtube.com/watch?v=OiNmTVThNEY - https://www.youtube.com/watch?v=OiNmTVThNEY

[3] Eric Cline | 1177 BC: The Year Civilization Collapsed - https://www.youtube.com/watch?v=hyry8mgXiTk

Many generational skills die out. Your grandparents would make their own butter, bread, many things today that are not so common. Then more, many skills lost over time and fact we are still working out some historical things got created, you can imagine rosetta code sites being a good thing to keep alive for the future. http://rosettacode.org/wiki/Rosetta_Code

The difference you're missing is we're still relying on COBOL, but we aren't relying on grandparents to make the butter and bread we eat lest we starve.

Not ideal, but it's not like it couldn't be migrated to another language and COBOL docs are readily available to at least understand the language. We would find a way.

I've worked for a software migration company and for some languages and the bulk of code you can automate the migration. However you also need to migrate the data and then anything that touches that data, so whilst you may want to migrate one program, that program will be a suit of programs and systems making it a more complicated task than it appears on the surface.

If you can data warehouse aspects off, that's great and can slowly bite away, migrate chunks bit by bit. Though one of the last will often be the database itself. Report generation is often an area that will hit resources hard and can also be easiest to migrate away towards something better using a system that periodically syncs the database onto another platform with all the latest do it yourself easy accessible tools. That allows those wanting the reports to be able to make and change that themselves removing a lot of code development and main system overhead.

With that, knowing the language is not even half the battle, the business knowledge would be the lion's share and to integrate the two skills effectively, that's where the money is.

You could have a payroll system, migrate line by line, logic by logic from COBOL to C. Yet the way the data is handled, the nuances of the way the languages round numbers, store data, formats, handle exceptions or early termination. It's not just a case of setting the right flags in the compiler and job done.

But yes, there is always a way. As always, the devil is in the detail.

One solution I saw somewhere here on HN the last time it came up was to carefully virtualize the existing systems so they have snapshots to restore from when things go wrong. That opens the possibility of moving the data to more modern systems with a translation layer between the 40 year old systems and the new systems that slowly replace them.

If you think of the butter as software running upon a compter and the process of manufacture the programming language, then you start to see it differently.

Though my main point I was making is that skills get lost over time for various reasons and how later you try to rediscover those skills for various reasons. I dare say that being able to make butter would be handy in some situations, imagine being isolated and access to a cow milk easily and your local shop not having any butter in stock for a while.

> Your grandparents would make their own butter, bread, many things today that are not so common.

I've never made butter, but I have used raw milk. And I've failed at whipping cream, by overdoing it. So from what I've seen, "making butter" is basically just agitating cream until the fat clumps, and then pressing out the whey.

Making bread, I admit, takes more skill.

My mother knew how to make blood pudding. But that's not something I miss very much.

Given enough time I'm sure we can figure it out. The problem is that they're not looking for archaeologists, they're looking for people who lived among the dinosaurs.

For all of these large, old organisations that still use COBOL, the risk isn’t that they run out of COBOL engineers to hire. COBOL is a very simple programming language, it was designed to be as simple as SQL. SQL was designed to be so easy that non-programmer business analysts could use it, COBOL was designed to be the same thing, but for non-DB business logic. Though you could debate how successful either of them were.

The risk these organisations face is finding people capable of maintaining their particular decades old COBOL spaghetti. Which is really just an ordinary key person risk.

The irony for me is that SQL is the language that stood up the test of time the most. All other are/were gamble.

Obviously it's not fit for all domains, since it's a DSL for data management, but I'm often tempted to push as much business logic into the DB with PL as I'm allowed to instead of having to do things in the application code itself, and it paid off many times, as I have rarely seen a project moving from a database to another, while application code is often rewritten in many different languages for no other reason than management's whims. I probably saved some businesses millions of dollars in code porting/rewriting/migration.

> The risk these organisations face is finding people capable of maintaining their particular decades old COBOL spaghetti. Which is really just an ordinary key person risk.

In my experience, the biggest issue is the complete lack of code documentation, as organisations relies on a handful of developers to maintain their codebases and when these developers retire, suddenly, nobody can understand an architecture.

I actively avoid companies that put all of their logic in the DB. It’s harder to version control, harder to unit test, much more work to rollback. You can’t keep it in sync with the source code etc.

I can’t just switch branches and know the state of the system at the time the branch was created.

You're making good points.

First, I'm talking about a PL "middleware" layer than does not necessarily involves restructuring the data itself (basically, instead of querying tables or views, you query stored procedures or functions).

When it comes to version control, wouldn't a migration system suffice to assess the state of that layer?

As for unit testing, I kind of agree, it's possible though, like any other programming language.

That doesn’t help. You can’t imagine the times I see

  Get_Customer_1
  Get_Customer_2
  Get_Customer_3

As far as version control, if someone does make a change to the unversioned “Get_Customer” and I want to switch branches. How do I make sure all of the stored procedures are in sync?

This is pretty much my experience with stored proc codebases too. I do consider SQL to be a very simple language, but store proc codebases tend to become incredibly complex. Finding an engineer that can maintain SQL code is very easy. Finding one that can maintain a particular maze of 1000+ stored procedures, views, functions, table triggers... is a completely different question.

Like Gibson's remark about the future already being here but not evenly distributed, the collapse is already happening but unevenly distributed.

Puerto Rico was without electricity for months. Some parts of the West will recover from the economic effect of the pandemic quickly. Others won't, without help from the center.

How is that different from say 1918? Centralization?

Jonathan Blow has a point that we cannot assume things will keep getting better. Although he says that we cannot fix software because we have chronically not done it for decades, his thesis is that the solution is to make software simpler. The issue I have with making things simpler is that if we don’t know how to fix things because we haven’t done it for decades, how can we make things simpler, when we haven’t done THAT for decades either? His argument rejects human exceptionalism, while also relying on it for his solution.

My point is that this absolutely is a funding and priority issue. I work as a developer who maintains legacy FORTRAN code. It was written in an era where global variables and Go To statements were the norm. Everyone who wrote it is retired or dead. It’s a pain to work with and there are parts that I haven’t yet gathered the courage to go in and change. However, me and my team have made substantive changes to it that are robust, that we have rigorously tested. This is not impossible, but it’s also not cheap or fast. It took me a year to understand how this blob of code works.

My team is young, we’re in our 20s, with backgrounds in CS, mechanical engineering, and physics. Certainly all systems are different, but if we can understand old FORTRAN, similar people can understand old COBOL. If people made it, people can understand it. We also know how to fix many broken things and how to fix many bugs. Often were told by management “not right now” and “we don’t have funding for that.” It’s frustrating, but that’s how the world works. They at least have funding to pay us to do what we’re doing now.

The point is that you can fix legacy systems, you can hire new people to do it, but it is expensive and it takes time. The whole issue is not whether we can or can’t, we can. The issue is: who can pay for it and will they?

This is a very hard problem. Not a silver bullet but I’ve often found that metrics are the best way (again not guaranteed) to get the money (wo)men on board. If you can demonstrate concretely how the legacy code affects your operational readiness or agility, it might make it a better sell to invest in refactoring. However the lack of standard tools to do this is a problem.

The problem with code is that most non technical companies don’t have a clue how this thing works. They use analogies. And for most folks the analogy is that of a machine that once built never breaks down. If you can demonstrate viscerally how bad the system is, it can help get better support.

> then civilization as we know it becomes one more step closer towards total irrecoverable collapse.

I love how pop culture always portrays the collapse as coming from our inability to maintain or repair some mythical machine or technology. Never once did I imagine that it was the state welfare software.

This will sound weird, but they should call up Northwestern State University of Louisiana for an alumni list. They were one of the last 4-year places that had a COBOL-focused degree program, through to at least 2012.

Probably CGI in Lafayette hires them all

Yet I’m sure they’re offering $60K/year.

I whole heartedly agree. The problem with government tech positions is they are not even competitive. Instead of hiring three mediocre or less than mediocre engineers at $60k, why not hire an amazing $150k - $180k engineer? I’m pretty sure they have ceilings on positions just so people don’t get outraged at government salaries, but incidents and times like these prove government needs to be competitive with private business to attract talent. Especially in national security and healthcare roles.

About ten years ago I worked for a consulting company that did lots of work for our state government. One particular state agency wanted to hire me to maintain the system we built for them but it would have been a nearly 50% pay cut. The only people they were able to hire on that pay scale were those who moved on the first chance they got from the private sector. Each time they'd end up giving us a contract to take care of things during the interim period and then train whoever they hired. Went through three cycles of that until I changed companies. Who knows, maybe they're still going through cycle after cycle of hire, programmer leaves, big contract the consulting company, train new person to take over. There's no way it was cheaper than just paying someone the going open market rate but the state pay schedule for that job classification simply won't allow it.

They aren't equipped to vet talent. If they hire a mediocre engineer for $180k, their situation has not improved.

> Instead of hiring three mediocre or less than mediocre engineers at $60k, why not hire an amazing $150k - $180k engineer?

In many cases, it's literally not legal for them to do so.

For example, at the Federal level: https://en.wikipedia.org/wiki/General_Schedule_(US_civil_ser...

(These restrictions don't tend to apply to hiring one via a contractor at $250-500k/year, though. Sigh.)

> These restrictions don't tend to apply to hiring one via a contractor at $250-500k/year, though.

They very well may have to go that route, i.e. some greybeard independent contractor that can name their price. It's a very limited skillset and God knows this ain't the kind of work that gets the super talented motivated.

Dad started out as a COBOL programmer before he got distracted by management and decided being a PM was a better path. This was his explainer to me once:

To hire someone who can do the job reasonably quick and well is really expensive no matter how you slice it. Consider it a 'job well done' if you can persuade them to:

a) Un-retire

and

b) Come out of said retirement to do actual work in the trenches again - as opposed to management.

In the private sector, these people have probably made the equivalent of GS-15 Step 10 for years if not decades. Good luck trying to convince them to jump back in for what is essentially a pay cut.

Can’t hire a contractor at that rate either. DCAA controls that very closely.

If a defense contractor wants to pay someone that kind of money they generally have to pay the difference between what dcaa allows and that price out of their own overhead.

What's the difference between "overhead" and "salary" for the contractor? If they charge 500k for the job and give 250k to their employee, it doesn't really matter whether they billed him at an hourly rate of $30 or $300.

Overhead comes out of company budget. The company has to make it up. They can't charge 500k for the job, they can charge whatever the DCAA approved labor rate for the engineer is per hour. Anything else they want to charge the gov for they have to pay out of their pocket.

Actually I didn't phrase that right. Anything else they want to pay the engineer they have to pay for out of pocket -- they can't charge the government for it.

I have a pipeline of good to great engineers wasting time in NJ State Universities and Govt agencies moving into my company. Great hunting grounds for people who can grow.

> why not hire an amazing $150k - $180k engineer?

If you're making that kind of money, you have a lot better options than working on some ancient COBOL system that processes unemployment claims.

Conversely, if you would choose COBOL in response to being offered $150K, you might well be a worse hire than someone who would do it for $60K.

> If you're making that kind of money, you have a lot better options than working on some ancient COBOL system that processes unemployment claims.

There's more to life than money and tech. I'm sure there are plenty of people who would be happy to work on a boring tech stack in helping to address possibly the largest public crisis many of us will see in our lifetimes vs. working with the new fad of the week in pursuit of selling ads marginally more efficiently.

Then the strategy of undervaluing the work of creating and maintaining robust systems will be validated.

Why spend the money and invest now for the future, when you can lean on people to donate their efforts in times of crisis?

> Why spend the money and invest now for the future, when you can lean on people to donate their efforts in times of crisis?

First, no one said anything about donating effort...the scenario was getting paid a decent, if not quite FAANG, salary to work on an older tech stack. And sure, you _could_ take that approach. We tried it with pandemic preparation, let's see how that works out.

It's really not sustainable or realistic to rely on altruism. Taking a huge pay cut to work on awful systems nobody wants to deal with exceeds how much nearly anyone is willing to help.

Just giving a frigging UBI for at least the duration of this crisis would solve all of these problems and work much better.

150-180k total comp isn't that high of a salary for an engineer. It's above average I'd imagine, but probably not even in the top 20th percentile. As such, for 150-180k you should be able get a pretty decent (yet unexceptional) coder.

Especially since NJ has the highest property taxes of any state in the US. $150-180K is a very good salary for where I live (Indiana), but not IMO for NJ:

https://www.cbsnews.com/news/property-tax-which-homeowners-a...

A little better than that, though not Google salaries. Skimming through some open NJ civil service jobs, I'm seeing advertised ranges like $60k-$105k for tech jobs [1] [2]. The pensions are also a lot more generous than most private-sector ones.

[1] https://info.csc.state.nj.us/Vats/WebAnno.aspx?FileNumber=30...

[2] https://info.csc.state.nj.us/Vats/WebAnno.aspx?FileNumber=30...

But you have to pay $25 to apply!?

Ya pay ya money and ya takes ya cheances.

If you’re valuing an NJ government pension at more than $0 to be received more than 5 to 10 years from today, you’re in for a rude awakening.

Maybe $0 is hyperbole, but there is no chance NJ makes good on all of its defined benefit pension promises, and I would rather just get all cash/401k as compensation.

https://www.truthinaccounting.org/news/detail/financial-stat...

The point at which the government should have rewritten their COBOL systems into a language that would have a talent pool to draw from was at least 30 years ago.

There's a claim in one of the reply threads that they contracted HP to build a replacement years ago, but the project didn't succeed: https://twitter.com/tradel/status/1246526465207869440

This feels like government thinking around infrastructure or military hardware where you open up bids and accept the lowest offer.

While there are problems with lowest-bidder-win, the real issue here is that systems like this are more living and breathing than one-off; you need a teams that tend to them, not bringing in a contractor.

The linked Twitter thread includes replies from someone who worked on the intended-replacement system indicating that “in the field” data entry accuracy was one of the biggest problems and not something that could be easily solved in code.

This is what happens when you have typical business moron-types get elected roles that treat complicated systems as though it's construction work.

It is, but you can't have any joe schmoe off the street coming in and learn how to pour concrete in a day. This is something that requires practice and problem solving skills. And years of it.

Judging by how a lot of construction projects have gone, this might also be usefully applied to pouring concrete [1]!

[1] https://www.nbcwashington.com/news/local/major-concrete-issu...

And these are the same people that people on HN think should “regulate tech”.

Needs a design-build contract, like the Highway 99 tunnel in Seattle. You get one set of money and only if it works in the end. No change orders or cost plus.

It's funny to see the viaduct replacement brought up as a success, but I guess if the state wasn't on the hook for the cost overruns it was probably a pretty good outcome, all things considered.

I'm still astonished we haven't yet been made to pay for that disaster. (Other than a billion more than fixing it would cost and with increased congestion...)

Easier said than done.

UI is mostly federally funded so they need to coordinate with them, and the complexity of the business rules and minimal funding during good times makes it a difficult task.

Also these systems generally work well, and it’s hard to fix things that aren’t broken. They just cannot handle the never before seen volume — it’s unlikely that a more modern enterprise system would either.

>>it’s unlikely that a more modern enterprise system would either.

sorry but that is just false. a Properly designed modern system would handle horizontal scaling just fine.

Granted the government would likely pay a billion dollars to have a screwed up system developed by on of the "big guys" that has the same problems as they current system, most likely because it would be designed by committee and not by engineers.

However to say that modern systems could not handle this kind of problem is just wrong. There are all kinds of ways to handle a sudden increase in load.

Many things are possible. But a core premise of government procurement is that you award to the cheapest vendor that minimally meets the mandatory requirements. Nobody would deliver a system that would just work with a claim load exponentially higher than the Great Depression.

If New Jersey landed a replacement UI system in 2010, it would be some sort of J2EE monstrosity on some outdated Websphere/JBoss/BEA/Oracle platform probably still dependent on some mainframe component.

The desperation for COBOL people is almost certainly because the (very complex) business rules are embedded in code that has been evolving for decades. Just changing anything is hard because it is a security nightmare (touches tax data, social security, dmv, and more) and regulated by all sorts of security/compliance frameworks that are difficult to implement and sometimes conflicting!

Yes I am well aware of how inefficient and ineffective government is, that is one of the primary reasons I remain a libertarian...

As the saying goes, Government: if you think you have problems just wait until you see our solutions

In SF permitting/entitlement system is being rewritten for already 5 (8?) years. Funding is only ~1M, while this is an easy task such small budget for for 5 years of work does not worth it even if you can build this by a single developer.

And this is a department that have billions.

> The point at which the government should have rewritten their COBOL systems into a language that would have a talent pool to draw from was at least 30 years ago.

Lack of foresight, lack of understanding that codebases are a living thing and are never "done". But hey, let's all work for ad companies and put our best minds at solving how we could make people see more ads!

Isn't it the power of the executive branch to requisition corporations or people in time of deep crisis in order maintain the continuity of government and its critical systems? Isn't unemployment claims such a critical system?

I assume that every state unemployment compensation dept. has the same basic set of requirements as New Jersey with a slightly different set of business rules. Here's a crazy idea: all 50 states could cooperate and hire a company to develop a flexible, secure, modern system that's easier and cheaper to maintain.

The basic set of requirements are the easy part.

It’s when you have to account for the differences between each state (50 factors!) is where the software becomes O(n^n) complex over the development of this product.

Even the best private companies struggle to build similar systems for a single entity.

I think each state should be accountable for their own programs. Avoids the “single point of failure” issue as well.

There are still benefits to choosing the same ecosystem. If everyone uses Java or Go and the same web framework then they can invest money into libraries that are useful for all 50 states. I've been at organizations that simply decided to use the same stack as another state (not USA). You end up using the same plugins and libraries. Even if you choose the wrong ecosystem because it is not mature enough you can just build it yourself and everyone benefits.

The problem is, you have to get the politicians to agree to the same policies so that the code works for both states. Politicians don't like to be told "You have to do it this way because of the computer system". Politicians want to decide the rules, then make everyone follow them. They don't follow rules themselves.

How hard is cobol to pick up and learn at the level needed to contribute? I'm sure there are many no longer employed programmers in the tri state area that might be able to help

I imagine it's pretty difficult. I'm not a mainframe programmer, but from my understanding you not only have to understand the COBOL languages, but various mainframe environment parts like the OS, APIs, databases, etc.

I did mainframe COBOL for a few years in the 90’s (Y2K fixes). It’s actually not that difficult - I could probably bring a Java developer up to speed on it in less than a month. COBOL is actually extremely easy to program in, but it’s also limited in its expressiveness, so you end up repeating yourself a lot, which leads to complex debugging sessions.

If you are already a decent programmer then switching to cobol can be easy. Cobol still uses the same logic, just with less fancy tools available.

That other stuff really isn’t that hard. Most of it is copy and paste-able once someone explains it to you. It’s really easy. It could also likely be handled by someone else for a situation like this with new-comers simply writing cobol.

The thing about old COBOL systems is, the language isn't the problem. Snark aside, it isn't that bad.

The major issue is the code written in it. Remember, a lot of COBOL code comes from a time before a lot of programming best practices were discovered. Structured programming-- as in, actually having function calls instead of just GOTO statements everywhere-- was a novel hoity-toity concept when COBOL first existed. Relational databases? Forget about it, they won't exist for another decade and a half. You get flat files; if you're lucky there might be field separators, if you're unlucky you'll have to find the column widths in the code somewhere. And don't even think that any of it is documented.

That's the sort of thing you're taking on when you jump into a legacy COBOL system.

That, and I don't imagine there's any tests written for a lot of these systems. :(

There are plenty of tests written for them.

They are just manual tests written up on literal, actual TPS reports (TPS = Test Procedure Specification) to be executed by humans.

My god. Are you serious? TPS reports are real?

https://en.wikipedia.org/wiki/TPS_report

And were widely used, at one point.

One advantage of a legacy system is that you can create your own great test data just by making appropriate extracts of your existing legacy data.

Where I worked, original test data was sparse and incomplete, and it avoided edge conditions. I, on the other hand, loved breaking stuff.

That's absolutely terrifying.

Come on now. That kind of go to programming was primarily pre- 1980 (remember the Alter verb - pretty powerful and used a lot in the late ‘60‘s/early ‘70’s). Ha! Alter was used to Change a goto statement to go somewhere else. I only saw it where the key driver paragraphs only had one instruction, the goto. That was before goto depending on existed in the ANSI instruction set. But that stuff was wasshed out when new compilers no longer get supported the verb. Point is, languages aren’t static, they grow and change as do programming practices.

I saw another post claim that COBOL programs primarily used flat files and had no concept of ISAM/VSAM/DB2 or relational constructs. Nothing could be further from the truth! In the late ‘70’s COBOL became the primary langauge for processing in CICS or IMS real time transactions. Flat fileS, eh. Ha

Youdon structured design was introduced about 1976 and 1977 - by 1980 only the very smallest IT shops weren’t coding in compliance with structured programming standard that did not allow goto’s. A programmer was unlikely to find a job in a good shop without structured coding experience.

Given New Jersey only began constructing applications in the 1980’s, then evolved for 30+ years - it is very likely most of the code containing go to’s and like code that were difficult to maintain were updated and rewritten in conjunction with other enhancements. Major enhancement projects were often used to fund rewrites in this manner. I am not saying New Jersey did this, I am saying any competently managed IT organization would have done this because that was considered best practices.

Let someone who has actually examined the code tell us how bad it actually is. Heck, if they are really an SDLC shop, there may actually be documentation! Lol

depends on the codebase, some people showed me long lasting, semi patched, semi generated code and it was not pretty

the language in itself is archaic but there's nothing difficult in it

I didnt find it too hard, I played with it a little in the IBM master the mainframe challenge.

The hardest part for me, is IO was not handled by the application, so all your file IO was setup by the JCL job that called the cobol program, which just feels weird.

It's actually quite easy to read, and much closer to plain english than say, C.

They came to my computer science course to pitch that challenge. I got as far as logging in to the mainframe and doing the first task before I stopped and thought “wtf am I doing? Why am I wasting time on this!”

Despite this story and others, still think I made the right call for me!

Programming a net new isolated Cobol may not be difficult for experienced programmer.

Difficulty (and/or downright hate, for many;) lies in

1. Ecosystem - obscure libraries rules and os that don't jive with posix-like mindset

2. Legacy code - having to dig into a ginormous monolithic program and figure out what some obscure pieces of code do, untouched and undocumented by anybody in your team, with ridiculous interdependencies and complexities.

So it tends to be more of "how hard is Cobol in this particular application for this specific client" :-/

Is there any way to learn the ecosystem? Last time I looked, you couldn’t just spin up z/OS in VMware and start hacking, which made it hard for me to figure out how I’d actually learn.

Intern or junior-level in a COBOL environment, unfortunately. Like others are saying, the problem with COBOL isn't (just) the language itself, it's all of the decades of very delicate cruft that has grown up around it. COBOL-the-language is absurdly simple to the point of being a little brain-damaged; but you have to be able to work in an environment where there's really close coupling between COBOL and its host operating system, and where minor code changes can have unintended consequences in seemingly unrelated applications, and the database isn't transactional, and there's no version control, and...

apt search cobol found this.

  open-cobol/bionic 1.1-2 amd64
    COBOL compiler

Also, I was hoping to find COBOL on learnxinyminutes.com, but didn't :/

Open-cobol works, but be aware that it translates your source to C and then compiles that. If that doesn't matter to you, great!

There's also tiny-cobol[0], but the project is dead and it only targets x86-32.

[0] http://tiny-cobol.sourceforge.net/

That's not really going to give you a taste for what it's like to "do" COBOL in a real-world scenario. Realistically, no one is running COBOL on a modern Linux. They're running it on something like z/OS and there is so much utterly bizarre and incredibly tedious stuff you have to deal with on those mainframe OS's that you'll run away screaming. And that's not even mentioning the code itself, written over several decades and likely not well documented.

When I worked at IBM not TOO long ago there were times when I had to fix a bug in WebSphere that only manifested on z/OS or zLinux and those were hands down the hardest bugs to fix. And that was a relatively modern codebase, by mainframe standards.

My way of learning and exploring something new (to me) is to ignore the bizarre and tedious system specific stuff that you mention. I have never even seen COBOL code before. Toy programs that I can easily run on my own computer are more than enough to satisfy my casual (and short-lived) curiosity.

https://kurthaeusler.wordpress.com/2016/11/16/learning-the-m...

That's not entirely true, a lot of people have migrated across to, for example, Microfocus COBOL, which runs on Linux and Windows.

It is true that a lot of legacy COBOL apps are on mainframe.

However, not all - I've personally been dealing peripherally with COBOL on UNIX/Linux via IBM or MicroFocus compilers for decades - and it even works on Windows :-O

e.g.

https://www.microfocus.com/en-us/products/net-express-server...

https://www.ibm.com/ca-en/marketplace/cobol-compiler-aix

Not too hard but a little unforgiving. JCL was harder to deal with.

This is the only book you need for JCL. Called, either the Brown-Blue book or Blue-Brown book.

https://www.amazon.com/System-370-Job-Control-Language/dp/04...

When I had to learn it it was on a s/390. Would the 370 translate well?

The ecosystem would be the harder part. It's either a 360 mainframe, AS/400, HP MPE, etc. The editors, compilers, runtime, batch schedulers, etc, are all different.

Here's a taste: https://qntm.org/zomg

>System z is a living fossil; it is computing archaeology; computing palaeontology. System z is an example of the principle "If it ain't broke, don't fix it" carried to its logical extreme. System z and its predecessors haven't broken for forty-five years: the "z" stands for "zero downtime" and some of these machines have been running continuously for decades without a reboot. Nor, therefore, have any of them ever been "fixed".

>It uses EBCDIC, a character encoding wholly incompatible with ASCII featuring such delights as non-contiguous letter sequences. It was, from 1983 to 2000, a 31-bit operating system. (Yes, thirty-one.) It does things with virtualisation of system resources and operating systems themselves that were thirty years ahead of their time, but all the interaction with z machines is still performed through an archaic green screen system from the 1970s, running in an emulator, with a 24x80 character screen. The screen is 80 characters wide because that is how many characters you could fit on a punch card.

>The output from the Job Entry System, however, is 132 characters wide, because that was how many characters you could fit on the line printer where the output came out. Nowadays, of course, the output comes to the green screen. So you always have to scroll left and right to see all of it. The keys for scrolling left and right are F7 and F8. For scrolling up and down? F10 and F11. No, Page Up and Page Down will do nothing.

>To enter a command? You hit the right Ctrl key. "Return" will do nothing. "Enter" will work, of course, but, AS YOU KNOW, "Return" and "Enter" are separate keys on your keyboard. "Return" is just above the right Shift key. "Enter" is way over next to the numeric keypad.

"AS YOU KNOW, "Return" and "Enter" are separate keys on your keyboard"

Just FYI, my laptop has "Enter" clearly printed on both keys.

Things were different in the old days, young padawan: https://commons.wikimedia.org/wiki/File:4800-52-mainframe-du...

I know they were, it just occurred to me to check and whaddaya know.

> JCL does have subroutines ("procedures"), but get this: every line in a JCL proc has to have a label, and when you call a procedure, you can add extra lines below the procedure call to alter the contents of the procedure on a per-line basis.

Mind blown.

I've got COBOL experience, but Everytime I check into these positions they aren't paying very well. I don't know about this particular case, but in general COBOL programmers aren't valued very much, despite the high demand within the narrow niches that still use COBOL.

I imagine the COBOL programmers are in the high-risk group for coronavirus.

Unless working in a Halon-protected, 50-degree, raised floor environment confers immunity. It's only fair.

I'm sure there is a lot to be said about the absurdity of this situation, but somehow this cartoon came to mind: https://abstrusegoose.com/323

I did COBOL for about 18 months in the Army in 1967-1968 on a RCA 501. Wonder if I would qualify. None of that other fancy stuff, though. Also RCA 301 assembly language.

Allan Sherman, the Weird Al Yankovic of his day, turned the tune from "Fascination" into "Automation" in 1963, a song about his RCA 503 sexually harrassing him:

https://youtube.com/watch?v=RsDFYUD5KUA&t=20s

For all the legacy Cobol out there and the amount of money everyone always says there is in this part of the industry, does there exist any cross compilers targeting Cobol lang or would it be possible for interop with other langs to slowly move away from the old systems?

Wow, if they don’t have folks who can lead this redesign development project (SDLC? Ever seen a highly critical emergency project where the SDLC was not the first thing sacrificed in support of expediency?) 2nd is QA & testing. Their dates never change as slipping development dates are a given. This thing sounds like something begging to come off the rails before it even starts (Governor begging for COBOL coders) while having degree requirements for that kind of legacy experience? What’s the definition of a legacy system? Anything in production!

COBOL was one of the few languages I was introduced to in college back in 90s along with C, BASIC and FORTRAN. Even back then people used to say that COBOL is a dying language.

I was supposed to learn COBOL as part of class curriculum in the mid 2000s, but it was replaced with HTML and CSS instead.

Interesting. My family is pressuring me I should help out. I’m a retired expert with Cobol68, Cobol74 and Cobol85 (w/embedded SQL). Entire 42+ years career was on Burroughs/Unisys (early years) and Tandem Nonstop. I was on a Y2K project for a very large telecommunications company and had to change A LOT of COBOL code. In later years I managed software development teams. But at 64 years old, I ask myself do I want to work in any Covid-19 epicenters.

Video clip of gov: https://www.youtube.com/watch?v=nlJqlWSPJeI

Would it be feasible to put the code up somewhere, along with an emulator of the ancient system, and some dummy data for processing?

I mean they're not going to pay enough to make it worth it, but some people might actually think it's fun, and maybe it will get more people interested.

Ah, I've always wanted to put my software archeology skills to the test with something like this.

So. The problem is. The rate you quote, if you were a cobol programmer, is that price gouging too?

Has anyone ever built an interface ( software and / or custom keyboard ) specifically for interfacing with the terrible UI / ergonomics of COBOL / mainframe systems like this?

Basically, has anyone ever built modern dev tooling or custom hardware to reduce the pain?

Depends on what you mean.

For example, my editor (which I use for work like this)

  MODE-'COBOL'
    72.b -.a -.t -.h 8.h 12.h
    .r

Roughly: for COBOL - ring alarm bell if I type past column 72. Do not autoindent. Do not use tabs (spaces only). Remove all tabulation except at column 8 and column 12. For FORTRAN 66 (77) work, I just set a single tabulation at column 7.

But that is just me. You should look into THE: The Hessling Editor on linux, etc. Look into Regina Rexx.

which would be all the "tooling" needed. You could run a mainframe emulator (look into Hercules)

How is 80 column "punch card" style editing and printer output terrible?

Relevant: https://twitter.com/Taytus/status/1246586694461202436

How does anyone apply for this job if they know all these skills ? Covid19.nj.gov does not seem to post the Job at its site. Can someone help with where did you apply or saw the JD?

They should just build a bridge gate to connect the island of old COBOL code with the rest of the world. But then Chris Christie would shut it down during morning rush hour. ;(

I taught COBOL in grad school and used MVS/JCL/VSAM. I was an IBM systems programmer. I got turned down for a COBOL job a few years ago. Sigh.

They could move to ReactJS... Move fast and break things

In the 24th Century.... https://abstrusegoose.com/323

Their own search for Cobol on their own site returns "Your search did not match any answers we have" - are they serious?

Unemployment putting people to work.

https://en.m.wikipedia.org/wiki/Z/OS Wikipedia: "However, z/OS also supports 64-bit Java, C, C++..." Managers: "Why do we need budget spendings for that Java, everything works fine..."

This reminds me of the article "COBOL and Legacy Code as a Systemic Risk." Like the pandemic itself, this was a foreseeable risk:

https://www.nakedcapitalism.com/2016/07/cobol-and-legacy-cod...

Hmmm. Sounds like every other mission critical emergency project in dire need of leadership, design and development. Perceived need way ahead of actual need, and legacy hiring practices (degree Requirements for legacy coders with 10 years experience and a degree, ha) not realistic to effectively recruit talented resources.

Job interview should include - what’s the definition of a legacy system?

Has anyone ever seen an emergency mission and time critical project where the SDLC WASN’T the first thing sacrificed for expediency? And the second is QA & testing - their delivery date remains fixed while development slips and slips.

The point is, if you don’t have leadership that has been there done that, knows what is coming, is solving tomorrow’s issues (like finding and hiring resources) so when the project gets to that point, the resources are there and ready, then you are in the false start phase that will continue until you admit it, and begin again with refreshed plan and clear minds.

Most SDLC’s do not recognize the false start as a very important first phase that must happen when a time critical emergency project is the only thing that can save the day. How long that phase lasts is dependent on how long it takes you to bring in the right leadership team to take control and begin anew.

Since they are in panic mode with the Governor pleading for “COBALT” programmers you know this is in the Coming off the rails / false start phase.

Some things you can’t short circuit around and luck out. not on really important stuff.

It takes some good minds with excellent experience to noodle out what needs to happen, how to make it happen, line up the political and technical ducks and make it happen.

This is a technology project using a rock solid mainframe programming language, COBOL. Don’t belittle it. Many of the nation’s largest banks still use COBOL based systems to process their mission critical applications, so quit whining about old mainframe technology being so obsolete.

Those of us who have been there fully understand there are no longer many or any users who understand what entire systems do anymore - because those legacy systems have evolved over the years, projects all justified by headcount reductions in the user community. That means all that knowledge is now encapsulized within that cobol code in those legacy systems.

So. If someone needs help responding to critical needs that were not anticipated (like an increase 100 or 1000 or 10,000 times the workload - New Jersey?), technology alone is not going to solve your problem, not quickly enough. You need many more people to do the processing of paying the unemployed until technology can catch up and take over.

Pay me now or pay me later I always say. If you don’t believe my predictions, Call me when you become enlightened and I can help you then.

Until then, good luck. PS: A legacy system is every system in production!

Companies and government agencies need to “ante up” on the refactor of their old systems.

I hope the position goes unfilled. Worst case scenario, the unemployment agency has to do it the analog way (“by paper”)

Where do we sign up?

Where do I sign up?

It’s a trap!

Reply on news.ycombinator.com