How the Pentagon's payroll quagmire traps America's soldiers | Hacker News Reader

How the Pentagon's payroll quagmire traps America's soldiers

reuters.com

59 pointsturar30 comments

This is both awful, somehow unsurprising, and COBOL is a piss poor scapegoat.

I've talked on here in the past about how crazy it is that some of my customers had huge financial systems that were written in COBOL, quietly still plugging away.

The most important of which is a system which processes billions of dollars of transactions a day, and has done so since the early 70's (when they purchased the what was then off-the-shelf software). You can imagine what a maintenance nightmare it must be (both because it's old, very important, and written in a programming language which predates the cotton gin).

Assuming that they can't just rewrite the system (which might be a reality), they can certainly sit down a bunch of developers and have them learn COBOL. COBOL isn't even a particularly difficult language, we were writing pretty decently-sized programs after a few months in high school (and if the system is that old, it's probably not even using any fancy language features).

If only the military had some means by which they could take people without past exposure to some skillset and train them to be competent at that skillset. Maybe part of that training could involve having to climb over a wall, or some sort of rope course.

It might not even be an issue of training COBOL programmers. I'm aware of more than one large-scale legacy system that people "somehow" lost the source code for.

Which means fixing bugs means disassembly, hot patching, and keeping your fingers crossed. And if you're lucky and the place is incapable of learning, they'll then lose all documentation for the patch, too. Fixing anything is a nightmare. On the upside, if you can stomach it, there are nice consulting fees to be had.

(I'm not aware if the Pentagon lost source code - my experience is with civilian systems)

> Which means fixing bugs means disassembly, hot patching, and keeping your fingers crossed.

This sounds like something with extremely high overhead, that would benefit from scale. That is to say, instead of trying to reverse-engineer and patch each module every time you need to make a change, just:

1. run the entire project through a decompiler, making sure that each generated code file on its own recompiles to something that works when linked back together with the rest of the existing modules;

2. and then use that decompiled result as the new codebase, and start refactoring.

In a reasonable world, yes. In a world where IT is a cost center, this is what happens:

1) People work around it as much as possible.

2) When the pain is unbearable, you bring in a high-priced consultant, because your engineers say something about "recompile" and "refactor". Which you as VP don't understand, are scared of, and are not willing to spend the money on.

3) The consultant puts in a week or so to hack things together. It's much cheaper than the recompile/refactor thing, and so you as VP are a hero to your bosses.

4) Since the consultant is paid hourly, documentation seems kind of pricey. Let's skip that.

5) Lather. Rinse. Repeat.

I wish this was just Dilbert, but in quite a few companies, that's reality.

Who knows how big the program is? Perhaps it can reach a size where refactoring is just way more expensive than patching as needed.

I love the last line!

I also agree that the language is probably the least of their worries. I can understand reluctance to change a system when it works perfectly, but this obviously doesn't work perfectly. Sounds like a good project for challenge.gov IMO.

I remember ~10 years ago they were trying to get the community to learn COBOL on it's own so they'd have access to cheap programmers. Isn't that how you build a business or successful organization, by exploiting the everyman for cheap labor? It worked for the ____, so I guess not since nobody fell for it.

Problems with pay were one of the primary reasons I didn't reenlist. This was an issue in the 80's. I lost days of personal time talking to yeomen, chiefs and officers just because every other check was late or missing. Turned out for the best, since I would have ended up on the USS Stark since my current ship was going to be decommissioned.

For those who don't remember, this is what happened to the USS Stark in 1987:

https://en.wikipedia.org/wiki/USS_Stark_(FFG-31)#Missile_att...

Readers may be interested that Britain's joint armed forces pay system, part of JPA (Joint Personnel Admin) is also recognised to be something of a clusterfuck (although not quite as bad now as it was back when it arrived), dragging in its wake myriad unlucky sods who found themselves paid incorrectly and struggled to get anything fixed, including an RAF chap who ended up simply suing the MOD for the money they owed him but found themselves unable to actually give him.

JPA was put together principally by the no-longer existent EDS, whose reputation for being able to handle large complex systems is well known in the industry.

It's worth mentioning that in the industry, smart cookies recognise that payroll is very much a wicked problem, and payroll for something as complicated as the armed forces is massively complex.

Less smart cookies do not recognise this and routinely massively underestimate the work involved and the support required.

Technically EDS still exist - they were bought out by HP and are now HP's Enterprise Services division. Their reputation was also entirely deserved.

We have HP contracted support at work (formerly EDS). We call them alternatively Helpless People, Hopeless People or High-Priced People.

I had a similar experience with Accenture. We called them Accidenture...

Everyone Dresses the Same... (referring to the earlier incarnation).

And now the Navy is buying their next set of massive Intranet services through HPES. Ugh.

It makes me furious that no one bats an eye at spending a BILLION dollars on a single presidential campaign but they can't sort out pay and benefits for our armed forces and veterans.

Surely a team with a background in large systems could take a set of requirements and produce a working system in 12-18 months.

> could take a set of requirements

I think that's the key problem: nobody has a complete big-picture idea of what the program is doing. Sure, there are people in each service branch that are familiar with subsets of the overall technology, but I suspect that at the overlap of these subsets there is significant disagreement over interpretation & implementation that has never become a visible problem because nobody runs consistency checks.

Since compiling the mutually contradictory payroll policies into a non-probabilistic master policy would immediately shed light on these issues, it would be a political minefield. Every choice you make would be the wrong choice and nobody is simultaneously high enough on the totem pole to say "live with it" and low enough on the totem pole to actually implement it.

Paychex seems to handle payroll policies for a huge range of businesses without too much trouble. Also I'm not quite sure why the DoD would have several different payroll policies? It seems like everything is determined by the relevant laws, similar to how you can look up the pay frequency and amount of pretty much any pay grade.

Laws often say things like 'significant' which are generally interpreted at the agency level. The DoD has a lot of agency's so these interpretations are unlikely to line up.

The problem is that DoD institutionally is shit at IT now. It didn't used to be this way (DoD was at the forefront in the beginning), but now we just can't do contracting for IT to save our lives.

Everytime someone tries to start a system to fix pay they realize that they will need to change all the personnel systems to match, and figure that they'll go and fix the personnel systems too.

They tried that with DIHMRS, which actually did involve billions, and the project never even got finished it was such a C.F.

Previous discussion: https://news.ycombinator.com/item?id=6022247

I thought that I read this article here once before... But a search turned up nothing relevant.

The story pisses me off to no end. The fucking Pentagon pisses away TRILLIONS of dollars yet they can't even get deserving soldiers their just pay?

It's completely unacceptable!

Take money from the NSA to pay these people.

this made me think of an editorial I read during the Iraq war (maybe someone else will remember the author's name).

The author was a university professor who was urging his son not to join the military. One of the points he made is that if his son was injured the military would not take care of him. It was tragic ... the son ended up joining anyway and was killed in Iraq. I remember feeling like shit when that happened. Just kept thinking how awful the Dad must have felt.

I can't quite remember the fathers name .. but I'm pretty sure it was Scandinavian. I keep thinking Uwe S.....?

Two things are striking: - The amounts under debate are small - The DFAS agency's budget is huge (1.36B)

Why not eliminate the agency altogether? If we divide $1.36B by the number of active military members 1,429,995, that results in $951 / active military personnel.

If we assume payment issues happen to 10% of the military, then that means we can just institute a rule that would give every soldier up to $9,510 in free "payment error insurance."

But I guess having a government body of 12,000 people with the task of giving soldiers a hard time makes more sense to some federal bureaucrat.

> If we assume payment issues happen to 10% of the military, then that means we can just institute a rule that would give every soldier up to $9,510 in free "payment error insurance."

Sounds like another scandal just waiting to happen.

A sad read indeed. Unfortunately this is not new. No country ever loved there soldier as much as their soldiers loved their country.

I remember reading about pay issues in the roman army when I was at the university. It hasn't changed in 2 000 years. Apparently the roman soldiers experience that their families back home didn't get their pay, and wives was given receipts where there was subtracted xx day for "leave" because their husbands was dead at the time.

We can only speculate if there is any motive and reason her, or only incompetence…

> This agency, with headquarters in Indianapolis, Indiana, has roughly 12,000 employees and, after cuts under the federal sequester, a $1.36 billion budget. It is responsible for accurately paying America's 2.7 million active-duty and Reserve soldiers, sailors, airmen and Marines.

It often fails at that task, a Reuters investigation finds.

Ripe for disruption, says I.

I heard the same story on NPR a few weeks ago (interview with the reporter): http://www.npr.org/2013/07/16/202360167/investigation-reveal...

Just imagine Walmarts chief accountant would request 17 billion to modernize the payroll system. The org is of comparable size.

Shameful.

Reply on news.ycombinator.com