Federal Agencies Need to Address Aging Legacy Systems (2016) [pdf] | Hacker News Reader

Federal Agencies Need to Address Aging Legacy Systems (2016) [pdf]

gao.gov

62 pointswallflower18 comments

There is an unfortunate culture prevalent in the government sector that biases towards "buy" vs "build" to the extreme that only vendor-supplied solutions are even considered. What this leads to is pathologically bad solutions that result from the RFP process because of the odd incentives that favor private sector profit over public benefit.

I don't know the "right" solution, but the current one overly favors corporate/private interests heavily, and this has caused a strong feedback loop that lobbying allows to strive.

The deeper problem is that powerful corporate interests yield so much power in the first place and allows them to prop up systems like this that continually favor them over, say, governments developing and retaining competent internal teams that can actually make or are empowered to make the right technological decisions.

Phrases like "government waste" and such are thrown around, but even a cursory examination of the current situation should be enough to demonstrate that problem hasn't been solved and the current situation is probably worse than if we allowed governments to run more like the private sector is allowed to.

If that means the government can compete with private businesses in the service of the public, I am struggling to see the downside to that compared to where we are at now.

Which is weird because government workers are, at least by reputation, very stable and static in employment patterns. So a custom system would be less risk than for most companies.

You'd think so, wouldn't you?

I have no idea what happens in general, but I can share that several experiences I've had is that the attitude is "we buy it and the vendor comes and makes it work" which, by necessity, is hands-off. Do that for enough years and you lose whatever skills you might have had to start with.

I would hope that my small experience does not represent the overall situation. I don't know. Just wanted to point out that there are side effects of the build/buy decision if you stick to it over a number of years that might not be noticed by folks outside looking in.

Is it paradoxical for a “stable and static workforce” to be risk averse? Maybe the stability has something to do with outsourcing risk?

Yes but: 1) relatively few government workers have marketable IT skills 2) they get paid the same whether they do any work or not, and they are required to spend the budget. Therefore, farming out the problem to contractors and writing checks is the incentivized path.

Federal Government IT workers are paid significantly less than private sector. It’s easier for federal workers to buy into private sector products because they can blame these products/companies if they fail. They would not be able to do the same with open source “building”. Investment needs to be in a digital team. Disclosure: I’m a federal IT worker. I have argued for building as opposed to buying and people think I’m crazy.

As a technologist whose built for both private and public, I'm always amazed at how little a contact can be for the public sector to purchase, only to realize that the price tag did not reflect the true cost. Government usually will go with the lowest bidder, but that project always will balloon way beyond expectations. They could've easily built in house, but that would've cost too much up front... It's easier to get a yes on a cheap, long tail expensive project then a yes on an extensive short term project...

I helped migrate an application from a mainframe to a modern microservices architecture. I'm not sure what happened but they forgot or intentionally didn't include provisions to train end-users to move the front-end to a modern architecture (like an SPA) so we had to...reproduce terminal green screens in Angular.

I had a very similar situation with a tax collection system written in Oracle Forms around 1999 that we rebuilt in ASP.NET Core. We had a bit more leeway than your project, but essentially we kept the navigation scheme and placement of the buttons and content panes in the app exactly the same.

The big upside of this whole thing is we were able to swap the database from Oracle 11g to MS-SQL with no impact on users. Something that just wasn't possible with the legacy tech.

For those that want to help directly, I'd like to mention 18F:

> 18F is an office of federal employees within the General Services Administration (GSA) that collaborates with other agencies to fix technical problems, build products, and improve how government serves the public through technology. 18F is part of the Technology Transformation Services, which is within the Federal Acquisition Service.

> https://18f.gsa.gov/about/

Interested in joining?

https://18f.gsa.gov/join/

I don't work there, but perhaps someone from 18F could comment?

Given that this was written in 2016, I wonder if these agencies ever successfully migrated to the newer systems? For example, the VA migrating to the new HR system by 2017.

This is something that has been long overdue. The usage of mainframe technologies from IBM just shows how much power and influence IBM wielded at the time.

Personally, I have seen the effects of “big blue” in a company that used IBM tech as their foundation and the increasingly poor attempts to modernize on top of that infrastructure instead of re-writing it.

From the management perspective, rewriting code that “already works” seems “unproductive.” This shortsighted perspective that appears to be the standard across the industry has caused billions of dollars of technical debt.

Unfortunately, this is not limited to the government sector but also affects the private sector. Hospital systems, airlines, hotels, banks, and many others have at least one system that is reliant on some form of mainframe technology.

Often times, these legacy systems were custom built, as far as the applications go. Decades of institutional knowledge that is documented no where, without the application running, no one could replicate it. Couple that with all the other systems that are integrated to read and write from the 'legacy' system.

Couple that with the fact that the government has almost no in-house development talent for most agencies. Couple that with the fact that they don't have someone that can understand and articulate the requirements to a vendor. What you end up with is an open-ended pay-as-you-go contract that runs up into the billions. Oops! This field was supposed to be 16 bytes, not 32, so there was an undetected buffer overflow on some down stream system and now your last 6 months of transactions were invalid? Oops, the government didn't specify 16 bytes in the contract, so the fault isn't on the vendor because they have no control over the downstream? Rinse, repeat. It's a real boondoggle out there.

> rewriting code that “already works” seems “unproductive.”

It is, but you need to make sure there's people qualified to maintain it when you need.

Can confirm VA has several mainframes still in operation. They also have a container working group evaluating kubernetes, and some active cloud deployments. Two extremes in a gargantuan organization.

I have wondered whether there is room for a new paradigm in government IT contracting: the worker-owned public benefit corp. The focus would be on building and applying “open first” government solutions.

This type of entity would feel to employees like a tech company and look to governments like a firm such as Mitre. Direct compensation would be higher than direct government employment but less than FAANG. Sort of like an “outside the house” 18F.

Crazy?

No, I don’t think you are crazy. You must look how big government is at the federal Level. It’s so interconnected with other agencies that moving to fast will break thousands of systems.

I worked on a contract to upgrade a legacy federal system to a modern architecture and it was a nightmare. For one thing, the previous contractor that has been managing the system essentially refused to cooperate in any way, dragging their feet producing any documentation or providing engineers to explain any aspect of the existing system, so a lot of it ended up being reverse engineered. In addition to that, the federal side of the house chased whatever technology they heard about at reinvent or whatever so the specs would continuously change to add cool new pieces to it from day to day.

On top of that, the contractor I worked for was very much agile and move fast and break things which is fine if you’re a startup but is a disaster if your working on a federal system that hundreds of thousands of people depend on. There were regressions and outages regularly pushed to production and the engineers were just like whoops we’ll roll that back, but they weren’t appreciating how bad it looked on the federal side of things. I left the company after a year or so, and they lost the contract soon after.

Does anybody know if there is an updated report?

Reply on news.ycombinator.com