Automated Refactoring of a U.S. Department of Defense Mainframe to AWS

> Once modeled within the tool engine, SMEs employ an iterative process of applying rules and tuning to output the transformed code into the target Java language.

Seems like they manually construct an IR of the source system, then translate the IR to Java Code.

Automatically transpiling COBOL code to Java will lose all possibility of maintenance moving forward. So it is not considered in the first place, nor it is what they need. They need a semi-auto REWRITE of their legacy system, not just target it to Java.

even if it $100 millions, it seems a successful project , given DoD mentioned potential savings of $25 millions/year

>Customer Benefits

> For the DoD, the component has been transformed from an expensive mainframe COBOL legacy system to an affordable, modern maintainable Java-based system. All valuable existing business rules have been preserved in the modernized system, while development, test, and production environments were migrated to AWS providing flexibility at reduced cost.

>The DoD’s projected cost savings is $25 million per year, and they are now able to use popular Java programmers to maintain and enhance the critical component.

They already did that, they bought SoftLayer, but it's not working very well since instead of going for "a cloud offering" they are doing a "same kind of ibm offering as usual, but in the cloud !", there is a reason no one is talking about using them on sites like HN their entire target market seems to be company moving from big "enterprise" hosting to big cloud "enterprise" hosting.

And they're not buying AWS any time soon.

COBOL is a very niche language that is very well specialized to banking. I don't envision applications rewritten in something else easily for any significant benefit.

But maintaining and extending the thing was only going to get more expensive. And once you've paid the price to modernize, you've bought yourselves a lot of savings for 1-2 decades, maybe more.

And they ripped out the COBOL Data management system and replaced it with a RDBMS. This alone probably is worth it since you can hook it up to business intelligence tools now to create reports and dashboards.

Fair point given how things usually go in the private sector. In the public sector, especially DoD, the rules of the game are changed.

What happens if or when AWS or Amazon ceases to exist?

Well guess what, they can't. It's now a national security issue so the necessary services will continue to be provided by a KTLO crew or DoD would facilitate a transaction where another company could absorb the needed technology and be contracted to support it. DoD is the biggest company on earth. FY 2019 budget is $686B.

DoD has a pattern of keeping mission critical end-of-life systems running and supported.

I'm currently advising the US Navy on moving some COBOL systems to Java. Those systems have been lovingly supported by lifelong Navy civilian employees for 40-50 years. Those people are now in retirement age and so they're "replatforming" to Java.

There really is no corollary in the private sector. The default for government systems has been the people who built it stick around and maintain it until they retire. For DoD the biggest risk isn't old technologies and changes in the private sector, it's brain drain when people retire.

You might ask yourself, what 20-something wants to work for the US Navy, as a civilian, and work on a Java implementation (ported from COBOL) for the rest of their career? Well, I'd like to introduce you to them. These guys could work anywhere but they like the Navy. They like that their work matters, it's important, and it's stable. The pay is below market but it isn't terrible. Keep in mind DoD isn't all in DC, many offices are in inexpensive places to live.

Frankly I'd love to work on systems with the meaningful impact that theirs have (non-combat in this case). Read about 18F and what it was like for the people who joined and how they stuck with it anyway.

This underlines how so much of the work and contracts at the DoD don't really revolve around defending the US than being another way to hand out tax payer dollars to already well off corporations.

> [Wang] says he applied to become a GS-15 or Senior Executive Service member so he could see through the assembler-to-Java project. But his approval didn’t come through until a week before his employment authority expired.

They lost a guy responsible for software that could save taxpayers how much -- tens of millions of dollars? More? All because they couldn't pay a guy a GS-15 salary, which translates to $100k-140k.

Man, people are really dumb about paying programmers well. They'd rather see a project go off the rails than pay someone (still slightly below) market rates.

Also:

> In many ways, assembler is still excellent for this application. Milholland said of the code, “The assembler is well written. It’s incredibly efficient and effective.” But a shrinking number of people understand it. And it’s not optimized for the online, transaction mode to which the IRS needs to keep moving. Java, relatively inefficient as it may be, is the current standard and has legions of people who know it.

That's like arguing mixing concrete by hand is "more efficient" because doing so allows you to be more frugal with concrete.

How true is that though? They'd still have software licensing, they still need an OS to run on Hercules after all.

You should be asking where is the current documentation of the system specs, and in some cases you might even get something reasonably current and complete when you ask that, depending on how to use the change management process has been.

> When you do a re-write you have to go back to the customer and whatever original documentation you have lying around and you have to figure out and document what the system does.

Both what it does and (and this is where the customer is involved) what it is currently supposed to do.

They spent 12 months making it less-COBOLy / autogenerated feeling

The other hidden cost is that these ancient software systems are extremely efficient in terms of cpu / memory utilization compared to modern equivalents so the hardware requirements go up considerably which impacts power / energy efficiency and direct deployment costs for replacement systems. Not saying modernization is not a win in the long term but there are direct and indirect costs with these big transitions.

Next step is to fix the bugs, and get all the tests to run.

The last step, to get to "clean" involves going through the code base, looking for project-specific idioms, then hand-tuning the syntax matching engine with rules that output idiomatic target language code. This might also involve refactorings to make the scaffolding libraries more idiomatic as well.

Since one is coding on the transformation engine, instead of directly on the application itself, production changes and maintenance can proceed, and no code freeze is needed.

I did find it interesting that the original system, and delivered overhaul, both ran Oracle databases. Overall transactional volume is low (500k/day). Perhaps the DoD has significant investments in stored procedures? Or perhaps Aurora is not ready for prime time in AWS Gov Cloud? (It’s no secret that AWS really struggles with keeping feature parity between Gov and commercial.)

As one example, when I was a systems engineer, I got a call from a team that was trying to run a business application from z/OS in emulation on an x86 server. It was running incredibly slowly, even on a fairly beefy server. I looked in its data directory, and there were hundreds of thousands of individual files in a single directory. Not a tree. One directory. That might work fine on a mainframe, but just doing a dir or ls (depending on the host OS) took something like ten minutes.

Similarly, I remember some folks being dumbfounded at the idea that their batch processes needed error-handling when they were doing things that involved talking over the network, because when everything was running within that one giant IBM system, failures were pretty rare.

Those both sound like they should be easy to tack on when running in an emulator, but they're just the tip of the iceberg. When you have code that's 30-50 years old, written for a vastly different platform, there are going to be a lot of gotchas like that.

> "Difficulty finding qualified, affordable COBOL programmers."

Continuing to emulate the COBOL environment doesn't quite solve the problem that finding COBOL developers is becoming harder and harder.

disc: I'm Amazon employee, not working in any of the facial recognition. I'm also an immigrant, ~5 years in USA

That's just a lone hacker doing shit outside of work... And my solution can go through directories of images, realtime video stream, videos, and ipcams.

So yeah the cat is out of the bag. The bag was shredded, and the cat has had 3 litters of kittens.

Any video system could be retrofitted as a facial recognition system... Or even process videos after the fact.

AWS has 13 US-based availability zones, each of which has, as I understand, multiple DCs.

GCP has a similar architecture with 16 US-based zones and 3 more standing up imminently.

Azure has a similar architecture, but separate AZs within regions are newer for Azure, and I can't find detailed information as readily, but they look to have 8 US regions,at least four of which are multi-AZ, so at least 12 zones, each with one or more DCs.

And that's 41 zones, many of which have multiple DCs, with just three big public cloud providers; I don't think 10 DCs gets you most of the computing power.