The IRS’s Effort to Convert Its Assembly Codebase to Java
federalnewsradio.com
federalnewsradio.com
I've always thought that this type of government work should be open to bids from anyone. If the government is worried about trojans, malware, etc., then they can easily hire an auditor to audit the code and vouch for its authenticity. The fact that only very large, well-connected corporations get a crack at these types of problems is insane and a complete waste of taxpayer money.
I know a handful of small players, myself included who have won IT contracts with the gov't and it is not at all like you stated.
I've worked for the government on small jobs here in the EU a couple of times, as long as you stay below a certain amount you can bypass a ton of requirements. But once you go above that threshold the number of players drops very rapidly and there is no escaping the formal process.
If you are capable of selling large contracts to the US Federal Government without having to submit a typical bid proposal then you are in a very fortunate position, but the people that I know that have done this in the past have all worked on stuff that either required their fairly unique skills or they were working on extremely small contracts (< $250K).
In 2005, Packouz (23 years old at the time) joined Efraim Diveroli (19 years old at the time) in Diveroli's arms company AEY Inc. By the end of 2006, the company had won 149 contracts worth around $10.5 million.[1] In early 2007, AEY secured a nearly $300 million U.S. government contract to supply the Afghan Army with 100 million rounds of AK-47 ammunition, millions of rounds for SVD Dragunov sniper rifles, aviation rockets and other munitions
https://en.wikipedia.org/wiki/David_Packouz
EDIT: added quote from Wikipedia
An acquaintance with a setup similar to yours does well with a virtuous cycle of rolling SBIR and STTR results into his commercial products, which fuel more SBIR wins. He also does lots of legwork in the form of hand delivering white papers he’s written in office calls with customers and potential customers during site visits.
People do business with people — particularly those we like, know, and trust — not companies and agencies. Find someone whose headache you can make go away. Keep the conversation moving.
This is a patient person’s game. Sometimes you’re planting seeds that will bloom later.
I do grant writing for nonprofit and public agencies, along with some research-based businesses, and people interested in getting into this sort of thing will contact us, or people like us.
We're a little like lawyers: it is possible to do everything right, but it's very hard and unlikely if it's your first rodeo.
To take one small example: when you submit a budget, is it a program budget or a total project budget? Err and your application may be disqualified. Or consider indirect cost rates: http://seliger.com/2016/05/16/federally-approved-indirect-co... .
Again, either of these things are small, but multiply them x1000 and suddenly you'll understand why organizations hire us!
If you think big projects are given on merit, then you are naive.
The career path at my former defense contracting gig went from software engineer to someone who writes proposals. They clearly valued winning work over executing it. Rent seeking economics at work.
But, my understanding is that the bulk of the work done in this case was done by a conversion tool that was designed and developed by a group of 8 people led by the gentleman referred to in the article (Jian Wang). The conversion tool project was the work that I was referring to in my comment.
I would disagree here -- like timezones, tax software has the disadvantage of being both extremely boring, and very fiddly (thus requiring close attention). This combination makes it very hard to maintain the concentration that's required to write exhaustive tests.
In some ways it reminds me of the project to programatically translate the golang compiler from C to Go. Though I imagine the codebase is quite a bit messier!
This often proves to be much harder than expected which is /why/ people almost never touch systems like this where the creators are long gone
I assume this assembly program does the tax calculation, so the IRS tax code can be consulted to verify the functionality. Functional tests can be written against the old code as the functionality is documented. The functional tests will guide the new implementation development, and serve as the acceptance tests of the new code.
You can even structure the process such that there's a team doing just the functional specification and functional test writing, and another team takes the result fed to them and does the implementation in parallel.
I'll agree with that heartily.
> the IRS tax code can be consulted to verify the functionality
You're not wrong, but the problem is often not that. The tax code helps when verifying the system end to end. But it's not like there's one single program that "does the tax" that you can verify that way.
The individual bits they're trying to replace are probably more like "fetch all these records from this one mainframe with format X and convert them this way, except if it's Feb 29th in which case fetch it from this other server and convert it another way and then pass it along, except for resident aliens which come from an entirely different place, fetch those from tape storage". This is much harder to get right without formal specs, which there almost certainly aren't.
In this case what we have is a "reference implementation" (a.k.a the implementation is the specification), and you can guess how well those go.
All this is not to say that they shouldn't have been doing what you're suggesting, but to say that you're making it sound easier than it is now that they're here.
Yes, in that projects which try to reimplement large existing systems from the ground up by nailing down the requirements and working from there, almost invariably fail (even if they formally “succeed” in the sense of being accepted and then facing years of remediation.)
You always want to do a Ship of Theseus replacement rather than all-at-once, if possible, and finding a way to lift-and-shift the existing implementation to a platform that supports the way you'd like to replace thing in the long term is a way of getting as close as possible to that when you can't practically do it directly.
Love it. It appears engineers have resolved that particular philosophical dilemma with "yes" :P
This why java seems an odd choice; I can't imagine trying to run a codebase that's a split between assembly and java (keeping behaviour the same as more and more code is shifted).
The best idea I can see for that, is start with an interpreter for the assembler in java, moving procedure calls "up" into java.
But I imagine getting the interpreter good enough to run the same as the original target machine would be a huge task in itself.
[ed: leveraging clojure on the first iteration(s) might actually make the project feasible, though...]
I imagine you might get the interpreter running up to spec by flipping actual mainframe to debug, then writing a supervisor at the interpreter's side that checks for parity at each instruction, then single-steps the physical machine.
Simply extracting the control structure of the assembly via automation is a huge win. This type of assembly is incredibly dense - it was written for machines with at most, 100's of K of memory. It relies on highly non-local side effects which are very dependent on the data.
Things like a reference to a status register bit that was possibly set as a side effect of an instruction 30 or 40 instructions back, but then could have been modified by three or more other instructions, none of which are control flow instructions. All depending on knowing that certain patterns do/do not ever appear in the data.
After the control flow is extracted and made visible, then you can get to things like numerical emulation of the exact oddities of S360 code.
Once you have all those things done, which you will not ever, ever, be able to do by hand, then you can take the resulting generated Java and begin re-implementing.
The work being described has to happen before the suggestion above can even be approached.
You must be new here. What specification? Oh dear.
It's like these large companies take on the work, then instead of doing it they hire lawyers to figure out loopholes in the contract. Then they just pay lobbyists to get politicians to award them more contracts. All the while doing just enough to not get sued into oblivion.
All great ideas are spoiled by great bureaucrats and even greater lobbyists.
Not only assembler, but IBM mainframe assembler. That's way cooler than x86.
Not that I disagree with the general idea of democratizing this sort of big government contract, but, well, this is a much bigger hurdle than you are making it sound like. Audits can be good at finding unintentional flaws, but a skilled adversary can often create code that looks safe, but in fact, isn't. Consider the underhanded C contest: http://www.underhanded-c.org/
On a personal note: I once worked on a large research project for detecting malicious behavior on Android apps, where the idea was to produce tooling to find underhanded/undocumented behavior automatically. The project had a red team. By the final rounds, we were getting code from them where the malicious functionality was extremely hard to find via either tooling or manual inspection. Keep in mind these were simple programs, rarely over 10k lines of code, written in Java which is a remarkably transparent language to read, and that we were allowed to ban features like reflection or raise the alarm on anything that looked like intentionally obfuscated code. Additionally, we knew each of those programs had malware in them. Reporting 'clean' was often just saying 'you win, we give up, couldn't find it after staring at this 2k lines file for a week'.
In theory, when working with a large contractor, you can put controls on how the software is built, not just the final code, and you can hold them accountable for backdoors discovered long after the fact. Now, not saying this always works that way, but it might still be better than accepting a system built by someone you only know from their Github handle and who might or might not live within your jurisdiction. The state of the art in software verification would need to change a great deal before that is a good idea.
My thinking is that the issue isn't actually technical, but one of risk management, and that's something that markets are very good at dealing with. I would guess that there would be quite a few companies out there willing to step in and certify/insure code quality, for the right price.
Of course, DOD and other more sensitive systems are a completely different animal with respect to risk, so this wouldn't necessary apply there.
Its not the flashiest language, but I think its the right choice for them.
FTFY
It might be a peripheral plus for an applicant, but it's more important they are a competent programmer.
"Tax law" changes constantly - you truly cannot reasonably expect a developer to "know tax law" and be a competent programmer.
Germany figured this out a while ago and built a rule engine with GUI and small DSL where the lawyers can input tax rules to the database - meaning better code and you can emit for any platform. It’s called ElsterRules (Elster is the online tax filing program) and runs even in your browser... I wonder if there’s something similar in other countries.
https://books.google.ca/books?id=3iEDAAAAMBAJ&pg=PA88&lpg=PA...
HN discussion here: https://news.ycombinator.com/item?id=13464675
Actually this is the best analogy I could possibly imagine.
If anyone could land a contract just because a senator owed them a favor you'd see a lot more diversity in who the government does business with.
The translation would have some minor benefits: interop with java libraries would be nice, it could help to plug the system into better infrastructure like a modern testing framework, and it might run a little more securely. (I wonder whether you could craft a tax return to crash this system.) But it wouldn't have magically produced readable modern java code.
If (re)writing software has taught me anything is that sentences that start with "simply" or "just" actually mean that you don't fully understand the problem.
A huge amount of fraud happens by exploiting a timing attack against the batch processing architecture of the existing process. The system only has visibility into data from previous batches, it's blind to what's been done during the the currently processing batch. One exploitable artifact of this is that if you try to claim someone as a dependent (and potentially get child tax credits, EIC, etc) and they've already been claimed in a previous batch, your return will get rightfully rejected. But if multiple returns claim that person in the same batch, and they haven't been claimed in a previous batch, they'll all get accepted. The IRS will catch it after the fact, but by that point they've already paid out the fraudulent returns. If it's something like a misunderstanding between spouses on who claims a kid, it's likely they'll straighten it out and get their money back. If it's outright fraud, then they'll probably be out that money completely (plus investigative costs).
That's just one example, but there are billions of dollars worth of fraud that happens every year which could be prevented by migrating the IRS systems from offline batch processing to an online/real-time transactional architecture.
> He says he applied to become a GS-15 or Senior Executive Service member so he could see through the assembler-to-Java project. But his approval didn’t come through until a week before his employment authority expired. By then he’d accepted another job.
(GS-15 is $103k-$134k. https://www.federalpay.org/gs/2017/GS-15)
So, basically: the IRS can't fix its software because it can't pay market rates for software engineers. It can't pay market rates for software engineers not because it lacks the budget for it, but because Congress won't let them. This is how it ends up with huge armies of bureaucrats doing things that could be easily automated.
Elected officials have little incentive to make governing any more efficient.
Bureaucrats themselves have some incentive to be more efficient because then they can spend more of their budget at their discretion.
Less bureaucracy and lower costs sounds like something anyone would agree on, but remember that a large part of congress has run on anti-tax promises. Hampering the efficiency of the IRS is one way of making good on that.
In general, it does seem true that Republicans haven't been interested in funding the IRS, even though they seem to believe testimony that more funding would increase government income.
Eg: http://billmoyers.com/story/how-lobbying-by-tax-preparer-hel...
The GS-15 range for the DC area is $135k-164k. For federal government employees, salary only represents about half of their total compensation, which includes a lifetime inflation adjusted pension and medical benefits.
My employer's healthcare plan costs my employer ~20k/year as long as I'm employed, more if I had children, yet for some reason, nobody ever counts that as part of AMAFAGOPL salaries.
I'd be curious how often Googlebot hits HN.
The other reply is a bit dated too, there's no longer a real pension available to new employees. There are probably still long-term employees or retirees kicking around with those older pre-1987 retirement packages, but all recent hires are on a plan that's pretty much equivalent to a 401k (Thrift Savings Plan) + a Basic Benefits plan (which is pretty basic and you still pay into. It roughly equates to getting two Social Security payments instead of one). The bulk of a federal employee's retirement funds will come from contributions to their TSP plan.
And guaranteed pensions are no longer a thing? What is left as a justification for below-market salaries? Employment opportunities in government-dominated towns (which do not have healthy private-sector economies), and more job security?
Yes.
>And guaranteed pensions are no longer a thing?
They pitch it as a three legged stool: social security, your 401k, and then a pension that pays out according to a formula (the amount you get in a year is calculated somewhat like pension := (1% of the average salary of the three highest paid contiguous years of service) * number of years of service, up to 20 (after 20 years the percentage goes down). Between the three legs of the stool you can probably get to about 60% of your income, which is supposed to be reasonable for retirement.
> What is left as a justification for below-market salaries? Employment opportunities in government-dominated towns (which do not have healthy private-sector economies), and more job security?
The salary isn't at all bad. For some specialties it's below market, but for the average joe it's pretty decent. But it's worth noting that, for me, the job security is key. I've made more money contracting, but contracts end, sometimes abruptly, and the peace of mind you get from having security is very valuable.
You're also expected to do so, while wading through a waist-deep morass of technological, beurocratic, and managerial stupidity.
In that agency, she can't just zone out and phone it in.
There is still a small guaranteed basic pension that federal employees pay into, but it's not really enough to retire on by itself. It's one part of their 3 part retirement plan. Couple it with Social Security (the second part) and you could probably scrape by and remain reasonably well fed and housed if you relocated somewhere rural and cheap, but if you plan on retiring and staying put you're going to have to save using the TSP (the third part) like anyone else saves with a 401k.
I can't provide a blanket statement for why people are federal employees though, it's probably different in each industry. My wife is part of the scientific and regulatory side of the oil and gas industry and [benefits included] she doesn't earn substantially less than her private industry peers in similar scientific roles. And that industry is notorious for frequent and cut throat boom-bust cycles which brings it even closer to parity over the long haul. As a software developer though, not a chance. The desirability of the roles they have and the compensation even with benefits is not usually very attractive and the federal hiring process is bonkers compared to most companies.
I'm also guessing that the government's pension scheme is defined benefit: the risk that the scheme loses money is borne by the government, and you get your pre-determined payout unless the government goes bankrupt.
Defined contribution pensions are typically much more valuable, since they can obviate the need to save for retirement (or at least part of that need).
In the UK, it's extremely competitive to become a civil servant despite the mediocre pay. The reason is security, both in your job and in retirement.
I'd hedge, and prefer splitting my money between both.
What's going to happen is that paid shills will start publishing op-eds citing a handfull of executives and upper managers who gamed the pension payments system with overtime, resulting in massive payouts for themselves, drum up public outrage about lazy government workers, and pass an across-the-board haircut. It will, as always, be turned into a red vs blue issue - and half the time, red's in charge.
The fact that it is inflation adjusted is critical. To calculate the value of the pension plan, you would need to compute the NPV of an infinite series (assuming advancing healthcare).
The medical benefits are also quite significant and, for lower grade workers, will dwarf the value of the pension.
No, nowadays you have to pay in for retirement and medical benefits. Retirement money is partially matched, but health care generally is no cheaper than a good private plan.
A relative of mine has been a federal employee for over a decade and gets neither of those. She has a TSP (like a government 401k) and that's it.
Maybe some federal employees got something like this at some point, but it's very misleading to just drop that into conversation like it's true across the board.
https://www.investopedia.com/articles/personal-finance/06251...
Public sector employers need to compete in the same job market as private sector employers. I can only speak about my department but we're hamstrung by bureaucratic and legislative inertia that is completely out of our control. I believe the mission of my department is an important one but if the private sector software market was more robust in my city and pay scales were just as disproportionate as they are now I would choose a private sector job 10 out of 10 times. The only ones here not constantly checking LinkedIn are the lifers that already have a fat pension and are reasonably close to retirement.
That's 15 years of "streamlined critical pay" authority. And it appears that some number of these slots continued to exist until recently. Two decades of special authority to modernize this software, and nothing to show for it, except perhaps some patents...
At some point you have to cut your loses.
These slots were used to fill 168 positions over the 15 years.[1]
[1] https://www.treasury.gov/tigta/press/press_tigta-2014-51.htm
It is above average for the area (across all levels of experience, for a very experienced developer it probably isn't), but it is not very far above average, so I would expect someone capable of handling such a massive and critical project to be making significantly more.
Plus, would you really want to endure the months of their hiring process (which intentionally filters out people not wanting a long term job) just to join an agency that has no promotion potential for you over the long term?
You'll get some of the best candidates from India and China lining up.
I work at Amazon in Seattle right now, but I'd take the pay cut to jump the India green card line. Probably go back to Amazon right after I got the green card tho.
GS employees almost cannot be fired, government-employee unions protect their members from not the private sector but from the government, they get better than ACA health care, and their retirement is guaranteed by an entity that can print all the money it wants.
Government IT is rife with corruption and incompetence. I have worked for a number of agencies and worked in federal government for many years. Projects are often steered in the direction of contractors and federal employees that benefit from technologies. This is often done to bias incumbent legacy stakeholders and technologies that benefit the federal employees and the contractors that collude to control the systems for their own gains. I have seen the sleaziest contractors enrich themselves at SV IPO levels.
Then came 18F, the hipster government IT revolution. As a senior developer, and maybe I am just some cranky old developer, but the thought of being vetted by this douchebag, fuckin blows my mind: https://18f.gsa.gov/2015/04/08/day-in-the-life-of-talent-man...
Yep, most government and corporate IT is fucked.
I wonder if I get 2x by expected refund the IRS can come at me to claim the extra amount :)
It's Gradle build scripts themselves that can now be written in Kotlin instead of Groovy. I haven't tried that yet, but it sounds pretty fresh.
And there is Gradle.
Why do you regularly get frustrated with groovy?
What is considered elegant and functional about assembly?
But yeah I don't see how it is elegant in any way that's useful
This is similar to high-level languages like Lisp and Prolog, whose code is readily represented by built-in data structures in the respective language.
A fun fact about this is that extremely low-level (like assembly) and extremely high-level (like Prolog) languages are homoiconic, but there is a large gap "in the middle" (like Java, C), where there are many languages that lack this property.
Author has no idea what assembler is.
It isn't just old and forgotten, its very style and function is extremely limited in comparison. Shakespearean English is just about as functional of a language as modern English (ignoring the vocabulary for things that didn't exist back then), which can't be said about assembly.
I think that's addressed in the, "No one really uses it anymore" part.
"Shakespearean English is just about as functional of a language as modern English (ignoring the vocabulary for things that didn't exist back then), which can't be said about assembly."
I would disagree. Arguably, the main function, if not the only function of English, or any other human language, is to express ideas. Shakespearean English being not commonly used, and many of the colloquialisms and terms used not being in use anymore would make that function more difficult. As an example, I'd use the "Romeo, Romeo, wherefore art thou Romeo?" line. To most people, that would be Juliet asking where Romeo was, but it's not. It's her asking why Romeo had to be who he was, which was a Montague, the rival/enemy of her family.
Even with that, the junior engineers (familiar with assembly!) were completely flummoxed by it. Domain knowledge encoded in assembly rapidly becomes highly obfuscated, even when written in a literate programming style.
I simply cannot imagine calling large-scale enterprise business logic written in assembly "elegant and functional" for any reason. Even the best written, best documented assembly obfuscates so much meaning .
> There was once a programmer who wrote software for personal computers. "Look at how well off I am here," he said to a mainframe programmer who came to visit. "I have my own operating system and file storage device. I do not have to share my resources with anyone. The software is self-consistent and easy-to-use. Why do you not quit your present job and join me here?"
> The mainframe programmer then began to describe his system to his friend, saying, "The mainframe sits like an ancient Sage meditating in the midst of the Data Center. Its disk drives lie end-to- end like a great ocean of machinery. The software is as multifaceted as a diamond, and as convoluted as a primeval jungle. The programs, each unique, move through the system like a swift-flowing river. That is why I am happy where I am."
> The personal computer programmer, upon hearing this, fell silent. But the two programmers remained friends until the end of their days.
And I have heard cool things about mainframe assembly programs (at least on z/TPF). Like a system to do code upgrades without hardly any downtime by pausing the VM, doing brain surgery on it, then resuming it.
With time register sets grew, ISA grew, this got muddied by usage.
Although, not all assembly code is like that. Further, you depend on the device it written for to still be around.
However, I don't see a problem with assembly so that fact they are just converting to just convert it. Seems a bit frivolous. If they are having trouble hiring assembler programmers that's kinda a different issue.
However, one nice thing is that any can program in assembler and well tend be good programmers. They know what they are doing. So by switching languages they open themselves up to worse programmers.
However, Java why? If they are coding in assembly wouldn't C make more sense?
I think I still have my banana book somewhere. https://i.pinimg.com/originals/fb/85/89/fb85890d32df071cf842... http://cbttape.org/~jmorrison/s370asm/html/tut-REFSUM-001.ht...
In addition, some of the automated translators I have seen also apply automated refactoring: They tend to merge similar sections of code and minimize the delta (similar to diff) that is factored out. This can create a maintenance problem especially in the context of a system like the IRS's, where presumably the code that performs calculations for different years or other periods (legislations etc.) is similar but not identical, but must be retained as fully as possible to reliably perform calculations that are subject to earlier regulations.
Such portions of code should be kept distinct, but an automated conversion may conflate them, and it may require additional and error-prone fiddling to enforce the separation.
Wat?
Compiled languages are more likely do thing under the head you don't expect especially a language ran in VM with garbage collection like Java.
Also Java why? I would not trust a language system dependent oracle. The days of Sun Microsystems is sadly gone. =(
https://en.wikibooks.org/wiki/360_Assembly/360_Instructions/...
For example, our computers nowadays are much faster and have much more memory. So suppose you have some long, tricky section of assembly code that you know computers some sort of function of a single 32-bit input.
You could simply run that section, in a simulator, with all possible 32-bit inputs and make a table of the results. Then instead of trying to understand what is going on by following the convoluted assembly, your task is to implement the function given by that table. With the function in table form you can do things like look for sections that are linear, or constant, of follow other common patterns.
Hopefully you can get some idea from context in the assembly code what, at a high level, the function is doing, and that should further help you see patterns in the table.
> elegant
> functional
Good one, mate