IRS’ 60-Year-Old IT System Failed on Tax Day Due to New Hardware
nextgov.com
nextgov.com
How do we separate business logic in a self-contained way that can be independent of hardware and (from a certain level) software?
I wouldn't consider someone who spent 5 years on AWS, especially if they moved internally, out of date.
Whereas I've done nothing remotely close to where I want to be in about 3 years. I feel very scared about my skills, especially since I'm not even 10 years into my career.
An aside, even when you spend that long in a single position, you do need to keep learning on new and relevant technology; I attended every on house training seminar I could, which helped a lot, and spent a good portion every day reading. I spent 7 years at that company in the same effective role, and never changed job titles, but saw a >200% increase in take home pay in that same period.
To me, it's the cool-app-of-the-month-that-harvests-private-data-off-peoples-devices that is brain-killing.
Fixing bugs and babysitting all day isn't going to convince people to stick around. Best you can do is get them something they can try to work with.
As strange as this may sound, documentation. Software is at it's core rather brittle it's designed to operate inside of a very specific system and trying to make it portable is only reasonable when you know the target ahead of time.
As for the hardware, I/O is done through channel programs, which can have any device as a backend - it's a generic way to do DMA.
There is a lot of legacy stuff (like CCKD, VTAM) but more or less it's all emulated today by the hardware/software.
The nut here is that these systems evolve in a very haphazard way do to funding availability, scaling stresses, and changing requirements.
For the IRS, I think it would be useful to engage the US Digital Service to see if they could design a system based on a cluster of non-specific hardware with only network interfaces, that could host a 'tax computation engine' and a growable data store. While it is a 'big' problem, it is not as big as some of the problems I have seen hosted on 25,000 machines as a 'small' cluster. That is 800,000 cores (assuming 32 core machines) so for 140M tax payers that is 175 payers per machine. Something that could no doubt process every return in a single day, and run fraud analysis as well.
Clearly the government needs warehouse scale computers that their vendor (IBM) is not in a position (yet?) to deliver.
We can emulate pretty much any old system, but emulated "perfectly" is at best not demonstrated and at worst already proven wrong. Perfect emulation implies emulating the undocumented bugs that end up working--and there are a lot of those. Getting the correct behavior in the case of parallel components (note that this applies even to single-core systems, since dedicated hardware constructs can race with the CPU) is particularly tricky. Even without parallel issues, you can have cases where software blatantly violates the API but it works reliably and consistently, necessitating complex emulation to make it work correctly (there are some games that don't work on WINE for this reason).
Heck, even if you emulate the documented performance perfectly, you can still suffer regression behavior if the existing program just happens to rely on misbehavior of the original hardware due to less tolerant exception handling, stricter memory alignment protections, etc. in the later deployment environment.
I've seen this a lot in memory misuse / misalignment when porting between various UNIX implementations. If your development environment just happens to forgive a lot of sloppiness (cough Solaris cough), misbehavior may only show up later on other platforms.
As for the rest doing cycle exact emulation of 50 year old system is and probably always will be doable and from some PoV trivial: you can just simulate the hardware logic of the original implementation on RTL or even gate level and it still will be faster than realtime.
[Edit: "reserved for implementation" was missing from the first paragraph]
Too many legacy migration projects fail to account for the active-active nature of migration.
the method by which it works is that code compiled into a new form which can be translated on the fly to match the hardware level below it.
now what is probably trapping the IRS besides a base that not everyone full understands but the sheer complexity of what they have to deal with, the tax code and whims of those in Congress.
Is this a question the industry actually cares about? If anything, I'd say it got much worse at creating long-living software. It sucks to run a mainframe-based system from 50 years ago, but running a 10-year-old Web Forms application can be borderline impossible. I'm scared to think of what will happen with all the cloud-based stuff in the next 10 years.
>How do we separate business logic in a self-contained way that can be independent of hardware and (from a certain level) software?
I often say that there is no such thing as "business" logic. There is just logic. The least maintainable systems I've worked with were the ones with the most sophisticated attempts at abstraction.
In my opinion, the key to software longevity is making systems that allow for local reasoning, i.e. analyzing parts without knowing the whole. This can be achieved in a lot of different ways, but it needs to be a deliberate goal in system design.
Seconded. Enterprise patterns mandating layers and layers of abstraction, making it incredibly hard to reason about the far flung pieces of code that actually do anything.
But something were you can describe something like "Read this value from this field, do some math based on fields in another table, return the result"
That's why I raised the question, how do we do that over long periods of time without rewriting it in several different ways? (ASM to flat files, COBOL and JAVA to different SQL flavours, what else is next?)
And if you migrate, let's say from Oracle to MSSQL it will work for 90% of the things, for the last 10% it will be a PITA.
Cloud deployments, especially modular systems (and of course microservices) are often sold to companies as the best way to reduce technical debt. I don't think they've been around for long enough to truly test it but would be curious if a 30yr old system of microservices running in containers is easier to update than COBOL software running on a mainframe.
50 years ago the IRS had to build a webserver, the servlet application code, it had to build in fault tolerance without SQS or Kafka.
If built today, the IRS' entire system would probably run on Rails with JRuby, deployed with Docker and K8s, and be 1000x more maintainable.
Speak softly into my ear, these little lies you like to tell me.
The system ran on a mainframe not 50 but 60 years ago. The only high level language at the time was FORTRAN, but this system was apparently written in assembly language.
Does anyone know the type of computer it ran on? The article it linked to says IBM mainframe but which model?
No direct insight to the actual system used, but FWIW IBM mainframe assembly IMHO seems 'surprisingly' user friendly due to the large amount of standard macros/routines provided, clear/direct integration with JCL, and the minimalist ethos of the OS. Take a look at some youtube videos for some examples, it actually looks like a fairly interesting/productive, if arcane, environment. Writing clean/well architected applications in it doesn't actually look like it would necessarily be a ridiculous notion.
Of course, how well any given production system was constructed and what cruft has accumulated in it over 50y is probably another story..
[1] https://www.washingtonexaminer.com/tech-timebomb-the-irs-is-...
[2] https://www.bloomberg.com/view/articles/2018-04-17/the-irs-c...
The first COBOL compiler was available 2 years later in 1960. Algol 58 wasn't available until 1960 either.
The ALGO (Algol 58) manual for the Bendix: http://www.piercefuller.com/collect/bendix/algo6008.pdf
The first compiler for PL/1 was delivered much later, in 1966.
You could have suggested LISP 1.0 but that appeared in 1960 too (as an interpreter, the first compiler was written in 1962).
LISP, COBOL, ALGOL: 1960 was a remarkable year in the history of programming languages.
Come up with programming languages with limited domain specific functionality that don't allow you to do funny machine specific things. (or even ordinary things like pointers)
Use the restricted languages for business logic and have a second language to do the plumbing.
Or just target a generic x86_64 virtual machine.
It's also clear the authors don't entirely understand what they're talking about. In the second article linked to, which is written by an author of this article, he states "[t]he Individual Master File, a massive application written in the antiquated and low-level Assembly programming language" as if to imply there is but one assembly language.
A GAO report I found chasing the last link in the article indicates that the two oldest systems in the Government are the IMF and Business Master File (BMF) which it says both run on IBM Hardware. My guess would be that the IRS is running modern z/Architecture mainframes in System 360 Compatibility mode. The System 360 dates back to the early 60s and IBM has maintained backward compatibility with it all the way up through z/Architecture systems. This would also explain why they're so far over budget and behind schedule.
Joe Lunchbucket doesn't care if it's hardware that failed or software. The system failed. That's what his tax dollars pay for.
It is the job of the writer to write for the audience, not include every single detail, no?
Based on my experience regularly communicating technical information to non-technical people, the above suggestion works out very poorly. Each technical detail is a hurdle for the listener, causing misunderstanding, frustration, feelings of inadequacy, and doubt about your competence as an advisor and communicator. Generally, they respond like most people do when confronted with something unpleasant; they tune out your message. My advice is to include the minimum possible technical details and to express them in the clearest possible non-technical language.
Joe Lunchbucket doesn't care if it's hardware that failed or software
Consistently high-quality in-depth reporting is the exception and is not a threshold most media will accept as a publication bottleneck.
In other words, the authors are speaking with authority about something they demonstrated ignorance of by improper use of an article and they went out of their way to use language that would lead a non-technical reader to conclude the failure that occurred was a result of 60 year old code when in reality it was due to 18 month old hardware.
Once I did manual disassembly of compiled code back into a high-level (for the time) language. It wasn't very fun, but , thankfully, the compilers of the day didn't optimize much.
It'd be a fun project.
the compilers of the day didn't optimize much
That potentially puts them in the late 50s. That might be some of the oldest running code in the world. That's pretty cool.
I wonder if the person who wrote it is still alive?
> January 1962 Automated data processing was officially put into operation in the IRS.
https://www.irs.gov/pub/irs-soi/62dbfullar.pdf says
> --Dn September 19, 1961, less than a year following its establishment as a separate division, the Automatic Data Processing Division...
So the division was likely established in 1960. This https://books.google.ca/books?id=x1YESXanrgQC&pg=PA119&lpg=P... history corroborates by saying the treasury authorized the IRS to computerize fully in 1959.
They were 7074, we know that from http://blog.modernmechanix.com/big-brother-7074-is-watching-... Popular Science but also that some docs from 2016 says https://www.irs.gov/pub/irs-utl/scap-pia.pdf
> SCAP downloads Corporate Files On-Line (CFOL) data from the IBM mainframe at the Enterprise Computing Center, Martinsburg. The CFOL data resides in a variety of formats (packed decimal, 7074, DB2, etc.).
> That might be some of the oldest running code in the world.
https://www-03.ibm.com/ibm/history/exhibits/dpd50/dpd50_chro... says The IBM 9090 SABRE airline reservation system begins to be installed by American Airlines in 1961. That makes SABRE contemporaneous with the IRS Master Files or slightly older. I do not know, however, how much of that code is running today.
The instruction set documentation starts at page 42.
https://www.irs.gov/irm/part2/irm_02-005-003
> The scope of this directive is Servicewide. This includes software developed by contractors. Where the guidelines apply to Assembler Language, COBOL, C Language, C++ programming, and Java programming, these guidelines shall be followed respectively.
For a system that is supposedly so brittle, they were able to get enough of it back up and running to at least accept submissions within one day?
That's pretty amazing for 60 year old technology.
The data is not stored in punch cards, the programs are/were. The data is/was stored in tapes or in tape emulators.
Storing data on cards would make random access wayyyy too slow (the operator would have to retrieve and load the card and keep them sorted manually, which is impossible to do at scale)
On tape, however, you can predict the distance you have to rewind by computing the physical length each records take if they have auto incremented IDs. A tape can contain a lot of records. Especially if it's a modern one or an array backed emulated one. At least that's what some people I went to school with told me it's how they did it.
Wide adoption of tape was a lifesaver.
They call them "Beltway Bandits" for a reason.
Building an app to a complex and ever shifting spec is a much different process than the way that startups work. You can’t just release a half done app to friends and family. You need to do the whole thing at once, with 100% if the functionality and it has to be ready for everyone in the country to use it in day 1. And the federal government has fairly onerous requirements for getting stuff through security— you can’t just google for random shit from GitHub and expect to put it straight into production.
I can only speak to my particular project, but everyone on it is working hard at getting it done on time and on budget, using modern deployment and development practices, but there’s limits to how much you’re allowed to just move fast and break things.
They were working on projects to "elevate" it to Java, because it's pretty difficult to maintain it as-is.
Until, of course, they are all gone. At which point the system will need to be rewritten by IBM, Oracle or some other useless, gigantic consulting firm.
If you had to pick, Java does make a lot of sense. Java & C are safe bets for "languages that will stick around for decades".
Why is that scary? It has decades of real life testing applied to it if there were any obvious bugs they would have been found by now.
Assembly is just another language, it is slower to develop in but there are many examples of high level language issues that are at least as scary as what you could do in assembly. In the end it is all about the processes around your development as much or more than it is about what language you pick. The best way to reduce your chance of bugs is to reduce the size of your project.
https://www.mayerdan.com/ruby/2012/11/11/bugs-per-line-of-co...
I also disagree with Mayer's assertions about LOC. In my professional experience, the best ways to avoid bugs are optimizing for clarity as well as continuously grooming your codebase (including modernizing when possible). Obviously the latter is not an option for safety-critical software, and probably not suitable for the tax system either. But in general, those practices have worked better for me than have others.
Really, that's what they said about CPUs - at least if you're not in the fab industry.
See the Chrysler Comprehensive Compensation fiasco: https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens...
These kinds of systems are all about the exceptions and how you enable humans to deal with those, not the rules and how the computer deals with those.
The problem is that the system first needs a comprehensive test suite written. THEN you could actually think about rewriting pieces.
> They were working on projects to "elevate" it to Java, because it's pretty difficult to maintain it as-is.
Yeah, apparently an eight person IRS development team nearly developed a replacement, but the main developers were hired on a special program to pay them at above-GSA rates, and that program expired before they could put it into production:
https://federalnewsradio.com/tom-temin-commentary/2018/01/ir...
NB: Don't object any of the new languages, I've never written code in COBOL or for a mainframe. But I've also never developed software with that kind of requirements.
Never replace anything unless the thing doing the replacing is better than the thing being replaced.
The IRS' IT "upgrade" is just a huge money grab. Nobody has the slightest interest or desire to do anything properly - they simply want in on that lucrative IRS money.
It's like the saying: "Anyone who is capable of getting themselves made President should on no account be allowed to do the job." Likewise, the kind of people who are bidding on this project are certainly not the kind of people that should ever be allowed anywhere near the IRS' computer systems.
Of course the tax forms themselves are still insane.
Easy to say as a grunt, but businesses (and governments) are run by managers and accountants.
What you describe is upgrading for upgrading's sake. If a system isn't broken, there's no point or financial reason to replace it.
Imagine trying to upgrade a system with incredibly complex data on a BILLION users that has to work 24/7, then deal with massive data influx four times a year plus catastrophic data ingestion one time a year.
And not only take in and sort those records, but validate the data and perform calculations based on rules that change almost monthly, and that sometimes must be applied to years-old data and sometimes not.
Plus, you're dealing with people's money. You don't fuck around with people's money.
It would be easier to merge Facebook and Google's databases.
When you build software you should plan to maintain it. Bug fixes, features, upgrades, etc...
A better analogy would be if you were an airline and kept upgrading airplanes before they were needed.
Airplanes (especially private ones) fly for decades and decades.
No one is suggesting that the IRS systems weren't maintained. Obviously they were since they exist on virtualized hardware.
Software systems like this get “mainenance” of the bare minimum sense - think a new battery after the original is dead.
I think you're over-estimating the amount of data involved. There were about 150M individual tax returns and ~5.8M corporate returns in 2015. Even if we assume 50 years of storage in the 'active' dataset that is < 8B returns. Also, most returns don't contain a particularly large amount of data in them. Pretty sure we're talking about an in-memory scale dataset here, certainly for the active set in any given year. It actually would not surprise me if we (current co) ingest more data in a week than the entirety of the US tax return history.
> And not only take in and sort those records, but validate the data and perform calculations based on rules that change almost monthly, and that sometimes must be applied to years-old data and sometimes not.
I might believe that is an issue if the IRS actually calculated everyones taxes for them and sent out a 'do you agree' form, but they don't. It's more like they randomly sample some returns based on some hand picked 'warning' flags and have someone manually check them, maybe with the help of some excel formulae.
The one billion figure was from the article.
The .gov pay scale has not kept pace with either inflation or IT wages and the benefits have been cut by Congress at various points so that’s less of a balance than it used to be. The GS-15 position mentioned in the IRS staffing article is the top of the non-executive scale and technical positions tend to be rare and specialized; managers are overseeing a fair amount of staff and budget. In DC, that starts at $134k and tops at $164k - not bad but DC isn’t a cheap place to live either: https://www.opm.gov/policy-data-oversight/pay-leave/salaries...
Software is especially prone to this as many people think of it as a project with a defined end date rather than an ongoing endeavor.
Before embarking on a re-write, how about we create a simplified tax code (laws)? Not sure which task is tougher.
The problem is 'better' is subjective. Plenty of mechanics will tell you older cars are better because they're easier to repair, don't require specialist machine diagnostics, have manual buttons / levers / knobs that are easier to troubleshoot than digital versions.
To most drivers however, newer cars are better because they're more comfortable and convenient to drive around in.
'Better' means different things to different people.
Crossovers typically have equal or better rear visibility to the car they're based on because the higher seating position means you can look down at a steeper angle greatly reducing the maximum distance at which a kid height object is not visible to the driver.
... which is a very good example of Catch 22 logic:
https://en.wikipedia.org/wiki/Catch-22_(logic)
I personally prefer Groucho Marx :
https://quoteinvestigator.com/2011/04/18/groucho-resigns/
“I don’t want to belong to any club that would accept me as one of its members.”
This is factually wrong (at least for the most current effort): https://federalnewsradio.com/tom-temin-commentary/2018/01/ir...
An eight person IRS development team nearly finished building a replacement, but the main developers were hired on a special program to pay them at above-GSA rates, and that program expired before they could put it into production.
Why, because you're busy being condescending to teach?
Software should be considered a system that needs ongoing care and maintenance, just like anything else.
Legacy data which must be transformed?
Old business logic which must be maintained? I do not know if there is a need to replay old rules vs. just kerping a snapshot
Continuity? Maybe there are so many transactions happening, linked to existing data that an interruption is complicated?
Risk of accessing old data which is better left undisturbed?
I hope they bought a new hard drive to back it up...
(At first I thought that they upgraded IBM mainframe to z14, and that caused the crash, but it's not yet 60 years old.)
They're most likely running some z/Architecture in System 360 virtualization mode.
"Overall IRS maintains over 20 million lines of assembly code. These millions of lines of archaic software, and hardware, that is no longer supported become more difficult and costly to maintain each year, and poses significant cybersecurity risks. To IRS’ credit it keeps these old systems running during filing season, but relying on these antiquated systems for our nation’s primary source of revenue is highly risky, meaning that the chance of having a failure during the filing season is continually increasing." - David Powner, director of IT management issues for the Government Accountability Office [0]
"[IRS Chief Information Officer Gina Garza] explained the code for the IMF was written in the 1960s, but the hardware it runs on is modern." [0]
[0] https://federalnewsradio.com/technology-main/2017/10/a-sense...
[1] https://arstechnica.com/information-technology/2014/04/50-ye...
Sounds familiar.
Source: https://www.brainyquote.com/quotes/will_rogers_128073
No he wasn't, so my confidence starts flagging.
Not filing or, worse, making an error in your filing makes it a really really nasty game. No recommended.
I'll even break my own rule: "Not filing considered harmful"
So the patchwork 1960s-era systems are the fault of Republicans in the current decade?
Maybe "starve the beast" really is a better approach. Nothing else has worked.
The flat tax, file-on-a-postcard system proposed by 2016 presidential candidate Ted Cruz was mocked by the establishment because it seemed tha politicians on both sides have a vested interest in a complex tax code — it is the means by which politicians derive their power. See Milton Freidman on the subject:
Milton Friedman used to say that a flat tax of 15% would be enough to replace the entire scheme of income tax. There is also one major boon: lots of capital is spent on taxes, from accountants to lawyers to gubernamental organizations, etc etc.
An effective 15% rate without exemptions would fly very well with the right (simple flat taxes) an the left (no exemptions) but as MF said, the political stances on this topic dont trust each other and they are both right: even if we moved to such a system, the right would still advocate for exemptions and the left for increases in taxation.
Its a noticeable deadlock.
I also want to mention that flat taxes are not regressive by definition. The progressiveness or regressiveness in that case requires to see where the taxes are spent.
Secondly, I'm not a big fan of how you (and quite a few economists) conflate personal and corporate income taxes. It really muddies the water as even questions like "what even is income" is different between the two.
Additionally, flat tax doesn't go well with the left. On the corporate end, exemptions in a lot of ways are the carrot to the sticks of fines and penalties that we use to regulate business (encouraging positive behavior). On the personal end, you greatly increase the tax burden on those least able to bear it, with an increase in benefits being totally orthogonal (most flat tax proposals intend to be revenue neutral). Someone making $25k nearly doubles their tax burden with a 15% flat tax.
Personally, I'm fine with taxes having progressive rates but not the countless deductions.
Let's say today I make $24,900/yr. That puts me at the edge of the 20th percentile for 2017. The average effective tax rate for people of my income was -4.7% (or a tax of -$1170)[0]. You're suggesting increasing my taxes to 15%, taking $4905 out of my pocket as compared to today. Now, since I make next to nothing, pretty much everything I make is going to rent, food, and just generally getting by, and I have nearly $5k less real money (or ~20% of my income!!!) in my bank account to make that happen.
[0] - http://www.taxpolicycenter.org/model-estimates/baseline-aver...
Its a tautological affirmation on progressive taxes being progressive. A flat tax would make a $24,900 pay $3,735, and if you made $249,000 you would pay $3735, your taxes increase just as proportional to your income. Its just..flat.
With a flat tax:
* On the high end you're taking from ability to have luxuries.
* On the low end you're taking from the ability to pay rent and put enough food on your table for your children.
The utilitarian argument is for me one of the most practical and also one of the most scary. Are you ready to exploit the poor if it were proven that it was economically profitable to do so? I digress. My point is that seeing flat taxes as regressive is either false or inconclusive, regardless of the desire of being flat, regressive or progressive in taxation.
It was an example of how nominal tax rate is not the same as effective tax rate. For example, mortgage interest deductions are a regressive benefit which lowers the effective tax rate. Unfortunatenly, we cannot make good date on what the effective tax rate is because it is protected information.
> On the personal end, you greatly increase the tax burden on those least able to bear it, with an increase in benefits being totally orthogonal (most flat tax proposals intend to be revenue neutral)
That depends on what you cut. And thats why i say that you need to look at the revenue and the expenditures to know if taxes are regressive or not.
For example, Federal income tax is nominally progressive, but california has plenty of regressive taxes: sales taxes and property taxes that favor older tenants. In general state and city taxes are regressive, because those are the easiest to levy.
You're literally arguing against the accepted definition of a term.
Thats why you can have a flat tax, and a progressive government by spending for the poorer and levying flat. In fact, a flat tax is probably progressive simply because the rich dont use a multitude of public services the average poor does, like schooling, or healthcare or whatever.
The commonly accepted denotation:
> (of a tax) taking a proportionally greater amount from those on lower incomes
Has absolutely nothing to do with spending, but on how much proportionally you are taking.
You can have super progressive taxation and a wealth distribution from the poor to the rich, and a super regressive taxation that distributes from the rich to the poor.
With the reporting changes of a few years back, for any stocks acquired after the changes took effect, they also already have all the information needed for the stocks as well. All that is missing is the type of sale you wished to make (FIFO, LIFO, average cost, etc.) when you sell the stock if you don't sell all of it off at once.
You have (purposely?) mis-stated the previous statement quoted above. A patchwork of systems that includes some systems dating back to the Kennedy Administration is not the same as a patchwork of 1960'era systems. Could it be that the "starve the best" mentality hindered or blocked a number of planned migrations/upgrades?
It's easy to throw stones but I doubt anything built today by anyone on HN will rival problems as complex as DEWS, ATC, IMF/BMF or Sabre and be around in ten years, let alone 60.