Why US government IT fails so hard, so often
arstechnica.com
arstechnica.com
Talking with folks afterwards, I heard about systems that I could have rewritten in the late 90s in a few months that are still in the process of being rewritten, 15 years later. After literally hundreds of people have worked on them costing the taxpayer millions of dollars. And these are not unique or oddball edge cases. Stuff like that is standard operating procedure. I can't impress on the HN audience enough how terrible federal IT is overall. (Aside from some really great outliers). There's a vast and increasingly large gap between political rhetoric and the reality on the ground. This has been true for at least the last 20 years.
I take issue with some of the details the supporting argument makes here, but that's mostly a quibble. The author is definitely headed in the right direction.
I'm a veteran of several federal IT programs, and a real veteran. I have seen some really good federal teams, but never a good federal team of teams. It all falls apart once you get past a few people and start throwing some kind of structure and management in place. I understand that Agile is becoming required for some federal work, but I'll bet you a thousand bucks this kind of "Agile" isn't going to be the kind that the rest of us use. Think big tools, big management, lots of stupid measurements, and no accountability. All with the word "Agile" stamped on the front of it.
...The bottom line is that federal IT programs' success is measured by things that have nothing to do with how successful they are or by the metrics most of the world uses. While the business world (and Web companies in particular) now monitor user experience and productivity as a metric for IT success, the government keeps throwing out numbers that mask the truth: the only people who would use their systems are the ones that are forced to...
Yep.
We saw this same thing with source control. Who doesn't like source control? But the government, in an effort to make sure there was absolutely no risk, mistakes, or accountability associated with software development, started adding more and more mandatory forms that had to be completed each time a developer checked something back in.
This created such a mess for developers that it wasn't unusual for one check in to take hours to accomplish. You'd read a book while the system tried to accept your code changes.
Agile is all about failing fast and learning quickly. Most contractual work for the government does not include the possibility of failure. At all. In fact, the way the contracts are written is such that the contractor takes on all responsibility for everything, even things there's no way they could control. In response to this idiotic situation, contractors start bidding at crazy low rates on these things, say 40 bucks an hour, then actually make their money on changes. So it's always in the contractors interest to capture and execute changes, while it's the government is trying to get the world delivered to them on a platter for five bucks. People use the term "perverse incentives", but this is like that taken to the next level. It's a multi-layered thing. It's total systemic failure. We'd call it systemic collapse, but the money keeps flowing in anyway, so that term isn't applicable, although it certainly feels like it.
This kind of distrust doesn't make for anything but a living hell for everybody, government employees and contractors alike. So, with tons of rules and structures in place, the way things really work is through relationships. People who are easy to get along with and helpful get more government work. Both sides of the fence can't control the process, and nobody rocks the boat and everybody gets a piece of the pie. You end up from the government side with some sort of informal and unspoken version of I don't care if you make a lot of money, just make sure me and my friends look good Also it'd be nice to pick up a little lucrative consulting work once I've put in my 20 and can start double-dipping.
Your average developer has absolutely no idea about this. If they've worked in IT in federal work, they probably have horror stories, but the more you see the worse it gets. Oddly enough, in the best cases you end up with people who really like working together, are motivated, but spend most of their time fighting a Sisyphean fight against the idiocy of the system.
That's because "the government" is operating under different pressures than corporations. A corporation, big or small, screws up, only a handful of stakeholders are affected and raise a stink. If the government screws up, you're looking at headlines in the Washington Post.
Plausible deniability is a powerful force.
Plausible Deniability: Delegation of tasks and lack of records of communications letting the actual responsible party deny knowledge of the "insubordinate" or "rogue" underlings.
CYA: A process to change the process to define the process for tracking changes to the process, with 20 signatures or more required at each step of the process, digital forms filled out in triplicate. Or, making proper recordings of communications with underlings so when they screw up you can point out that they didn't act according to your direction, and that they failed to report issues back up the chain.
That isn't to say that state governments can't screw things up also but at least the damage is contained and state and local governments generally respond better to their constituents complaints/needs than the federal government.
Agile can't be mandated; you can mandate methodologies (e.g., Scrum), but by the very act of mandating a methodology you are not agile. Agile is a culture, and it is a culture of flexibility in methodology with focus on the goal.
And its a culture that is very much contrary to the entrenched culture of most large bureaucracies, government or private (and, in government, often in tension with binding mandates where the authority to make changes to the mandate is very distant from the locus at which one would normally expect efforts to adopt development approaches.)
That's hardly confined to Federal IT projects, however.
People like to blame contractors but I think a large part of the problem is that the federal employees who are responsible for these big projects are rarely held accountable for failure. In fact, the lack of accountability for federal employees is why so much work was farmed out to contractors in the first place.
Then there's the fact that a lot of the decisions in government IT are made by people who have little understanding of how successful software gets built. People don't understand the importance of defining the scope of the project properly. Or you get people that change requirements willy nilly. Or you find people that are just trying to protect their little fiefdom so you end up with wasteful duplication of effort or withholding of vital information.
The Ars article indirectly touches on the fiefdom issue. When you have dysfunctional IT departments or applications, often times instead of fixing the dysfunction, someone just creates their own little separate IT effort to get around it. So the VIP's might get something functional while the peons get classic ASP & Access.
What I find very interesting about the Healthcare.gov problem is that this may be the first time that a major federal IT project failure has major political consequences for elected officials. If Healthcare.gov ends up being a crash and burn failure, it could result in Republicans gaining enough power in the next election to overturn the whole ACA.
I agree that requirements are a big issue. In addition to the requirements issues you mention, there is the problem with the "business line owners" attempting to unnecessarily dictate implementation details, poorly defining the functional requirements in the first place and ignoring interfacing systems.
Not to say that FTE's are great or anything, contractors do tend to me more up-to-date on technology but I think that is a result of the government employee hiring practices and compensation scale.
Yeah I've seen a lot of "business line owner" issues like the ones you've mentioned. Another common one is that software is often built to mirror dysfunctional and wasteful business processes. So you either get software that is needlessly complex, or you build the software then have to re-do it because people realize how bad the underlying business process was.
When was the last time you saw that happen? The only major .gov IT screwup which I can recall having any consequences for the vendor was the FBI's virtual Case File system. Everything else tends to get brushed under the rug either to avoid disrupting the relationship (!) or because nobody wants to have to explain why they didn't say anything before the failure was publicly obvious.
Then there's the incestual revolving door between government and contracting companies. Government lead retires, then goes to work for the contractor they used to oversee. The biggest contractors use that as a prime mechanism for earning new business. In some companies the best path to get into senior management positions is by being a former Fed.
NYC had a runaway project where they spent something like $300M on an employee time card system (google "city time"). There was no consequence until some bank teller noticed that two guys were literally stuffing duffel bags of money siphoned from the project into safe deposit boxes.
The prime contractor was SAIC.
Contributing factors are, but not limited to:
- scale
- complexity
- legacy / installed base
- laws, regulations, rules
- organizational processes (workflows)
- quality of developers
- resources
- politics and agendas
Just trying to support all the entrenched chaotic mutant use cases and edge cases will wreck any project plan.
I much prefer working at younger, smaller companies. There's fewer roadblocks. But the healthcare and other benefits working for government rock, so here I am.
Probably why the government is so big and expense. I can only imagine what that $600+ billion gets us for the military. -_-
http://en.wikipedia.org/wiki/Military_budget_of_the_United_S...
Strange to contrast that with a tremendously active and efficient era of public works from the 1890s through the 1950s. Just thinking of New York City, I suspect that the total cost of all the bridges, tunnels and highways (adjusted for inflation) might add up to a lot less than the 800 billion USD that disappeared into our recent stimulus bill. What changed?
edit: Apparently the Brooklyn Bridge cost ~15 million USD to originally construct, which now would be somewhere between 300 million to 400 million USD (http://historical.whatitcosts.com/facts-brooklyn-bridge.htm). Even though these are rough figures and inflation over large time horizons isn't a perfect measure, I'm still astounded. The government's totally broken healthcare website cost more to build than the Brooklyn Bridge.
Vague platitudes about society at large aren't crashing government projects. Society at large aren't really involved in them, after all. Something specific between the interaction of the government and the government-chosen contractors is probably the problem.
In a way, you're probably proving far more than you mean, accidentally; if the problem really is "society at large", then there is no solution and these large projects can't work, and therefore the most effective thing the government could possibly do is stop trying and wasting money. (A project with a prerequisite of "change the larger culture's attitudes" is simply a guaranteed failure, after all.) I suspect that's not the point you're going for. You probably really want to be arguing for a much more local reason.
Bad oversight: whoever's managing the project, doesn't understand the project. Not just, "I don't know how to read <some language>" lack of understanding, but "I don't know the difference between a spec and code". They don't specify terms that keep contractors in line which allow projects to run years and millions or billions over budget. Most of these contract managers aren't engineers, they may have engineers on staff, but it's not uncommon for those engineers to be employees of the same contractor that receives the benefits of the contract. The system is both incompetent and corrupted.
The companies receiving contracts that habitually run late and over budget are also some of the largest. Small contractors don't make it when this happens, but NG, Lockheed Martin, Boeing, they can survive the political fallout thanks to their clout.
If the companies have no oversight, and no accountability internally because they have no real concern for the other stakeholders (ie, the people paying them, the citizens/taxpayers), and the desire to make high quarterly profits, they'll rig the game by rent seeking, rather than completing a project effeciently and effectively and moving on to the next one. See NG and the maintenance of the JointSTARS aircraft as one such example. A rushed, but militarily valuable, project that had no maintenance plan in place, ends up contracted out to a company that destroyed one of the (few) planes and received no real punishment for it.
Thanks to the tenure-like nature of civil service, a large chunk of government employees are retired in place, new employees have no place to start (few positions opening), and those that get in have a terrible example in place and become retired in place after a handful of years themselves. They may start off dedicated to the job, and many do remain dedicated, but they are also frustrated by the processes involved and either quit or ride it out to retirement.
And regarding the platitudes, if companies and citizens (and consequently many of the elected and appointed officials) see profit as motive, rather than the future of the country or the society, then they make decisions based on that altered morality. Decisions based on short-term profit may not be harmful to long-term objectives, but they often are or are neutral to such objectives. A modern works project on the scale of the construction of the interstate system would likely never see the light of day, and if it did it would be mismanaged so as to never achieve its objectives.
Normally I type these up in emacs and copy them over, this is not a well formatted message, sorry about that.
Would it help to know that the same government (altho Federal Reserve is private but heavily regulated) print the money for their own use?
Imagine you have a small house with a garage inside and printing money machine in it. Now you want to expand the house and you know you can pay for the materials labor, etc, thanks to your printing machine. Wouldn't you do it?
The answer is those who care about US, about their outlook on the world would be very careful with using this machine. Those who would "own the house" for just a little bit would abuse it. We are at this stage where government human resources are so overbloated, ,you need so many signatures to buy a roll of toilet paper that eventually at the end of the day nobody is responsible for anything.
That can least if you own the world and don't have to answer to anyone. Un/fortunately this is not US case. If we default, we lose credibility with other major player such as China. Imagine if China stops producing for US (would be insane embargo if you think about out), can you imagine what would happen with us economy? all of sudden the soap dispencer you buy at wallmart cost $45 (made in USA) instead of 4.99 (made in China). Unfortunately, like I said there is no credibility even Chief in Charge Barrack Husain Obama is not aware (in most part) what is really going on under his watch.
And;
Chief in Charge Barrack Husain Obama
Weeks of development time have been spent dealing with how to keep our memory load under 1Gb, since that's how much RAM we have. Granted, 4GB of RAM would have been more than enough, but all that developer time saved the tax payer $20.
On the reverse side, I've seen half a million spent on hardware for a project where the sys-admin was making $35k.
The guys who built the Brooklyn bridge were day laborers who made nothing and died young.
http://articles.latimes.com/2006/jul/02/news/adna-sand2
The young guys probably make 125-160k, the old guys get overtime preference. IMO they deserve it -- it's a really dangerous job.
In any large scale project, any issue cascades and costs a shit-ton of money. For government contracts in many states, you have stuff like "prevailing wage" (where the gov pays too much for labor), "project labor agreements" (where the gov agrees to employ X number of people), minority and women owned business quotas (a big pit of corruption).
Also remember that governments are run by politicians, and politicians embrace magical thinking. Painting a rosy picture to get something approved and blowing huge money with no tangible result is a good way to raise capital in govt. (whoops, $X didn't do the trick, but to avoid wasting it, give me $X+Y)
The USA, like many countries, has rules about simply seizing land that are of the form "well, OK, if it's for a good cause, but you better pay for it".
http://en.wikipedia.org/wiki/Interborough_Rapid_Transit_Comp...
http://en.wikipedia.org/wiki/Brooklyn%E2%80%93Manhattan_Tran...
Busses and ferries were private too:
I agree though that the mix of in-house expertise vs outside contractor effort does play a big part in the outcome. Look at Boeing's 777 vs 787 aircraft development for different sides of cost to use internal resources, vs leaning more heavily on external resources. The 787 with it's heavy use of subcontractors has had a much more difficult introduction.
The government in general has been upping the contractor content in a lot of projects with the belief that more "competitive private contractor" content automatically means better efficiency. But, in truth, it's no silver bullet, often increasing costs if it's blindly applied - especially if you fall below some threshold of in-house expert judgement.
I can't vouch for the American case, but New Zealand has a similar history of disastrous government-run IT projects - from a terrible teacher payroll system[1] to a police information system[2]. The IT community hear is desperately trying to stop the same mistake with a massive tax system overhaul[3].
Nevertheless, civil infrastructure projects appear to be very well executed. Recent major roading engineering projects like the Victoria motorway tunnel in Auckland were completed on time and under budget, for example. There has been an active discussion in this country about why the IT industry, in particular as manifested as major systems projects executed by foreign software companies, are such consistent clusterfucks.
It's a big discussion and I won't go into many of the points raised here, but the issues is, IMHO, very much a "software industry" one, and not a "government" one.
1. http://en.wikipedia.org/wiki/Novopay 2. http://en.wikipedia.org/wiki/INCIS 3. http://www.stuff.co.nz/business/industries/9261063/Row-deepe...
Construction management is a serious discipline that is studied by civil engineers. It helps that construction methods don't change very often. Software technologies can become obsolete in a matter of months.
I wouldn't start calling IT procurement a failure compared to civil infrastructure until software engineering has matured to the point that civil engineering has.
I live in Perth, Australia. Almost every civil project there is blowing out like a balloon. Opera houses, hospitals, fancy developments. All going over budget and schedule.
I was in Toronto a few days ago. What's on the cover of the newspaper? A civil project. A subway station running way over cost and time.
Industrial and civil megaprojects routinely go way off course. Bad project management is universal, good project management is hard, any project of any serious size will require owner-commissioner organisations to not be useless idiots.
But, obviously these are all anecdotal, I just know from my time at a large civil engineering company that the failure rate appeared to be very low. Part of that is excellent risk management, but also long-term client relationships, clear and long established success criteria and a distinct lack of political meddling (once the project is underway, at least).
http://marginalrevolution.com/marginalrevolution/2006/08/the...
Large projects fail or run over budget because of organizational failures: poor goals, lack of accountability, internal gridlock, etc. Nothing about that is specific to government – and if we wanted to change that in the United States we could reach the level of, say, Scandinavia if we were willing to make significant cultural changes.
However, I suspect the true extent of the flaws with Healthcare.gov and the exchanges is not understood yet.
If anything good comes out of this, I hope it's that the government finally starts paying attention to the woeful state of government IT projects.
...The IRS has failed twice to modernize its system, in the 1970’s when taxpayer privacy issues surfaced and in the mid-1990’s when the IRS spent ten years and $2 billion on the project and had little to show for it. The current project launched in 1999 with a budget of $8 billion. The project is late and over-budget; problems have been well publicized after the release of the IRS Oversight Board investigation...
http://www.kellogg.northwestern.edu/student/courses/tech914/...
AFAIK, the IRS dropped $10 Billion in IT money in just over a decade with little to nothing to show for it.
I used to joke around and say heck, they could have paid me 2 Billion and I would have also provided them nothing, but look at the money they would have saved!
So NSA, which is fairly young, stays out of the political eye (until recently), and small? They do some really cool stuff. But the older, more controversial agencies, covering things like immigration, defense, taxes, health, and so on? Not good.
This is why you run into legacy codebases that are 10-20 years old and will keep on going.
What disincentivizes talented (specifically young talent) people is the ageism inherent in the system and its perverse method of rewarding people (specifically talking civil service here, contracting's another beast). Unlike the ageism you hear about in the occasional article about Silicon Valley, its the young that have a hard time in civil service. Positions aren't available because too many people are waiting to retire (for various reasons). When they do get in there are few leadership roles open for advancement (see previous sentence). Their pay is strictly time and not merit based (well, not 100%, but NSPS is gone and only a handful of places use its equivalent of merit based pay increase). Career fields are disincentivized, for instance engineers are treated better than computer scientists and mathematicians even in offices where the latter make more sense (bonuses, faster promotion schedules for trainees, pay differential for the degree). Lack of proper training programs (this is office by office, so not sure how universal the problem is, but it's pretty shabby in a lot of places). People are expected to be hired in and, within months, become experts on arcane and outdated systems, with no support because the actual expert has just retired (that's whey there was an opening). People leave quickly, similar to the Teach for America article earlier in the day, because they burn out or realize there's no where to go, neither up nor down, just steady.
On the other hand, in some regions older, talented folks can get a good deal. They can come in as a high grade GS employee with a middling step, automatically enrolled in a decent 401k (5% matching), good insurance (less critical with Obamacare?), and, after a couple years, effective tenure so they have stability which they might value for their family.
The size of healthcare.gov is actually well under the average:
http://www.webpagetest.org/result/131010_ZG_X5C/1/details/
(see http://www.webperformancetoday.com/2013/06/05/web-page-growt...)
For comparison, look at WashingtonPost.com: it uses considerably more data and more total requests and yet the page loads much faster:
http://www.webpagetest.org/result/131010_GG_XDK/
If you look carefully, though, you'll see a) that they aren't setting Cache-Control headers and are setting ETags (both performance killers) and b) there's this weird staggered response behaviour which indicates something is seriously wrong. It seems to handle much better in Chrome - 3s vs 70! - so presumably there's some interaction with the browser going on as well:
So why hasn't the likes of puppet and chef caught on? Genuinely curious.
Who is selling support for Puppet and Chef?