How Healthcare.gov Botched $600M Worth of Contracts | Hacker News Reader

How Healthcare.gov Botched $600M Worth of Contracts

bloomberg.com

75 pointspetethomas72 comments

Hey HN! I worked on the "tech surge" to save healthcare.gov, and as many of you have noticed, this report gets everything backwards. A 96-hour training course would not have saved healthcare.gov, and you can't fix bureaucracy with more bureaucracy.

If you want to understand what went wrong on healthcare.gov and how it was fixed, Mikey Dickerson's talk here is the closest thing we have to a postmortem:

  https://www.youtube.com/watch?v=0albm_hhQzM

healthcare.gov has also given rise to new initiatives to fix the way the government writes software. Within the government, there is the US Digital Service, 18F, and the Presidential Innovation Fellows:

  https://medium.com/@USDigitalService/an-improbable-public-interest-start-up-6f9a54712411

  18f.gsa.gov

  http://www.jasonshen.com/2015/when-did-you-do-your-tour-of-duty/

  http://www.jasonshen.com/2014/everything-wanted-know-presidential-innovation-fellowship-afraid-ask/

There are also new types of government contractors, including Nava and Ad Hoc LLC:

  http://www.fastcodesign.com/welcome.html?destination=http://www.fastcodesign.com/3047856/innovation-by-design/meet-nava-a-startup-that-wants-to-fix-the-governments-crappy-design

  http://adhocteam.us/

Please forgive the directness, but I need to ask you this because I have noticed a pattern. Whenever HealthCare.gov gets mentioned, there's always the most over-the-top enthusiasm for how it was such a great turnaround.

I've worked with dozens of large projects -- including ones that were turned around -- and this is new to me.

The only thing new I can see here is the political environment around the healthcare law. Hopefully I am wrong.

So when the Republicans take control over the executive branch, this time or whenever in the future, are we going to keep hearing such great boosterism? The Republicans decide to automate voter identification and the project goes south, will we see such great turnaround stories? It would be very comforting to know that the new Digital Service and the influx of enthusiasm is not limited to one party or large government project. Some assurance would be most appreciated.

"So when the Republicans take control over the executive branch... are we going to keep hearing such great boosterism?"

WHy are you asking him? I don't think anyone knows the answer to this. Obviously the major factor would be whether the next president (be they Republican or Democrat) decide to continue support for these kinds of teams, both in terms of funding and autonomy to do their job. This seems like something we should be asking the candidates and their parties about.

Because the topic of this thread is the large scale system of project delivery and changes to it. The poster made some sweeping statements about how the article was not accurate and provided great reference material to what he considered more relevant material.

Given the topic of how these large systems operate and could be improved, and given the poster's claims to have special insight, seemed reasonable to me to continue the dialog. After all, this was an example of behavior associated with the topic at hand.

And my question was about enthusiasm and public support. Why would presidential candidates have anything to do with this? If it is partisan, then my question has been answered.

The US Digital Service started with healthcare.gov, but nowadays they work across the government on projects that matter to people all over the political spectrum: VA, immigration, department of education, and more.

That said, I have no way to predict who the Republican nominee will be or what they'll do. I think the impact of USDS relative to its budget is huge, and I'd like to believe making the government competent at writing software is a bipartisan issue, so I would hope this initiative crosses over into the next administration, whoever it is.

Right now, I think everybody at USDS is just focused on getting results over the next 18 months.

This is one of many audits conducted as part of the IG's review of the healthcare.gov rollout. The purpose of this audit was to determine if existing rules regarding contract management were followed.

It in no way claims that taking a 96 hour training course would have fixed the rollout of healthcare.gov or even that the contract management failings were the primary reason for the failure of the rollout.

I hate to say that the (lousy) article claims exactly what you say the audit doesn't claim. It specifically describes "how shoddy oversight by the Centers for Medicare and Medicaid Services (CMS), which manages federal health programs including Obamacare, contributed to the website’s early struggles."

"contributed to" ≠ "caused"

The other reports suffer from the same problem -- they're focused on whether paperwork was filled out correctly. That's not what killed the project. To the extent these reports are read, I think they lead people to focus on the wrong things.

I think the IG reports would be much better if they just did the following: interview the engineers who worked on the original version. Ask them what the problems were. Then, interview the engineers (from both the "tech surge" and the contractors) who lived through the process of getting the site working. Ask them what changed. Write those things down.

I'm a college student, how do I get into doing the work that you did? I'd love to spend some time on the side helping out my country.

I'd recommend either Nava (navahq.com) or the USDS reserves (https://www.whitehouse.gov/digital/united-states-digital-ser...).

I wonder how this affects the many IT companies which rely solely on government contracts. I looked for some of smaller the companies that are on the list of the report. There was one which has its "HQ" in a single family house. This whole business sector looks shady and more like scheme to earn easy tax payer money through government contracts. Is there some form of götterdämmerung for these companies or are they worked with in order to improve?

The solution proposed to fix this mess is insane. A '96 hour training course' at the end of which you are able to handle 100+ million projects? That's 2 weeks of work + change. Does anyone think that they could end up qualified to handle contracts of that size after a generic 2 weeks course?

Of the people that enroll in this program, what % fail? If it's serious, the % of people qualified should be at most 50% - yet it's obviously going to be just a rubberstamp where you put in the hours, then get the certification.

As someone who firmly believes in the power of the state to do good, these things make me go insane. The solution to failures of this magnitude are not to pile on more useless regulations and certifications, or to put even more emphasis on useless credentialism. Healthcare.gov got fixed when they gave carte blanche to highly qualified engineers and managers that gave up very lucrative jobs to help fix things.

I am convinced that lots more people would be willing to give up their cushy tech jobs for a job where they genuinely made a huge difference in the lives of people, provided you got rid of all this completely meaningless hoop jumping. Give clear lines of responsibility, take a serious look at your hiring process, and jesus, if your solution to a 600+ million dollar failure is to mandate a 2 week course... you really need to think things over.

Glad I wasn't the only one who balked at the idea it would have gone well if everyone had more training.

I think we really need intelligent, experienced people who have a proven record in large product management. However, they're rare and more expensive than the government will pay for. Compounding the failure is the fact that you can't get fired from government jobs for incompetence. These projects aren't going to get any better.

I mostly agree, but I don't think that it's just a matter of money. Money is a way to motivate people - but for lots of people, the ability to work on a system that will cause a huge improvement in the lives of hundreds of millions of people is reward enough. However, nothing saps motivation more than pointless credentialism and bureaucracy.

While I agree that the process for hiring should be better than "pointless credentialism", it must also be highly selective, or else the increased salaries will go to the same people, resulting increased costs and unchanged results.

And therein lies the rub, because the people qualified to hire such teams also do not work for the government.

Perhaps there could be a "expert panel" of external people who can be called up to at least give input into hiring gov employees. Perhaps you can have two hiring panels - internal and external. When they agree, hire the person.

Of course this raises the question of who hires the "expert panel".

>you can't get fired from government jobs for incompetence. //

This seems highly unlikely, can you expand further, some proof? [I assume you're talking about US state and federal government?]

An easy one, how about the ongoing fiasco that is the veterans administration?

It seems to be a property not just of government, but any large organization that becomes driven by politics over results. I just quit a job in a large corporation where I had the displeasure of observing this up close. I have noticed several key dysfunctions in a political hierarchy.

1. You cannot be smarter than your superiors.

After several layers of this, this cumulative effect is that the people on the bottom (those that do the actual work) are not allowed to be smarter than a sea sponge.

2. Reporting bad news to superiors is punished.

Problems are covered up instead of solved. Reports up the chain must hide problems without being provable lies so they are incomprehensible - they must appear legitimate but be devoid of all fact. Using as much jargon and acronyms as possible is effective.

3. Blame for failure will be placed upon the least popular person.

Typically this will be whoever is drawing attention to problems. The reason for project failure will be because of "bad attitude" and not being a "team player" leaving the next project with a greater proportion of ineffectual suck ups.

1. You cannot be smarter than your superiors.

At one company I worked for, you couldn't be taller than your superiors either.

I've seen that effect, but I don't think I've ever seen it as rigidly enforced as this. It does illustrate how most hierarchical organizations are nothing more than adult versions of king-of-the-sandbox.

It was never a formal policy - but it was common knowledge that you shouldn't even hire someone taller than yourself because they'd surely leapfrog you for promotion.

Maybe we need the government doing fewer huge, complicated projects?

I find it interesting that this audit only covers administrative contract oversight. The failings uncovered seem to have been of the variety "Contractor A didn't submit monthly reports for February and March". I don't claim that contracts of that size should have poor administrative oversight, not at all. Even movies with big budgets mention the assistant producers and accountants in the credits.

But what about technical oversight? Were there design-review meetings where somebody said "our system emits data in HL7" and somebody else said, "we haven't defined the XML schema for our data acceptance module yet." In a meeting like that somebody in tech oversight should have said "wait! is it HL7 or XML? Who is going to sort this out?"

(I have no idea whether this actual issue came. Probably the stuff that actually came up, or should have, is far more intricate.)

The purpose of administrative monthly reports' "issues" section is to track stuff like like this. That's why contracts require them. If the government can make the Apollo program work, the government can figure out technical oversight. It just takes the will to do so.

I see it as a high/low level oversight. If people don't know who controls the project and how to measure the progress on it, then who would even care about technical reports? That's like checking the details on a body with a head missing - maybe the guts are OK, maybe not. He's dead anyway.

Lesson for government: Software is not a typical infrastructure project (e.g. bridge). It is almost impossible to specify all requirement upfront. That's why waterfall methodology is prone to spectacular failures.

Training government officials how to manage those project can help, but I doubt it will make a difference. It can rather give false sense of "fixing it" without actually fixing it.

I would rather focus on hiring competent ppl and give them freedom to do that right. Such as: https://www.whitehouse.gov/digital/united-states-digital-ser...

Competent people with a long history of developing whatever you are actually looking for. Software is the wild wild west. And there are many caveats because lots of people claiming to know what they are doing who know nothing. And even corporations who don't know what they are doing, but at good at marketing to get the contract.

US gov has been doing software projects for as long as there has been software. So it's not like there _should_ be new lessons. They could just look at the vast number of previous catastrophes, and the few that went reasonbly well.

I've been involved in a few myself as an employee of a government contractor. The gov people overseeing the projects were always in over their heads. I think that's a common issue.

I wouldn't lay blame at the government over this. I have seen countless large implementations fail within enterprise companies i.e. massive overrun in budgets, not meeting requirements.

And unfortunately waterfall is really the only approach for projects like this i.e. you are having to select different third party vendor products up front and integrate them with custom applications. You can't iterate in sprints when you don't control the software.

And if only hiring competent people was so easy. Really talented IT professionals are very hard to find and keep. The Digital Service you mentioned really is a great step towards retaining talent and long overdue.

I'm a bit shocked that anyone could believe there was actually $600M of software to build. I know prices are inflated by the whole federal contracting guidelines but I'd say these contracts were at least two orders of magnitude greater than what they would have been in the private sector. And nothing against our friends to the north, but why wouldn't we bid more of this out to US companies? Perhaps those that satisfy the WO-MO requirements?

It boggles my mind that they spent so much. CGI Federal got almost 300 million. The company I work for has an IT budget not much smaller. We employ over 1,000 people around the world in IT alone.

They must've had an army of people working on crap apparently. I'm sure they higher ups padded their wallets and walked away happy whether it worked or not. The first mistake was not finding a department with big budget experience.

Part of the issue is everyone who is present at every meeting gets billed for the time at the meeting for the project.

Mikey Dickerson talked about how there were ~50 contractor firms and nobody seemed to be in charge. This is good for all contractors because if nobody's in charge then nobody gets in trouble. https://www.youtube.com/watch?v=7Vc8sxhy2I4

I think that is the one thing about all of this that saddens me the most. There's definitely enough talent to make such a project function, and yet we totally botched that up completely. I've known teenagers in my younger days who've created much more complicated projects in shorter spans of time.

> I've known teenagers in my younger days who've created much more complicated projects in shorter spans of time.

No you haven't, and the hyperbole doesn't help. It's quite possible the specification of all the legislation is actually logically unsolveable since it was created by thousands of different people over the course of decades with very little coordination. When the time comes to build a unified system it's no wonder the job is very very difficult.

> It's quite possible the specification of all the legislation is actually logically unsolveable

The specification was made by people as part of the project, it's not a law of physics. The teenager wouldn't create so much complexity.

Not a law of nature, but needing to accommodate thousands of man-laws that predate the project by decades, yes. The teenager is also not responsible for solving the health needs of 300 million people.

Okay but aren't you implicitly assuming a sane set of specifications? I've seen it claimed that the input legislation was forty thousand pages. Can't confirm that of my own knowledge, but if it's true, far from criticizing the management of the implementation, we should be holding it up as a miracle that it was ever successfully implemented at all for any price.

I've seen that too ... and it probably cost many millions of dollars to come up with the ambiguous and likely contradictory specification. People always want to blame the waterfall method for this, but there are a couple aspects to BDUF that I think can help even the most agile project. From the DOD world, Interface Control Documents (ICDs) are a great way to make sure components (especially those developed by different teams) inter-operate. There should also be a reasonably cohesive architectural vision. From that starting point, write some user stories and adapt as needed for each sprint.

You need to know approximately where you're headed, but you almost certainly CAN'T know everything you'll learn ahead of time (as this sort of specification implies.

As an aside, contractors love ambiguities in specifications and it's actually in their best interest (unless operating on a fixed-price contract) to purposely produce something that meets the specification but is obviously not what the customer needs. There's good money in ECRs.

This seems to put most of the blame on the gov workers managing the contracts, and very little on the contracted companies themselves. I would find it hard to believe that they also weren't absolutely sure what they were milking.

It sounds like the people hiring the contractors didn't exactly do their due diligence.

> unclear who is really in charge, sometimes it's no one

> vague deliverables from people who don't know what they're talking about

> tons of cash, regularly, with no end in sight

Sounds like an incredibly profitable and easy contract. It's an ethical grey area, but it's not surprising that companies took advantage.

"easy contract?" I doubt it. Ten million users on the go-live date? Interchange of health-care data from system to system? Getting Oracle's identity / authentication management system integrated as a door-one requirement? Hostile congresscritters doing everything in their power to torpedo the project? The only difference between this and a typical giant cluster#uck is the vast and visible scale.

> I would find it hard to believe that they also weren't absolutely sure what they were milking.

It was pretty clear that the project was doomed to start with. Maybe some companies didn't bid because they knew that, leaving only companies that were either naive or willing to get paid for a broken product and then blamed and abused after.

In other words, I completely disagree. The buck absolutely needs to stop at the people putting out the contracts and spending the money.

This is one of several IG audits covering the healthcare marketplace roll out. The purpose of this particular audit was to review the government's management of the contractors.

Looks like they should look at who stole the money.

Even a very generous 100 engineer team paid at 300k/year for 3 years only costs 100 millions.

If no one is looking into it, it's reasonable to assume that whoever is supposed to look into it is the one who stole the money.

> If no one is looking into it, it's reasonable to assume that whoever is supposed to look into it is the one who stole the money.

While this isn't exactly unreasonable, I don't see why it's such a slam dunk compared to "whoever is supposed to look into it is lazy" or "there is no one who's supposed to look into it".

its a joke software and governments here in Australia checkout out this mess.

1.5 billion on a Public transport metro card system https://en.wikipedia.org/wiki/Myki#Issues_and_criticisms

1.2 billion on a payroll systems WTF!! http://www.theaustralian.com.au/business/technology/queensla...

UK's Connecting for Health System

> Originally expected to cost £2.3 billion (bn) over three years, in June 2006 the total cost was estimated by the National Audit Office to be £12.4bn over 10 years

Which is roughly 27bn Australian Dollars, when it comes to this stuff our Government is in a class of it's own.

IIRC the system was then abandoned as unworkable?

Most of it was, some bits where salvaged (I think, I followed it at the time) at huge cost and then they built a newer version which sorta works.

In any base £20bn over 10 years seems like quite a lot on an initial budget of 2.4bn over 3.

Wow, even including the cost of outfitting all the readers, that metro card system has got to be at least two orders of magnitude higher than reasonable.

1.2 billion on a payroll systems WTF!

http://c2.com/cgi/wiki?WhyIsPayrollHard

holy shit that site is like tvtropes for programming.

This will go into the big stack of incredibly large technology project failures, including the FBI case management system and the IRS modernization program.

From a systems point of view, the problem here is the publicity. For each public disaster, there are many more that never get that much attention.

Just got through reading a biography of John Boyd. Great narrative about how huge systems of people act politically to resist change and make decisions.

Botched health IT projects aren't new The UK has had some horrifically overspent and underdelivering health IT projects. It's a shame that people don't appear to learn from other people's earlier mistakes.

http://www.telegraph.co.uk/news/health/news/8780566/Disastro...

£11.4bn !

You presume the collectors of that $11B and those that benefit politically and financially indirectly (revolving doors) want to "learn."

This is why I laugh when people complain about the cost of a Trident replacement... You could pay for it out of the NHS IT budget and neither patients nor medical staff would even notice.

You can watch a very entertaining talk from one of the Google engineers who went to fix the website at the time: https://www.youtube.com/watch?v=GLQyj-kBRdo

The most ridiculous part, they didn't have a way to run distributed commands over ssh. When a filesystem mounted in read-only mode, somebody had to ssh into each of the 175 servers manually and run commands to fix it.

What do you mean by "distributed commands over ssh"?

Say you want to run a command on a group of machines using SSH. Rather than manually running the command on each remote machine you can automate their execution over the group using what is known as distributed commands.

http://www.tecmint.com/using-dsh-distributed-shell-to-run-li...

http://www.ibm.com/developerworks/aix/library/au-satdistadmi...

600m and the gov still has to take the contractor with the lowest bid still? Makes me think the requirements were so broken no one really wanted the work.

Tangential question, how do software firms get contracts like this? Seems like a good business to be in.

Navigating the byzantine contracting system is the only skill these companies have. They can't do the job, but they can fill out the forms, which is a rare skill.

You mean reading and following the instructions manual is hard work beyond the reach of a normal human being??

If you think it's easy to win government contracts then I invite you to do so. It can be lucrative.

This is why I wonder how much we need more and better coders versus better managers?

You need both.

The biggest issue I take with good coders lead by non-technical managers is they invariably convince managment they need to rewrite everything in the flavour of the month or architect in requirements so they can use technologies they're interested in.

Inevitably, these guys are now qualified with a whole new bunch of tools, so they move on and hopping about the industry writing a whole bunch of software that relies on a datastore that doesn't actually save data consistently, for example.

Could this project have been completed by the likes of Google?

I expect nothing less from the federal government.

Disclaimer: Highly political rant.

You mean to tell me a company that got selected a no bid contract from their connections in the Obama administration couldn't do an adequate job?

Crony politics at its best.

Reply on news.ycombinator.com