Comment by GVIrish | Hacker News Reader

Comment by GVIrish | Hacker News Reader

Parent Full thread

I don't see how giving 20 startups $15 million each would have led to success instead of failure. Even if you found a better company to do the implementation than CGI Federal, there were several enormous problems that no company would have had any control over.

1. Requirements were delayed so much that development didn't start until March of this year. That spells doom for a system with this kind of complexity, regardless of who the implementor is. You need several months of functional, load, and integration testing so that effectively means you would only have had 4 to 5 months to code healthcare.gov. And that's assuming there weren't any big requirements changes.

2. The people responsible for integration (Center for Medicare/Medicaid Services) had no large IT project integration experience. These are people that thought 1 week of full integration testing would be enough.

3. Healthcare.gov had to integrate with legacy systems from the IRS, Medicare/Medicaid, Social Security, in addition to the various state exchanges. Any number of those systems could have serious flaws that would make it extremely difficult to interface with. On any project a poorly implemented legacy system can dramatically affect the effort needed to be successful. Again, even the best companies would've had a sizable challenge dealing with that.

4. One of the biggest challenges in government IT is the customer. The government decision makers often don't know enough about software engineering to make sensible decisions on requirements, timelines, testing, you name it. In this case there was the added political pressure of, "This cannot fail," even though it should've been clear at least a year ago that there's no way they were going to make the deadline. But you get people who think that you can just deploy and fix it as you go along. Or you get people that think you can just add developers to make up for lost time.

5. And what does being a "startup" have to do with it anyway? Either a software company can do the work, or they can't. Whether they are a startup or an established entity really has nothing to do with it.

It looks like you and jroseattle are demonstrating the main two competing views about how to approach new software projects: 1) deterministic, controlled, strong planning, avoidance of failure vs 2) nondeterministic, flexible, embracing uncertainty, expect and handle failure gracefully. There are many different versions of this in technology: one big expensive server vs many commodity servers, waterfall vs agile, ACID vs BASE, BigCorp in-house R&D vs distributing risk across startups.

Large parts of the technology world operate according to the latter model, and they do so for a variety of very valid reasons. Obviously, the government and government contractors do not.

jroseattle's comment presents a speculative model for how to apply a distributed, fault-tolerant model to this kind of technology project.

I'm not saying that you can't or shouldn't distribute risk across many small projects, I'm saying that in this case doing so would not have helped you at all. If you had a competition to build a MVP of healthcare.gov where the requirements didn't get delivered until 48 hours before it was due, then the requirements changed substantially 30 minutes before the cutoff time, none of the teams would have succeeded.

The distribution among startups is about spreading risk for the project. The existing approach was one integrator, and they failed. Because there are so many reasons that are provided for why this had no chance of success, i.e. "Even if you found a better company to do the implementation", it makes strategic sense to me to try multiple approaches.

But I would contend the problems weren't unmanageable. Problematic, yes, but manageable. I will try to address your points:

1. Requirements delay - sure, specifics were delayed. But, there was a basic premise for what this system would entail, and core work could progress without incurring much risk or waste. (I'm unsure when project work by CGI was started versus development work.)

2. IT integration experience - this is dependency management 101. If this is a recognized risk, then CGI should have raised the red flag. This looks like very poor project management.

3. Dependencies - yes, there is a possibility that external dependencies can affect your system. You design and architect for these scenarios, including failure.

4. Technical know-how - I would not expect anyone in government to know one iota about software development. CGI's job, in this scenario, is to elicit those requirements in a way that's useful.

5. Good point, but this project would benefit from some competition. Why not let more teams try this, rather than wrapping it all up with an entity who's only proven competency is simply receiving more contracts?

The integrator in this case was the government (CMS is a division of HHS), there's no getting around them as a single point of failure.

1. Having a basic idea of what the system will do is helpful, but in a system that has a lot of very specific business logic that has to be defined and HAS to be right in order to comply with the law, you're not going to get very far without good requirements. Furthermore, depending on how the contract was structured, CGI Federal may not have been allowed to start development work until it was authorized by the customer.

2. CGI may very well have raised multiple red flags, and CMS and HHS could have promptly ignored them. That is one of the maddening things about government IT. You can tell the decision makers that they are heading for disaster and they can simply ignore you.

3. Let's say you plan for every one of your dependencies to fail. What useful product could you deliver in that case? If you don't get certain information you cannot give the user a health insurance product according to the law. This is system where you cannot reasonable ignore a component if you can't get it to work in time. Let's say the interface with the IRS is going to take longer than you thought. You can't just cut that out or substitute dummy data. That would amount to failure to deliver a functioning product.

4. Again, you can elicit as hard as you want, if they don't give you the requirements because the law is still up in the air, or because the president might change his mind, or because someone wants to meet with all the stakeholders in the other agencies first, there is absolutely nothing you can do. I've been on projects where the stakeholder with the final say couldn't be bothered to give their input until the software was done. I've also been on projects where the decision maker prevented us from getting the input of the employees that would actually be using the application. Sometimes the people you're dealing with are not rational actors.

5. Fixing the contract award process is certainly part of the solution. Normally many teams could submit a bid for the work, but due to the insane time scale it was skipped for healthcare.gov. I think having a proof of concept competition after some sort of initial proposal selection gate could be useful. It could sort of be like how the military has contractors build prototypes for weapons systems competitions.

Comments added inline here.

> The integrator in this case was the government (CMS is a division of HHS), there's no getting around them as a single point of failure.

This makes it easy to identify where the roadblocks are, and to present that information to interested parties at the appropriate points in time.

> 1. Having a basic idea of what the system will do is helpful, but in a system that has a lot of very specific business logic that has to be defined and HAS to be right in order to comply with the law, you're not going to get very far without good requirements.

Without requirements, how would one expect very specific business logic? And what is so incredibly precise that it derails any possible planning for it? I'm not familiar with that type of problem.

> 2. CGI may very well have raised multiple red flags, and CMS and HHS could have promptly ignored them. That is one of the maddening things about government IT. You can tell the decision makers that they are heading for disaster and they can simply ignore you.

This is very true. Of course, with basic project management 101, these things are duly noted so that when you're hauled in front of Congress, you've documented the situation. Based on CGI's responses the other day, they surely didn't do that -- otherwise, they would have raised it. Unless government IT work of this nature is classified?

> 3. Let's say you plan for every one of your dependencies to fail. What useful product could you deliver in that case?

One that allows you to fix those problems independently, rather than tying the success of the entire project to the success of all integration. In a project where the requirements are constantly shifting, I'd say the "useful product" definition is dynamic and iterative. A strict all-or-nothing definition makes little sense in that environment.

> 4. Again, you can elicit as hard as you want, if they don't give you the requirements because the law is still up in the air, or because the president might change his mind, or because someone wants to meet with all the stakeholders in the other agencies first, there is absolutely nothing you can do. I've been on projects where the stakeholder with the final say couldn't be bothered to give their input until the software was done. I've also been on projects where the decision maker prevented us from getting the input of the employees that would actually be using the application. Sometimes the people you're dealing with are not rational actors.

I get it; I've worked with federal agencies as well. It all goes back to project management and documenting interactions. As for logistics, there is always someone else to go to when a stakeholder is a bottleneck. Here's something that I found that worked in my experience with some fed agencies: identify those who are considered a project dependency, and that project progress relies on specific people. When facts are presented simply, and project management details are documented...individuals get motivated to NOT see their name in lights. As always, your mileage may vary.

> 5. Fixing the contract award process is certainly part of the solution. Normally many teams could submit a bid for the work, but due to the insane time scale it was skipped for healthcare.gov. I think having a proof of concept competition after some sort of initial proposal selection gate could be useful. It could sort of be like how the military has contractors build prototypes for weapons systems competitions.

Absolutely. Basically, it's an iterative phased approach, and let multiple teams go for it.

"> 2. CGI may very well have raised multiple red flags, and CMS and HHS could have promptly ignored them.

Of course, with basic project management 101, these things are duly noted so that when you're hauled in front of Congress, you've documented the situation. Based on CGI's responses the other day, they surely didn't do that -- otherwise, they would have raised it. Unless government IT work of this nature is classified?"

Depends on the stakes. At the moment, they're kind of low, I doubt sufficient to piss of your monopsony customer. I mean, when CGI's VP was handed this straight line (http://www.washingtonpost.com/politics/house-panel-grills-co...):

"Later, Rep. Leonard Lance (R-N.J.) asked the contractors whether they could conceive of “a more incompetent administrator” than CMS."

And replied:

"“I have no opinion on that,” Campbell replied."

Yeah, I'll bet she has no opinion on the clowns that put her front and center before the nation's Grand Inquisitor (the US Congress).

What it boils down to for me is that the best contractor in the world can't make up for the worst government decision makers. After a project goes down in flames the contractor can point to specific failings at the government level, but at that point it's too late, you've already wasted a lot of money and you don't have a working system.

In my opinion, multiple facets of government IT have to be fixed:

A. Government leads need training on how to successfully manage IT projects.

In practice this is a huge undertaking because there are a lot of organizational challenges that have to be tackled before a lot of IT projects have a chance of success. Good managers in government will also have to tackle dysfunctional bureaucracy in order to be successful

B. Government employees need to be held accountable for their failures.

C. There needs to be more full time IT professionals in government in positions of leadership

D. Hard deadlines should not be determined politically before it has even been determined if that the deadlines is achievable

E. The contract award process needs to be reformed.

Reply on news.ycombinator.com