HealthCare.gov’s glitches prompt Obama to call in expert troubleshooters
washingtonpost.com
washingtonpost.com
I will, at my own expense drive to D.C. and spend two days of my vacation evaluating the architecture of the failing system, and spot check the source code quality. Additionally, I'll add the project to my Sonar source code quality analyzer so that there's a focus on what code isn't up to industry standards and which sections should be addressed first.
It's still astounding to me that we spent $700M on such a pathetic LITTLE system ... there simply isn't that much code to write. If there are others who'd like to show the government how it should be done, I'd be happy to start an open-source project to rewrite the whole project, as well as a CI/CT/CD/CM system to make sure that the code is maintainable, that changes are delivered in a reasonable amount of time and that servers are constructed and clustered in a consistent way.
This really isn't rocket science (and it's a good thing ... there would have been deaths).
EDIT: If someone in the government a) reads my offer and b) is willing to let "the industry" voluntarily help, you can contact me via the e-mail address on my profile page.
The "Federally-facilitated Exchanges IT" contract awarded to CGI Federal Inc, which is the one that includes building the Healthcare.gov website among other services, was $55 million. The numbers are directly from the Government Accountability Office's report:
http://www.gao.gov/assets/660/655291.pdf
That's still a big number, but it's over an order of magnitude less than yours. It should also be noted that not all of this money is coming from taxpayers. The GAO estimates that in fiscal year 2014, $450 million in fees will be collected from the insurance companies participating in the exchanges, and this money is directly credited to the account used to manage the FFE program.
NBC News has released a report on what the actual numbers are. It's $196 Million as of 5/15/2013, and the price appears to be well on its way to $292 Million: http://www.nbcnews.com/video/nightly-news/53309356/
That was five months before the failed launch and the emergency overhaul that's now underway, so it looks like the final number could be even higher.
The ACA was upheld in the Supreme Court on the basis that it was a tax. Unlike other taxes, which are paid to the government, you will be paying the tax to an insurance provider. They, in turn, will be using the money to partially fund the exchanges. So technically you're correct, but it's just a redirection of tax dollars.
Both sides are choosing to use the exchanges rather than one of their alternatives, so the fees paid are user fees, not disguised taxes (unless you consider all user fees disguised taxes, which is a separate philosophical discussion).
I shouldn't have focused on any specific number as I'm really more interested in the huge failure in both the PM and execution of the software project. And I'd like to see contractors that fail miserably on one project removed from the list of companies that the government can use.
EDIT: I've seen a lot of companies create more complex software at a small fraction of that price ... and many of those companies exited well below $55M with smiles on their faces.
And I'm assuming said company billed by the hour? And of course, there's no way for the government to claw back the money?
However, the biggest problem I have heard of is simply stupid requirements from the government. Rather than have an open market where an individual insurance customer or any curious person could look at all of the products and prices, they make you verify your identity first. Why would they do this? They were afraid that people would get sticker shock from looking at insurance prices that didn't include the money they would be getting from taxpayers to defray the cost. This lack of trust in us has caused a lack of trust in their system.
It would seem that one thing that independent developers that want to help out could do would be to extract all of the useful data from the site. Gather all of the data about the plans and the pricing, and then make that available in a static searchable, filterable, browsable fashion without having to log in. Then, once people were able to do that, they could then go through the arduous identity verification process to qualify for a subsidy and get one of the plans. If it takes to long to crawl it out of the system, maybe you should file a FOIA request?
Once you get a basic information system up and running, and ad funded, you could then hook it up to connect you to the live site.
There are subsidy calculators out there that ask a few basic questions about income levels and you can get this information without requiring somebody's exact identity.
The cynic in me suggests that this system was specifically designed to get somebody's identity because at some point this information is going to get funneled directly to the IRS with the goal of going after and hassling people who still can't afford healthcare.
And don't forget that the "law" has changed literally every single week via policy changes from HHS.
I was a government contractor for years and found it miserable enough.. but our milestones and requirements weren't constantly changing. I feel sorry for some of these guys.
But even if that was the case, there is no excuse for not having an adequate architecture to at least let people register for an account.
It just seems like common sense to me that if someone gave me a 44K page rule book on how to build a site that serves millions of people, the first thing I would worry about is how to, at the bare minimum, get people through the door.
We can't keep letting them get away with the 44K pages of regulation bullshit excuse.
The code is broken, but not the broken part of this system.
b) The selected contractor won the work as payback for some political benefit, and the person who made sure they won can't allow them to be perceived as incompetent.
c) They don't believe it's really broken so much as it just needs to be "tuned" (almost "the myth of sunk costs").
d) No one in the government cares ... so they're not reading our thoughts and comments on the system.
Why don't they vet the results instead of vetting the people? Isn't that what matters? Not even the Soviets believed that the grain imports from the US were poisoned in order to kill them.
http://en.wikipedia.org/wiki/Antideficiency_Act http://www.gao.gov/legal/lawresources/antideficiencybackgrou...
It was created because of tricks and workarounds past administrations have used to get around the legislative appropriations process. One of its provisions prohibits federal employees from accepting voluntary services for the government. From what I understand, the idea was to prevent people from volunteering today with the expectation of pressuring Congress to pay them tomorrow.
And, to be clear, if it were legal I'd already be trying to find ways I could help.
If you were a Linux specialist, for instance and setup some system that ran solely on Linux, you would in effect be picking a platform and preventing vendors like Microsoft and IBM from competing.
Obviously, paying some government contractor bazillions of dollars doesn't necessarily result in a great outcome, but these rules are there to stop the casual corruption that impacts the faith of the public in government.
Because formal, institutionalized corruption doesn't breed lack of faith, I'm sure.
Something like this doesn't need more grandstanding, it's either good architecture with some bugs and scaling issues which are probably solvable or it's crap. Likewise is there some sort of jacked up insurance industry rules engine?
Unfortunately, you do not have an email in your profile. And your website is throwing a 503 error.
And thanks for pointing out the issue with my server (I'm in the process of replacing it!).
“One highly unusual decision, reached early in the project, proved critical: the Medicare and Medicaid agency assumed the role of project quarterback, responsible for making sure each separately designed database and piece of software worked with the others, instead of assigning that task to a lead contractor.
Some people intimately involved in the project seriously doubted that the agency had the in-house capability to handle such a mammoth technical task of software engineering while simultaneously supervising 55 contractors. An internal government progress report in September 2011 identified a lack of employees ‘to manage the multiple activities and contractors happening concurrently’ as a ‘major risk’ to the whole project” [1].
It is speculated that "facing intense opposition from congressional Republicans, the administration was in a bunker mentality as it built the enrollment system, one former administration official said. Officials feared that if they called on outsiders to help with the technical details of how to run a commerce website, those companies could be subpoenaed by Hill Republicans, the former aide said” [2].
[1] http://www.nytimes.com/2013/10/13/us/politics/from-the-start...
[2] http://www.politico.com/story/2013/10/obamacare-aca-website-...
> "Officials feared that if they called on outsiders to help with the technical details"
Aren't the 55 contractors outsiders? This sounds like typical DC finger pointing, blame game, and the-buck-doesn't-stop-here tactics.
As for having medicare/medicaid quarterback it.. honestly, that makes sense to me too. It's healthcare in the government, it involves deep integration with medicare/medicaid, if they're not capable of managing those contractors then they need to learn how.
I'm not defending the execution in general, but those 2 specific decisions seem ok to me. There weren't a lot of political problems building the site, so those decisions paid off -- they just needed to execute better on the technical stuff.
The Obama Care fiasco continues, almost like it was deliberately on purpose in a way.
A cynical person might think first contractor failed on purpose, so that the second bunch of "experts" could be brought in to fix it. One system for the price of two (or more, "expert troublshooters" don't come cheap). I'd really like to see a transparent accounting of who is getting paid, and how much, to implement this so-called website.
I've seen program managers getting pretty nervous when senior bureaucrats start breathing down their neck - I'd have to admire the balls on one who intentionally fucks up a project while being personally scrutinised by the President of the United States.
It's not evil, it's just incompetent and embarrassing.
And anyone who has worked in the industry over a long period of time will know that for big projects this closer to the norm than the exception.
I've personally worked on one massive banking project in the 90s that turned into a billion (in today’s terms) dollar right-off and in that time I read of many other such disasters.
Here's an alternate scenario:
1. The government agency that hired the contractors does not completely understand the business requirements in a way that allows them to effectively communicate them to the contractor. This could be due to numerous factors outside their control such as changing legislation or perhaps because the ACA is so large (900+ pages) and relatively new that there are few experts who understand the minute details sufficiently to specify how the exchange should work.
2. New requirements emerge at the last minute, requiring the contractor to make significant changes. While it is recognized that additional schedule time would be needed to sufficiently implement and test the required features, there is enormous pressure (political and otherwise) to adhere to the October 1 release date, so the implementation of the features is rushed and QA is skipped.
3. There are dependencies on legacy government and commercial systems that are outside the control of the contracting agency and the contractor. The functionality of these systems impact reliability and performance, introduce strange bugs, etc.
5. The software is delivered on time but is woefully inadequate.
While I wouldn't rule out the incompetence of the technical staff, these types of issues can often be traceable to problems that are more in the realm of management.
All states were supposed to have their own exchanges. 14 ended up doing it. New York after all sorts of political machinations agreed to run their own exchange at the 11th hour. (Supposedly "unprecedented" traffic also screwed up the NY exchange, which was also built by some big government contracting firm -- CSC)
I think like most IT projects you need to look at the management team in charge and the decisions they made to understand if this failure could have been identified and avoided.
Do big failures really hurt people or companies all that much? At some point, you're dealing in a realm in which relatively few people even play (or perhaps even want to play) and you may be the only game in town anyway.
Bankers/CEOs who lost millions were still given large bonuses with the reasoning that "we wouldn't be able to find anyone else to do their job". Well... I could certainly lose millions faster than many of them, and I'd do it for half their bonus, thereby saving the companies money :)
Some folks do get black balled[1], and companies will liquidate from time-to-time, but in the end, I really cannot think of anyone that didn't end up richer except the poor souls the companies hired. These days, most of them just switch to another company and continue the cycle.
1) if your name is know to a career, high-level administrator in a bad way, expect a lack of contracts
And since there wasn't much airtime, there won't be much pressure to get to the bottom of it.
But to play Government's advocate a little: That's also kind of how we got awesome things like GPS systems kinda fast.
1) which should make you read very closely some government findings when it was a pet project of some longtime admin or Senator.
0. http://developmentseed.org/blog/2013/06/25/healthcare-launch...
Maybe #56 is the charm.
I take no pleasure in watching this train wreck. It could, however, be a teaching moment for the country on what's wrong with Federal IT. Probably won't be, but it could be. So the story is worth following.
They're so moronic that they built the thing such that registration of the userbase would take more than a hundred years.
Stop coddling the state, they're not going to do it right, they don't welcome the debate, they don't care, they won't present a balanced budget, the debt default is constitutionally prohibited, the war on drugs is a war on people, their wars are so obviously wrong that there are more soldier suicides than deaths in combat, the military burns more oil than the entire population of india, socialism doesn't suddenly work when the current regime tries it.... Oh never mind, go ahead and keep wasting your time listening to sociopaths and participating in silly, futile politics; I'll just make a greasemonkey script to filter out all stories with "Obama" in them, and be on my merry way.
Because government. They get paid the same, by you and me, whether it works or not.
* Has to pay more for drugs (less bargaining power vs a big plan)
* Has more statistical risk (# people goes with N, stdev in # people with a given illness goes with sqrt(N), so risk/person goes with sqrt(N)/N and larger plans have less risk)
* Has larger legal/consulting overhead
* Forces doctors to incur overhead with nonstandard forms
* Is vulnerable to the bandwagon effect (if, by accident, it happens to favor X group of costly individuals, the costly individuals proceed to pile on and sink the plan)
* Has the ability to discriminate based on fine-grained geographical knowledge and other factors
* Has an under-leveraged customer base (if the plan is "secretly evil" and dumps people who need expensive treatment and 1% of people require expensive treatment, a 100 person plan will have 0 or maybe 1 person being screwed at any given time, while a 10000 person plan will have ~100 people being screwed who are much more able to band together and fight a legal/PR battle)
Previously, small plans were able to flourish because of #6 and #7, incurring #1-5 as costs of doing business. This was bad for everyone except insurance companies. The best way to turn a profit in the insurance industry is to sell insurance priced for a medium-risk person to a low-risk individual(or hi-risk price to a medium-risk person). If you look closely, this kind of "innovation" is actually just restoring the pre-insurance reality of charging sick people a lot and healthy people very little. It defeats the purpose of insurance while increasing costs (#1-5) and letting the people in charge of the purpose-defeating process pocket a hefty cut of the proceeds by fooling people into buying overpriced coverage.
Exchanges are designed to put an end to the tragedy of the commons. The savings from #1-5 can be recouped at the expense of eliminating #6 and #7, which were perverse incentives anyway.
Even if you disagree with my explanations, it is objectively true that employer plans are much more efficient than individual plans (E(payout)/E(cost) is larger). You can then see the ACA exchanges as an attempt to reform individual plans to emulate employer plans. Of course, it remains "cargo cult economics" unless you can figure out why you expect the emulation to close the gap in value, which is what I was doing with the list in the first place. In other words: I have provided 7 possible explanations that I think make sense but there are even more out there which I don't agree with and haven't listed.
I think there are a variety of reasons for that (e.g. keeping better tabs on the large amount of money spent on subsidies, easier administration of the subsidy tax credit), but the bottom line is the law says people only get the subsidies if they go through an exchange, so...
Let me be concrete. According to [1] (which gets its numbers from the CBO), individual plans have 29% administrative overhead as opposed to 12% for employer plans. If the exchanges successfully emulate employer plans, they save 17%*$300b = $51b per year in health care costs. Unless the exchange program is 10,000% more efficient than its alternatives (tax subsidy, payments to companies, etc) then the effects I listed dominate.
In any case, the exchanges were designed with the explicit intent of making the individual insurance market more efficient, not merely making subsidy disbursement more efficient.
http://www.americanprogressaction.org/issues/healthcare/repo...
Thats what keeps me up at night.
I can only imagine how these conversations go...
"We need more QA! STAT! It's going to take at least $50 million to test this system... it's awfully broken!" chuckle
face/palm
That said, I think there should be a lot more concern about the security of this site if there's been such a mishandling of the basic functions and operational flow.
Seems only reasonable to suspect this will be leading to a horrible leak of many people's private records at some near point in the future.
But after all getting reelected is more important than making sure your signature piece of legislation is actually working at launch.
If I know one thing, and I say this from firsthand experience, it's that Accidenture is absolutely the worst choice for any kind of contract.
Accidenture isn't an "outsourcing" company. They're an oversized roleplaying project, where the overlords (controllers) have the power and whim to kill projects and just pay the termination penalty, and where everybody strives to become a partner because it's up or out. Any problem inside Accidenture is seen as a nail, and every solution is a hammer, meaning a combination of ISS, MSSQL, Sharepoint, Windows Server and copious application of ridiculously overpriced "consultants".
One major problem slowing repairs, people close
to the program say, is that the Centers for Medicare
and Medicaid Services, the federal agency in charge
of the exchange, is responsible for making sure that
the separately designed databases and pieces of
software from 55 contractors work together.
More broadly (and apologies for the non-HN bent to this), the project serves as an example for the utter shitshow that is the United States healthcare system. The exchange is intended to serve as an abstraction for the process of figuring out what public and private options are available to you. The motivation for building the exchange was "Hey! Figuring out what options exist for healthcare is complicated!". Of course, building such a system doesn't make the complexity go away - it just makes it someone else's job. Perhaps that this determination is so complicated that it requires a ridiculous system is an indication that we're "doing it wrong".
Perfect for any tyranical statist type.
Does anyone really believe the whole website thingy wasn't full of graft, corruption, kickbacks and paybacks? Come on, seriously.
<script src='//assets.healthcare.gov/global/js/lib/jquery-1.8.2.js'></script>
https://www.healthcare.gov/css/all.css https://www.healthcare.gov/js/all.js
Their main problem is the hard deadline for October 1st launch. Things should have been rolled out state by state - but I am not sure whether this is in the law or just a political decision.
$95 Million apparently buys everything but new management and proper coders.
sigh
People are not idiots. Large problems are never easy. Politics, management, and salesmanship are genuine parts of the problem, not useless distractions from technical stuff.
I'm not saying that improvement is impossible, I'm saying that improvement is hard. The healthcare.gov system could have gone live, and been functional, on October 1 - if you could hire better technical people. Can you hire a top-notch team of techies in this environment? Ever tried it? It could be working if the political appointees had nailed down the requirements earlier. Have you ever tried getting hundreds of legally-mandated decisions pushed through the federal bureaucracy? Many of those decisions were pushed back for political reasons, which sucks - but could you made the unpopular decisions early, then won enough elections to keep Republicans from dismantling the entire thing? The system could be working if the law made more sense in a dozen different ways, but all of those compromises were necessary to get the law passed in the first place. Have you ever tried to write legislation, and get it through congress?
Don't get me wrong - the healthcare.gov system is a clusterfuck, and there are easily identifiable reasons for that - in retrospect. And hopefully we can learn from the failures and move forward. But laying it all at the feet of some nebulous government employee is completely unhelpful.