Poorly Managed HealthCare.gov Construction Cost $840M, Watchdog Finds
online.wsj.com
online.wsj.com
A couple of my colleagues have given talks on healthcare.gov from the perspective of the engineers who worked around the clock to help get the site running again: http://youtu.be/0albm_hhQzM?t=3m40s http://www.youtube.com/watch?v=GLQyj-kBRdo
A small team is also re-writing the site, starting with the most troublesome components for the next open enrollment period. Wired covered some of what is going on here: http://www.wired.com/2014/06/healthcare-gov-revamp/
If any of you out there want to help, please email jobs@hcgov.us!
A bigger challenge is ensuring people get the right coverage for their situation and understand the plan and provider network they're buying.
Disclaimer: I work at Stride Health, we help individuals buy plans in-context of their doctors, drugs, and health conditions they're treating. We're also hiring, sans the $840MM budget (https://stridehealth.com/jobs)
1. Tipsy Deloitte associate chatting at a Chicago bar
Why don't they contract someone to come up with a better solution?
Because:
1) There are lots of people ideologically motivated to minimize the "size of government" as measured by public payroll, and almost no one ideologically motivated in the opposite direction, and so contracting out helps to improve a metric that is important to a large number of stakeholders even if it is substantively meaningless, and
2) There's lots of money in firms that are specialized in taking government contracts, and they both lobby government and engage in public political advocacy effectively.
And, no, those two factors are not independent -- #2 contributes directly to #1.
(Also, outsourcing blame as adamnemecek points to in a sibling comment, can be a factor where the people who would potentially be at risk of blame actually have discretion of whether to do something internally or contract out, but that's often not the case.)
Also, most governments have made a feeble attempt to move toward a 'one size fits all' type of IT structure. This includes state run data centers, state-wide Active Directory structures, procurements services, buying from approved hardward and software vendors, yada, yada, yada. They didn't care that software from approved vendor A met 48% of my needs list whereas software from vendor B met 98% of my needs list at a fraction of the cost. Vendor A had been approved by procurement and vendor B hadn't. How are we to know your Cousin Billy doesn't own and run vendor B? How do we know you aren't receiving a kickback for buying from vendor B?
Which leads me to another ridiculous rule under State Government...I wasn't allowed to receive anything from a vendor (worth > $10) as a gift, if it wasn't also available to the general public. So say I am interested in buying a server for $4500 and I'm on the fence. The vendor wants to throw in 2 years of 7X24 technical support for 2 years if I buy the server from them. NOPE. If the general public can only buy the same server and get 5X8 for 1 year. I wasn't allowed to even (on the fly) negotiate a better deal.
EDIT: But not the initial comment about anti-corruption/anti-nepotism -- contracting out everything isn't a fix for nepotism/corruption -- indeed, statutory rules restricting contracting out (which, IME, state agencies are very good at working around when they aren't just ingoring them outright) are attempts to restrict nepotism/corruption. More often, the contracting out is because budget rules make it administratively simpler in practice, even if the theoretical conditions that should apply are harder, to contract out then to get additional state positions. (An example of how this can manifest is rules which require legislative action both to appropriate funds and to assign positions to an agency to perform that function, but where using appropriated funds to contract out doesn't require separate legislative action, even if it requires the agency to certify that the work cannot be done by workers in any civil service classification.)
But having said that, even with companies on price agreement, I wasn't allowed to take anything else that they might offer me if it wasn't also being given to the general population. A quick story: I received a really nice backpack from HP when I bought one of their very first "convertible" tablets (this would have been around 2008-09). When I purchased it, the site made no mention of the "extras" and I had to tell my boss that I received these without being aware that I was going to get them.
Unfortunately there are still a LOT of people in state government and elsewhere, whose eyes glaze over at the simple mention of IT. For them, it's a relief when you hear names like Oracle or IBM or Microsoft who win contracts. "Surely I've heard of Oracle. They're a big technology company and surely they'll know what they're doing". Relief ensues because NO REGULAR PEOPLE WANT THE BURDEN OF MANAGING AN IT PROJECT.
[1] http://it.wikipedia.org/wiki/Italia.it [2] http://www.marcosansalone.com/creativeantblog/?p=310
Healthcare.gov is at about 3 years, $840M, and I can't find a number for how many people were involved, but it's likely also in the thousands. Healthcare.gov has delivered life-altering value to about 6 million people and cost about $150 per user (soon amortizable over at least two years). Twitter has about 200 million active users getting everything from minor titillation to revolutionary aid and has cost about $6 per user. Neither of those "prices" seems shocking to me, and I'd pay $6 to use twitter and $150 to use Healthcare.gov.
If anything, healthcare.gov seems cheap in that context. In terms of scale of effort; if I took a cursory look at twitter's functionality, and even taking scale into account [1], I'd spitball it as taking maybe 3 or 4 small dev teams for the website and API, and another dev team per app; maybe 100 engineers total. I'm sure they have more than that though; because a cursory spitball guess is an insult and there's likely a lot of hidden complexity and business logic discovered only you if try to do something like twitter very well. I'm also sure there's a huge mass of employees dedicated to wet, human, problems too like sales acquisition and support and what direction the shading should be on the twitter eggs and why the logo should just be a bird.
And so it likely is with Healthcare.gov, interfacing with hundreds of different healthcare providers, handling different administrations and so on; probably a real deep mess. I'm impressed they got it working at all; especially as when they set out it was never intended to be used much (as I understand it the idea was that the state exchanges would handle the larger burden). The frontend seems to have been shoddy, but how much of the expense was that?
[1] I have built and operate services at the same scale as twitter.
My point is that you're comparing apples and oranges. Equating raw user numbers alone is not a fair comparison, because that's not what causes the engineering problems that make Twitter hard to scale. It's the usage patterns of those users that create the challenges.
Healthcare.gov is one step up from a static website, and customers rarely engage with it throughout their lifecycle. It's absurd to suggest that developing it should cost anywhere near the same as developing Twitter.
I may have a bias because of my background, but I absolutely don't agree. Twitter is a very busy service, a planet-scale multi-way messaging bus isn't trivial by any means, but that part is still a relatively well defined, small in scope, engineering problem and something a small team of talented engineers could tackle. That's probably the easiest part to account for.
Healthcare.gov on the other hand is gigantic in scope and faces many ill-defined problems; "integrate" with 50 states and various other administrations and insurance providers and so on to deal with, all of which are a moving target.
In my experience the cost of having things poorly defined is something like 100x having something well-defined, and increasing scope is a lot more costly than increasing scale.
Still, I get the sense that the government threw way more money and engineers at this than necessary. The fact that a few relatively small "repair teams" could join the project after development and fix all the problems, makes me wonder if those problems really needed to exist in the first place.
Perhaps it would have been better to group these 1000's of engineers into smaller groups of ~100, and have them compete with each other to implement specs in the fastest/best way. Pay each engineer a base salary, and then allocate the remaining money to a bonus system that incentivizes fast and quality development.
All that said, I'm talking out of my ass. :)
That's 3.8 trillion dollars, or 11900 USD per person per year (first figure divided by 300 million). Wikipedia lists the per capita expenditure as 8500 USD per person per year for 2011, not sure why there is such a large discrepancy.
Total amount of people on healthcare in the US is listed as 11.7 million jobs.
Something else I should have mentioned before; it is also very tempting to look at the outward functionality of something and not see the mistaken paths that led to nowhere, the research and measurements and experimentation it took to arrive at a working destination. That's another area where I'd guess the two sites have costs.
It looks like most people in HN have never actually worked on large systems and have no idea just how costs increase exponentially, how internal politics and policies affect those projects, and how infernally hard it is to think in terms of systems that have to be used by more than a handful of use cases.
There's not doubt that government projects have a huge amount of internal overhead, but if you compare them to the stuff that goes on in multinationals that have working systems in the 60s, it's really nothing out of the ordinary.
I know a guy who worked in a project related to one of the largest insurance companies in the US. It's a 60 year-old system running on IBM System Zs, AS/400s, Unix and Windows systems, with varying degrees of hardware and software obsolescence, running custom COBOL builds and compilers, integrating with thousands of external APIs, on half a dozen data centers with completely different specs. There was a literal library of documentation spanning millions of pages of requirements to describe how policies flowed through the systems, how batch processes should run, and what tests should be used to make sure a lady who took out an insurance policy on her TV in the 70s can still cash out on it now and withing 20 years in the future.
All this does not even include the network topologies spanning TCP/IP, custom frame relay protocols, proprietary NICs, SANs for storage and backup, tape libraries, electrical layouts for data centers, etc.
There's more than a few reasons why large software projects fail; I have not even begun to include the intricacies of the aforementioned industry and have not even counted the regular issues of any project.
Some might say that those do not amount to an acceptable excuse.
The cost only makes sense when the ultimate goal is worth more than getting 15m uninsureds onto cheap health plans. The ultimate goal might be arrogating more power over individual health to the federal government.
For the same insurance company I quoted, the people in question estimated that modernizing their system would cost $500 million USD in 1997 dollars and take 6 years without considering continuous upgrades of code and infrastructure. And it's likely that this company's infrastructure was probably much cleaner and uniform that goverment agencies'.
Then click the story.
That's $150 per registered user.
http://www.businessinsider.com/how-many-people-signed-up-for...
While we're on the subject of waste, the USA's ongoing experiment with market-based health care costs ~20 weeks of coverage per user per year compared to its social counterparts.
http://healthcarereform.procon.org/view.resource.php?resourc...
So what is the problem? The way I see it, there are two features that really distinguish the American system structurally. First, there's the state-by-state laws and regulations, resulting in effectively 51 different legal regimes. Second, American insurance is provided primarily through employers, rather than purchased individually. In fact, non-employer insurance is prohibitively expensive and often impossible to get.
I think the employer role is a much more likely cause of problems than the state laws, which are mostly consistent. Worse, tying insurance to employment traps people in jobs they hate - a distinctly un-American outcome.
So anyway, I used to think a more socialist system was the right answer too, until I looked more closely at the working systems in the world. It's not that one particular model works, because there are lots of models in use in other countries and they all work fine. It's that one model doesn't work - what we have in America.
I'd call them "social".
They're all two-tier systems. The basic tier is heavily regulated and meets my definition of social: government dictates terms, price, profit margins, and provides funding such that everyone is guaranteed to be able to afford it. It's no more private than your typical govrnment contractor. The supplementary tier is private in name and practice. I agree that this is the best arrangement and I suspect we mean the same thing and are just defining "social" differently.
http://www.commonwealthfund.org/~/media/Files/Publications/F...
http://en.wikipedia.org/wiki/Healthcare_in_Singapore
http://en.wikipedia.org/wiki/Healthcare_in_Switzerland
http://en.wikipedia.org/wiki/Health_care_in_Germany
> The way I see it, there are two features that really distinguish the American system structurally. First, there's the state-by-state laws and regulations, resulting in effectively 51 different legal regimes.
Agreed. This is a huge problem that needs to be fixed by regulating at least the bottom tier of health care provision at the federal level.
> Second, American insurance is provided primarily through employers
Employer-provided insurance does have a very strong purpose in the free market that adds value vs individual insurance: it dramatically decreases the risk of the bandwagon effect (people with need X pile on to policies that favor need X and sink them). Put differently, it chops off the long tail of P(claim rate in insured population deviates significantly from that of national population). Risk=money, so employer-provided care actually does decrease the cost of providing health insurance. It also increases the effective bargaining power of employees by allowing them to act as a bloc and making it economically viable to actually analyze and compare the nuts-and-bolts of a large number of alternative plans.
Of course, there are other ways to achieve the same thing, but they're all "social": the government can dictate terms for the bottom tier of care (so all insurance plans look the same to someone maximizing personal utility of an individual plan) or the government can outright provide the insurance (no choice = no bandwagon effect).
I agree that the freedom-reducing effects of employer-provided care are decidedly un-American, but I contend that they occur as a direct result of allowing insurance companies the freedom to set different terms in the basic tier of their individual plans.
So if the purpose in the free market is to add value, it's pretty much a dismal failure.
The thing is, though, that employer-based insurance wasn't there for some dream of free market efficiency. Rather, it's a legacy from mid-20th century labor negotiation. Back when insurance was largely nonexistent, labor unions negotiated for it as a benefit, and it became widely accepted practice. This, along with commie-phobia, kept America from adopting a more socialist model. Today, it's a political problem, because unions are actually against eliminating employer-based insurance, as it's a powerful negotiating tool for them. So it's hard for Democrats to attack it.
That conclusion would follow only if employer-based insurance were the only plausible explanation we had to explain the increased cost. Off the top of my head, here are a few alternatives:
1. Pharma lobby has neutered efforts to centralize bargaining for drug purchases for private insurance companies (what is the bargaining power of BCBS Wyoming vs the NHS? Nothing.)
2. Doctor's guild has created an artificial scarcity of doctors by squeezing the training pipeline.
3. Differences in entry-level health plans prevent price competition because the valuation process is too difficult for consumers (allowing insurance companies to differentiate based on ads and such).
> So if the purpose in the free market is to add value, it's pretty much a dismal failure.
No kidding. Sometimes it works miracles, sometimes it falls flat on its face. Here it has fallen flat on its face. Repeatedly. And we keep throwing money at it hoping it will fix itself :/
> employer-based insurance wasn't there for some dream of free market efficiency. Rather, it's a legacy from mid-20th century labor negotiation.
That's the narrative they taught it a health care economics class I took. I don't buy it. Say what you will about the market, but it does not hesitate to tear down economic structures that have outlasted their utility. In particular, if you think that individual insurance plans are more expensive due to cultural legacy rather than due to economics, I invite you to create your own insurance company and undercut the incumbents ;)
> Today, it's a political problem
Yup. The only real question is how long the circus will continue. ACA was a step in the right direction, but it doesn't go far enough. Time will tell. My fear is that stopgap solution is the enemy of the good solution.
So for a "normal" state, Switzerland actually has a bigger healthcare market. California, about 5 times the size of the Swiss healthcare market, still manages to screw things up royally though.
People are often blaming the US legal rules. In Switzerland, if you want to win a court case that a MD acted improperly (justifying damages), you have to get a jury of his peers (ie. MDs) to agree on that (which is not impossible, but ...). In the US, you have to get a jury of patients to agree on that. Needless to say, there are many, many more medical cases in the US, with spectacular damage payments (damage payments are also standardized in the EU. If an MD kills you through negligence or outright a mistake (not satisfying the level of a criminal act), his employer (even though they're mostly freelancers, this is an exception in the law) owes your family 300k euro (inflation adjusted since 19xx, so it's a bit more). In the US, it's not fixed, judges decide, but it seems that number runs into the tens of millions of dollars.
edit: Or Nimitz class nuclear aircraft carriers for $4.5 billion each, with 10 of them built.
My opinion based on the fact that nothing is free, and people who currently don't have insurance, but still require medical attention, still cost the rest of us money. Also, since people without medical insurance tends to go to emergency room only in critical cases - problems they had could be much cheaper if they had insurance from the beginning.
Anecdotally, I know such people :)
http://blogs.marketwatch.com/health-exchange/2014/07/30/gao-...
"Confusing and constantly changing marching orders from Centers for Medicare and Medicaid Services managers will get the blame for last year’s HealthCare.gov debacle under an independent government review, the Associated Press said Wednesday."
You've provided no evidence of this. Including that article.
I am just disputing your assertion that this is comparable to your typical Silicon Valley web company who rarely have to deal with legacy systems. Not whether it was a well run project.
On top of that, consider that the system needs to serve populations that are, shall we say, less sophisticated in the ways of technology and finance than the average startup's customer base? When I worked on a state system, we called it the "Amish problem", because our state has a significant Amish population.
There are people who don't read. People who don't speak English. People who are absolutely terrified of authorities. People who have religious beliefs that isolate them from computers. It's difficult and ugly.
FWIW the current enrollment is 6mil so if you assume a very poor conversion rate of 1% then that is 600 million page views over the signup period and that comes down to about 38 views per second. At peak load that would only be like 1k concurrent views ( assuming a poisson distribution )
Have you ever worked on these projects before ? It DOES require technical expertise.
Working with legacy and enterprise systems requires knowledge that most developers think they could handle until they are actually in the weeds trying to understand some arcane domain model or file format. Combine that with the integration and scaling challenges and it is quite a technical problem.
Is it expensive sure. But to dismiss these projects without any visibility or understanding of the challenge requires some arrogance.
Yes documentation is always lacking and there is some reverse engineering that needs to be done, and yes there will always be some stupid wrinkle in the process, but this is a well understood problem and claiming its a challenge due to "Legacy Systems" is just a way of justifying the over 100% overhead that the government runs.
Legacy integration is huge and horrendously complex task. The coding is the least of it.
I can absolutely guarantee you that they do not have the utmost of security. For a project to be really secure, you need extremely talented and meticulous developers. Judging by the clusterfuck that was Healthcare.gov's original launch, that does not describe the people involved. For example, this sentence (lifted from Wikipedia) raises a million red flags in my mind re. data consistency checking and sanitation: "Even for those that did manage to enroll, insurance providers later reported some instances of applications submitted through the site with required information missing.". No project like that can feasibly have the "utmost of security".
this probably was the source of the initial failure.
Getting that to work every time with all failure paths handled correctly is definitely architecturally challenging.
You're right. Good thing they completely failed to handle all failure paths correctly. This is extensively documented at http://en.wikipedia.org/wiki/HealthCare.gov#Launch_and_techn...
I am just disputing anyone who thinks projects like this aren't technically challenging.
On the side, I'd first hand experience with healthcare.gov to help someone else. I can imagine myriads of integration points between insurance companies, keeping track of their plans, state level requirements, credit checks, other govt agencies like social security, customer service backend etc. From my experience in working with this kind of complexities I think 200-300 engineers may be more than enough for 2 year execution plan. So still it's about 3X-4X waste when government gets involved.
PS: Per head cost needs to include base salary + bonuses + stock grants + hiring fees + employee events + office expenses + subsidized cafeteria + health benefits + 401K etc etc.