Healthcare.gov crashing because backend doomed in the requirements stage | Hacker News Reader

Healthcare.gov crashing because backend doomed in the requirements stage

forbes.com

45 pointsblurpin54 comments

> In my case, even the most expensive plan for our family in the State of Maine, where we live, is half the cost of what we are currently paying, exclusive of any subsidies.

The elephant in the room is that this is not the typical case - http://www.forbes.com/sites/theapothecary/2013/09/25/double-...

> The elephant in the room is that this is not the typical case - http://www.forbes.com/sites/theapothecary/2013/09/25/double-....

It still boggles my mind that people are surprised by this. Mathematically, including people with pre-existing conditions will mean either that those people are required to pay for the (expected, not actual) costs of their own care, or that other people will be required to pay for those costs[0]. This effect is much larger than the effect of mandating insurance for younger, healthier people who otherwise elected out of getting insurance[1].

Note: this is not a normative statement about whether this is a "good" outcome. This is just a a mathematical observation about the way insurance works, from a mathematical and economic perspective.

[0] Subsidies come from taxpayer money, so they have no net effect on this accounting - it changes how the money changes hands, but the money still has to come from somewhere.

[1] I'm waving my hands around the math here, but if people are interested I can dig up the more detailed blog post draft that I did a while back explaining this in full detail.

The effect that you're ignoring is that insurance premiums have a hefty amount of margin built into them, and the ACA produces a collective bargaining effect that gives insurers an incentive to offer a competitive price on the exchange.

> It still boggles my mind that people are surprised by this...

I was surprised because I was told that this new coverage affordability came from getting insurance companies greed in check.

I didn't know that insurance companies were actually lobying for Obamacare, not against it.

http://www.policymic.com/articles/62541/here-s-the-proof-oba...

Of course they are; this bill feeds them a guaranteed customer base. This is the largest issue I have with ObamaCare. The sole job of an insurance company is to assess risk. With a government mandated customer base there is no longer any risk to asses.

So... Why are the insurance companies involved at all? Why are they still in business? That's rhetorical, I know the answer, and I don't like it. We're all paying a middle man to do nothing for us.

> Of course they are; this bill feeds them a guaranteed customer base. This is the largest issue I have with ObamaCare. The sole job of an insurance company is to assess risk. With a government mandated customer base there is no longer any risk to asses.

I'm no fan of insurance companies, but this statement does not compute. Of course there's still risk involved - the risk that the patient's costs will exceed premiums. If you set the premium too high, the customer will flee to a competitor. You also can't raise your premiums too high, because you have the 80% medical loss ratio - you can no longer just eat the rest as profit [1].

Totally agree that government should be the single payer or even the sole provider of healthcare (though that model isn't as widely implemented (UK NHS, US Veterans Admin) and has it's challenges as well).

[1] http://www.cms.gov/CCIIO/Resources/Forms-Reports-and-Other-R...

Because Obama was unable to garner enough support for Single Payer... at least, not yet. While the ACA is hardly a perfect law, its quite possible that the greatest thing it accomplishes is showing Americans how much worse our current system is than the Single Payer systems the rest of the 1st world employs.

While the insurance companies were generally lobbying for the ACA (although there's definitely clauses they did not support), they are vehemently against Single Payer, since it puts them all out of business.

> It still boggles my mind that people are surprised by this.

I agree with your mathematical observation, but maybe part of the surprise that such math was often underplayed or outright ignored in the claims that premiums would be going down, ex. [0][1])

[0] http://www.washingtonpost.com/blogs/fact-checker/post/presid... [1] http://www.nytimes.com/2013/07/17/health/health-plan-cost-fo...

It still boggles my mind that people are surprised by this. Mathematically, ...

Its an interesting social phenomenon, that large-scale opaque repricing of a commodity is met with opportunistic ~scalping. This happened accross europe during the conversion to the Euro. US consumers are probably rightly suprised.

Despite the expectations of economists that the euro change over would have no effect on prices, European consumers perceived the opposite. To shed some light on this puzzle, we develop a model of imperfect information in which cheaper goods experience higher price growth after the changeover. Retailers, aware of consumers’ difficulties in adopting the new currency, use currency changeovers to increase profits by in creasing prices. The lower the price transparency after the changeover, the higher the euro–related inflation. Using data on inflation (Eurostat) and price levels (Economist Intelligence Unit), we show that although the euro changeover did not significantly increase inflation, it nevertheless had distortionary effect on prices. {etc}

http://www.kellogg.northwestern.edu/faculty/dziuda/personal/...

Just want to add another data point, that here in Washington State, a comprehensive plan for my family is going to be around $600 less per month under Obamacare than what I can get right now. This is without any subsidies of any kind, so while I know it is just 1 extra data point, I consider myself a "typical case" (3 dependents, 1 pre-existing condition).

I can say that the Manhattan institute's numbers (what the Forbes article is based on) for Washington appear to be incorrect (based on just my experience though), as they list a premium for a 40 year old man to be currently $181. I've been buying individual health care for myself and family for the last 8 years, and maybe back in 2006 it was about that price. A "comprehensive" plan in Washington would easily set me back ~$1300 per month for my family of 5 - under Obamacare the same (or even sightly better) plan would be around $750. I currently only carry a "catastrophic" coverage plan, and even with a $7500 deductible it's still $600 per month.

So for me I'm looking forward to Obamacare.

The problem here is that for some people's coverage to be lowered, impoverished people to be covered, and pre-existing conditions paid for, a bunch of young and healthy individuals (without families) are now required to sign up for insurance. And the cost of that insurance for those healthy individuals is now 2-4x more expensive than it would have been before Obamacare took place (obviously the money has to come from somewhere to balance it all out). *This is of course for the lower and mid level coverage plans, the higher end stuff reduces that margin between the pre and post levels - but not for everyone, and in some cases people are going to pay 15-20% of their TOTAL income for worse plans. And that's going to enrage those people.

Of course - that's the whole reason for the mandate. Similar to social security, there needs to be a pool of people paying into the system that are not using it. I don't agree with you that the plans will be worse though - maybe in some states, but certainly not here in Washington.

The Medical Loss Ratio part of the law also helps with that. Used to be that plans paid 75% - 80% of their premiums towards care. Now they're mandated to pay 85%, or 80% for individual plans.

The other thing to factor in is that most of the people that are experiencing premium increases are on plans that don't meet the "minimum level of care" as mandated by ACA. Those plans that no longer qualify are often the types of plans that lead to the horror stories of people being kicked off of plans after becoming sick (rescission), lifetime maximum amounts, etc. Insurance companies were saving a lot of money due to those brutal practices. Part of the intent of the ACA was to eradicate those practices, which ultimately protects the people that were subscribed to those plans, and who didn't understand they were at that type of risk.

In other words, if you were subscribing to a crap plan, your premiums were artificially low in the first place. It's kind of a degraded case of the people that are complaining that they can't any longer go insurance-free. The law of the land now is that everyone has to participate, in order to help take care of our sick and injured - so that means that our insurance plans all have to have a basic level of quality.

I would bet that if you restricted your search to the people that originally had plans that met that basic level of quality (which is really not all that high), you would find that their premiums have gone down.

(Obligatory data point: I'm self-employed in washington, have had a Regence BCBS plan for $540, and my monthly premium will go down about $150-$200 / month for similar care.)

But keep in mind they are using it, since the insurance reduces the risk they'll be saddled with high medical bills (or more likely that they'll saddle others with those bills). They are healthier, but can still have an accident of course.

> Just want to add another data point, that here in Washington State, a comprehensive plan for my family is going to be around $600 less per month under Obamacare than what I can get right now.

I get your point and I congratulate you. However, please don't be prematurely happy, "because $600 less" is not what will fix you -- the coverage itself will.

Let's hope that when you need that coverage, you won't be treated the same way as ObamaCare website right now is. You see, before you could complain to your health plan, make a big stink out of it, even sue them, because there were private entity. When to GOV takes over, all you can is pray. Go ahead make a big stink that their system is broken, but don't be surprised to get heavily audited, for example.

My point is that I would prefer to pay even $6000 more knowing I will get the quality of coverage I pay for. I would not want to pay less to be hit with bureaucracy, abuse, lack of transparency and fairness the way government will serve me like they did with their website.

With ACA you're still getting insurance from a private company and you can still sue them if you like.

Adding another anecdotal Washington state data point, my private individual health insurance plan (Regence Evolve HSA) with a premium of $149/month will no longer be available come January. The plan most similar to my current one is $252/month (Regence Direct Silver), which is a pretty massive increase. As a healthy, single 25-year old though, I knew some version of this was coming.

Are these two plans equivalent? What are the deductibles involved? What's your copay?

Some of this is because ACA prevents unreasonably high deductible, unless you get "catastrophic" coverage. It was quite possible that your plan was a "catastrophic" plan. You might be able to get a similar catastrophic plan via state exchanges, I know that it varies state to state.

Definitely have a catastrophic plan and the new options are essentially the same as well. Here are some more details:

Old plan: Regence Evolve HSA Plan - Premium: $149, Deductible: $3,500, Coinsurance: 20%, Out-of-pocket max: $5,000

New plan: Regence Direct Silver - Premium: $252, Deductible: $3,000, Coinsurance: 20%, Out-of-pocket max: $4,900

I would lose the HSA benefits from the new plan, which I could keep if I wanted to accept significantly worse terms (with a premium increase of only 33%!):

Regence Direct Bronze HSA - Premium: $198, Deductible: $5,000, Coinsurance: 30%, Out-of-pocket max: $6,250

I don't see how I can come out ahead on this one, once I was able to finally get quotes on the exchange, no other plan was comparable in terms of value and cost. :(

Citing Avik Roy does not help your claim. See http://www.washingtonpost.com/blogs/wonkblog/wp/2013/06/01/t...

If there is anything that can come of good of my taxes (a ridiculous amount of it in fact) is that we should educate people on the difficulties of software development. This would make a fantastic case study on what I presume are the issues at hand:

- Maintaining complex code (+1mil LOC)

- Classic "waterfall" approaches to software development (and the inherent problem with trying to blueprint requirements up front)

- The mythical man month[1]

- Change management

Funny thing is...these are all human problems.

[1] - http://en.wikipedia.org/wiki/The_Mythical_Man-Month

Thoughtback [https://thoughtback.com/] sent me a pertinent reminder this morning. No affiliation, just a great tool.

    Organizational metrics, which are not related to the code, can predict software
    failure-proneness with a precision and recall of 85 percent. This is a significantly
    higher precision than traditional metrics such as churn, complexity, or coverage 
    that have been used until now to predict failure-proneness. [1]

[1] The Influence of Organizational Structure On Software Quality: An Empirical Case Study

http://research.microsoft.com/apps/pubs/default.aspx?id=7053...

edit: formatting

"If you have four groups working on a compiler, you'll get a 4-pass compiler" -- esr

I know the people who worked on this. This is not what happened. Just yet another propaganda piece by Forbes.

Can you elaborate on what did happen?

Well something happened. You might want to tell those guys that 500 million lines of code is a bit ridiculous.

You know what's cool? 1 billion lines of code.

I wonder what those LoC were for? XML parsing remote API calls between various disparate systems? Sounds like it's well beyond operation system complexity. Linux kernel is currently 15M LoC.

Then again, kernels and operating systems are quite logical and laid out by architects. Systems like these are rats nests defined by bureaucrats and integration nightmares.

> the interface kept asking me to reenter the information for our dependents. Farther on in the process it doubled the number of our children (to 6!) Even though it listed our children’s names twice (identically) and even though these clones had identical SS numbers, the validation software was not smart enough to realize that the duplicate records were describing the same people!

This didn't happen? Are you suggesting he made this whole story up?

The article, as the headline states, is about poor requirements dooming the project.

Can they elaborate on what actually did happen? If I were your friends I would be trying to contact news agencies with corrections if at all possible because this is shaping up to be the highest profile government IT failure in US history.

Per the NYT: "But the government was so slow in issuing specifications that the firm did not start writing software code until this spring, according to people familiar with the process."

Source: http://www.nytimes.com/2013/10/13/us/politics/from-the-start... (October 12)

And from today (Oct 21): http://www.nytimes.com/2013/10/21/us/insurance-site-seen-nee...

In a previous life I worked on similar software development projects at the US State level. This project was several orders of magnitude larger in scope and in complexity than what I worked on. Assuming a generous start date for Spring - say March 1 - this boils down to ~7 months to design and develop to the Oct 1 hard deadline. Nearly impossible in my opinion.

I'm not discounting that there couldn't be issues else where in whole process. These would include: communication overhead between all the various agencies and contractors, integration nightmares, and not enough skilled folks at Medicare to manage all of this. But having the requirements delivered several months before roll-out seems to me a recipe for disaster.

You don't need 100% final requirements to start building the system. If nothing else start working though the interfaces with other systems, and try and get as close as possible to a working prototype. Granted funding is often an issue, but even a tiny team given a few months head start can get a lot done.

On government contracts that's actually a problem. You can't bill for any work you did without 'authorization'. Which means you can't do anything until a finalized spec, signed in triplicate, is physically delivered to you.

I have wondered for some time now if it is legal to develop tools and libraries that are relevant to an upcoming contract, but which are also general purpose and could be used elsewhere.

Totally legal.

You just won't get paid for it.

I don't know if I would use the word propaganda...but speculation definitely fits the bill.

I bet you half the problem is the software being specified in a synchronous and not asynchronous manner.

If a flow chart is expressed synchronously, it immediately bottlenecks around the slow backend systems. e.g. check the applicant has 4 kids, and if that checks out, present them with these options.

I attribute some of these problems to inadequate advances in today's mainstream programming languages.

For instance, if futures/promises were common parlance in programming, some of these bottlenecks would have been identified earlier.

I once read an interesting essay on what future programming languages could work on. One of the interesting ideas is a language that allows programmers to implement undo/redo easily. Recently there was a blog post about fuzzy computing, where a function is able to return approximate values when it is busy. The Knight Capital incident teaches us that if we architected our monitoring systems the same was as intrusion systems are done, we could have stopped activity when suspicious incidents occur outside the normal operating range. There was another incident where a Japanese trader accidentally keyed in the wrong number and promptly lost his employers a lot of money. Languages that incorporate ideas like validation and expected values as first class constructs might be able to help here. The Adriane reminded us how badly humans do when reasoning about systems where multiple agents perform the same task.

I can't wait to see what the report of it didn't work actually says. It's quite possible that one of the services that the website hit couldn't handle the traffic which in turn caused a bottleneck in other systems.

Wow, this particular excerpt from the article is stretching things a bit too thin:

"Clearly there are problems with the government’s IT procurement processes, but if the same level of dysfunction had been present in the software behind Obama’s political campaign (as it was, apparently, behind Romney’s) the President would have never been reelected."

Really? I doubt the campaign websites were the deciding factor.

Anyway, a lot of speculation in this article...not very insightful or informative.

The software in question was the software used to determine the best media buys, the volunteer organization software, and similar that was known to be part of his advantage.

See http://arstechnica.com/information-technology/2012/11/built-... and http://awsofa.info/ (https://news.ycombinator.com/item?id=5542368) as well as http://arstechnica.com/information-technology/2012/11/inside... (https://news.ycombinator.com/item?id=4764864) for comparison.

They aren't talking about the campaign websites. Both campaigns had complex internal software systems to help target field worker campaign activities, get out the vote efforts, monitor and respond to turnout data on election day, etc.

According to various reports, the Obama campaign was hugely successful in this endeavor [1] while the Romney campaign's election day voter turnout system was a disaster [2].

[1] http://www.drdobbs.com/architecture-and-design/software-deve...

[2] http://www.businessinsider.com/romney-project-orca-disaster-...

But that is a very agile, very incentive aligned project. You don't need to negotiate compromises and tip toe around various laws to run some stats for your media predictive analytics. Your vendors want you to continue in the campaign, so they still have contracts. Everyone is running for the same goal line.

Health care and insurance entitlement programs are clearly orders of magnitude more quadmired before a single line of code is written.

Why government software projects go wrong [http://www.mrobinson.ca/2012/09/why-government-software-proj...]

ibid - Part 2 [http://www.mrobinson.ca/2013/08/government-projects-going-wr...]

Could anyone share which platform / framework they used?

There probably isn't any one "platform/framework" they used. This beast touches too many systems, many of which are 'legacy' for the phrase "platform/framework" to really apply.

If it's anything like what I saw at the one (and only) government contractor I ever worked for, the site is a frankensteinian nightmare of COTS ('Commercial Off The Shelf') crap (which federal contracts encourage, when they don't outright require it) and sweatshop-style integration development run by contractors who hire anybody with a heartbeat and a handful of competent people, and pay them all peanuts while billing $200/hr in labor to the government.

I'm sure tons. The web site's code is pretty straight forward, it's that interacting with so many other parties that is not (besides anything having to do with actual insurance, Healthcare.gov interacts with the IRS, social security, homeland security, Equifax, etc etc).

I'm not sure what the solution is since these checks are not optional. It would be a huge political scandal if just one illegal immigrant was able to buy insurance using Healthcare.gov and there are surely many groups trying every day to make sure this happens.

Is there any large benefits system that works well? All that I have seen from MegaCorps are also awful and they don't even have all the complex stuff to make sure you qualify (since you work at the company, you qualify).

mnsure.org was throwing springframework exceptions the day it launched

healthcare.gov itself was a Ruby Jekyll static site. It was on GitHub[0] for a while, but mysteriously disappeared. Here[1] is a clone of it, and what seems to be the currently agreed upon canonical version.

My state appears to be using Joomla based on the meta-generator tag in it's head tag. http://www.avenueh.com/

[0] https://github.com/CMSgov/HealthCare.gov-Open-Source-Release

[1] https://github.com/blencorp/HealthCare.gov-Open-Source-Relea...

I didn't see anything that meant it was doomed from the requirements stage.

This was more of an Obama love fest and defense of ObamaCare than a technical article describing who the backend was doomed.

For what this cost to build it could have been much better, much simpler, and handled the scale gracefully.

Yes privacy concerns keep it off the elastic cloud. (not a point mentioned in the article)

Yes authenticated users limit caching and make scaling harder. (not really mentioned)

But when you are talking about everyone in the US, you only have so many combinations. Income in something like 32 ranges, number of kids 1 through 12 (greater than 12 they are all free) a bunch of ages.

We just saw a post that only 15 searches are new to google each day, so the number of combinations is much less than search which does lots of caching.

The requirements were messed up, the implementation was. Clearly not built by people who had ever done "Facebook scale" stuff.

Well, some reports are that the requirements were changing until 1 week before launch, and that development didn't start until March of this year due to requirements delays. Just those two factors alone would doom just about any project of this scale.

The primary contractor has had plenty of large gov projects including some for Medicare and SS. Of course so have all large consulting firms and they fail often as well. What the issue is isn't clear from random musings. Unless someone who worked on the project at a high enough level can explain all we have is guesswork.

They screwed the pooch, clearly. However I wouldn't assume that the insurance landscape is so simple. You're talking about an insurance industry and legal system that has had a fifty year head start making the process of insurance buying as complicated and obfuscated as possible.

Reply on news.ycombinator.com