Inside the race to rescue a health site, and Obama
nytimes.com
nytimes.com
"Software that assigned identities to enrollees and ensured that they saw only their own personal data, known internally as the EIdM, was being quickly overwhelmed. Customers could not log in to create accounts."
"Experts disagree on what went wrong. But several said that errors in the software code written to stitch the Oracle product into the online system and improperly configured hardware trapped users in endless technological loops. It would take eight days to resolve just that one bottleneck."
Fool me once, blame the contractor; fool me twice, blame the management.
Towards the end of October they got "fired" from this role and replaced with QSSI ... who are responsible for the "data hub" and this Oracle EIDM system (which is part of the data hub? Development has been so politicized that they refused to publish architectural diagrams, not even supplying them to the state exchanges).
Government workers have interesting employment metrics. They are, for lack of a more profound term, extremely risk averse. If you, as a contractor, propose a new solution to an existing problem, you are looked at like a watch practitioner, or perhaps a voodoo doctor. All claims are treated with a level of skepticism that can only be described as paranoia and sadly, given the general trend on government projects, that isn't necessarily unwarranted.
Building a website to service a total user base of 30,000 users is, for most of us I'm sure, pretty easy. That's not exactly a difficult scaling target to hit, and could easily be done on commodity hardware. However, in implementing something similar, I was forced to order 4 Sun Fire boxes, at a cost of over $10k each, to mitigate concerns of speed.
Certificate based SSL is also very easy, but we were coerced into purchasing a dedicated SSL module to offload SSL handling to hardware, to mitigate concerns of security, which were then compounded by anecdotal experience that "ssl is slow". So, not only did we have to implement SSL-only for a site with absolutely no public facing endpoint, we had to do so with hardware crypto, that we had to hustle to learn at the last minute -- all reasonable sounding objectives, but compounded with the notion that the contract was a firm-fixed price, bid with different hardware and software SSL, and you can see where the project was trending towards either failure, or a lawsuit after an overrun.
The phases of a government contract too, are curious. It starts with skepticism, in which everybody wants the project to die before implementation because its public failure would make everybody look bad, then, if it's somehow able to assuage enough fears to get a go-ahead, you're generally assigned a COTR (Contracts Officer) that ostensibly wants the project to succeed, but would happily wager a failure before implementation than a bigger failure post-implementation.
Assuming you can get to the point of actually convincing somebody that the project might actually succeed, and then it might actually deliver good results, the exact opposite happens -- EVERYBODY wants to become a stakeholder, so that they can have their names attached to the success. This has its own faults, because now everybody wants to put their stamp on the project -- which means week-long meetings with various groups of stakeholders all arguing over what image should be used in the page header -- I've had day-long meetings in which the only topic of discussion was "In the picture of the building, the summer photos have too many leaves on the trees, and we can't see the building, but in the winter photos, there are too few leaves, making the scenery look dead... how do we fix this?"
Assuming your project hasn't failed yet, you get to a point where now, all the other contractors in the building hate you, and want you to fail. In a normal business, that would never be an issue -- but in the government, the DNS changes are handled by company A, while the active directory is managed by company B. You're sitting there, company C that you are, requesting changes from company A and company B -- both of which will happily obey the exact letter of every request, so that they can indicate full compliance, but which are happy to interpret those letters in such a way that set you up for failure, so that if the project is at this point, you've gotten to the level where the idea is sold, and failure can be blamed on the contractor -- YOU! This means that if it fails, Company A and Company B will swoop in with the exact same idea, only they'll offer their superior experience and intimate knowledge of the project, its failures, and the existing infrastructure to why the project will succeed in their hands.
This should be a blog post -- I'll shut up now.
The rest of it sounds like they botched making this system scalable, to handle both the number of possible users (every adult in the US) and the rate at which they'd be signing up. "Endless technological loops" might refer to either a circular reference in an overly-complex schema, or to normal dependency checking in the schema which takes too long to evaluate within the database when the database is under enormous load.
>The rest of it sounds like they botched making this system scalable
these systems just don't scale. In a BigCo with local replication to regional sites it takes noticeable time for such systems to perform. Trying to build such a website - well, it is a government procurement, though i don't think this is a worse one around (Halliburton's and the likes' cost+ in Iraq/Afganistan definitely bigger :).
>normally a single table with fields like
It is "normal" from another world :) The simple way you described would be done by some startup, and they would horizontally partition it when time come to scale. It is not the way the "Enterprise" things are done.
> "Software that assigned identities to enrollees and ensured that they saw only their own personal data ..."
I guess the simplest EIDM is a where clause in your SQL query :P Just kidding - I do wonder what this thing does more than just a regular authentication / authorization system though. Maybe sandboxes data so that it's impossible to even query for data that doesn't belong to you?
Actually, Enterprise systems are my bread and butter. My current company's product is a base framework and code generation system for producing Enterprise-class systems, and among the many features we include is identity and access management. It uses a few tables, as I described. We also integrate with IBMi authentication and Microsoft Active Directory, neither of which are logically more complex than I've described. At my previous employer we designed and built an authentication system that was a lot more complex, with federated authentication managed by dozens of systems that were outside of our control. That took a lot longer to design and build (most of a year for a two-person team) but once deployed it could process a login in a couple of seconds at most, even with the authentication system halfway around the world from the web application's server. And, it could scale to many more federated systems than a few dozen; that's just the number of countries that were using it.
http://www.oracle.com/technetwork/middleware/id-mgmt/overvie...
...
Services available from EIDM have been grouped into four main services areas (registration service, authorization service, ID lifecycle management service, and access management service). [1]
Optum/QSSI used the EIDM to make accounts. Their part of the system crashed, and they blamed it on the feds, for making people register to compare plans.[2]
This reddit post describes the infinite loop: i'm now stuck in some infinite loop where it can't identify me, asks me more questions, verify fails, repeat. [3]
1 - http://www.civicagency.org/2013/10/learning-from-the-healthc...
2 - http://www.inforisktoday.com/obamacare-website-security-ques...
3 - http://www.reddit.com/r/AskReddit/comments/1oq0ds/has_anyone...
Some of the companies building the system opposed an early decision by the Medicare agency to use database software from a company called MarkLogic, which handles data differently from systems by companies like IBM and Oracle. Some suggest that its unfamiliar nature slowed their work. By mid-November, more than six weeks after the rollout, the MarkLogic database -- essentially the website's virtual filing cabinet and index -- continued to perform below expectations, according to one person who works in the command center.
The company's response is recorded as:
In interviews, MarkLogic’s executives faulted inadequate computing power and instability at the site’s data center, as well as the failure to properly integrate their product, problems repeatedly cited by other website vendors.
Any insight on what is meant by how MarkLogic "handles data differently"?
"Schemaless" aka "Schema on read" is the biggest difference between Only "SQL" and "Not Only SQL" systems. Scale-out on commodity hw vs. scale-up on shared storage is another. Customers often choose MarkLogic instead of other NoSQL systems because of it's ACID capabilities, ability to run multi-statement transactions between multiple databases (including Oracle and MarkLogic in the same transaction) and a multitude of enterprise capabilities from government grade security to point-in-time restore.
Read more: http://developer.marklogic.com/pubs/architecture/inside-mark...
Put any database on top of the resources that were provided and I think performance "below expectations" would have been likely.
I haven't worked with it directly but at several places it has a reputation for high memory requirements and requiring you to scale by getting single massive servers rather than clustering – I'm not sure if that's still true or if it's an issue for healthcare.gov but it could definitely make it harder to scale in a hurry.
Why? Scaling by requiring massive servers is Oracle's story, not MarkLogic. We scale out on commodity servers in a cluster just fine. However, we do not magically create more capacity when adding additional VM's on top of over-subscribed network and storage systems.
We don't require proprietary shared storage systems (local, NFS, HDFS - just give us bandwidth) or proprietary networks (TCPIP please, 10G preferred, don't need FC).
Cheapest way to scale is a cluster of decent sized machines with a decent amount of IO capacity each.
{Q: What do you call a database without indexes, transactions or security? A: A file system!}
Not learning it well enough would likely go along with not provisioning the necessary resources.
Usual lesson of "don't try to do too many new things if you're on a tight time and/or money budget".
Unfortunately, given that this is almost entirely a political exercise, I don't see how you're going to avoid getting some scapegoating.
"...the root causes for these site flaws to be hundreds of software bugs, insufficient hardware and infrastructure."
The infrastructure flaws cannot be chalked up to "not learning it well enough"....There is no database - not Oracle, not PostgreSQL, not MongoDB....and not MarkLogic - that would have handled the traffic on this hardware and infrastructure.
That's what I meant by "not provisioning the necessary resources". I'm a programmer who dabbles in small scale systems building (from scratch, as in mount CPU on motherboard, etc.), so I'm perhaps not using the word "provisioning" as domain experts do.
But to the extent MarkLogic was the major database used (I'm getting that impression the more I look into this), unfamiliarity plus all the bad management could have contributed to not procuring beefy enough infrastructure. Or helped contribute to the widespread magical thinking, i.e. an experienced Oracle DBA could say with authority "this won't work" but not have as much weight when talking about MarkLogic. And I'm sure you had at least one field engineer helping them ... or I should say trying to help them. Ditto, I'm sure, the people or contractors in/for CMS who were already using MarkLogic, assuming the ones who really knew their stuff were even consulted.
Engineers not being listened to/respected in favor of magical thinking is obviously one of the biggest problems with this project. What can you say when the integration testing is delayed for the last 2 weeks before launch, proves it can't work the week before, and launches anyway?
So far, there have been two contractors that have been changed out - QSSI took over from CGI to lead rebuild and, recently, Terramark to be replaced by HP. Maybe this has absolutely nothing to do with not being familiar with MarkLogic.
http://kellblog.com/2013/12/01/the-pillorying-of-marklogic-w...
There was a healthcare exchange built 100% on Oracle in Oregon (Oracle team, Oracle packaged software including Siebel, Peoplesoft, IDM, Oracle integration SW + people,Oracle infrastructure, Oracle hosting). It's not going particularly well.
I don't think familiarity of technology has one thing to do with it.
I do think that MarkLogic's ability to be agile - programming (EasyApp), infrastructure (speed of transition) and performance - have a great deal to do with the speed of the team being able to deal with the software bugs above MarkLogic and the weak infrastructure around us.
(For those joining this conversation in progress, I'm a product manager at MarkLogic - I'm in charge of infrastructure like storage, performance monitoring and cloud platforms.)
The latter could be related to MarkLogic; I don't know about Terramark, but I hope HP isn't in this business except to provide at minimum full hardware solutions, not just a co-lo's power, cooling, connectivity and so on. I.e. better able to provide today's "big iron". Even if the current database instance has been beefed up by 12 dedicated servers, I've seen complaints the storage system is not up to snuff, and I know that's an HP focus (in fact, I'm doing a monthly full backup of my home systems to an HP LTO-4 tape drive right now...).
Or maybe Terramark isn't in a position to provide a backup site, something neglected to date....
My Google Fu isn't up to finding the technical firewall issue, do you have a pointer handy? It did find mention of the Administration freaking out when insurer giant United Healthcare bought QSSI before the election....
Your former CEO has the usual good things to say by anyone with a clue; I just hope the meme doesn't build up that MarkLogic was at fault. Especially since it was forced on reluctant contractors, that's enough to get a lot of the learned one thing and not about to learn another types to bad mouth it, apparently from a position of technical authority. Then again, how many of these technical people are in reporters' Rolodexes?
"I do think that MarkLogic's ability to be agile - programming (EasyApp), infrastructure (speed of transition) and performance - have a great deal to do with the speed of the team being able to deal with the software bugs above MarkLogic and the weak infrastructure around us."
Hmmm, I'll be looking for evidence of that in the serious postmortems that'll be coming out over the next decade or so, assuming it continues to be a nightmare if the backend connections to insurers fail to deliver quality data, as various of these articles are suggesting. Not to mention the previously mentioned not even working on the software to pay them, a very complicated thing....
Good luck.
400 bug fixes ~= 200 new bugs inserted? (Well, at least initially.)
Page 5, hardware upgrades: "Deployed 12 large, dedicated servers; upgraded storage unit", with a resulting ">3x Database Throughput"
Don't like reading that, although maybe it's not the critical path slowdown now. "upgraded storage unit" sounds like they're using a SAN, NAS, what have you, not the ideal recommended cluster architecture you described. I wonder if they've got enough IOPs....
Ah, reading this: http://online.wsj.com/news/articles/SB1000142405270230333290... confirms what we've been hearing about it running on VMs on machines shared with other Medicare services, vs. the dedicated hardware now.
System stability is bad at 95.1% exclusive of scheduled maintenance. Per this table https://en.wikipedia.org/wiki/High_availability#Percentage_c... that's a bit over 8 hours a week of unplanned downtime.
Hmmm, maybe not a total disaster now. And the fix-it czar said his highest priority was (correctly) to stop sending garbage to insurance companies. We'll see ... especially since they evidently haven't started working on the really hard problem of paying the companies....
Sorry, I wasn't trying to say that was a negative – just different in ways which many developers don't expect. With similar systems (e.g. zodb, MongoDV) I've seen developers use ORM-like patterns where they store properties in separate records or do reports by walking millions of records and then complain about performance rather than using it wrong.
As for memory, that was awhile back so it might have been an old version or poor usage. I just heard about trying to max out server RAM as a key requirement.
Mmmm... I'll take ZFS over MongoDB any day of the week. :)
My understanding is that CGI was already handling various health care related work and was selected to do the work without a proper competitive bidding process. But having seen these processes, they definitely wouldn't have resolved the kinds of widescale technical issues.
Even if there was a highly qualified company to do the work, the FAR requires it be divided up. Top that off with a silly idea that a company needs to have past performance doing something that's never been done before, and well you end up with selection committees picking the "safe" (I.e. Big firm) choice. And you're guaranteed to get a firm that doesn't know how to innovate, they just imitate and throw money, hardware and sub contractors at the problem.
They were on a list of already vetted vendors, plus if "a proper competitive bidding process" had been done it could easily be hung up in the courts by losers lawsuits.
As you note, and as I've been noting, given the structure and processes of the project, especially it being run by political and bureaucratic types from the White House on down to CMS (the integrator prior to getting replaced in late October), there's no way they and the other contractors could have possibly succeeded, especially with change orders continuing though the week before launch. Heck, the integration tests CMS ran in the last week or two before the launch told them it couldn't possibly work, but it was launched anyway....
All I can say is thank god CSC wasn't chosen! And to the extent CGI Federal retained some of the qualities of the civilian part of AMS the company bought, again, not an obvious bad move (I worked for a short time in a national security part of AMS, which went to the American firm CACI because foreigners couldn't own it).
Looks like they had planned that downtime, but I wonder how messed up this whole project must be that they can't even manage to have a static maintenance page up!
1. This is already too complicated. ( why a username when you know I am going to need to use my SSN ) 2. This is not a complicated web app. I am now fully convinced that this could have been done with better results by anyone other then the government.
The fewer times you have to enter an SSN, the better. Especially as username-type data is usually shown in text on the screen, might need to be read out over the phone, is vulnerable to phishing, etc.
But it wasn't built by the government. Do you think Obama asked for a username field?
It was built by a private contractor ie. your usual capitalist, profit-oriented enterprise IT company.
It's nothing like a private company building a system. It's more like being a peasant to a lord. Sure you can do what you want, as long it's exactly what they want to you, when, where and how they want you to do it.
If you're going to have to go outside the exchange for coverage, the open enrollment period is probably closing very soon. My work ones have traditionally needed a decision made by mid November. By now, I'd have had to tell my employer yes/no on his healthcare - regardless of whether I'd been able to log in to shop around.
I do not think this is system is well thought-out or particularly good, but this is the only explanation I can find.
>the idea is that restricting the enrollment period will cause uncertainty (or fear) that the citizen may 'miss out' on the opportunity to save money on treatment throughout the year.
I believe this logic shows that enrollment periods are reasonably related (, Supreme Court language for constitutionally permissible,) to the free rider issue. Whether it is a sufficient justification for government intrusion into independent decision making by citizens is a completely different, and entirely normative judgement.
But lots of people are angry about the new health care laws and are linking the laws with the excecutive branch's ineptness at website building, and trying to imply the two are related.
Others just like reading about conflict and struggling against a problem, and getting "insider" information.
The way this article is written helps both groups get that entertainment.
I can't see anything useful about the article though. It's not clear what the website is trying to do, how it interacts with the insurance companies, or to get anything else out of the set of metaphors the author is using to try to describe a website.
Has anyone found any good articles detailing the problems or, better yet, any description of how governments can create or have created successful websites, or how future administrations might approach creating a highly visible website?
Of course the article isn't useful, that's not what it's for. As a description of a problem that has affected people in terms that they may actually understand, I think it works pretty well.
Hopefully we'll get some good detailed post-mortems eventually, but they certainly won't be in the newspaper.
FYI, not having "prescription drug coverage" does not mean insurance does not cover any medication. It is a pharmacy benefit, in other words, over-the-counter medication. In my personal experience, I pay significantly less without the coverage than with it. I also don't have to worry about formularies, prior authorization, or generic swap-outs.
To give you a specific example, in 2013 I paid $107/mo for a private individual policy with a $2,700 deductible, $5,250 max out-of-pocket (inclusive of the deductible) and unlimited lifetime benefits. Next year under ACA I'll be paying $283/mo for a policy with a $4,500 deductible and $6,350 max out-of-pocket (inclusive of the deductible) and unlimited lifetime benefits.
The most important clause in the ACA is limiting the rating factor due to age to a maximum of 3x. That clause alone makes it the single largest tax increase ever passed on young middle class families with children. By my estimates, it will cost families like mine about $50,000 in increased insurance premiums over the next 10 years, because the least we can now pay is 1/3rd the rate of unhealthy 64 year-olds.
Am I parsing that incorrectly?
"Prescription drug coverage" has always covered prescription drugs, in my experience. I've never had a policy that would (to my knowledge) cover OTC meds.
edit: Hey downvoter, let me educate you - the worst thing about a coverage gap used to be the potential to have an illness classified as a preexisting condition, and then to never be able to get private insurance coverage again. The ACA prohibits that practice by insurers as of next year. Here, I even dug up a pre-ACA article on the practice for you, served up on a silver platter: http://articles.latimes.com/2006/sep/17/business/fi-revoke17
edit 2: You know what, prospective downvoters can kiss my butt. Sorry that you don't like the health care reform law. It may be a total mess, but it's no worse that it was before. The only people that are pissed off are healthy, young people (hello, HN) that are paying next to nothing for individual coverage that is garbage. Get sick or old and then tell me how bad the ACA is.
Beneficiaries of wealth transfer are fans of the wealth transfer! Film at 11.
... And actually, I completely reject your characterization of it as a wealth transfer. It's insurance. You have no idea if you're going to need it. You may be in a car accident and badly break your legs, and require corrective surgery and rehab. You may get pneumonia or a bad MERSA infection. If you haven't saved up a few hundred thousand dollars and you show up at an ER, then the wealth transfer is to you when the hospital eats the cost and passes it along to properly insured people. That's not even going chemo/surgery should you develop cancer, god forbid.
I know less about the age rating bands. Off the top of my head, it's to smooth costs over a lifetime. So it's a wealth transfer from your young self to your old self, if you have typical health care costs over your lifetime: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1361028/
edit: http://www.rwjf.org/content/dam/farm/reports/issue_briefs/20... for more on age banding.
To be clear, I am not saying wealth transfers are bad, but we should be honest about what is happening.
Unless we want to decide as a country to not subsidized healthcare for those that can't afford it and to allow hospitals to refuse treatment without upfront payment... then let's be honest about what's happening there, too. The money comes from somewhere, it's every bit as much of a wealth transfer.
Now, yes, you can argue that it's good to smooth costs over a lifetime ... but that also hits the young very hard when they can least afford it. As does their FICA taxes for Social Security and Medicare.
As we've seen in pretty much every other developed country, this whole welfare state approach is a self-correcting dead end, as enough of the young decide not to play the rigged game and decline to have children.
More importantly, Mr. RacerX cannot call the insurance company from the scene of an accident, say that he needs auto care, and expect to pay the same rate that you do. Yet this is the financial equivalent of knowing that you have cancer, and signing up for a new insurance policy.
Finally, there's catastrophic health care insurance (which the ACA has more or less outlawed) which is in fact similar to auto insurance... then there are fancier plans with more coverage, which are more like warranty / service plans for your car. If you have an old car with 300,000 miles on it, it is likely to need more service or develop a crazy expensive engine issue than a brand-new car. If a warranty service plan or similar contract is available to you, you would expect to pay more for it. Likewise a human being, but the ACA has both mandated these plans and limited the amount that older human beings may be charged.
Please note that the moral rectitude, overall desirability, or similar characteristics of the wealth transfers associated with the ACA, or of other hypothetical wealth transfers or health-care related scheme... are not addressed by this post, which examines only financial structures. (The grandparent post examined only incentive structures.)
AFAIK, at least in California and I think this is true more generally, you don't. In order to drive legally on public roads, you must demonstrate the ability to meet a certain minimum amount of financial liability in the event that you are responsible for a collision either by posting a bond or by demonstrating that you have adequate liability insurance.
Most people choose to meet that requirement through insurance, because it has a lower up-front cost (though likely higher lifetime cost), and because they also want other coverage besides the required liability coverage along with it.
> ... And actually, I completely reject your characterization of it as a wealth transfer. It's insurance.
It is insurance, but rules like the "no exclusions for pre-existing conditions" and controls on what factors can be used in rate setting also make it a wealth transfer. (At least part of the idea of making it a wealth transfer is to improve overall access to insurance and, through that, preventive care, to make the overall system more effective for everyone.)