The Obamacare software mess
economist.com
economist.com
I think the tendency in these Monday-morning quarterback sessions is to blame the people or the tech. If they only used X,Y, and Z! people will say, or Why can't they find qualified people who can do this, it's not that hard!
When that fails, we start blaming groups of people. Those damn contractors. Always screwing us over! or Dang government bureaucrats! Lazy bums!
These patterns of analysis are almost always mistaken. You don't drop a billion bucks and get screwed because you hired dumb people, because you're lazy, or because they decided to use WhizBang 1.0 instead of 2.0 You get screwed because you have smart, motivated people working inside of a system of delivery which is hosed. Incentives are wrong, feedback loops are loose (or non-existent), and so on.
So while this is a nice political topic, and sure to get everyone's emotions roiling, be aware that we see this same stuff all of the time. (I would make the case that government is far worse, for exactly the reasons I mentioned above, but I won't because it's only going to push many of your buttons).
It's not the simple stuff that kills projects. It's certainly not tech. It's particular configurations of people that choose bad tech, or that refuse to change bad tech once it's identified. Most all of technology development problems boil down to social problems, not the other stuff. If you've got a good social structure with ten qualified people, you can rule the world. If you have a bad one with a thousand geniuses, you'll be lucky to get your shoes tied in the morning.
No matter how it looks at first, it's always a people problem.Also, before anyone gets all "hurr durr government!" on this - this is the exact same situation at any major bank or company founded before about 1990.
> Why does it take so long to send money from one bank account to another?
http://www.npr.org/blogs/money/2013/10/04/229224964/episode-...
I can help thinking that a few good Perl programmers, and a few high-level bureaucrats to force open the maws of the respective agencies, could have had this sorted in much shorter, and more cost-effective, order.
Also, as I commented elsewhere before, first make it work. Then make it pretty.
I guess I'm being a bit snarky. But I'm also being more than a bit serious.
Contractors who do shitty work are putting most of their effort in to landing new clients to replace the ones who will never repeat.
http://www.economist.com/blogs/economist-explains/2013/09/ec...
And on universal health care they're generally on the left.
It's 2013 and there are tons of technologies that allow this type of system to be set up fast and scalable. The fact that I can't even login (with a registration I made a month ago) and that it doesn't even give me an error message, and that support (phone and chat) just say "it's experiencing problems right now" is inexcusable when you consider that they had half a billion dollars to work with.
Is a secure registration and login system that at least gets people started on the site too much to ask?
Evidently so, even given 3 and a half years after passage of the bill that's this Administration's signature accomplishment to date. While it's hardly unknown in other sectors, I've read one of the problems was HHS/Sebelius making important decisions way late in the process, and I think I recall the usual required changes as well.
But unless the signup process hooks into outside live databases for verification or what have you there's as you note no excuse, especially since disallowing window shopping is one of the biggest modern UI hindrances, something pointed out to the people doing this (some information gathering is needed to offer the right plans, but others and I have compared it to the medicare.gov Part D sub-site, which collects that but doesn't require giving all your information prior to you officially selecting a plan, after which the site informs the chosen provider).
Anyway, I am a "right-winger" who's upfront about wanting to ... eliminate many parts of government, and just from memory I can point out several massive and notorious IT procurement failures that would kill or cripple most private concerns. E.g. last time I checked (which is some years ago):
The FAA is still using their 1940s (sic) techniques, and every attempt to really modernize the initial system architecture that automated it has failed. (They famously hired a couple of retired IBM engineers to make microcode patches for some of their old machines for Y2K). This one has really serious economic consequences.
Every comprehensive attempt to upgrade the IRS following their very first system has failed. They're still using code for the 1959 IBM 1401 (https://en.wikipedia.org/wiki/IBM_1401) in their front end processes.
The FBI is legendary for their failures to ditch paper. I remember at least one failed attempt pre-9/11, there's one acknowledged failure after it, and I don't know if they're really using the Sentinel system as they claim as of 2012, after 7 (!) years of troubled development.
These are the really big ones that come to mind. Which of course is not to say the government is always incompetent. The Census Bureau was famously cutting edge, using punched cards in 1890 (sic), buying a UNIVAC I in 1951 (sic), and when I talked to some of their people in the mid-90s I got a sense of serious competence.
As noted in previous discussions, the same contractor operations CMS.gov and Medicare.gov, and I can attest that the latter site, while a bit clunky, works just fine. CMS's backends seem to work as well.
Off the top of my head, the CDC and FDA do well in getting out timely info. The DoJ's statistics sections are good, the NIH's sites are stellar. Heck, the BATF has a nice web page where you can type in some of the numbers of an FFL (firearms dealer, gunsmith, etc.) and confirm it is legitimate and still valid. Etc.
That is an order of magnitude less traffic than I expected. 33 visits per second isn't child's play, but my WordPress blog hosted on a 1GB Linode VPS has easily handled bursts of that kind of traffic in the past.
That's totally an apples and oranges comparison, of course, but it's more and more difficult to understand how this project has failed so miserably. I half bought the excuse that it was one of the highest traffic site launches in history, but apparently that was more speculation than truth.
> but it's more and more difficult to understand how this project has failed so miserably.
So, imagine someone estimated 8.6m visitors over three days, and decided to load test the site for 33 visits per second... =)
Then it becomes really easy to see why it could fall over.
Disclaimer: I've been doing load testing the last few weeks, and have been beating up on other peoples server side applications. It's a lot of fun when it's not your code getting hammered. =)
That's why telco systems are sized as "BHCA" (busy hour call attempts) and "CAPS" (call attempts per second), which are peak rates, not average rates. Well, they are both averages, but not over the whole day or year!
Then, as others have pointed out, there's a difference between a static site and a non-static one. This is mostly determined by I/O. A single disk can do between 100-150 IO operations per second (IOPS). Reading a row from a database will take several IOPS. In the systems I work with, we have to keep the entire database in memory to meet our performance numbers, and even then we are still O(database reads).
Doing a SOAP request to a legacy platform that was designed to only support batch requests? Lots of opportunity for slowness. Particularly when you start to run into TCP timers controlling port reuse.
Doesn't excuse the 404's though. That stuff is easy to test automatically.
I do remember reading articles six months ago or so that the federal government was disappointed that they would be managing the exchanges for so many of the states. They were originally hoping that they'd only be managing the exchanges for a handful of states, rather than the 36 they're having to shoulder the load for. I can see how that level of political uncertainty adds an extra challenge for estimating load.
A number of Blue or Purple states are also punting to the federal exchange, or doing a "partnership" where the federal exchange is doing the heavy lifting:
New Hampshire, New Mexico for the moment for individuals, Utah permanently for individuals, Arkansas, Iowa, Michigan, Illinois.
In many of these cases of partnership, whatever the governor of whatever party wanted, the legislature didn't buy off. New York is the only case I know of where the governor told the legislature to jump in a lake and set up one for the state through an executive order using the temporary money the Feds are donating early on.
And there's Missouri, who's people outlawed a state run exchange, and New Jersey's governor vetoed 2 bills to set up one.
I would have thought California (note: I live in CA) would have had a good healthcare site, considering all the techies, but I guess in this case that didn't work out.
map of state/fed/joint-run sites: http://www.dailykos.com/story/2013/10/03/1243855/-Kentucky-s...
Kentucky is as about as "red" as the current Federal government; even the DailyKos article you link to is self-refuting, first claiming it's "one of the only reliably red states" and then claiming "the biggest factor of all [in its claimed success] is Democratic Gov. Steve Beshear".
As I note elsewhere in this thread, Idaho is the only truly Red state implementing a full state exchange, although the map you cite claims it's not full. And that turns out to be in theory temporary, further reading my source (http://www.healthinsurance.org/idaho-state-health-insurance-...) it's in the process of building its own and depending on the Feds for 2014.
Mitch McConnell and Rand Paul would disagree... (although of course, given that Obama is somewhere to the right of Nixon on most issues, it's not so far off)
Do I really have to explain that state officials run state governments, not the state's elected Federal representatives who do their thing in far off D.C.?
http://www.cgi.com/en/CGI-selected-build-US-wide-competitive...
1. Jason Lahoz couldn't sign up.
2. @Ginnyproffitt was frustrated.
We should not be so quick to jump on the government without evidence that there is a problem, especially when you can just go look at the site yourself and see if it works.
I realize that the plural of anecdote is not data, but there's got to be some multiplier of plural that is data.
How else would you do it? Public executions of any public-funded program that looks bad? We have an accountability system -- it just might not be designed to optimize the things we would like for it to optimize.
Something like a NASA for non-space software development?
My question would simply be: how would this be any different than using a small contractor? Or having really great people in a small team do the same thing?
In other words, what does a new organization and standards bring to the table that's not already there? Simply a new name? What do they do that you currently cannot do separately if you wanted to? Are you really suggesting something new, or just adding a bunch of buzzwords together in an effort to try to make something sound good? Or, to be even more provocative, is the goal here to actually do something, or create something that sounds like it might do something? Because they're two different things. We've been down this road of "Ok, let's set aside this special group, with these special criteria, and a new name" a hundred times. Best case -- it works at a very small scale. Worst case, same old, same old, just with new titles over all the stuff.
Walter Schirra; Fifth Astronaut in Space
By Patricia Sullivan
Washington Post Staff Writer
Walter M. Schirra Jr., 84, one of the original seven astronauts and the only man to fly in the Mercury, Gemini and Apollo space programs....
[...]
[His second] mission was delayed a second time when the Titan II engines beneath the space capsule ignited at countdown and then shut down.
For several heart-stopping minutes, Capt. Schirra and astronaut Tom Stafford, sitting atop a highly explosive mass of rocket fuel, chose not to pull the ejection handle, which would have scrapped the mission. It was a calculated risk. Capt. Schirra trusted that the booster rocket would not explode and that the first attempt to rendezvous with another spacecraft, Gemini 7, could still occur. The risk paid off, and three days later, the launch was successful.
Asked later what he thought while sitting on the launchpad, Capt. Schirra replied, "This was all put together by the lowest bidder."
E.g. the story of the reasearch and developemtn of the legendary Apollo Saturn V first stage engines by Rocketdyne, first for the Air Force and then for NASA, is also legendary: https://en.wikipedia.org/wiki/F-1_(rocket_engine)#History. They used explosives to debug it! That's got to count as a hack....