Health Exchange Tech Problems Point To A Thornier Issue
npr.org
npr.org
As someone who has never worked on a huge site, I don't want to naively say that it could have been done better. But I feel like those are basic issues that could have easily saved them a lot of bandwidth.
Also, it seems to me that if they had so many issues on the front-end, it's a hint that maybe they also had many similar issues in the backend as well.
edit: Nevermind: http://newsroom.accenture.com/news/accenture-chosen-to-imple...
Vomit everywhere.
1. At the dinner their representative spoke about how life at Accenture was a breeze and how as a undergraduate he spent most of this time drinking. He drank on the weekdays but his favorite time to drink was on Sundays. He said he used to do some SQL but now he had moved onto the business side of things. One of the candidates told us how his hobby of skydiving let him see the big picture and another told us that his desi background would enabled to come up culture specific features for software services. I probed this question further, and he said that in India showing your muscles was considered hubris, while in America it made you look strong. On his facebook page he wouldn't show his bare chest... Nobody at the table `looked` like they could do any programming work.
2. During the technical interview I was asked what `clubs` I was in. I told them I wasn't in many clubs and instead spent my time doing research work (which by that time had yield publications) and administrating websites. I told them my hobby was programming. I wanted to tell them how I dealt with EC2 sight outages, worked with clients and their clients to write WordPress plugins. I wanted to talk about my experience engineering our production and outsourcing workflow. I wanted to summarize the techniques I learned overseeing our production staff. They didn't comprehend and asked he what `clubs` I was in.
I was scheduled to have 3 interviews, I walked out after the first. Don't work for Accenture, they are a joke because their employees are a joke, which is much in agreement with this article.
They also have 275,000 employees.
Is my goal to improve the ability of the developers? What does that gain me? If the ability of my talent is only loosely coupled with landing contracts, then I'd try to maximize the gulf between the salary I pay and the value they provide. If fresh grads can produce good enough work to satisfy contracts, and they cost less, they're my target.
Many firms work this way, and it seems to be a fairly successful strategy if you can land the contracts. Having a few senior, competent people in the mix probably helps hedge against any serious problems. Also, I'd make sure to get everybody as many useless certifications from big companies as possible, as they improve our ability to land contracts.
I'm just playing devil's advocate though. My real answer is I'd change my job.
...then I would be well compensated for not changing a thing. They've had this model since they were "Andersen Consulting" (the rest of us on project teams with them called them Androids, before "android" was cool [this was also before "Andersen" became exceedingly uncool]), and they bank by the truckload. Apparently there is a type of customer that wants legions of safety-school graduates with lots of highlighter in the first three chapters of their "For Dummies" books.
This caused a number of people to worry that this was one of those scam exchanges people had been warned about.
Total cost: $54.8 million
[1] http://www.latimes.com/business/money/la-fi-mo-california-he...
BTW it looks like Accenture is not the only one. This page: https://www.coveredca.com/shopandcompare/#calculator
has this in the source: Produced for Covered California and the State of California by Kiefer Consulting, Inc. www.kieferconsulting.com
(and then for some reason Apache license - copy-paste from somewhere?)
So 180M might be just one part of it, the whole sum may be much bigger.
http://www.theatlantic.com/technology/archive/2013/06/health...
Especially these paragraphs:
"When we worked with the World Bank, they chose a plan from Rackspace for 16 servers," said Gundersen. "That added tens of thousands of dollars, with a huge hosting bill every month."
HHS had similar strategic plans for the new site, at least at first. "They were planning 32 servers, between staging, production and disaster recovery, with application servers for different environments," said Cole. "You're just talking about content. There just needs to be one server. We're going to have two, with one for backup. That's a deduction of 30 servers."
It's a sad, broken system that will almost certainly never change - in part because we don't vote for the people in charge of these things. It really doesn't matter who is in the White House or in control of Congress. The fiefdoms within the government that control these issues have maintained a culture of mediocrity for decades, and it will likely remain that way.
Why? It's not fair now.
For most of the problem areas, the answer isn't to mend it, but to end it. Stop contracting out core functions.
I have a feeling one state's site will be the best and will perhaps be mirrored to other states within the coming decade. This is clearly not an overnight problem, but in the end, once each user gets their insurance they should never need to visit again unless they want to switch.
The other side of the problem, arguably the bigger side, is that most of the people in the acquisition process do not know how to write a contract that is flexible and really gets the government what it needs. For such a large project as the health care website, the people at Health and Human Services most certainly did not know how to articulate what they wanted. They could've hired a company to help them determine their requirements and write a proposal, but that's a significant (and long) undertaking itself.
In short, government systems will never be user friendly, scalable, or secure because we don't know how to acquire such a system.
1) Python is not allowed
2) Open source software is insecure
3 Perl is allowed
4) there is a list of approved software.
5) I can't see it
6) Enthought is on at least one version of the list
7) I can verify there are several python projects on forge.mil
The key is to write good quality Perl, so it's fun:
• http://modernperlbooks.com/books/modern_perl/chapter_00.html
• http://my.safaribooksonline.com/book/programming/perl/059600...
1. Maybe they know of vulnerabilities that you don't. Remember the bug in Debian where cryptography was being seeded by the process id?
2. They might have put a lot of time into ensuring that one particular product was 100% secure and trust only that one. This is often the case for hardware, where vetting for security compliance can be difficult.
They are dealing with scaling issues that are almost completely unprecedented. Every other e-commerce site that deals with many millions of visitors a day has had months, if not years, to figure out the touchy parts of their infrastructure and make adjustments while the traffic ramps up.
Given this, I can't help but think that 90% of the scorn heaped upon healthcare.gov is politically motivated. When Twitter spent YEARS going down on a regular basis, or Simcity just completely shut down on launch day (with a fraction of the users of Obamacare), nobody demanded the companies that running them be shut down. Shit happens when you build a website that gets incredible amounts of traffic. It will work out just fine.
And how much those months or years of experience were leveraged in figuring out the touchy parts of the infrastructure? This shouldn't have been the first rodeo for anyone involved. There are numerous blog posts, books, videos, etc with ideas, what works, and what doesn't, for running websites of all sizes. Many of them are posted and debated on this very site.
I can think of two that I've worked on personally:
- Yahoo!'s tribute site for the first anniversary of 9/11 - Ticketing for the 2008 Beijing Olympics
Yes, in both cases, the companies had experience & infrastructure for building high-scale website, but that should have been a requirement for the entities building healthcare.gov as well.
In both cases I've cited there were a lot of significant changes to the technical stack (e.g., it was first time we used PHP for a large scale project at Yahoo!).
TL;DR: This isn't really excusable.
Malpractice is endemic in the IT space and nobody, either technologists or managers, are held accountable.
For instance, the issues @vinhboy talks about are "bad smells" that reek of unprofessionalism -- like the primary keys that aren't unique, the predictable race conditions, and all of the problems that recur again and again.
"One might look at this and go: Why can't we get the smartest people from Facebook and Google and from Twitter to come and work out these problems?" Johnson says. "The problem is that the way that federal contracting works is so burdensome that the only people who get contracts like this are experts at lobbying and experts at regulations that require you to get these sorts of contracts. And they're not experts at doing the job of building these websites."
This is precisely the sort of shit that make people believe that the government is incompetent. It really is for certain undertakings that require some degree of operational flexibility.
It doesn't "make people believe", it is incompetent. And, unfortunately, the very capable and competent people are working hard at making it worse by electing exactly the people who would perpetuate and expand the problem.
1) Our government has a huge set of rules about procurement. Who's allowed to get their bid approved, the right kinds of projects, who's allowed to manage them, who's allowed to set project requirements (different than the former), who's allowed to pay them, etc. In theory this is all really smart and awesome: you don't get someone giving their brother-in-law a swell multi-million dollar contract, physically impaired people can access your site, etc.*
2) Governments and campaigns face different incentives. Campaigns don't have many cross cutting concerns: the goal is to get a candidate elected. Sure, some high powered consultants are simultaneously angling for a new job after, but IT and most staff are relatively immune from that. Your job is to get something done, and you want to get somethng done. Government, though, has no such incentives: you can do shitty work and not face consequences, and the mission is inherently less aspirational and more shoveling shit. Much of the time government bureaucracies also double as a social welfare program, or a private welfare program if a well-connected corporation has a well-placed surrogate legislator or bureaucrat.
*Reality, needless to say, can turn out a bit different.
http://www.digitaltrends.com/opinion/obamacare-healthcare-go...
With all the state exchanges included, the websites for this program must be over a billion dollars spent. Amazing.
There are over 100 million families in the US and they build for 50,000???
Is that number at least 50,000 pages / second? (Which would be reasonable.)
I totally agree that the government needs to become more lucrative to smaller more agile groups. I think it could ultimately get better products at lower prices if it opens up bidding. That said, the government has massive needs and massive legacy systems that many small shops may have a hard time reconciling. There's always ways to theorize about how the government could standardize and modernize its disparate stuff, but that requires strong leadership, a good direction, and money. None of these things appear to be in abundance right now. It's really unfortunate.
The asynchronous alternative is a menu of plans you may or may not qualify for, at prices that may change after you apply, based on inputs many people will get wrong (unintentionally and otherwise), with a true cost that won't be known until someone does your taxes up to a year later and computes what subsidies you qualified for.
Even if there's only a day between signing up and verifying your information, that turns an hour of "getting it done and over with" into a multi-day ordeal. Considering this is a mandated purchase for a lot of people, they want to devote multiple days to it about as much as they want to stand in a long line at their local DMV.
I know it's nicer if it's instant, but I don't think anyone expected it to be instant; private health insurance is a multi-day process. Nobody is getting this health insurance until January. It sounds like the system is asynchronous anyway, in that it then goes off for further processing (no matter what the website promises).
In any case, we're talking about a delay of a week at most; I'm not suggesting that nothing be verified until tax season.
The only other two options I can think of would be:
-- Pre-verify all of the potential applicants, many of whom may already have coverage and never come to the exchange. -- Ask users a bunch of questions that they may or may not know the answers to.
If you batch processed the delta from the other agencies every night (or as soon as they could if better), it could work!
[1] http://www.latimes.com/business/money/la-fi-mo-california-he...
The amount of money spent on building healthcare.gov (many millions) just makes that incompetence even more infuriating.
To me these incidents, the broken procurement systems, the mediocrity (or worst) that permeates government and many other factors only serve to reinforce the idea that we need massively less government reach into our lives, not more. Frankly, the combination of healthcare and the IRS scares the living daylights out of me.
Government is successful with complex public facing IT in areas where the processes are well understood. The DMV, tax collection, and unemployment are good examples in most states. Medicare billing is a success story in the federal space. Most other things are not very successful unless they are limited in scope.
The whole system is a screwed up farce. "Obamacare" is a flawed system to begin with, to the point that nobody really understood what was needed for months. So the Feds and states were forced to roll with some highly generic requirements and specifics were made up as they went along. That scenario is a dream for Accenture (Cali) or CSC (ny) to churn hundreds of millions of dollars.
That doesn't mean that this site isn't a mess, but I don't see where the "screwed up farce" bit comes from. I rather suspect it's driven more by your choice of political tribe than any real intuition about what the government is capable of.
Like all website launch problems in the months and years from now people are going to forget they ever happened. And instead will find having an independent marketplace to be incredibly useful. The states that have exchanges have already seen insurance premiums drop significantly.
[1] http://www.mercurynews.com/nation-world/ci_24248486/obamacar...
It's NoSQL!
[1] http://www.informationweek.com/smb/services/uk-pushes-govt-i...
[2] Nuts, cannot find the link. Was an article on theregister.co.uk for what it is worth
Really?
No wonder we are going broke. It's nothing but special cases, all the way down.
You missed a step.