How Healthcare.gov Changed the Software Testing Conversation | Hacker News Reader

How Healthcare.gov Changed the Software Testing Conversation

blog.smartbear.com

47 pointsBaustin38 comments

I tried to get coverage on the 22nd. The process is very long and tedious.

I spent a solid 5 hours going through the entire process about 6 or 7 times, from start to finish, only to get told each time afterwards that it's not complete for some odd reason, and to start over again on step #1!

I tried multiple things, deleting the application, using Chrome instead of IE, etc.

On the 23rd you could not even log in - they took that option away.

What bothered me the most was, besides your social security number and a slew of personal information, they even demand that if you are a naturalized or derived citizen, you locate your naturalization or citizenship certificate and enter numbers (Alien # and Cert #) from it that -- get this - their Javascript refuses to validate (it kind of looked like their rule match is off by a digit). And you get stuck on that step.

I also noticed that if you enter your income as below $10,000, the system tells you that you qualify for no benefits. But the moment you enter $11,500, you get a $260 tax credit. Go figure that one out.

> I also noticed that if you enter your income as below $10,000, the system tells you that you qualify for no benefits!

I know that the common plight there is that some are poor enough that they qualify for Medicare vs. Obamacare, though I admittedly don't know what the distinction is. If you need insurance, and Healthcare.gov is telling you you're too poor, it might simply be failing to tell you to go check with Medicare and see if you're eligible there.

No. It was rather -

Enter $10,000 - you get no credit.

Enter $10,500 - you get big credit.

Enter $11,000 - you get no credit.

Enter $11,500 - you get big credit.

Etc.

Ah. I misread then. That's... odd, to say the least. About in line with the expectation though, I suppose.

The ACA expanded the income range for Medicare eligibility. Eligibility for insurance through ACA starts where that expanded range ends. The Supreme Court said that states could choose not to expand Medicare, thus leaving a gap in which people make too much for Medicare but not enough for ACA coverage.

23 states have taken that route, even though it is more expensive overall than expanding Medicare.

> I know that the common plight there is that some are poor enough that they qualify for Medicare vs. Obamacare

Medicaid, not Medicare. Completely different programs.

> though I admittedly don't know what the distinction is.

Medicaid is state run, largely federally-funded public insurance program (sometimes delivered through capitated private plans contracted with the State), "Obamacare" is just private insurance with specific standards, with subsidies for people between 100% and 400% of the Federal Poverty Line.

> If you need insurance, and Healthcare.gov is telling you you're too poor, it might simply be failing to tell you to go check with Medicare and see if you're eligible there.

Everyone too poor to get subsidies for private insurance (and some people who aren't, because the subsidies kick in at 100% FPL, and the Medicaid expansion was beyond that) through the ACA would be eligibile for Medicaid if all states had expanded Medicaid as set out in the ACA, however, half have not after the Supreme Court rules they could choose not to and still get funding for existing Medicaid programs.

You are 100% correct. Thank you.

Isn't that part of why that Supreme Court decision was such a big deal? It's possible to make so little that you won't qualify for Obamacare subsidies, but the assumption was that those people would always qualify for Medicaid due to the Medicaid expansions. However, the Court ruled that states couldn't be forced to expand Medicaid. So for many of those states that are refusing to accept the Medicaid expansion, there are people that make too much to qualify for Medicaid, yet not enough to qualify for ACA.

From what I've been able to gather, since the majority of the States did not expand Medicaid to additionally cover those making straight poverty line incomes, and since Obamacare has increased the cost of insurance premiums across the board in all but a few states (300% in NC alone), the poor will simply not be able to get or buy insurance - period.

Only those making over $11,500-$15,000 or so will get the credit.

The real kicker to all this is that if you don't end up making that amount this year (2014), even if you are off by $20, you'll be forced to pay back the entire tax credit the government allowed you!

23 states have not expanded Medicare and do not plan to. 25 and DC have expanded it for 2014, and 2 more are going to but post-2014.

> From what I've been able to gather, since the majority of the States did not expand Medicaid to additionally cover those making straight poverty line incomes

25 states plus DC are participating in the ACA expansion of Medicaid up to 138% of the Federal Poverty Level and two additional states are working to implement expansion late [1].

> since Obamacare has increased the cost of insurance premiums across the board in all but a few states (300% in NC alone), the poor will simply not be able to get or buy insurance - period.

Yes, its true that in states that have chosen to reject the -- completely federally funded in the short term and 90% federally funded in the long term -- Medicaid expansion, people making under 100% of FPL will generally have neither Medicaid nor the ability to afford individual insurance.

[1] http://kff.org/health-reform/state-indicator/state-activity-...

> if you don't end up making that amount this year ... you'll be forced to pay back the entire tax credit

My understanding is that the repayment amount is capped based on your income, so, for example, a person whose income came in just below the FPL (or just below 138% of the FPL in states that expanded Medicaid) would be liable for at most $300 of the advance premium tax credit.

Likely outdated source: http://kaiserfamilyfoundation.files.wordpress.com/2013/01/81... (2011)

With so many problem's, I doubt it's just an issue of testing. They probably have serious design issues and other engineering problems.

Exactly. The cardinal rule of competent software development is that you don't test quality into the software. The root causes were requirements churn, split development, and no project coordinator. Those would be nearly insurmountable on an average size project but were hopeless on this project. It's a marvel they were able to stabilize it to the point it's at.

I chuckled when we had the president of the United States of America announcing software delays on national television.

Anything that is done by a public official at that level (not just the President) is well thought out and implemented after much discussion to achieve a particular goal. You can reverse engineer it and draw your conclusions as to why they said what they said.

In this case my interpretation is that it's to shift the blame a bit and also to appear to be out in front of the issue and let people know that "things are being handled help is on the way and we're on it".

An analogy I would use is this.

Let's say you are hosted somewhere (say Rackspace or AWS) and you lose connectivity and your site is down.

Better for RS or AWS to "appear to be on top of it" (whether they are or not) and to be "doing something" than to have this black box of "hey exactly what is going on and when will it be fixed". The anxiety is what kills you.

Once you know that at least (using another example) "a spare airplane is on the way but it is currently delayed by a snow storm that should clear in 2 hours" you are feeling much better than if you don't have any answers at all.

"School children and grandparents alike are now worrying about whether their passwords are being passed in the clear now. Imagine that."

Even if they are "worrying" (and I really doubt that they are actually) they would only be worrying the same way they do about flying right after a major plane accident. That anxiety tends to go away very quickly.

Yep. With every password breach the word gets out that you should have unique, strong passwords for each site. And yet with every breach we learn of more shared, common passwords. Not to mention how little attention most pay to entering a password on a site without SSL. Credit cards they'll worry about but passwords, less so. And imagine explaining session cookie hijacking as we tried to when Firesheep launched? ;-) Once the big players launched with SSL, problem "solved". That said, most do now secure their WiFi, so the public can learn, and I think has learned from healthcare.gov.

"That said, most do now secure their WiFi"

Then there is the linksys wifi camera which you can access over the net but doesn't have https. So in order to get to it securely I have to ssh in to a machine on the network with a tunnel and connect by proxy. A real pain.

Or the devices you don't check. In my case, I've yet to intercept traffic from a Dropcam to see if it's SSL'd correctly everywhere. Let's not start on the alert screenshots it emails me....

Interesting post, probably true enough, but I don't think testing is the major problem.

The law itself was rushed through, to the point that the President and members who voted either way didn't really know what was in it. It was huge, and rushed because they had to beat the clock of the potential coming of a hostile Republican House. Even champions of the bill in Congress admitted that it would take years to fully understand what the bill does. The mind reels.

The law has many hostile stakeholders, and many turf grabbers, and the real requirements were not managed by a single entity who wanted to get it done the best and most economical way possible. Members of Congress (keepers of the purse) were and are fighting it at every turn. Government agencies had to have a piece of it, providing data for deciding customer eligibility and policy level; law enforcement and immigration concerns play a part. The requirements were a result of compromise (in the political sausage making sense) and turf dominance, some of which are part of any project, but at a hyper level when government agencies are involved. There was probably no technical adult in the room telling political people that this and that just can't be done in any economical and working way.

The government does not normally provide web sites to manage and coordinate anything between government, citizens and corporations on this scale. The most you'll usually see, from the outside as a citizen/consumer, is listing of information in text or pdf format, possibly filling in a form to make an appointment or communicate with a representative. They don't know what they're doing, in a profound sense and on a grand scale.

The government just doesn't have experience managing and running such a site with such requirements. They're trying to do something like Facebook in its current incarnation right from the start. It's like never having gone to the moon and strapping some astronauts on the top of an ICBM and hoping for the best, which is a formual to "fail fast" in the worst sense.

Because they didn't know what they were doing, and because it was probably thought of as just another IT contract effort, they followed standard procurement practices and used standard ("we know them") vendors. And that failed miserably.

And because of all of the above, and general inexperience for such a project, and relatively unlimited funds to get it as wrong as possible, they started actual specification and implementation way, way late, with the added complication that the possibility of a time overrun was just not there, because of a hard political deadline. It's no wonder that testing was done so poorly and ineptly, they just had no time at all.

I don't see how Congress can be responsible for most of the reported problems. If the law was just too complicated, then I could see how portions would be inaccessible.

But what we see here are major UI failures, availability problems, losing applications partway through the process and forcing people to start over, and inexplicably inconsistent results and behaviors. These are failures of execution.

> These are failures of execution.

They are, but in my opinion they happened because they tried to do a full on web site in a rush with outdated federal procurement and hostile stakeholders. The implementers and implementation were doomed.

There's a lot more to a web site fronting a huge federal program beyond mere technology.

Obama seemed very surprised that no one told him the project was going so badly. Delaying the project would have been bad for his reputation but not nearly as bad as the fact that hardly anyone could log in on the day it launched! Somehow he had managed the project so badly that he hadn't even communicated this basic set of priorities to the team.

The unavailability of the site made for great headlines for a couple of months, but ultimate it will be nothing more than a footnote in the history books.

I hope the law itself will be nothing more than a footnote in the history books. It's maddening to me that we still have insurance companies in charge of and gatekeepers to health care.

Laws are messy, that's pretty much a given. A nasty set of requirements didn't screw this up -- an outdated process for software development screwed it up. It's also solvable. So, rather than throwing our hands up and saying "government can't handle big problems", we should demand reform in the bidding process, and in how technology processes are managed and executed.

The health care law was passed in Obama's first term, and implemented literally years later during his second. If they were confused about what was in it, they had plenty of time to figure it out.

What apparently was a genuine surprise was how many state governments were unwilling to set up their own exchanges, which was how things were originally envisioned to operate; having the federal government operate the exchanges was intended as a last resort. But even there, it was apparent in plenty of time how the wind was blowing.

And as to what it took to operate an exchange --- there was prior information there, too, in the form of a functioning exchange in Massachusetts. (Which hasn't been functioning so well this year in part because it's now required to integrate with the federal system, and that part has had severe teething trouble. But again, none of this should have been a surprise.)

This isn't meant as commentary on the specifics of the law. But as a left-leaning guy myself, I don't think there's a reasonable excuse for how they blew it (and, to his credit, Obama's not trying to make any).

It was huge, and rushed because they had to beat the clock of the potential coming of a hostile Republican House. Even champions of the bill in Congress admitted that it would take years to fully understand what the bill does. The mind reels.

The law was a horrifically complicated moronicy for a very specific reason that it is important to call out. The law was formulated to satisfy the numerous semi-monopolistic rent-taking interests already well entrenched in the field (especially the health insurance companies but also including other players. The final bill was written by an ex-high official of some big insurance company as I recall).

It is important to not ascribe anything like good intentions to those who went through this exercise. If we are very lucky, the new regime will allow hospitals and insurance companies to extract our money in a more orderly fashion and thus we may find ourselves benefiting in a relative sort of way. But we should never speak of better organized vampirehood as having a similarity to good intentions (the US will continue to spend twice what other industrialized nations spend on health care going forward).

> The government just doesn't have experience managing and running such a site with such requirements.

They still don't. They outsourced it, first to Verizon, then to HP

http://online.wsj.com/news/articles/SB1000142405270230356290...

Software developers should better appreciate the uniqueness of their testing situation: like any other field, writing, maintaining, and executing takes time. But in no other field that I can think of is testing so relatively cost-free to continue doing..."regression" testing is either not possible or feasible elsewhere. The ability to automate and near effortlessly run tests with every code change has a fundamental impact.

To give a non-programming example...journalists make silly mistakes all the time about dates and misspelled names. It's not always, or even usually, because the reporter is incompetent. Sometimes they hurriedly typed in a fact from the top of their head and forgot to put a "TODO" near it (some in-house CMSes do not make meta-comments easy). Or, just as frequently, something got changed as the text moved from one editor's desk to another...there's pretty much no such thing as diff software, which is fundamentally different than keeping revisions.

Spellcheck can help detect and auto-fix some problems. But generally, you need to manually proof-read things to verify them, and at some point, you just assume that no one is going to change what you've verified, and then you press "Publish". It's not that continuous-proofing isn't possible, it's just not feasible.

It can be frustrating working with people who think testing software is to keep "proofreading" it over and over again...user testing is vital, obviously, but I'm talking about people who test the wrong, already-verified things, and then sap their energy not checking for other variations, and this is understandable and very human, of course.

I guess the big picture to understand is that screwups are frequent in every field, all the time. The best surgeons forget to wash their hands...not because they're idiots, but because emergency surgery will cause all kinds of things to go haywire, including basic procedure.

And obviously, very basic mistakes can occur in production code. But as programmers, we uniquely benefit from what we can do to prevent that. And this superpower of ours is something that I wish was more conveyable to the greater world.

As a counterpoint, software development arguably suffers most from the combinatorial explosion of behaviors. In other words, testing software is easier but less effective (or at least less complete).

It should be noted that testing consists of such a wide array of tasks that it's often as hard to find the right people than it is to identify the right type of testing they should perform.

While performance and load testing on Healthcare.gov should be obvious tasks we get into murky territory when asking questions like: Does the UI need to be tested more than the APIs that power it? Will the value of automated end-to-end testing be deprecated after a UI refresh takes place? (Will there be a UI refresh?) How much time should be put into scalable UI tests? Are automated smoke tests enough?

I could go on, but the question I'd like answered is what did the actual testing plan consist of?

"Hey Frank, you test this?"

"Naw, Larry, government contract."

"Shit, I think this one's important though."

"We'll fix it later. It's easier to beg forgiveness than permission to change spec."

Anyone have a link to the github repo she mentions? Also was there an announcement of the open sourcing of code that I missed (been travelling and off the grid for a while).

Thinking that this is about TDD or software management is to grotesquely miss reality.

This whole fiasco has demonstrated, to an incredible level of clarity, why we've reached a point where government is too big to work. This has nothing whatsoever to do with who's President or which party controls what. This is a simple case of an entity that has grown so large, complex, ignorant and bureaucratic that it simply can't figure out how to produce anything useful, from laws to websites.

The difference this time around is that this has been very public. This is a problem that forces people to pay attention and get involved. Healthcare affects everyone directly and people care about it.

Most of us who have understood the devolution of government over the years have recognized the incompetence of the organization as a whole for quite some time. Some have been more vocal than others in trying to highlight the issue.

It's a difficult position to hold because it is relentlessly attack by those who, through indoctrination or religious-like following, stick to their respective parties and simply won't even admit they are being screwed despite mounting evidence to the contrary. The President lies about healthcare to the country and the world dozens of times and Liberal media contorts itself to try to figure out a way to spin it into some alternate reality that makes sense. We had exactly the same kind of thing happen with Bush and his wars, but this is about the ACA. We let them lie. Some of us see it and call them on it while others shoot us down based on party loyalty and continue to support the effectively criminal behavior.

The difficulty in gaining mass awareness for these kinds of problems has been in that most people, at the end of the day, couldn't care less about what's going on inside the sausage-making factory. They are too busy trying to earn a living and going through their daily lives. There are things in most people's lives that are far more interesting to them than what government does and how it works. True to this I think it is fair to say that most US voters are utterly uninformed and get their opinions (and voting decisions) from the media --mostly TV.

Now things are different. This is something that is important to everyone. Health, food and housing are top-level concerns for everyone. This ACA/Obamacare mess is achieving something no government critic could possibly buy for any amount of money: Bring to the forefront the incompetence, waste and mismanagement that has become part and parcel of what our government has been about for years.

Everything in our government is done this way. Everything. You just don't see it or don't care to dig into it for other issues. Healthcare just made it first page news for everyone.

Everyone now sees how the sausages are being made. There is no way to hide it. Everyone can now balance the equation of what politicians said and promised them against what they've actually delivered. A family who's healthcare premiums doubled, who's deductible skyrocketed and who lost the ability to see their doctors at their hospital has no way to satisfy a promise of lower premiums, better coverage and keeping what you like.

I still remember Los Angeles Mayor Villaraigosa deciding to change the outcome of a vote in a very public way right in front of the cameras while a woman came over to him and said "Let them do what they are going to do", all of it picked-up by the microphones and cameras. There is no shame or respect any more. Politicians know they can get away with murder because there are no consequences for such things as publicly and visibly changing a vote or spewing out lies.

Hopefully people are starting to think about just how ridiculous it is to give a blank check or a pass to anyone in government. I also hope they are starting to become more convinced that if we don't make those who lie to us responsible for their lies we are never going to improve government.

If it had not been for Snowden who in government would have told us what was going on with the NSA? They've been lying to us for years and they would have continued to do so had it not been for him. They lied knowing there are no measurable consequences for their actions. At worst they move elsewhere in government and live goes on. It's a joke.

Imagine if you were involved in a contract negotiation for you business, signing the papers for your home purchase, auto purchase or a lease rental agreement and the other party said something akin to "Don't worry. We have to sign the contract to see what's in it". How fast would you run from that one? You would, right?

Then, why is it that we let those in government play such games? By extension they --the governing class-- over time, start feeling they can do anything, to anyone at any time. I would not be surprised if this is the sort of thing that leads to mass surveillance programs not being given a second thought within that community. If you can get away with murder you keep murdering, plain and simple.

The solution, I am afraid, is not technological, it's political. I've said this before: If you are in tech you need to be smart about politics because bad government and bad policy can be more destructive to your efforts than bad code. The NSA fiasco alone will probably cost US companies billions. Trust is a very difficult thing to regain, whether it is with your girlfriend, boyfriend, wife, husband, company or country. Once trust is broken it takes orders of magnitude more work to regain it. If ever.

So, there you are, becoming a highly skilled technologist thinking that government and political issues are of no importance to your mission, only to find out that your international trust and market potential keeps getting boned by ignorant fools in government who really have no clue what they are doing. Government is almost the only entity that can completely destroy your startup with the stroke of an ignorant pen. Look at the startups fighting against the transportation and housing unions, rules and regs. How different would things be in terms of competition and progress if those barriers where not there?

Bottom line: You have to be informed and get involved. You have to fight against indoctrination to see what reality looks like. Stop taking other's opinions as your own. Question everything, research, critically observe, measure and use your CS and mathematical skills to model. In other words, be the high information citizen.

The devil isn't in the details, it's in politics.

I'm not sure what people that earn below the required threshold do if their state isn't expanding on medicaid. How do they apply for medicaid?

I am just an undergraduate, so what the heck do I know about agile in big project. Please feel free to critique. What do guys from big projects have to say?

Here is how I imagine people arguing about agile vs water fall.

WF: It probably will take the same amount of hours and people to hash system together in agile, making sure all components are integrated properly without a strong and detail system requirements.

AG: But you don't wait until the end to test your software and wait until the end to discover shit and changes. You can iteratively change stuff.

WF: Ah. Fine. I will just tell my co-workers to test more often. Now, you agile people just want everything to be short-term. How do you ensure people don't block other teams from completing works? you see when blockers appear, the time requires for completion is the same as everything done at once.

AG: But if you finish 80% while waiting on the 20% to block forever, you still have 80% done. Plus, in agile you have some short term and long term goals. You order them by importance and blocker. When the whole sprint is blocked, just do more testing and making sure existing systems are working as said.

WF: Fine. Now what about people going monkey on their own? The point of water fall is to ensure no one is outside the design and minimize risk. Now everyone goes agile and shit going to come because some teams are not playing nice.

AG: This is a tough problem. But the only way out is by scrum, well, at least meeting daily or weekly. The benefit of agile plus regular meeting is that everyone knows what the heck is going on and changes can be made. Once you made a specification forever, it's hard to change.

WF: But it is the same in agile that a chance can be tough to make because people didn't make the right decision at one of the early sprints.

AG: Yeah. That certainly is. I won't lie agile people don't make that easier. But I argue it is easier if you just work on smart parts.

WF: Are you sure it will work for multiple-billion dollars project? With a lot of legal blockers and a lot of other software to work with? You need to do a thorough analysis. You need to conduct research how existing healthcare registration works, find out the pros and cons. That's a lot of time. Now if you go agile, you will just keep doing research.

AG: ....

YEUKHON: I will fill in the blanks, but hey what the heck do I know about big project software engineering. I think agile doesn't require people to start right away. Agile just ensure nothing blocks forever and that goals are more or less short-terms. it is always required one to have good knowledge of how existing solutions works. So if you need to find out how to integrate the new healtcare.gov with the rest of the solutions out there, do that research first. Obviously you can't just develop a site like a baby punching keyboard, hoping the baby compose some legendary music. It takes real talent to do that. So I say myth buster, agile still requires professionals to do priori research, have lengthy discussions about how to go about implementing the system, give a pretty good sketch about how things supposed to work, and then go off and start implementing something. But don't come up with the full solution so detail that you ought to obey that contract.

WF: Yeah, but your client is spending 100 millions and this is impacting tens of millions of people. Any example out there like healthcare.gov did finish on time and running successfully using agile? I know WF sucks, but real examples of agile out there? You know, people tend to do a lot of talking and sketching when they are working with so many people and so many things involved. "Oh Bob and his teams are doing research on existing solutions works and my team is working on consumer APIs? Okay, let's talk about what fields we want, what schema we want in the database, etc"

Reply on news.ycombinator.com