Healthcare.gov failed despite agile practices
gcn.com
gcn.com
My theory is that Ken Schwaber and Jeff Sutherland came up with Scrum in order to limit management's ability to screw up software development in corporate settings. That was mixed with some observations about the need for iteration, not doing upfront work when it's wasteful (and it's not always wasteful) and visibility that came from the Agile manifesto.
However, one lesson I'm constantly re-learning is never underestimate people's ability to screw things up and I think Ken and Jeff underestimated.
The whole agile thing has been subverted and if indeed Healthcare.gov was done using "Agile" practices then this is another bit of evidence.
We, as software engineers, need to move away from these and figure out some way to qualify what makes good software engineering. You don't see non-technical hospital managers going around telling doctors where to place an incision and you don't see non-technical aerospace management telling engineers what bolt to use in a rocket engine. But for some reason "programming" has been demoted to something that just being "agile" is enough to do. Badly.
In his YouTube Scrum talk Ken Schwaber says something about if you have a poor team with bad engineering practices than Scrum will generate crap for you sprint after sprint. Somehow no one listens to that part.
I doubt we'll ever be able to distill it to a formula.
People ran successful software projects decades ago, long before any methodologies were formally described. And they have failing projects now, even when they use a supposedly good formal methodology.
Instead of another alternative to waterfall, scrum, or agile, I'd propose the following ingredients for success: Organized people, a good working relationship with plenty of trust, a clear understanding of roles and responsibilities, realistic expectations, and colleagues who honor their agreements with each other. To name a few.
All of this is peopleware, not process. Unfortunately, there's no silver bullet to get this stuff right.
The best answer is probably to get people who are experienced in similar projects to manage your project, and has little to do with the process they choose to use.
I know there's no silver bullet but those of us who have been through a few projects have a pretty good idea of what's going to work and what isn't. We can tell, well in advance, what's going to work and what isn't both in terms of the technical approach and the management process. Agree?
If that is the case how do we convert that intuition into something that can work in the real world? Is certification the answer? Specialization? How do we recognize excellence?
But e.g. how many times have you told a superior something like "This won't work because of math" and then been simply ignored? Until, of course, the system/project goes down in flames.
We need stature that can get around Pournelle's Iron Law of Bureaucracy: (https://en.wikipedia.org/wiki/Jerry_Pournelle#Iron_Law_of_Bu...):
"In any bureaucracy, the people devoted to the benefit of the bureaucracy itself always get in control and those dedicated to the goals the bureaucracy is supposed to accomplish have less and less influence, and sometimes are eliminated entirely."
And this amplified restatement:
"...in any bureaucratic organization there will be two kinds of people: those who work to further the actual goals of the organization, and those who work for the organization itself. Examples in education would be teachers who work and sacrifice to teach children, vs. union representatives who work to protect any teacher including the most incompetent. The Iron Law states that in all cases, the second type of person will always gain control of the organization, and will always write the rules under which the organization functions."
Lawyers frequently don't get that. Doctors? Not so sure anymore, the same masters of the universe have been revising the way they have to do medicine (e.g. the Obamacare foretaste in the stimulus bill requiring "meaningful use" of Electronic Medical Records). Scratch that, Obama's utter contempt for doctors was strongly on display while he was barnstorming the nation, e.g.:
"Right now, doctors a lot of times are forced to make decisions based on the fee payment schedule that’s out there. So if they’re looking and you come in and you’ve got a bad sore throat or your child has a bad sore throat or has repeated sore throats, the doctor may look at the reimbursement system and say to himself, 'You know what? I make a lot more money if I take this kid’s tonsils out.'"
Or what about the conceit that a generic MBA can manage anything?
MIT considered this the #1 problem for it to address in the '80s.
Of course not everyone wants to be pushed this hard, wants to be part of something like this, and is paid enough to take the obvious abuse. It's just something nice to fantasize about.
* As an example of an "arbitrary group of people," consider the set of contracted engineers, contracted managers, government employees, and insurance company personnel that had to work together on healthcare.gov. These are people who were not previously on a team together, and who cannot be expected to be of a uniformly high quality.
Yes, those of with experience can usually spot this stuff early on. But that doesn't mean we're in any position to do anything about it. Most often, software projects come pre-populated with people and institutional cultures. We don't get to pick the people and design the cultures from the ground up. At least not when we're contracting.
> If that is the case how do we convert that intuition into something that can work in the real world?
If you're building a startup that sells a product, then you hire good people and foster a healthy company culture. You're not beholden to any one outside entity, so you're not forced to import any such entity's preexisting pathologies.
If you're contracting? I think you just have to accept that projects will involve destructive people and cultures at times. Sometimes they'll cause projects to fail. In that case, the best you can hope to do is to protect yourself and your reputation.
A good product a day late is still a good product. A bad product on time is a world of hurt not just for the programmers, for sales, the users, getting next projects etc. In the end it is good project management and programmers/contractors that can build a solid product, not just deliver on weekly Agile tasks. In the end the quality and product result from where people place their demands. If it is time as the main demand, over shipping a good product, then we know what happens.
If you have an excellent team with superb engineering practices, why would you be reading up on agile anyways?
Organizations rarely want to change a working process willy-nilly, they only go shopping around for a better if the current thing doesn't work.
When the estimating meetings happen without you and you're told "we think this will take you five story points" (that is, two and a half days) and then the build is broken for three weeks (because of all the people checking in code in a panic) and you're called on the carpet for being behind, then it's an atomic clusterfuck.
You're in a circle of managers and PMs.
"We're concerned about your burn-down rate."
"I'm concerned about your ability to manage a project more complex than calling the elevator to your floor."
"Are you going to be done . . . today?"
"Is the build going to fucking work . . . today, or any time in the near future?"
Then the circle jerks (none of whom has written a working line of code in five years, yet they've somehow "shipped stuff") gabble amongst themselves for a minute, look confuxed, dismiss you and call in the next developer.
"Spartacus, you're next!"
This'll be good.
We are a community of motherfucking programmers who have been humiliated by software development methodologies for years.
We are tired of XP, Scrum, Kanban, Waterfall, Software Craftsmanship (aka XP-Lite) and anything else getting in the way of...Programming, Motherfucker.
We are tired of being told we're autistic idiots who need to be manipulated to work in a Forced Pair Programming chain gang without any time to be creative because none of the 10 managers on the project can do... Programming, Motherfucker.
We must destroy these methodologies that get in the way of...Programming, Motherfucker.EDIT: To clarify, his style is incredibly annoying.
It can be hard to find a balance when communicating something you're passionate about.
Any significant programming project involves more than one person. Being arrogant is probably the most serious flaw in a programmer. You will be wrong. You will under or overestimate something important. You do have blind spots. Most important, you are not as smart as you think you are. If you are not careful, your mistakes will ruin your project. They'll delay you or create liabilities you'll have to deal with.
Methodologies exist to help you manage yourself, not for your managers to better manage you (although they can be misused for that).
You don't have to feel "humiliated" by them (neither methodologies nor managers). You don't have to be insecure.
I cringe in embarrassment on his behalf: http://zedshaw.com/#/fuck
1. Using user stories to define your requirements does not make your process agile, just like writing a traditional requirements doc does not make your process waterfall. It's not about the tools. If you're not capturing the requirements you need to capture, your process is just broken.
2. Design documents do not magically make your project successful. Agile doesn't like design documents because it typically uses more lightweight artifacts to capture design decisions--but it still captures them! Again, if those decisions are not captured at all, your process is broken.
3. Wat? Asynchronous vs. synchronous is a project design decision; how does this demonstrate the failure of agile methods?
Multiple sources ([1], [2], etc.) have reported that the requirements for the system were not known until Spring of this year, and that they were in flux until weeks before the release. That's not a problem that would be fixed with use cases, UML and design documents.
[1] http://www.nytimes.com/2013/10/13/us/politics/from-the-start...
[2] http://www.huffingtonpost.com/2013/10/22/obamacare-website-p...
To say it failed "despite agile practices" misses the mark. It's like blaming a roof for leaking despite the use of hammers. Process is a tool, and if used properly, it may be a solution. But pixie dust, it is not.
Building a database to track the communications of ever single person in the US and ever foreigner they talk to? Piece of cake.
Building a database to track the health insurance policies for a subset of Americans? Over budget, behind schedule, still doesn't really work yet.
Priorities.
When it comes to algorithm design, reverse engineering, or crypto, I know exactly who to call when I need help. When it comes to a layout that works, or in many cases, even just coding up some fairly simple HTML with CSS, my NSA buddies aren't in the top 50 of people I'd call.
Yes, they're brilliant. The best of them though, produce websites (if they're capable of coding on the web) that look more like stallman.org than anything like healthcare.org.
Flip side is, doing this in the black budget does allow them to avoid some of the usual government contracting insanities. While that's no gaurentee of success, e.g. look at the NRO's recent expensive stumbles, from the dozen years I spent Inside the Beltway I've gotten a strong impression the intelligence community does a generally much better job than run of the mill open Federal IT.
That's a reasoning that's simple, easy to understand and entirely wrong. Contrary to what the evangelists' tell us, process has small (I'd almost say minimal) influence on software development failure or success.
The responsibility is on the lead project manager. He's not been good enough. A failure on a project of this size doesn't just happen overnight. If you just look, you can see it coming from miles away.
The solution will always be highly uncomfortable though. You'd have to stand up to powerful people who hold authority over you, fire sub-contractors or employees, significantly reduce scope, move deadlines. Big, high impact actions that won't be popular, hell they might even get you fired. Not a job for the faint of heart for sure.
It's depressingly typical of big corporations and organizations, to allow people not capable enough to be put in charge of such large projects. And it's even more depressing knowing tax dollars are being wasted in such huge amounts on relatively (big emphasis) simple software.
There is no prohibition in agile from using UML, and the process is certainly not hostile to design. It may be that some people believe that using agile implementation processes relieves them of the responsibility to design their software, and that it's OK for user stories to be generated off the top of their heads without analysis. Those people are mistaken. Nowhere does agile promise to design your system for you.
As a user I would like to be able to use the Healthcare.gov with at least 50.000 people or more at the same time
I think it is the same deadly cocktail of empty slogans in politics that meet the naked truth of booleans in software that wreaks havoc among government IT programs all around the globe.
We should ban the use of booleans in any governement project from now on. Politics can't handle booleans.
The details of the methodology don't matter. If you look at what http://semat.org/ is proposing for the standard engineering methodology for developing software you will find that it IS agile, but it is not specified down to the nth detail and more importantly, it is extensible and adjustable so that any significantly sized project can build their own compliant methodology and process within the SEMAT framework. That part is so often missing, where the developers themselves discuss how to go about building the software, and management listens to them and defers to the developers because they are software engineering professionals.
Contrary, a minority of national-level projects that succeeded here were always almost things that were initially planned to be very small, experimental, with no hard deadline and implemented without a big-bang in media. The biggest distinction is that the small systems tend to be put in production only partially but much earlier, often as an opt-in, working in parallel to the old often paperwork based systems. So there was no huge switch to something new and the risk was pretty small. This is what agile means to me.
The didn't need a bunch of "nimble", independent teams. They needed some early, stable requirements. Some early, hard deadlines that they had to hit. And a focus on making the fucking back end work, first.
The last thing this project needed was some sort of last minute "agile". Of course, now they'll attempt some variation of it, out of necessity.
Corporations act like using the word 'Agile' instantly brings the benefits of flexible planning and proper management to a team.
Most people never really interact with a proper agile process - its kind of a unicorn in the development world.
The failure is not necessarily the process but the managers not understanding that it isn't a silver bullet. It is often used as an excuse to delay requirements gathering or understanding of requested features from stakeholders. This leads to churn and failure.
These were security applications, real time data backup and mirroring services.
Agile in the wrong hands (or heads) doesn't work and in many cases is just a buzzword.
First, agile principles don't imply that an agile approach will always succeed where a waterfall approach will fail. It merely asserts that agile increases (not guarantees) the likelihood of success.
Second, the article's three sources didn't "proclaim that HealthCare.gov would not have failed if it had just used “modern” software practices, called agile development". The closest that comes is the third source that says, "None of these missteps would have occurred if the contractors had taken a gradual, agile approach." But in general, the articles were arguing that an agile approach would have improved matters, not guaranteed success.
Third, agile is a spectrum. People attempt agile within larger waterfall restrictions all the time. It doesn't work very well, but it still might work better than being 100% waterfall. But it doesn't change the fact that this was a project with guaranteed high traffic on launch day and a hard deadline, and a prohibition to deliver a smaller MVP early. That right there is incompatible with the purest forms of agile.
That's not my understanding. I understand Agile to assert that it's better able to accommodate changes to the project during development.
Talk about process all you want, but everyone is just waving their hands around spouting buzz words. A technically experienced (in large scale systems) management team can make sure the project is done right regardless of the process they use.
What I wonder is how many political types are still telling the new fix-it czar, Jeff Zients, what he can and can't do. I.e. right now all features that can possibly be punting into the future should be.
Although he said his #1 priority was to stop sending garbage to insurers, which isn't particularly political at this point, one would hope.
"I have seen some of the developer documentation, and it clearly discusses sprints, user stories and incremental testing — all of which are hallmarks of an agile process."
For a project with reportedly 50+ subsystems, where the main failures seem to have been integration, that doesn't tell us much either about this project or agile practices.
Getting tired of 'agile' or 'XP' getting the blame when in reality they are half-assed versions of some scrum masters cook up.
No company in the club of companies governments hire does it. There is no agile way to cut through all procurement rules, all documentation requirements and all miscellaneous bureaucratic processes (required in order to prevent government funds misappropriation) present in any big project, much less one so publicly examined.
I suspect that the answer is none.
But if I'm wrong, it'd be interesting to know what tools they tried to use and where there processes worked, if they ever worked at all.
Is there an aspect of this project that would be resolved with a specific version control system?
These aren't rhetorical questions, I'm genuinely curious why you're asking.
a symptom of a more systemic problem.
We need a real post-mortem on this process. The congressional inquisitions aren't getting to the root of the problem. We just need the individual who managed this project to explain, from square one, how the project flow went.
(Believe it or not, I intend no sarcasm with this post.)
that didn't help T.T
I am perhaps naive in my belief that they at some point had a combonatorial solution that worked at scale.
so i am waiting for someone to say "we are rolling out a leaner version of the site till the other more major issues are worked out."
Instead, it feels like the only bathroom at the mall has something vile leaking under a door marked "out of order."
So I won't critique ACA/Obamacae, the website or the development process. That's pointless. There are people who, just as in the case of various religions, will stick to their belief system no matter what anyone says. It is only when something external and utterly impossible to argue against presents itself that some of these people are willing to reason. So, none of that talk. It's pointless.
In thinking about this whole healthcare/website issue while kayaking (great way to unplug and just think) I came to the conclusion that what we are seeing unfold in front of us is of incredible value. In many ways, up until this ACA fiasco became front page news everywhere most US citizens dealt with government incompetence by simply ignoring it. It's one of those out-of-sight-out-of-mind deals. And, while most probably knew that ugly things have been happening behind closed doors and underneath each and every bill passed (regardless of party affiliation) most voters had better things to do than to stop their busy lives to go kayaking and critically think about these matters. Having solitude to just think is a rare commodity these days.
What the ACA has done in a major way is open the doors so we can all see the sausage being made. Plain and simple. We can see government incompetence in action. You don't have to be a programmer to recoil a the idea that a bad website that does not work cost taxpayers $600+ million dollars. And, you don't need a degree in computer science to probably guess that the final cost might reach double or triple that. And you don't need to even look at the code or know how it was written to start to understand that it is probably a complete mess and that three hundred million people are going to be forced to provide their most intimate data to this ill-conceived and ill-executed monster.
Let's not even discuss the tens of millions who are predicted to suffer significant collateral damage due to the the letter of the law and the intended or unintended consequences. I don't care who you are. Write a law consisting of 30,000 pages and an ever-expanding set of rules. Don't take years to debate, analyze, simulate and evolve it. Then go and try to implement it. Well, there's an absolute certainty of having huge problems.
Again, there's no partisan bend here. I am not arguing for or against the contents of the ACA. I am simply stating that something so massive that was created as the ACA ultimately was is almost guaranteed to be a minefield (or have one hidden inside somewhere).
Imagine writing 30,000 pages of code in your favorite language and not testing it until you release it to your customers. I know, I know, this isn't a perfect analogy. I am just using it to stress the point that something that large, without suitable testing and "quality control" is bound to be a mess.
Yet government operates this way. And we are the ones who get to eat what they cook-up. They certainly don't. Virtually nobody in the very top layers of this country's government has to live by a lot of the laws they pass. The Supreme Court certainly does not. Neither does congress or the office of the President. That should tell you something. They like to talk about the 1%, yet nobody talks about the 0.003%: The top 1,000 people in government who, to some extent, get to live above the law. The easiest way to have people not focus on you is to create focus elsewhere.
The ACA, from my admittedly Libertarian, perspective is great in that here we have something that irrefutably exposes how bad it is to put almost anything in the hands of government. This is not what you probably heard in school. This is not what you probably heard your professors proselytize to the captive audience of a classroom. To those who have never lived outside the security of government jobs or jobs where real-world consequences are, well, not real, their view of reality is vastly distorted from "real" reality. And that's what they spew out to our youngsters. And, because they are young and impressionable, they take it in and believe it. Very soon they are repeating the mantra without every having set foot in the real world.
Take this as an opportunity to watch and learn what reality looks like. If you are a reasonably accomplished coder you know, without a shred of a doubt, that $600 million dollars to implement the ACA portal --brilliantly or dismally-- is nothing else than obscene, if not criminal. A number in the range of one to, at worst, ten million (still sounds ridiculous) is what I hope most of you would think, no, would know, this should cost to do it right and to do it better.
Leave the details of the 30,000 pages and all of the collateral damage they will surely cause out of the equation. Just look at the sausage being made and form an opinion. Now realize this:
Everything. Absolutely everything in government. Is done. JUST. LIKE. THIS. You simply don't get to see it every day. Nobody reports on it. And it becomes water under the bridge. Now you can see it. Learn from it. Does this seem sensible to you?
Now, you tell me why this is such a great system because obviously I must be missing something. Because US as the only country from the list above has free-makret system while the others are 100% taxation sponsored, a.k.a. free. Tell me. Dying to find out.
What I am highlighting here is that, due to the ACA, for the first time our nation is seeing in all it's splendor how things are done in government and how ugly all of it is. Add to this the horrible NSA stuff and other issues and, well, I just can't see how anyone with a true and solid moral and ethical compass could be for more government control of our lives rather than less.
I want younger HN'ers to realize that this, exactly this, is how EVERYTHING, in government is done. It is wasteful. Incompetent. Inefficient. Intrusive. And it should scare all of you.
Take the US budget and imagine that nearly every program and every line item is executed in exactly this fashion or worst.
What do you think we could have done with six hundred million dollars. I am not talking about the ACA. I give you six hundred million dollars to spend --efficiently-- in anything you want. Would you burn it this way? Would you spend it on a pointless war? I would hope not. Only government can pull off something like this and then go on TV and make it sound like they are doing a good job. In the real world people would get fired and a number of them would face fraud and other charges.
Again, I don't care about what you want to discuss. For all I know you are absolutely right. I am simply using the parade of insanity marching before us --all sides, all political corners-- to illustrate that for the first time we all get to really see how things work.
That's it. It's the only take-away.
https://www.cia.gov/library/publications/the-world-factbook/...
Talking point ignoring:
Different countries can have different populations with markedly different health characteristics. That is, "race" in the sense of genetics; in the US, not just African-Americans but plenty of mestizos with a lot of Amerindian genes. Not that I know anything particular about what the latter brings to the table, just that it's sure to be different ... and, heck, the former is a unique population, right?, and neither are in large numbers in any parts of the 1st or formerly 2nd Worlds.
Different countries use different methods, e.g. to my knowledge no other country in the world tries to save premature etc. babies as much as we do, and scores failures to do so as deaths. Neonate deaths are a very major factor in overall life expectancy calculations, since so much potential lifetime is lost.
Those who are interested in bourgeois truth look at things like life expectancy after being diagnosed with X type of cancer. And even then you have to adjust for different populations.
And you compare us to Japan, as genetically homogeneous as you're going to find after they disconnected from the rest of the world and most kept that in modern times??? Please....
Plus plenty of those systems you refer to have free market characteristics, which you weasel with by saying "100% taxation sponsored" ... which I'm not sure really includes any of them but Canada, and they have the obvious fudge---part of their official system, too---of crossing the border. (By that I mean they're the only ones I know of to outlaw private healthcare.)
Of course, you might have bright spots here and there. Like premature babies are so common for women after 30s and 40s that sure you have to do something about once women started to work and be pregnant much later in life. It's not like we need it as much in Poland where still majority of pregnancies is for women in their 20s.
All I'm saying you are 51 on the list. You may have all kinds of excuses for that, but it's very, very weak.
In my experience he blames one or more of his subordinates. "His process" implies ownership of something responsible for the failure.
Shifting blame is pretty much by definition a core competency for bad project managers, it's that, get better or get fired.
Right now working on "sprint zero" which exists to 'determine deliveries'. I mean they still operate it as they have always done. They just stole another buzz word that sounds cool. Yes Mr. President we do agile. And because Mr. President doesn't know what agile is, the agency doesn't know what it is, so we can continue business as usual, just make sure that our contractors speak sprints instead of iterations. Stupid.
dumb + agile = agile dumb