A $300M IT flop
fcw.com
fcw.com
When I was consulting, I responded to a few RFP's but they were always either (a) obviously tailored to the capabilities of a single firm so there was no way to win unless you were them or (b) so ridiculously vague or out of this world impossible ("We need you to basically rewrite GitHub for us and the budget is $30k") that it wasn't even worth the time. Even now where I work, we have to deal with government agencies literally telling us they can't tell us any requirements at all because that would give us an advantage over other firms somehow. We just have to pitch our product to them and hope that it fits, even though at this point in our development cycle we could (and really, want to) make adjustments, feature additions, etc. to fit them if we were allowed to.
Government contracting in IT, from my point of view, is nearly 99% about "playing the game" and about 1% actually delivering something. Especially since it seems that even if you fail spectacularly, you still get paid (possibly paid even more to fix your stupidity).
Any small competent development team can built almost anything imaginable with 6 years of time, proper funding, and without unnecessary restrictions.
$300 million to develop anything is insane. I can't imagine what happens when the check is cut, "let's hire 100 programmers, 10 managers, 5 accountants, 10 support staff and they'll figure it out."
Maybe there needs to be some kind of "insurance" company that oversees the project and has to pay back a portion of the money if the project fails. I'm not sure if that's reasonable, just an idea. If the project starts going off-track then they start letting people go and find competent replacements.
Plan projects out, have milestones, goals, backup plans. If they spend 6 years and $300 million dollars on a computer system and fail then they were just winging it from the beginning. The managers responsible should be identified and go on a blacklist. Right now it's either "oh well, hopefully we'll do better next time".
You have a prime contractor that then subcontracts the actual work to small companies and then the prime just handles project management and the government interface.
However, this ends up causing problems with the motivations of the individual actors not lining up.
I worked at one for a while, we were held in very high regard amongst the Government. When they doled out new contracts they basically asked us our opinion for which company to choose, based on past performance. We would also review other contractors work and give it the thumbs up / thumbs down.
This was on a much smaller scale.
http://en.wikipedia.org/wiki/List_of_federally_funded_resear...
How much staff do they need to make sure everything is okay? Oh you have a database? Well you need a DBA on this. Security? Who is making sure the application is secure? Where is the security officer? Etc. etc.
Many of the boxes can only be checked by having more people.
This is true for almost all business, especially consulting work.
The "official" process is likely a very difficult set of hoops to jump through and you're not going to be promoted in that company more than likely, and fired at the first opportunity.
Unfortunately, I did and this was my first professional job out of college. Morally I made the wrong call by not speaking up even though truthfully I don't know if what happened was actually illegal at the time or not.
We had a competitor who had copied a bunch of our stuff, including our typos, but they were missing one essential feature. Yet they still got the bid somehow. A mystery for the ages.
Add in special consideration (the owner of the company I worked at was a woman and my state gives preference points to woman-, minority- and veteran-owned businesses) and it was about as close to a no-bid contract as you can get.
It's absolutely disgusting that he'd craft his bid to the supplier he knows isn't taking shit, right?
<citation needed>
So what's an administrator to do in that situation? Take the "best" bid, see the project go over time, see the contract and specification disputes start, see the costs go up, and end up with a 300M failure after 6 years?
What you call "corruption" and "a lack of accountability or transparency" I would call "exercise best judgment and experience" to save the taxpayers their money. Basically, a procurer doing his job.
You would have some point if software were more amendable to requirements specification and bidding, but the entire industry is still terrible at it, so it's a bad process, because it doesn't actually promote "accountability and transparency" at all.
And to your earlier comment, you are assuming both
> your company can deliver on time and in budget, with little fuss
and
> your competitors are known to underbid, over-promise, deliver very late or never and discuss endlessly over contractual fine language when they start failing.
This may very well be the case in other examples of this going on, but I can assure you neither point was true in my case (but is obviously specific to me, the plural of anecdote is not data, etc).
To be fair, what that really means is that the contractors that they want to win already know the requirements, and that's the closest way they can get to impartially telling you to fuck off.
Generally, that response connotes that the bid is tailored to contractors already doing work with the agency in question, who are not only intimately aware of the request for proposal, but likely had a hand in its creation.
For what it's worth, this isn't necessarily corruption at work.
As I've related on HN before, the procurement arms of these agencies are often woefully unequipped for handling anything IT related. It's typically a process made for sourcing completely interchangeable, relatively simple supplies or services seeing terribly complex software or systems dumped into them - not 'shoe-horned' as that would imply an attempt at fitting - simply dumped so that requirements spill on the floor and everything gets jumbled up onto the existing conventions.
Given that, providing an agnostic SOW is a great way to end up with stuck with a lowball bid which makes the most self-serving, word-splitting assumptions possible to stick you with something utterly apart from what you need.
>'(b) so ridiculously vague or out of this world impossible ("We need you to basically rewrite GitHub for us and the budget is $30k") that it wasn't even worth the time.'
Related to what I said above - these requirements are not as hard and fast as they might seem. Writing a defensively worded response, getting your foot in the door to talk with the people who actual request and implement is not a terrible way to start doing real work.
>'Even now where I work, we have to deal with government agencies literally telling us they can't tell us any requirements at all because that would give us an advantage over other firms somehow.'
Right.
It's not unlike any sort of courtship, business or otherwise. You can't just ask outright, you have to make friends and hear the information gush over lunch or volunteer some no-fee consulting and take notes.
Big vendors and VARs have people who do this nearly full-time so it's certainly much harder for a small company, but not impossible.
>'We just have to pitch our product to them and hope that it fits, even though at this point in our development cycle we could (and really, want to) make adjustments, feature additions, etc. to fit them if we were allowed to.'
Something to be aware of, when dealing with a Lockheed terms like 'adjustments, feature additions' are euphemisms for 'one million dollars'. If you're pitching these possibilities and expecting/willing to make them to earn the business then it might pay to be explicit about it.
> 'You have companies like Lockheed Martin -- they're not web development companies, they don't do work in the private sector, they're not building start-ups,' Gershman said. 'There's very little in the market that forces them to adopt new technologies.'
All of these are true, but I think this glosses over some fundamental structural problems with government contracting. The SSA has almost 60 million beneficiaries, and deals with distributing an entitlement that is extremely sensitive and protected by layers of legal process. 8,000 Social Security checks get lost,[1] and its not a blog post, its national news.
Moreover, a lot of lean and agile processes just aren't consistent with political reality. There's no minimal viable product, there's no invite-only beta testing, there's no iteration, there's no A/B testing, etc. Technical decisions ("let's just roll out to northern Alabama first") become divisive political ones. Purchasing decisions ("we've had good luck with this vendor before") become the basis for accusations of favoritism and corruption.
Perhaps most importantly, there's no failing fast. The private sector accepts the fact that many engineering projects will fail.[2] Good companies take care not to punish failure unreasonably. But in the public sector, failure is just license for those opposed to project to crow: "we shouldn't have been doing this anyway!" If you fail fast, it's easy to pin the blame on you. This creates a perverse incentive to drag a project along so failure happens over years if not decades.
[1] https://medium.com/@jan.curn/how-bug-in-dropbox-permanently-...
[2] Most big IT projects fail: http://www.computerworld.com/s/article/9243396/Healthcare.go.... Most startups fail. Most internal prototypes never amount to anything. This is just how engineering works.
You are right, the development team never had a real chance at achieving success.
Congress writes the bills that somebody (higher than GS15s?) turn into implementing regulations that GS-15s turn into program requirements. If Congress writes bills that are simple, clear, and short, then you have a chance at simple implementing regulations and therefore simple program requirements (not certainty, but a chance). But if Congress writes convoluted, lengthy crap, it's almost impossible to write a program that will correctly implement it.
Yeah, I forgot to mention Congress and the SES(the higher than GS-15) guys. Yet, even when there is a simple set of program requirements, the guys I already mentioned are going to keep adding shit on until the very end. If you give a high-ranking government employee authority over an IT project, they start imagining that they are Steve Wozniak, Bill Gates, Johnny Ive, Steve Jobs, and Leonardo DaVinci all rolled into one. Then bad things happen.
This is not unreasonable. After all, projects in the private sector are funded by private funds. If you fail, you only waste your own money. Whereas public projects are funded by the taxpayer. When they fail, what's really happening is that taxpayer money is going into the pockets of some private company, but there's only a failed project to show for it. So yeah, of course the backlash is going to be much stronger.
The difference is that smart investors/owners are aware that the possibility of failure is inextricably tied to any attempt to excel, and allows people to try things that might fail in the pursuit of success.
In our current political climate (and probably in other climates as well), governments are dumber, and tend to drive their staff to manage towards over-safety and covering of asses rather than towards best outcomes.
Lockheed has their own set of problems, mostly hinging on the contract they've signed with SSA. They are forced to hire a certain number of subs per their contract - hiring a percentage of subs is not unusual, but the level in this contract is very high. As such, the subs know they can force feed trash into LM. The resumes coming from the subs were complete bullshit. Things like "10 years of experience administering X", but then in the interview, you find out they know next to nothing. Especially bad with databases when you even try to ask them how to write a join. One meeting we had while I was still there was a "skills assessment" - from 1 to 10, how well do you know X? There was no negative to this meeting, we just wanted to know who knew what. The number of "10s" in that meeting was beyond belief.
So sure, Lockheed may not have a lot of experience here, but that problem is exacerbated by the stakeholders and the contractors they are forced to hire.
On DCPS, the problem is that LM is forced to hire 45% (FORTY-FIVE) of the subs through one specific subcontractor. That company had a previous contract with SSA, and SSA liked them. Since the sub essentially lost their work with SSA to LM, and they know LM was contractually obligated to hire through them, they stuffed us with a ton of below average folks.
On the face of it, I agree with the idea of a large contractor having to sub out some of the work to the smaller players; it's just that LM overplayed their hand trying to get this contract, and got screwed for it. But that's another story on how LM management is trying to make revenue numbers for bonuses...
I always wonder when I hear about these projects, where exactly has the money gone? It can't possibly be in hardware, that would make some of the biggest data centres in the world just for the NHS. I can't even see how it can be manpower, how many IT people can you get for £10bn? And surely if it were corruption, someone would notice so much money being squirreled away...
There's just no logical explanation behind these things.
http://www.amazon.co.uk/Plundering-Public-Sector-Letting-Con...
[NB Note recommended for UK taxpayers who have high blood pressure and/or easily upset]
The biggest problem, though, was the lack of pushback. The government officials would ask for the moon, and the vendors would promise to deliver it. There was almost no one saying NO - the word that project most desperately needed to hear. Features outweighed core infrastructure reliability, so configuration management was a nightmare.
Generally, the disconnect is that management is under the assumption that you can manage your way to success, so they add more overhead to watch and prod at the proceedings. Development is almost always an afterthought in most business projects. The first thing they do? Get consensus, which is generally everyone putting in ideas so that they seem involved—see Atwood's Duck[1]. After you have a ton of often contradictory ideas, you need to have them prioritized. And since nobody managing the project is technical, they need to hold countless "prioritization meetings with stakeholders and implementers." Often times the meetings will grind to a standstill while process analysts and engineers argue over the wording of things. Usually the analyst will tell the engineer that the business has standardized on a set of terms, of which are not the trade names for things. Maybe everyone uses a set of terms which aren't the "correct" wording.
After things are prioritized based on the wrong criteria, they're then pushed off to an engineering group who are tasked with implementing things exactly to spec. Since the definitions are incorrect, they need to constantly circle back and check with the stakeholders to double check. Occasionally checking in will involve a "subject matter expert" who is on vacation, and the only one allowed to answer the question. (Thus stalling that item and anything depending on it.) Others will often say "it's above my paygrade" in an attempt to avoid culpability.
After the project has run long and only has a majority of the features complete, people are reassigned to other projects, and the app is sent to QA to hand-test features. Sometimes it will involve the "stakeholders" sitting in a room for days or weeks, manually clicking their way through the interface. Maybe the project is declared a boondoggle, deleted, and replaced with someone else's pet project.
So yes, not logical.
The tough reality is that certain projects have a scope such that a small team could never reasonably be expected to capture the requirements of the project. Also, remember that large enterprise projects are generally built by people who are not users themselves. This is where the technology crowd often draws false conclusions, reflecting on projects of their own, where they have intimate knowledge of the problem space. There are likely very few forward-thinking engineers who also have visibility into the intricacies of the various SS systems. This is where the layers get piled on and the efficiency vanishes.
Once a organisation gets to a sufficient size, organising and allocating resources and communication starts to eat up a disproportionate amount of time, if the organisation is left to drift into whatever shape it likes.
From personal experience in software development, dilution of responsibility can be incredibly expensive.
Scenario 1: I wonder about a minor detail, call up a random user, we hash it out in 5 minutes. Cost is about 10 minutes of total labor, figure maybe $10
Scenario 2: I have to organize a meeting with the exec stakeholder, the dept manager, and the entire dept staff. Which due to scheduling conflicts is in two weeks. Finally the meeting arrives and due to bike shedding we spend 2 hours talking about everything other than the topic until its hashed out. So the cost is two weeks of my salary doing nothing but surfing HN, lets say $3K, plus 2 hrs of mgmt for exec at $100/hr and the manager at $75 thats $350, plus double that time for them to make the "command decision" and later inform me, so thats $700 of management, plus the ten person working team has to burn an extra 2 hrs of overtime each at lets say $25/hr and 10 people that's $750 so we'll call it $4450.
So figuring out the default value, if any, for a blank of a form can cost anywhere from ten bucks at a personally responsible organization up to about five grand at a fully diluted responsibility organization.
Its obviously orders of magnitude more if you involve consultants, people flying in from out of state, etc.
This is the bit that jumps out at me as clear indicator of the sort dysfunction that dooms projects like this more so than anything else.
It's indicative of a the classic exploding scope and fragmented responsibilities leading to an overall lack of accountability that unfortunately characterizes IT in and around Government.
Gershman seems to agree.
"If there's not strong leadership, problems arise with coordinating work between contractors, and there's no one person looking at the system as a single entity -- everyone is making sure their piece is done, and fulfilling their responsibility up to a certain point,"
However, I'm not wholly keen on the the start-ups as a solution direction taken later. It's not that it's wrong, just beside the point unless the start-up savior is specifically targetting culture and process change as opposed to technical implementations.
The 54-armed kraken of a defined by committee project will drown and devour a fleet of confused start-ups lost in the mist of *BE requirements and contract mods just as readily as the Lockheed leviathan.
The filthiest part is that nobody but the taxpayer is interested in things being more efficient. The government agencies don't really care because there is no real financial accountability. Besides, if they use up all their budgets, they just get to ask for more next year. And contractors generally have their profits limited to a percentage of the project cost, so their goal is to balloon the cost as large as possible.
For several years I worked on a very large government project. When you're sitting in a two-hour, 100 person, sprint review every two weeks and everyone is fucking off on their phones the whole time, it literally makes you question your sanity.
But the system is rigged that way: if you do decide to hold them accountable, you'll find that they have documented every little specification change that the SSA put in after the project was awarded. So now the blame comes back to the SSA. Both sides will point fingers at each other; the consultants on this project will laugh (on their way to the bank), and things will remain the same.
In other words: the system of contracting in place today is broken. From labyrinthine requirements to bid for one, to obtuse specifications to lack of oversight; it's one huge mess.
And the system is corrupt to boot. Primes like LMFS, etc. employ former employees of these agencies (and vice versa), so it's very hard for an outsider to break in.
http://www.theguardian.com/commentisfree/2011/aug/03/nhs-dat...
> ...final cost to be as high as £20bn, indicating a cost overrun of 440% to 770%
I have worked on project like this. Usually it's the sign of management failure (poor communication, poorly defined requirements, bad governance, inability to retain knowledgable employees) rather than technical feasibility. It's not like the software is pushing technical boundaries.
"Echoing McKinsey, Gershman said that without a strong leadership element, IT projects of this magnitude are almost doomed to fail from the start."
This is a big problem. Because employees know these problems will fail, the smartest avoid them like the plague.
Ahhhh
Says the cynic in me...