NY Pays 230 “Consultants” $722M For Project 7 Years Behind Schedule | Hacker News Reader

NY Pays 230 “Consultants” $722M For Project 7 Years Behind Schedule

democracynow.org

83 pointspauldelany64 comments

UPDATE: Corrected misleading headline from article. $722m to date, not per year. Average consultant earns $400k / year.

Very common situation in government contracts and dealing with consultants in general. I am sure IBM and Accenture are in there some how. NYC government could have created a small crack-team of internal IT people who could build the system for less than a 5th of the current cost. Unfortunately most organizations view IT work as something fit for mercenaries and if you ask any King who has ever had to rely on mercenaries, they'll tell you it should be your choice of last resort.

The thing is here, no-one is incentivized to deliver. The bureaucrats who specify the system don't really care as it's not their money, and they justify their positions by sitting in meeting and writing reports that only each other will ever read. Then there are the consultants who are paid hourly...

Typically it goes like this:

Bureaucrats: Here is a 1000-page spec document that we have spent 3 years writing. We captured all the requirements at the beginning Consultants: We can do this project for $X. However it will take a long time and a lot of people for us just to understand your spec, so any changes to it will cost $Y. Bureaucrats: OK whatever.

Consultants: work work Bureaucrats: Wait, we've changed our mind about this! Consultants: lick lips

> The thing is here, no-one is incentivized to deliver.

Amen.

Incentives are everything, and if you look at most things without keeping that foremost in your mind, you'll see a hodgepodge of weird random effects ... but use that perspective, and most things make a ton of sense.

No, NYC couldn't have created a small, crack team for several reasons:

1) Government Software jobs pay less than civilian. It can take up to 6 months of paperwork to hire someone. While that paperwork is in process the job applicant must wait and can't be hired. Guess what kind of developers the government gets?

2) Requirements, they've never heard of them (sarcasm.) My last project had the language changed 3 times, went from a plug-in to a stand-alone app to an API, and during the 4 years had 3 different "final" decision makers. Feature creep was more like feature tidal wave.

3) Paperwork. There's massive amounts of it. One of my coworkers spent 1 month writing paperwork for 10 lines of code.

I could go on but the government does everything in its power to interfere with rapid development and then wonders why costs overrun.

> 1) Government Software jobs pay less than civilian. It can take up to 6 months of paperwork to hire someone. While that paperwork is in process the job applicant must wait and can't be hired. Guess what kind of developers the government gets?

Which New York City law mandates this? (not doubting that there may be one)

Hiring someone for the New York City government can take up to 6 months? What is it typically? (which is the only thing we should care about here without reason to believe it would be atypical)

>3) Paperwork. There's massive amounts of it. One of my coworkers spent 1 month writing paperwork for 10 lines of code.

Do you guys work for the city government in NYC? Somewhere comparable?

> NYC government could have created a small crack-team of internal IT people who could build the system for less than a 5th of the current cost.

So you're arguing that the NYC government is incompetent at running a project done by contractors, but it would be better at both FORMING and RUNNING its own team of contractors?

Sounds like nonsense to me.

Internal IT people would be able to "cheat" by talking to actual users and stakeholders, doing stealth requirements gathering, examining the current system and how it's used, doing rogue user testing, etc. Basically doing all the things that everyone agrees are required for success on this kind of project, but which in the real world end up being dropped or neutered by bureaucracy. E.g., your "expert user" is the manager of the users and doesn't know anything about how they do their jobs, and you aren't allowed to talk to the actual users. An internal person would be more likely to have the personal contacts and trust required to work around crap like that.

Yea the internal IT people know what the ACTUAL requirements are and how the current system REALLY works. The bureaucrats writing the requirements and other documents usually don't understand the nuances of how the systems work in real life and will fail to capture the details and idiosyncrasies accurately.

An internal crack IT team is the only way to make something meaningful happen. Consultants should only be brought in when there is no in house capacity/capability. The best way to make applications happen is to have the programmer intimately know the requirements by being on site and intermixed with the end users.

Yup, a buddy of mine works for Accenture and has been in NYC for 8 months now working on this project.

Yes a small competent team could build the IT system for a fraction of the time and money. However they would only be successful if the business processes were well defined which I'm willing to bet they were not.

New IT systems often come along with changes in process, a well defined IT project should produce a system that follows and complements those processes. Instead what we often get are "boil the ocean" projects that attempt to fix process issues under the guise of an IT implementation.

One reason companies hire consultants is not to outsource the project but to outsource blame and accountability to IBM, Accenture.

"How could the project fail? We hired some of the most highly recommended and expensive consultants. They didn't deliver."

What's that old saying? "No one ever got fired for going with IBM."

Yes a small competent team could build the IT system for a fraction of the time and money

A team that can do that AND is willing to navigate the treacherous minefields of internal politics at a government department doesn't exist.

^^^ This! I was involved on writing an invoicing system that ended up failing. The real intent of management was to try to bring more control to pricing using the project. Management was unable to negotiate this new process, so it ended up being an overpriced spreadsheet and was replaced a few months later when the company switched to Salesforce.

I don't agree. If they cannot control their contractors, there is no way they can choose and hire competent software engineers.

What they should do is just create an open bidding system for the work and make sure that the bids are well publicized, so that not only IBM and accenture will bid. And then they will pay the bid amount only when the whole system is ready and functioning according to specs and schedule.

Thus any cost overruns are not NYC's problem.

There is a million time keeping apps out there, I am sure there will be plenty of candidates willing to modify their app to the city's requirements.

I read Accenture (coughAccidenture*cough) and IBM and up-voted right away.

Hmm... sounds like an opportunity.

Juan Gonzalez says this is happening all around the country with cities and states converting to digital systems.

The consultants are being paid $400,000 a year. Nice. Because it is a capital expenditure no one knew what these people were being paid so you have to dig into the city records.

For what?

Basically a digital payroll and time-keeping system with features such as "biometric hand scanners to avoid city workers punching in for other workers" and digital timecards.

Anyone up for going after cities and governments who are doing this or looking to do this? It's taken 7 years in New York at a cost of almost $1 billion, I say we can do it for... oh, I don't know, 1% of that and in 1/7th the time?

The $400,000 dollars per year isn't all going to product development. $350,000 of it is to compensate the consultants for navigating the nightmarish government bidding process laced with procedural road blocks to newcomers, cronyism and incompetence. Been there and done that, would not do business with again.

I used to put together proposals for a huge general contractor as a marketing assistant. The completed government proposals usually consisted of between 15 and 30 three inch ring binders of information. It would take me 20 minutes just to load it into FedEx via hand truck from my car. (This would be for something like a military barracks, I'm not talking about skyscrapers or stadiums. Not to mention that it wasn't uncommon to have several rounds of RFP's.) Contrast that with a civilian proposal that would be two or three binders. Software is not construction, but I'm willing to bet the process of bidding is similar.

In other words, it's a fat payday if you can stomach the process, but few can. This is also doubly sinister in that it's a process that is going to be particularly distasteful to just the sort of person that would make a great developer; one who holds efficiency and ingenuity on high, etc... Which is exactly how you end up with terrible DMV sites and CIA, FBI and DHSA databases that are unsearchable, and won't talk to each other.

Addenda: I'm not trying to put people off. I just want to shed some light on exactly how much pain is here: A LOT. Maybe the game-changer isn't someone getting the contracts and doing great, but someone rejiggering the bidding process?

The best part is that the RNPs are designed to create EFFECIENCY in spending by creating a competitive bidding process.

I think you're wrong, and I don't think you can deliver on that promise. I had the opportunity to speak with someone in a similar sort of project in Canada who told me that one of the biggest costs involved in these sorts of projects are the legal expenses. Often times you have to clear all sorts of red tape that needs to be torn down. Lawyers aren't cheap, and laws will slow you down.

Another point to consider is that you're going to be building something that needs to last for many-many years and can take on new interfaces over the coming years (e.g. iPhones, web, portable devices, etc). You need to build something that is going to be durable, grow, and be maintainable for years to come. Along this point, you'll need to import, manage, and parse old data.

Did you forget the training costs for city employees and the programmers that will maintain these systems? How about documentation?

I know you're being flip and trying to make a point that the city is being milked (and they are), but there's a reason for this. These contractors are doing a difficult, high stress job with lots of visibility. Their compensation, though high, is understandable -- as are their delays.

It is unfortunate this is happening. Be aware, however, that the solution is not just a technology solution it is probably also a business solution--dealing with various agencies of the city and their processes. I bet most money is spent on meetings and business process consultants... [Edit: not to say it cannot be done cheaper, but may not be trivial.]

The problem isn't the product - the problem is the politics.

You would be surprised. A lot of these guys write the most horrific code you will ever see.

If you are a startup, leave government contracts for someone else to figure out. By the time it comes thru and makes you a boat load of money, you will be long dead (as a company).

This was said by PG or someone definitely smarter than me.

I do know of one Start Up that is making huge amounts of $$$ from the government: http://www.palantirtech.com/government

It seems that if you know how to navigate the red tape and can deliver a compelling product you can make millions.

My company as well sells about 80% to the government. Government contracts are not really that big of a deal.

I'd say that our commercial customers are quite a bit higher touch than our government customers. The government also seems to have a much better handle on their business processes than any of the fortune-500 companies we deal with.

The city is paying some 230 "consultants" an average salary of $400,000 a year for a computer project that is seven years behind schedule and vastly over budget.

This is misleading. The figures in the article are based on the hourly rates charged by the contractor that employs these consultants, not their take home salaries.

This is noted only later in the article:

The actual amounts individual SAIC employees took home are most likely lower than their stated rates, since computer firms typically take a cut of each consultant's charges. Nonetheless, these are breathtaking numbers.

This too is misleading because it suggests that the hourly rate is comparable to the individuals' salaries except for a "cut". In reality, this "cut" is likely 50 percent or more. 60-70% would not be unusual.

This is typical of newspaper coverage of "scandalous" IT contractor pay. The project may indeed be a disaster, but those consultants are not making nearly the money that the article suggests.

In consulting it's typical to get paid a quarter of your billing rate, as a rule of thumb. I've seen places where it's a tenth tho'.

As you ascend the ranks of consulting and get to the top 3 shops (McK, BCG, Bain) it drops down to 1/20th to 1/30th. That is once you take into account the disparity between number of hours billed and the number actually spent doing the work.

I don't know the figure now but ten years ago you were expected to bring in at least ten million in business annually to be considered for partner at Andersen Consulting.

And how is this justified, does the companies brand and support structure really outpace the value that consultant is creating by a factor of 20 or 30?

I'm not sure what he means here - the higher up you get, the better the ratio. For partners it gets inverted - they are making more than their billing rates because they're getting a cut of everyone below them's billing too.

When you buy consulting, you're buying a commodity. The premium is meant to insure you against natural variation, key people quitting, and so forth. Joe Accenture is meant to be interchangeable with any other Accenture consultant. A lot of the overhead is the extremely rigid, prescriptive methodologies that are supposed to make this possible. For multi-year projects, enough budget-holders believe that this is worthwhile for the entire Big Consulting industry to exist.

That type of strategy consulting is completely different from IT consulting. You don't "ascend" from doing IT work at Accenture or IBM into doing management consulting at MBB.

722m on a fucking payroll system and it's 7 years behind date. Have you ever tried to contemplate counting the amount of stars in the sky or the number of grains of sand in the world? Your mind just gets blown, because you cannot process the scale of information. The same thing just happened to me when trying to figure out how this is possible.

You're right, the scale is ridiculous.

Most of these consultants themselves are not making all of that money. They work for companies like Accenture which bill their time out at several hundred dollars per hour, but then pay the consultants much less.

For example, the 400k/year figure quoted only amounts to about a $200/hr bill rate. That's far from the worst I've ever heard on a project like this.

When I consulted via a firm, they regularly billed clients $200-$300/hr for the work of developers and operations folks who were getting paid ~$20-25/hr.

Granted, this was in fly-over country, but the gap between billable-hour and developer-hour is quite often massive.

The upside of this is that their customers are used to paying those rates. If you are self-employed or work for a small consultancy you can undercut them and still make a bag of money.

can you? My experience has been that large corporations, never mind the government, are perfectly happy to pay 2x or 3x the rates in order to deal with someone large.

This in a nutshell is why the new U.S. health care bill is really a stimulus package for the IT and healthcare industries.

Integration of systems can be a nightmare.

As the UK can attest, large systems fail from sheer complexity, if the politics, internal resistance, and silo mentality don't screw you.

One of the reasons SAP succeeded was that in order to implement the software you had to rearrange how your company worked - not the other way around, modify the software to fit your unique business processes. Yes, SAP consultants make a lot modifying the software but those modifications are constrained.

Enterprise software is lucrative but you give up the freedom to do what you ever want. Design by committee.

http://business.timesonline.co.uk/tol/business/economics/pbr...

Are you seriously suggesting that "complexity" was the root problem here? Rather than say cronyism, incompetence and greed?

edit: amused to see another comment on a different thread that uses the phrases "cronyism and incompetence". Did I subconsciously read that and copy it, or is it just the most apt description?

I'm sure incompetence plays a part, but two other factors at play are the fact that no one person really understands how things work, and when things start getting that distributed it's probably fairly likely that there are logical fallacies that have been built into the system and adhered to by strict processes for decades. It doesn't take much fudging for a bureaucracy to paper over these edge cases, the accounts shrug and say close enough, and the consultants are then put in an impossible scenario where different parties give them different business logic and no one has the authority or will to reconcile the issues from the top down.

Take a simple payroll system. Start with basic employees. Add in contract workers. And, this is the hard part, handle the exceptions - those who get paid more even though they are at the same level, those who get more vacation, those who have legacy benefits. Scale to an entire city.

Test. Because if people don't get their paychecks, you have a very angry group of customers.

Why does NYC need a custom payroll system? Government agencies are supposed to be staffed by employees who can be trusted by the public not to abuse sensitive personal information such as is contained in tax returns. Yet, these same employees are considered so personally untrustworthy by the government that it is justified in spending hundreds of millions of dollars to design a payroll system whose primary feature seems to be innovative techniques for preventing fraud. Is there any evidence that government employees are more untrustworthy than other employees so as to justify such a system?

I'm not sure biometrics is really worth it either, but let's not be naïve here, we don't live in a trustworthy society by and large. If you're hiring people on the scale of a city, you're going to get untrustworthy people. There's no use in them putting their head in the sand and then avoiding doing anything that might suggest that maybe they have a few bad apples.

Also, it's not as if anywhere near the majority of those people have access to any given personal information. How many people does the city employee... 100,000? Every single one of those people will use the payroll system. By comparison, how many people have access to your birth certificate directly... 50? 100?

Yeah. This seems strange to me as well. If you need biometrics to prevent people from reporting hours when they aren't even in the building, why would you even want them to show up in the first place. If you get them to show up in order to collect a paycheck that doesn't mean they are going to do useful work. I'm guessing the problem is that you can't fire public employees.

This is because you can't always trust management to report things properly. My girlfriend works at a liquor store (small business, <50 employees) and they use a biometrics scanner for their clock in/clock out. It is very efficient and prevents the problems I stated. If sizable groups of your organization are cheating the system and your organization is large enough, it's going to be difficult to figure this out.

Both of my in-laws work for the USDA and they have been complaining for quite a while about a new system that was also way behind schedule and really didn't solve the needs of the department anyway. It's a typical case of "gather some requirements, go off into a black whole and build it for a few years, than come back and act surprised when it isn't relavent anymore and half the features don't even work." There are a lot of companies out there making a lot of money for doing a really bad job, waste is everywhere, and unfortunately I'm not getting any of it :-)

Consulting: If you're not part of the solution, there is good money to be made in prolonging the problem.

or something like that.

There is a market for an off-the-shelf system or maybe customizable system with the same core that cities can purchase and install without the need to do development. Maybe with different levels of strictness as far as scanning in employees is concerned (biometric scanning vs. RFID cards, etc).

Most likely it's more than time tracking. It's also paychecks. And once you get into paychecks in government, there are LOTS of nasty little loopholes and special rules that apply only to this small set of people, etc, etc, etc.

How are these contracts being awarded? Are we talking public RFP's? If so, that should have produced competitive rates per project, no?

Unless these 'consultants' were already promised the job and the RFP was just a formality, as is in most city contracts.

> If so, that should have produced competitive rates per project, no?

What makes you think that those rates aren't competitive?

Govt projects have lots of overhead. Some comes from procedures intended to keep them from being cheated. Some comes from trying to make sure that the "right" people get the job. (Some sounds okay, such as"diversity", but is actually a cover for graft.) Some comes from all of the stakeholders. And on and on.

because reading this line is somehow not surprising at all..

"The problem is no one knew how much these people were being paid, because their salaries don’t appear in normal city records."

I've dealt with city contracts and it's a culture where the oldest members of the club always land the contracts, no matter what. Finding ways to bury the actual amounts through loopholes or channels is what keeps city officials at work. ha!

I've always said that 2 smart people can get way more done than an army of consultants in a shorter amount of time. (for the record, I'm an ex-consultant)

But this is business and this is why these companies make as much money as they do. When you are in charge of such a project and your ass is on the line who better to blame than a large Fortune 500 type of company (i.e. Accenture, Deloitte, IBM, etc). You can't get fired for failing a project with a vendor that is considered the "best" but you can for using "Jim's Custom Development Shack".

A friend of mine works as a trainer on this project and he does a great job because the software is so convoluted - you need a great trainer.

He likely bills $60/hr to a sub-contractor, who then adds in his likely $100 profit and bills SAIC (the main contractor on this project). SAIC probably bills the city $250/hr for my friend's services!

The numbers are guesses but I spoke to my friend and he said that the vast majority of the billing goes to the main contractor and subcontractors.

At any rate, its a big waste.

Government is a giant Principal-Agent problem.

http://en.wikipedia.org/wiki/Principal-agent_problem

corrected misleading headline - should be $722M to date, not per year.

State Government RFP opportunities can be frustratingly bureaucratic at first, but there are real opportunities for IT staff augmentation and IT project proposals, but there is a ton of competition for these limited opportunities. For instance: State of Texas has 400+ approved vendors on their roll, S. Carolina has 200+ vendors. Some states are more progressive and permit electronic proposal submissions. Others still require submitting 6 identical binders for each proposal plus a CD or USB drive will all documents in electronic form. Because they can.

You know what they say about spending other people's money.

There seems to be a huge opportunity for a website like ChallengePost to get local governments to post IT contracts online.

The story: http://www.nydailynews.com/news/2010/03/26/2010-03-26_city_p...

Accenture, one of the primary beneficiaries of this $722m, operates offshore.

They're now in Ireland after having been in Bermuda for the sole purpose of avoiding paying the US governtment taxes on their international business.

So in reality our tax dollars are going to fund a multi-national conglomerate that's making the spread between the $100/hr consultants and the $400/hr billings.

I can't wait to see the tricks these companies come up with to service the $100,000,000,000 a year that we just dumped in the pork barrel for health care.

Man it must really suck to be the guy thats only making 100k but doing all the work, and finding out that some guy in the next cubicle negotiated 600k, and he spends all his time dickin' around with the secretary.

Reply on news.ycombinator.com