NYC mayor demands $600M refund on software project
itworld.com
itworld.com
From $63M to $600M? I would have thought that ppl may have noticed when the costs doubled or tripled the budget. Almost 10x the original budget? That type of oversight must go pretty high up.
Clearly fraud and kickbacks kept this stumbling along much longer than a normal failing IT project would. I'm not sure why or how an expense like that on an overage like that on a major payroll system would not be on the mayor's daily agenda. The right people must have been getting those kickbacks.
I've been dragged into 'feature acceptance meetings' that literally dragged out for weeks when testing of a new chat component devolved into a days-long discussion on whether the graphic in the header (provided by the customer) was appropriate. The unspoken issue proved to be that they didn't want us to use 'barren trees' in the picture of their building. We switched the photo used from one taken in winter to one taken in summer. This took something like 400 man-hours to resolve, after which we achieved signoff, but at the expense of having met budget.
Another instance on the same product was feature creep to the extreme. We had originally won a firm-fixed contract, meaning we would build products a, b and c for x dollars, and if it cost us more, it came out of our pocket. If it cost us less, we pocketed the profits. The customer decided, on day 1, that they actually wanted a lot more features than were described in the project scope defined, and that we won a legally binding contract to perform. Out of accomodation, we changed their product to a time-and-materials contract, meaning we had a fixed markup on labor and a minimal markup on hardware / tangibles. This then involved them dragging the project out for years longer than anticipated and signing invoices all the way, then griping at us on the budget near project completion and asking for a bunch of free labor (which is illegal under their new contract). The problem in that case was that the project management and senior execs changed repeatedly. We were at first working with their Office of Science and Technology, then with Information Services, then with Public Affairs. Each of these department changes involved swapping our equipment out so that we could have appropriately charged-back equipment to the appropriate department, which meant weeks of non-work. Each department had a different spin on what they wanted out of the project. We spent 8 months and thousands of man-hours implementing a very elegant services oriented architecture that could surface all of their legacy systems to a single web portal. Then we changed departments, and they literally told us to turn that feature off, and instead focus on a visual redesign. We implemented a single sign on service that we integrated with our product, but could be extended to work with literally every system in their enterprise. It was abandoned due to a department change.
During the first phase of the project, we'd worked with the customer to build out their data center. They wanted a new data center because the division that owned the project was going to be adding a bunch of other programs to integrate with what we were building. They spent over a year on construction, converting a service garage into a data center. They bought millions of dollars worth of EMC storage hardware and Cisco routers. Then they realized that they didn't have any budget planned for out-year maintenance, so they moved the venue. That was somewhere in the realm of $50 million dollars that was literally just thrown away, as the last I heard, they were re-converting the data center back into a garage.
In short, government contractors catch a lot of flack, because they control the visibility of these things. The contractors often can't speak to details of these projects due to security clearances or NDAs, and at the end of the day, the government is always happy to blame the vendor. It's well practiced, and designed with a variety of scapegoat clauses built in.
Replacing the logo itself took all of about 30 minutes (receive new picture, digitally retouch, overlay onto background image, resize, play around with transparency, save to png.)
Also, bear in mind, the 400 hours is 400 man-hours, which is a cumulative number based on the number of people involved. If you've never done business with government, let me assure you, it was a lot more people than necessary.
I recently did contract work for an agency and had instances where a simple spacing or image resizing took days of work because the client or the agency would understand and then forget that I must crop and/or resize images for them all to be of one size.
And then they changed it back to the original.
Your assessment of the software and services seems fairly accurate to me (there are a couple of exceptions: CAS is simple enough to be easily understandable and works ok, and Atlassian Crowd looks ok too).
... it's just the article notes a subcontractor has already been convicted of fraud and kickbacks, so this might not have been a project quite like the projects you describe (though if this is a Terry Childs-type case, I'd love to hear about it).
I meant to say those things, but got lost in typing so much other nonsense and trailed off. Thanks for bringing it up.
They would bid on something if they weren't going to find a way to make more than what the original contract was for because in the bid wars they often lower the bids so much they have to recoup somehow. In this case they probably did not build the recoup method into the contract and did it illegally.
At least that's my theory.
...
The city is also planning to launch a request for information (RFI) process in January in search of alternative timekeeping systems.
Wow. So they eventually got a working system. Now they're throwing it away and going to purchase a new one? Meanwhile turning 83 contractors into FTE's to maintain the existing system? This seems like complete craziness for something as commoditized as timekeeping software.
Municipal governments are a large portion of our business, so I'm intimately familiar with the challenges. What's unfortunate is that it's infrequently the fault of the individuals running the municipality. We meet a lot of really great people out in the field. It's just that they're doing their job with one hand tied behind their back. They have to operate within a framework that is (mis)guided by public interest.
For that reason, I just started a new startup company: TimeKeep NYC. We specialize in timekeeping software for NYC area cities. Looking for employees now, especially ones who aren't snitches. Based on past products in this market, I think we can do quite well.
But, yeah, on the other hand that's just the usual business software story. 10x budget overruns are probably not called for.
- Have bloody awful UIs
- Aren't extensible
- Need a massive amount of expensive consultant days to setup
- Are very brittle
- Have an overwhelming list of features which make them very hard to use
- Have bizarre restrictions on certain data (e.g. max 100 chars for a comment)
- They're not capable of talking with common system X
- Don't expose an API
- Have atrocious db structure making reporting extremely expensive in dev time
- Use a home rolled DB (oh. god. no.)
As for the problem of timekeeping being extremely complex, it's not at all. It's difficult, but solvable with one or two decent developers/project managers in a few months. I know as I've seen it done twice now.
It's only extremely complex if you try and include every weird quirk and feature that every client has ask for. Sometimes the solution isn't extending the software, it's changing the business process or simply leaving them on the old paper system.
If you take a 37signals-esque path you've got something that'll sell to 80% of businesses, let niche software cover the rest.
It was actually my first task that I ever did "for real", though we did only the raw recording; the processing was done by older code that had all the quirks built in. It was a success, too. The problem itself is easy, but it's definitely in the class of things that can be made arbitrarily hard by added requirements with no thought given to costs/benefits. Fortunately on my project, the devs and the management were on good terms, so much thought was given to cost/benefit tradeoffs, but I can easily see cascading failure in a different environment. Even the easiest software projects can and are sunk by that.
Er, has it occurred to you that the insanely convoluted world of NYC government might constitute a niche?
The City of New York employs about a quarter of a million people, which is around the same size as General Electric or GM, worldwide, performing a wide range of job functions.
Also: the particular system under development included hand scanners, so you should include developing for that.
http://www.citylimits.org/news/article_print.cfm?article_id=... (article from 2005 when it was only 6x over budget)
The brazenness...only in NYC.
Also, if you ever get in the position of saying "X is extremely complicated and I can't explain it to you" it is because you stand to benefit by designing a system poorly, or you aren't bright enough or don't have the experience to realize that you can reframe the problem. Algebraic geometry is also complicated, but these days there are plenty of books that will explain it to you, and the tools (like sheaves) are well-established and standard.
Obfuscation is the MO of installed, configured enterprise software. If there were an incentive to create a simple, effective solution, then, lo and behold, one would be created. The incentive is almost never there.
Take Unix for example. It is simple and effective (and flimsy as all heck) but before it came along, obtuse operating systems were the norm. It is worth pointing out that A) Thompson and Ritchie are geniuses and B) there was no financial incentive.
Job Title Annual Salary
-------------------------------------------
Specialist, Level 4A $676, 031
Project Manager, Level 4 $661, 628
Project Manager, Level 3 $632, 653
Project Manager, Level 3D $601, 286
There are literally a hundred thousand requirements like that one.
As for the requirements that drive these projects, typically they are written specifically to present the appearance of an open bidding process while allowing the departments to know ahead of time who will actually implement the contract because they are so specific in certain areas that only one contract can possibly fulfill them.
Yet $600 million? I've built accounting software, and though it wasn't as complex as what NYC needs, I'm 90% sure $600 million is astronomically higher than what a reasonable price would be.
What about the original price of $63 million? That still seems awfully high to me, but I'm not sure. Does anyone have an opinion on that?
$600 million is extortionate, of course - hence the demand for reparation. That's why bloated consulting companies like this are able to demand such high prices. Their size provides a target for the inevitable lawsuit if things go south.
Mark Mazer, CityTime consultant accused of scamming millions, has checkered past of investigations http://articles.nydailynews.com/2010-12-23/local/29441786_1_...
CityTime Crook Pleads Guilty As Defendant List Grows http://gothamist.com/2011/02/11/citytime_crook_pleads_guilty...
Mark Mazer Indictment [scribd] http://www.scribd.com/doc/58405089/Mazer-Mark-Et-Al-S2-Indic...
It's really easy to waste money on big development projects of this sort. The bigger the development team the harder it is to manage, the less well it is managed the less productive it is in aggregate, and thus the less productive it is per salary-dollar. And these are not small effects. A small team of very talented devs can quite easily compete head to head with a giant team of hundreds of devs, precisely because of these scaling and management problems.
http://blogs.villagevoice.com/runninscared/2011/01/this_week...
The role of Bloomberg the businessman is independent of Bloomberg the public servant. He can sell overpriced software as a businessman all he wants, but once he becomes mayor his responsibility is as a steward of the public's money and to combat ridiculous overpriced projects such as mentioned here.
In a sales call, the Bloomberg sales reps will demonstrate endless functionality of the terminal : there's a huge amount there. The dirty secret is, though, that most people use only 4 different functions (different people, different functions, of course). The big switching cost is that the email/IM is entirely proprietary : Everyone has to have a BBg terminal, because everyone else has one.
The software is probably as well written as you would expect from a company in a monopoly position. One thing in their favour : If they fundamentally changed the way the 'MSG' button worked, 50% of their users would complain.
You're essentially correct on all points but clients tend to use many more than 4 functions though significantly less than the total number.
And with regards to IM and email, it should be noted that any user of a Bloomberg terminal can IM any other user anywhere in the world (there's no 'friending' or anything). Many, many trades are conducted over IM or 'IB' as it's called.
The most accurate analog I can think of to the Bloombeg terminal is Emacs. It has everything and the kitchen sink (News, dining suggestions, email, IM, sports updates, etc.) and clients are extremely particular about functionality. I have a cousin that's been a subscriber since 1997.
And with regards to code quality, I'm not sure how much I'm permitted to say, but the first Bloomberg terminal was released in the 1980s (this was before SQL existed). Also they maintain the largest private network in the world (they run a line to every terminal)... so all the things that go with that.
Of course, Emacs is free. Do you think Bloomberg is susceptible to competition from a much lower-priced and much leaner startup?