Hacker News Reader

Comment by seibelj | Hacker News Reader

A few years ago I worked on a project where my company was contracted to build a small piece of a much larger system by IBM. IBM was managing many different contractors. We needed a way to "securely" show a picture, aka not using the pre-installed image showing app. So IBM spent tens of thousands of dollars to build a new Android app that displayed a picture passed to it via an intent. I told our product guy that I could build it in an hour for free, but he said it wasn't our problem. In our twice-weekly status calls, we would ask about when the "image showing app" would arrive, and we heard all about the minutiae of building that app, from getting a PM, to finding a partner in India, to QA, to issues with the app, back to the contractor, and finally to us, where it was buggy as hell and never worked right. Completely mind boggling, and very enlightening about how IBM (and I'm sure other big vendors) operate.

View on HN

> I could build it in an hour for free

The big problem is that it's not beneficial for IBM/Oracle/Accenture/$consultancy to quickly deliver a stable solution that works well. They make much more money by having seven layers of middle management committees vote together a crappy solution that will need five years of support/upgrades and then a replacement.

In other industries, like construction, the same fundamental problem exists, but there human lives are at stake, standards bodies have been formed to prevent it, and non-software engineers aren't outsorced to people in India who have as little knowledge as they have pride in their workmanship.

If a contractor built apartment buildings the way IBM and friends build software, they'd be sued out of existence. I mean, have you ever heard of a building project going 3x over budget and then not actually delivering a building at all? It happens relatively often with software consultancies, usually on projects so trivial (like public transport ticketing systems) that it's hard to not attribute it to malice.

> usually on projects so trivial (like public transport ticketing systems) that it's hard to not attribute it to malice

Also something to consider in a circumstance like public transport stuff is that the multiple approval levels and meddling happens on the Government side as well. Like a perfect storm of bureaucratic bullshit that ends up making an enormous bloated mess that does everything but what was originally intended.

"I mean, have you ever heard of a building project going 3x over budget and then not actually delivering a building at all?"

Construction projects go over budget and are late all the time. Remember the big dig? https://en.wikipedia.org/wiki/Big_Dig

The Sydney Opera house was 10 years late and 1475% over budget: https://en.wikipedia.org/wiki/Sydney_Opera_House#Completion_...

Anyone who's had renovations done to their house can also probably sympathize.

Government construction projects, at least in the US, "go over budget and are late all the time".

Private construction? Not so much, certainly nothing like 3x the estimated cost.

Who performs the work on government projects? Private companies.

Yes, construction projects can frequently be late and over budget, but they always end up with an actual building when they're done.

I'm trying to be charitable in my interpretation, but "always" is a strong word:

https://en.wikipedia.org/wiki/Plaza_Rakyat

http://gizmodo.com/the-sad-fate-of-the-worlds-six-tallest-un...

But rather than split hairs: I interpreted the underlying assertion being that construction is more predictable and reliable than software. I've heard this in other contexts as well. And that didn't pass the gut check for me, because I can think anecdotally of many construction projects (and home renovations) that were way over budget and took longer than expected.

There's a name for software built with that kind of process and quality. Enterprise.

I'm not defending IBM here, because god knows I've had my experiences with their juggernaut, but in this case you are not considering the big picture. If IBM had let you donate this component, there would be:

1) financial implications (you were paid to develop a solution. we, the customer, have discovered that you were donated time & code. did you disclose your potential status as receiver-of-charity during your initial bid? do we need to hold a recompete because other bidders did not know to use free labor in their bid? how much code was donated? does this reduce the average wage of developers on the project below minimum acceptable standards? can you prove it?)

2) product liability implications (this free code you used is not working properly. people are dying or becoming impoverished as a result. we are going to the donor-developer for doing this, his employer for allowing it, and you for failing to provide sufficiently rigorous integration testing.)

3) support implications (your free code broke, and nobody can get ahold of the rando who wrote it, and the company he worked for at the time disavows responsibility. everyone is going to get sued.)

4) due diligence implications (did you use this code because it was free or because it was correct? can you prove it? was this a cost-cutting measure that may have deprived me, the customer, of a superior product? can you prove it? don't answer me; answer my legal team.)

I know it sounds extreme, but I've seen each of these things occur, at differing scales, over the years, and the reason companies like IBM exist is to deliver defensible solutions. This is not startup-land; it is not sufficient (or even, often, necessary) to deliver a working product -- the product must be able to withstand a Congressional investigation. You don't have to prove it correct, but you do have to prove you adhered to legislative guidelines intended to control the process.

This is also the reason behind the headline regarding IBM's "reputation." Nobody at IBM gives a shit what any of us think about their technological prowess or even their project management skills. IBM's reputation is an ability to operate at a massive scale without getting nailed to the wall by some government or another. If they get their ears pinned back in this census situation, the takeaway is not "IBM tech sucks" or "IBM PM sucks" -- the takeaway is "IBM is not a safe company for politicians to hire"... and for an IBM, that is fatal.

Again, I agree the whole system by which IBM operates is disgusting and wasteful, but to compete in this space you must become this space, as it were.

I don't think you're wrong, it's just amusing that something so simple became such a gigantic mess when put through the IBM process. It's food for thought.

Presumably that kind of process rigor is necessary in massive, high-scope projects, and the fallout we see is chiefly from the inability to apply a more lightweight process for tiny projects?

It's counter productive for big projects as well. The Too Big To Fail attitudes to process cover all manor of sins and corruptions to the point the projects is guaranteed to fail. See the Joint Strike Fighter for the worlds single most expensive example of this.

I'd buy that logic if it produced low defect rates and high uptime, but it doesn't. When we used SilverPop earlier this year it was down about 5% of the time and wrong 5% of the time, and even the basic way the interfaces we had to integrate with worked was a mess.

I don't think their process is heavy because it's rigorous, I think it's heavy because it's a reflection of their corporate structure (which is large).

I'd call it process theater rather than process rigor.

This is hardly limited to IBM or software development. Almost any business that relies on human to human "sales" is at risk of devolving in to a deceptive mess of complexity things that have nothing to do with delivering the desired product or service. That is dangerous because eventually competitors will be lapping you at a tiny fraction of the price.

I don't disagree with the description "process theater", but its a common part of the overhead of enterprise contracting (often as a result of the demands of the customer, and often central to the compensation structure), and even a bigger part of the overhead of enterprise contracting that involves subcontracting.

"Securing revenue through enterprise development contracts" is a different discipline than "software development", and optimizing an organization to perform on the former can involve compromising on the latter. (The entities that understand software development well enough to align incentives of contractors well with effective delivery of value are usually the ones that don't need to contract development out in the first place.)

This seems more like "spaghetti process".

Process meant to dissect problem to the right size for predictable execution. Right size does not translate to always the same dissection for vastly different sized problems.

For a tool as simple as the parent post describes, what IBM does is not process rigor.

The problem is I've worked with companies with a lot of "process rigor" (as you phrase it), which in theory should result in bug-free quality software. But it doesn't.

The process rigor encourages checking stuff off of lists, but discourages the little ad-hoc fixes that make a product good. Nobody's going to fix the minor bug in the date entry field if it takes 4 hours of "process" to write the 4 minutes of code.

The goal of it is not to result in bug-free software, the goal is to result in completely traceable responsibility. It's to protect the company.

I work for IBM and believe me, the process and paperwork gets super tedious, but I understand why. The number of times I've had a client threaten to sue for breach of contract is staggering. If we didn't have a ton of process and paperwork showing a trail of evidence and every communication that led us to the final result, we'd be out of business so damn fast. Unfortunately some people really like to take unfair advantage of their vendors.

Let me give you an example. Last year I had a client who asked us to implement a solution and configure it according to their specifications. We did so, but they then realized that what they asked for was not what they actually wanted. The real event was more complicated, but for the sake of anonymity, let's say they wanted a weekly report as a bar graph, and when we delivered, they realized a pie chart made more sense. Now, being the good consultant I am, I switched the bar graph to a pie chart. It wasn't in the statement of work, but it only took a couple of hours and I like to keep my clients happy. After they got the pie chart, they said they liked the format but they wanted it switched from a weekly to a monthly report. Sure, it only takes a couple of hours. And then they wanted the monthly report, but the data resolution broken out into weeks instead of days. Only a couple of hours. And then they wanted the name of the report changed. A couple more hours. Bear in mind, this is all 100% custom development, from scratch.

Eventually we ran over budget and over time doing these minor tasks, because what the client asked for wasn't what they actually wanted. They demanded we keep going until they were happy. They hadn't signed off on this work being completed, so they claimed it wasn't done according to the contract.

A few weeks later we got a letter from their legal department, and in return we sent them the statement of work with everything checked off, and all of the emails we had sent back and forth between us and the client, showing where they acknowledged the scope was changing and we had said we would try our best, but couldn't guarantee anything other than what was in the SOW. We also showed where they refused to sign an extension of the project timeline and refused to sign a SOW change to formally add the new scope to the project. And that was the end of that story.

Every step of the way when something changed, even if we discussed it on the phone, we had a follow-up email to confirm. Everything we did got documented, every change was clearly spelled out as to what we changed and what was the original agreed-upon scope. Every scope change was approved by the PM and sometimes, one of my managers. And at the end of the project, we deliver all of this (sometimes hundreds of pages) in a nice, neat report detailing to the client what they hired us for and what we actually did. It takes forever and it doesn't guarantee the end result will be perfect. What it guarantees is that if we get sued, we have all the paperwork proving we did exactly what was agreed to in the contract.

> Last year I had a client who asked us to implement a solution and configure it according to their specifications. We did so, but they then realized that what they asked for was not what they actually wanted.

This is essentially always the pain point for "enterprise" software, and essentially always it's because the actual programmers communicate with the actual users through seven layers of middle management at both client and consultancy. Because middle management has to justify their existence, and are mortally afraid of being cut out of the information loop.

It's a fair point. I've certainly seen the sales team promise something that, as an engineer, I have no idea how I'm going to implement. Some days I wish I could join the sales folks when they're giving their pitch just to make sure expectations are kept in line (from both sides).

That's why I always give a little leeway if I can. If they want their report changed from monthly to weekly, I'll do it. But sometimes it becomes clear they're just trying to take advantage of me and get extra work done for free. After all, why would they pay for a bar graph AND a pie chart, with both of them being shown as monthly and weekly reports, when they could just pay for one and then say "oh that's not what we meant, can you change that?"

I certainly will change that, no problem. But only if I have hours left after I complete everything listed in the original SOW.

Reply on news.ycombinator.com