IBM's infamous "Black Team" (2002)
t3.org
t3.org
After reading this article I have realized that even at large corporations if you make isolated and largely idependent small teams of highly competent people that can select their members themselves, you can create "startup" bubbles inside the organization that can deliver outstanding results. Maybe this is the way to go when growing large, from a medium-sized enterprise to a large corporation?
Consider: it would be in the best interest of any project manager to steer their project away from the Black Team. They aren't incentivized to ship good software, but to ship on schedule. And should avoidance be impossible, they absolutely will point corporate HR at any and every aspect of that strong team, to break-up, slow-down or demoralize them.
Similarly, with higher-up management. They'll need certain projects to be treated with kid gloves. So they'll scheme and conspire to either defeat the team and ruin its effectiveness or defeat the process, so that the team's effectiveness is rendered moot.
It's the perverted reward structure in large corporations that causes most of their problems. There's no lack of people trying to fix those problems. But any solution (including this one) will ultimately be defeated so long as that reward structure remains.
It's an open question how to adapt this to "infrastructure" groups that are shared by the whole organization, which may explain why there're so many horror stories about corporate IT and HR. I guess technically QA is also an infrastructure group, so it's surprising the Black Team survived for so long, but I guess the improvements they made were visible enough that they had senior management's protection.
Corporations today are worried only about the next quarter, or what things will look like maybe twelve to eighteen months down the line. But certainly no further than when the options vest and management exits.
And that's why divisions don't help. Because defeating the process to ship crap today has a better effect on this and next quarter's P&L, versus increased spending testing/refining the project for no real increase in revenue in the short term.
Down the line, when product quality would become known and helping the product sell better than it would otherwise, those responsible for making the decisions will likely be gone. (They'll definitely have planned on being gone by then.)
So shipping quality was still important.
I'm largely in agreement. This line from the start of the article stood out:
Customers do not pay for defective products.
However, if customers are happy to pay for stuff that doesn't work (eg, where the people who make buying decisions don't talk to the people who will be made to use the software - quite common is my impression) then sure, stay a hundred miles away from the Black Team.
If you are smart, you will develop an informal network of contacts/relationships throughout your organization. And keep your resume up to date. (I think that smart people develop such a network not only or even primarily out of a calculated decision to manage their career, but because they find interesting what other people know and do and struggle with and solve, and helping them.)
Perhaps this is one reason social skills (of a certain sort) tend to predominate in large organizations. Given the situation, they become the most essential tools to a career.
Also, regarding social equanimity, a large part of corporate life is "not rocking the boat". At lower ranks, this can imply a strong uniformity of treatment. Your increase will be between 0% and 4%, regardless, because those are the absolute limits that have been chosen. If someone were to receive 10%, there would be a lot of jealousy and unrest. The only way to larger increases is promotion. Promotion changes one's duties. Ergo, certain duties will never be compensated beyond "industry standards". The social structure caps their compensation and so, minus the individual who 'self-sacrifices', their effectiveness.
And this is, in part, where your "average or below-average" coworkers come from. Management won't sweeten the pot, even if, as the posted story implies, going after the top of the talent pool produces far greater returns than the additional investment.
I guess this comment is kind of thrown together. Large social structures tend to enforce norms and drift towards a lowest common denominator. Exceptions occur, but over time, they are fighting the tide.
A bad one will make you aware of every microissue their boss has to worry about, and makes you code in some terrible things in order to please them.
A good one acts as a firewall between their boss and their subordinates. Bad ideas from above are pushed back (as much as they can), lots of independent (and usually fairly random) requirements are combined into something logical, and my resulting problem set is something reasonable to create, test, and maintain.
A boss with programming experience will be able to negotiate a deliverable that's internally consistent enough to make for a good product. They do some of the up-front thinking to reduce the problem down before it's even given to me.
A nontechnical boss just sees a spreadsheet of checkboxes that we have to complete. They're like a simple IO controller: send requests, wait for completion.
What usually surprises me about large organizations is how little power the managers have over these two points. A manager in charge of internal applications development may negotiate the functional specs with the business, but have no influence whatsoever over the hardware that's supplied to the developers.
I couldn't find a good source for how many employees they have, but its in the thousands, so they have been able to make this work on a large scale.
A more nuanced view might focus on how corporate culture, a short term focus on sales due to stock market pressures, and the inevitable lack of accountability that seems to shield the management chain from harm provided they continue to take no risks around innovation combine to create some weird effects. Look in particular at how some big software providers make acquisitions, which are then polished and re-branded as a new version solution and then shoved into a customer pipeline.
These customer pipelines are the real magic - no matter what junk gets shoved down them, nine times out of ten, junk gets paid for. Nobody on either side cares. Takes too much personal risk in the big corporate world to point out that the emperor wears no clothes.
Growing small teams of highly competent people seems to naturally lead to duplicated effort and turf wars as well. Don't think that a team of magic snowflakes will outshine all the "drips" hanging on . . . I've spent too much time in big corporate IT to not realize that most people walking the halls in these places are damn smart. They're just hamstrung by the corporate structure. But - this is why small startups have an advantage. They're not nailed down by convention.
This had little to do with competence! The team was successful because it grouped together some people who actually enjoyed their job. Individually, they were only slightly better at it than their peers; as a group of "like-minded individuals" (quoted from the article), they spent much more energy collaborating on ways to improve their software testing process.
"Like-minded" is only tangentially related to intelligence or competence or any of the other traits that arrogant people like to ascribe to themselves. It's really about people that have similar priorities and styles and work well together.
It's probably worth the $95 anyway.
Actually, I think so. Yes.
You might also try a local brick and mortar bookstore.
http://www.bookfinder.com/search/?ac=sl&st=sl&qi=5je...
New for $55, used for $4-65, but several in the $28-35 range
Edit: I tried searching for anything related to IBM and their Black Team. It seems Google saturates my search request with everything linking back to this article.
Perhaps someone could help?
Basically, it can lead to having a rift between developers and QA, and in a big corporation, it can become a bureaucratic mess and a 'us' vs. 'them' attitude between groups. I've been in long heated meetings between QA and Dev arguing over individual bugs, not so much on behalf of the customer as opposed to face-saving exercise.
From the article: "And the things they did to software went beyond all bounds of rational use testing and were more akin to software torture." In a perfect world with unlimited time, it's great to find bugs through contortions but usually there is some costs associated with it either with QA not focused on other areas that matter, or having a developer focused on confirming that bug as opposed to working on something else.
Sometimes it's great in a bigger organization to have a unique culture so I agree with the jist of the article and to other HN comments. For instance, I worked at a company on a smaller Macintosh team that fostered a healthy competition with the much bigger Windows team which worked well. I'm just not so sure it works well when people are essentially working on the same project where interfacing with each other is a requirement.
But sure, if you had a terminal, a spare IBM system (the iSeries is probably the lowest end) set up to support SNA, and then a networking controller to attach the 3270 to, you could probably get Lynx working.
Alternately, if you just want the effect, you could just pick up an old green/black monochrome display and hook it up (via a physical adapter) to any VGA card. Buggy implementations excepted, all VGA cards should be backwards-compatible to MGA.
Add a nice IBM Model M keyboard and you'd be all set. I suspect someone has probably done this as a case mod before (if not ... I might); stick a modern mobo in an old IBM PC case with the original monitor and keyboard.
Terminal-based systems often have a very steep learning curve, but I've seen many cases where they were better designed and met business needs better than COTS replacements.
Granted, those terminal-based systems were probably phenomenally expensive when they were put in (mostly in the 70s or early 80s, that I saw), and I suppose it's possible that after 15-20 years we have a biased sample that only represents the best of the breed, but they got the job done.
I judge how good things are by the human element. That hasn't changed. The software industry is as inclined as always to engage in death marches, shortchange quality, seek imaginary silver bullets, and devalue experience. In short it isn't fundamentally different at all.
Djikstra and Hoare are the only ones who tried to advance a higher standard and yet beyond naming an award for one and singing praises of the other, the field has continued to accept lesser standards.
It's not apples-to-apples, but if you look at the number of lines of code executed before a noticeable bug it's in the trillions.
You could, but the argument would be false there. Building a car, constructing a house, sewing a shirt, writing a book, etc. are all not significantly more complex than they were decades ago. Software is orders of magnitude more complex than decades ago.
You would be incorrect - having worked in the industry for a short while, I can attest that modern automobiles are at least an order of magnitude more complex than something from, say, the 60s.
They still operate on the same mechanical principles, but we exploit these in vastly more complex and efficient ways. If you want to make this argument, then you also have to accept that computers are "just" current running across silicon.
No, the reason why we get away with this is because replacement is easy. You get a car right the first time because a large recall can cost in the hundreds of millions, and be logistically impossible to execute. For us it's a simple "Hey! We has update! Yes/No?" dialog box.
But that's compared to Moore's law that a computer becomes an order of magnitude more complex roughly each eighteen months for twenty or more orders of magnitude increase from the sixties.
Also, modern cars do have lots of bugs in sense of suboptimal behaviors. They just cannot fail utterly without people being pissed.
The complexity of code - not the compiled binary - the text you punch into the machine and what it semantically represents, has not really gotten that much more complicated over the years. We've introduced several new paradigms since the COBOL mainframes: object orientation, functional, to name a couple. It'd be hard to argue, though, that it follows Moore's Law. Not even close.
> They just cannot fail utterly without people being pissed.
This is an important point: people who expect flawless behaviour from software because other fields of engineering demonstrate it, IMHO, are misguided. Aircraft engineers work incredibly slowly because the consequences of fucking up is perhaps thousands of deaths, and billions in liabilities. Car engineers are the same on a lesser scale. There's no need to expect flawless, 100% perfect function when you don't need flawless, 100% perfect function.
We could spend 20 years developing the perfect toaster that will never, ever burn your toast. Or we can spend 2 months on something that will get it right 97% of the time, and just move on with our lives.
Software is different not just because it is more complex, but because computers are much more finicky than roads and people. A logic flaw in software can cost millions; an equivalent flaw in a book will likely never be noticed.
Love it!
The point is that they become loyal to each other, the company reaping a reward is just a side benefit of people working towards a common goal.
I was on a similar 'elite' skunkworks dev team in a company of ~ 35,000 that had executive support, and the results were insanely good. You can't manipulate people to do this; You can however give them the freedom to make it happen.
The amount of productivity, support, and "you grok me"-ness is so extreme that it's really a wonder to behold. As a regular dev now I really do look back on those days and wish I can do it all over again.