Five whys: The startup immune system, Part 1 | Hacker News Reader

Five whys: The startup immune system, Part 1

venturehacks.com

43 pointswheels20 comments

A few years ago I worked for a smallish company where the repeated "why?" would consistently release untold nightmares. The turnover rate was high, but there were a few old-timers who would tolerate a historical/political inquiry, and a few more whose reactions revealed things about history and politics in a different sort of way. At a glance, the product designs and manufacturing procedures appeared to have been born from pure ether. (This company made medical devices.) Design decisions were undocumented; what documentation did exist was withheld out of general mistrust.

Problem: Applying the drug to the latest version of the device in the usual way gives a messy coating.

- Why is the coating messy on this design? Because the device's surface can't hold this much drug.

- Why are we applying so much drug? Because the previous products used the same dosage.

- Why was that dosage chosen? Because, in an animal study using sample size N=1-3 for each of three different dosages, this did slightly better than the other two. No further investigation was ever done. EOF

- Why can't this device cleanly carry as much drug as previous designs? Because the surface properties and geometries are different.

- Why did we change the design? What properties did the other design have that resulted in a clean coating? Because a partner company wanted out of the biz, created a durability test no existing product could pass, and used that to torpedo our product. And, wait, that design had a problem with the coating, too.

- Why didn't we catch these coating problems in previous products? Because quality is fucked in this pilot facility, and our other manufacturing facility is still trying to get up to speed with the first product made with this manufacturing method, and unable to duplicate our allegedly flawless results from two years ago.

- Why are we unable/unwilling to do serious QC work here? Or even basic science? We think the CEO doesn't intend to pass regulatory requirements or even launch most of these newer products, so it doesn't matter. These projects are window-dressing for investors. Yes, of course we're all looking for new jobs on the side.

As it turned out, the real benefit to asking all of these whys was that I was able to abandon ship at just the right time. Shortly before I left for another job, having challenged the CTO about the scientific feasibility of his latest escapade during a meeting, a senior engineer showed me some years-old pictures of a device with exactly the same physical design as our flagship product. This young engineer was enlightened.

This is great advice and can be used everywhere.

Beware that if you work for BigCorp this will make you unpopular.

The statement "Because that's the way it is" springs to mind.

When I worked for a larger company, I had the rare pleasure of working with someone who was at once brilliant and patient (something he actively cultivated.) When trying to get to the bottom of things, I usually was granted a purview into the historical technical and political history. While this was illuminating for a while, eventually it boiled down to: lack of approval to pay off technical debt -- the compromise was made knowingly and was the best decision at the time. Of course, this was something I thought to be true, but only really learned it after coming to the same conclusion time and time again.

I've had the benefit of similar illumination where I'm employed, although without the same level of patience. The same explanation was given time and again there, too; incurred technical debt due to prudence. They were small, and thus didn't have the financial wherewithal to afford a long term outage should some of that debt sudden become subject to a margin call, and so I dug deeper. And from that the true reason came out of why things had gone off the rails so bad. The political history of the company had been one where ego and authority ruled. The technical debt would not be repaid, because it would be an admission that the incurring of such debt was not inevitable. It would be a sign that one or more wrong decisions was made. And, due to the history of authority rule, no one would dare question the decisions, assuming both that the management was approaching the situation with full information and rationality, and any one else should assume to have neither. From what I've gathered, I'm not so alone in the universe with these experiences. But when the term "political history" comes out to explain something, I'll generally start reaching for my fine tooth comb, to better understand who it matters to and why.

What exactly is meant by incurred technical debt? How do you have "margin calls" on technical debt? Totally confused.

I'd venture that it refers to bad decisions that lead to low quality of code, the problems of which are inherited by all code built on it. And so on. With every layer you pay for your past mistakes, and you quickly spend more time working around the problems than you would have spent fixing them in the first place. (or fixing the stuff below before layering even more code on)

For example, the amount of hoops I had to jump through to be allowed to fix some old code at my last job was astounding, even though the bugs had literally caused man-years of lost work, and eventually took me about 2 weeks to sort out.

I agree with pmjordan's answer, but thought I'd add an example: Your company's product log messages of some kind into a database. The developers were lazy, and put everything in one table with the classic "key, value" SQL antipattern. Subsequently, it was discovered that values weren't enough, you sometimes needed arrays of values (or hashes of values, or ...), so a (probably crappy) serialization format was written for the values so they could be arrays.

Suppose now you were hired in to this position and you actually do a lot about databases. Now, your company gets a call one day from your biggest customer that while they've been putting up with your product getting slower and slower over time (without telling you of course), it's finally gotten to the point where "queries take over an hour and we really need them in less than 5 minutes"/"our daily reporting process now requires more than a day to complete" (which means the system clogs up with nothing but reporting processes, eventually)/"we've been sued and the court has mandated that we turn over some data by next week and we compute it'll take two months"/"the CEO just tried a query that took an hour and now he wants to cancel the contract"... or so on, you get the idea.

This critical problem comes back down to you, and you realize that the only way to fix it is to entirely restructure the databases using better practices. Conceptually, this is trivial. However, since every single part of the system is built around this data system (and of course, there's no abstraction, everything uses the key/value store directly), the combination of the amount of time it would take to write the new schema, modify every program file in source control to use this new schema, write the migration script, and potentially even the time to run the migration script for the customer (during which time they will be off-line, which could be days, and which could be anything from unacceptable to impossible), oh, and testing, don't forget testing, could in fact be longer than your company can survive with a pissed-off major customer.

And if one customer is pissy now, it means every single other customer you have is a ticking time bomb.

(Everything in this message comes from things I've seen more than once in my professional career, except the hypothetical about getting sued (thank goodness). I'm not providing details because there are thousands of companies that could fill in the above!)

Your technical debt margin just got called. You couldn't cover it. It wasn't even your technical debt, but who cares? Buh-bye!

You just described my current day job. lol.

Incurred technical debt is the result of many "best compromises" made over a period of time.

"Why" applies perfectly well to that.

In certain circles, that becomes an endless feedback loop.

But then they grow out of it...

Which usually means that they find someone that isn't as "difficult to work with". :-)

Not that the person who states is the guilty party, but this strikes me as an immediate sign that someone got tired of thinking and was compelled to feel a sense of control and certainty by turning what was said into a truism. They might be right, they might be wrong, but I couldn't fault someone for feeling a little lost of respect toward someone who would assert this.

Beware that if you work for BigCorp this will make you unpopular.

Why?

It demonstrates a conflict of priorities between you and most of those around you, if my rather unscientific gathering of data is tending toward the true mean.

It demonstrates a desire to improve and understand, both toward yourself and toward those who you ask "why?". Superficially, improvement is a good thing, and worth the devoting of a lot of rhetoric. But it comes with a lot of extra work, and with the admission that the way things are done now aren't optimal. So, it rubs people who are either lazy, apathetic, or egotistical entirely the wrong way. You'll also chaff indirectly those who are authoritarian in nature, and put faith in the management above; it's the same reason people get irritated when you question their deity's righteousness. If they're coworkers, they'll get spiteful; you're making more work for them now, and it either doesn't occur or doesn't matter that it means less drudgery later. If they're up the management chain, they'll wave authority in your face; it's the easiest way to short-term tranquility and an ability not to adopt a similar change ready attitude.

This is not an endorsement; just a reflection. This is the way it seems to be, but it doesn't have to be a truism you just begrudgingly accept. Being unpopular doesn't mean you're doing the wrong things, and jeopardizing your future and well-being. Maybe at BigCorp, but how long before misery and desire to do better would drive you away anyway?

Long reflection, sorry. But I hope this helps answer the why, and I hope your question was not rhetorical. ;-)

it seems that the five why's would work better when asked by a consultant who was brought in by management than by employees from the lowest rung of an organizational chart.

however, myself being one of those at the bottom, I definitely like stirring things up in the bigco where i'm currently employed.

I really try to get along with everyone and see each side of a story. that said, there are times when people can be unreasonable and I figure I can't please all people all of the time.

I believe my colleagues and even upper management understand where i'm coming from when I tend to disrupt their plans a bit. my manager even reminds the execs who I am saying, "that's just the way tom is". I haven't figured out if this is a good or bad thing as of yet - good. I guess.

I think the people I work with value my opinion and I understand that once my voice has been heard there's nothing really more I can do that moment to push my ideas and get them implemented with management's blessing. sometimes, some things still go forward without their approval but this is not the norm with me.

I take the oportunity whenever it avails itself to remind them of my ideas, gently nudging them in my direction. I follow up with research i've done which makes it's way into their inboxes. from what I can tell, a little persuasion and a lot of repetition has it's advantages.

one thing is for sure. there is definitely a give and take in any dealings with people. I guess one must be prepared to lose a battle which can lighten the initial blow of rejection. just don't think the war (your cause) is lost due to one conversation across a conference room table.

I guess if the five why's don't work you can use the one big bye.

i really like discussing the personality of bigco management and simply wanted some more input than just 'beware'.

thanks.

>manager even reminds the execs who I am saying, "that's just the way tom is".

That sounds like a defensive statement to a negative impression, to be honest.

Looking forward to more.

Reply on news.ycombinator.com