Hacker News Reader

Comment by skmurphy | Hacker News Reader

When Frank Marshall was hired as VP of engineering at Cisco in 1992, he assessed after three months that we had a software quality problem and forbade any software engineering activity except "Fixing bugs." This went on for perhaps six months. It allowed key competitors to get ahead of us because we could not address some serious architectural issues in the code base because that would be writing new code, not fixing bugs.

Gerald Weinberg talks about "Software Stalins," managers who grab onto one indicator and think that driving it to zero (or 100, or 11 for you Spinal Tap Fans) will resolve all other problems.

If only things were so simple.

View on HN

This is a really interesting anecdote because (obviously) Cisco fared quite well on net over the last severl decades, is largely respected as a reliable vendor for critical infrastructure, and doesn't evidence even a hiccup in market assessments of its health during that time.

https://www.macrotrends.net/stocks/charts/CSCO/cisco/stock-p...

I don't mean to take away the truth of your experience or dismiss that you might better know the disconnect between what you describe and what outsiders might see in hindsight -- it's just the disconnect itself that's fascinating.

> If only things were so simple.

Indeed!

The renewed growth and increasing customer satisfaction--and our long-term success--were due to engineering abandoning a poorly thought-out policy before it crippled us instead of merely hampering us.

Part of the challenge in a "freeze spec" approach implicit in "only fix bugs" is that it assumes the market requirements are well established. That was definitely not the case for Internet protocols/functionality in 1992. HTTP traffic did not exist yet (along with a couple of dozen other things that would quickly become dominant by 1997.

The effect of the bug fix stall was to give our more nimble competitors more growth and more runway. The fact that Cisco did very well was despite the "just fix bugs" mandate.

I offered the story because I thought some folks reading the linked post would find it unbelievable that a company would decide to fixate on a single simple metric when operating in a complex and evolving environment.

You have to do both, on different branches. Rearchitect things in one branch, fix nothing but bugs in another. Forward port all the bugfixes you can from one to the other.

Though: version control ... 1992 ... haha.

Hey, that's when that unwieldy behemoth ClearCase was released.

I think that approach would be doomed even in the present day. Multiple "legacy" release branches is very hard to manage. Only the very largest such as Microsoft can do it, and they clearly hate doing so and want to push to rolling releases like everyone else.

Red Hat / RHEL does this on all EUS release every day, on RHEL 6,7,8,9.

Tell me more about how you would subscribe to a MS "bugfix only channel of updates" ? This is my first time hearing about that.

> bugfix only channel of updates

Isn't that what's meant by LTS?

Do you just 'install' the msi, and then thats LTS ?

> our more nimble competitors

Who was competing with Cisco in 1992? I struggle to remember. They were so dominant in the "dotcom" era (late 90s) that it felt like they were the only choice, the "IBM" of enterprise networking hardware.

> Cisco fared quite well on net...

Incompetent management is something of the norm in software engineering. Large successful companies have a deepish management hierarchy (4+ levels) and the competence of the org is capped by the least competent manager in the chain - who tends to be nontechnical and confused. Small companies have the ability to be more competent, but correspondingly tend not to have the resources to be influential players in the market.

I would expect that typical - maybe even above average - performance by software companies is accompanied by some breathtaking stories of management failure. If it were possible for a company to not make any mistakes in its software development it'd probably be a never-before-seen fountain of wealth creation.

The lack of nuance to consider exceptions appears to have been the problem.

Perhaps a lot of products needed bug fix campaigns because most corporations tend to reward SWE productivity based on so-called "evidence" like:

- KLoC

- # of commits

- adding features

- so-called "impact"

, rather than:

- reducing the cost to maintain code

- making code more robust

- making features actually work as users expect

- rip-replacing what shouldn't be salvaged because it's too expensive

So instead of adopting a funny mustache and making inflexible commandments, it would be less disruptive to have targeted bug fix campaigns where needed to boost nonfunctional requirements like quality, reducing support costs, and improving user satisfaction to their desired target levels.

It's too bad that Microsoft didn't take this same approach, and stick with it for a long time.

I'm wondering what things are like in the alternate universe where OS/2 became the dominant PC OS instead of Windows.

Architectural problems are bugs. Hopefully, most companies have VP of engineering who understand that.

Disagree. Architecture problems are different enough from software bugs that it's worth it to keep them separate words.

If bad architecture makes the system fall over, that's a bug. It's not black and white.

> "Software Stalins," managers who grab onto one indicator and think that driving it to zero (or 100, or 11 for you Spinal Tap Fans) will resolve all other problems.

Paul O’Neill famously did exactly this at Alcoa starting in 1987, focusing solely on worker safety resolved many other problems and multiplied profitability.

>The company's market value increased from $3 billion in 1986 to $27.53 billion in 2000, while net income increased from $200 million to $1.484 billion.

[0] https://www.forbes.com/sites/roddwagner/2019/01/22/have-we-l...

It's a reasonable counter-example, two other CEOs of successful firms who emphasized safety were Lawrence Culp at Danaher (he is now at GE) and David Cote at Honeywell. All three had tenures more than a decade where their firms invested in and introduced new products, so I suspect they focused on more than worker safety but I am not personally familiar with their policies and decisions.

I think it makes a very good story that a focus on worker safety--to the exclusion of any other objectives--is all that you need. But I don't think anything is that simple.

Reply on news.ycombinator.com