An Operations Mindset is at Odds with Innovation | Hacker News Reader

An Operations Mindset is at Odds with Innovation

spin.atomicobject.com

29 pointsgvb12 comments

The arguments advanced against an "Operations mindset" are mostly strawmen. Operations is primarily about efficiently operating an organization towards a goal. The type of thinking advanced in the article however is more likely to be pushed by someone with no operations management training.

For instance, putting people on multiple projects versus single projects. Multi-tasking and context switching cause known losses in efficiency even under the perfect scenario with no startup time. So if you want to emphasize getting projects completed, multi-tasking must go. This is an operations management mindset, though it runs against much current practice.

The points about efficiency are similarly misguided. You don't focus on worker efficiency, operations points you at throughput and global optimization rather than local optimization.

Those however are about focusing on improvements and operations of the current systems. It is true that innovation is different than operations. In the same way that learning to program efficiently in a language is different than inventing a programming language. They are different kinds of things. Many operations principles however can be successfully applied to innovation. For an excellent read on the topic, read "The Principles of Product Development Flow" by Reinertsen.

To get a better understanding of operations in general, and not the hyperbole advanced in the article a good starting place is "The Goal" by Goldratt.

One big revelation I learned while studying operations is that the things that largely drive developers crazy aren't good management practices. They aren't advanced by operations researchers. They are in fact what I always thought they were, bad management practices.

You might think it fair to call them "strawmen" but don't assume that all operationally-minded organizations are run my management with operational training. Some of us live these "strawmen" everyday.

Yes, and this is why a company really needs two technical teams (at a minimum): developers and sysadmins.

My goal is stability. The dev group's goal is growth. In a very small startup, it's hard to hold both these ideas in your head at the same time. If you don't grow enough, you will never need stability. If you grow enough, you will eventually need stability so that your customers do not desert you. Thesis, antithesis, synthesis: supportable growth.

My ops group meets with devs for discussions of supportability and performance well before the actual handoff. We make sure that monitoring is allowed for from the beginning, that there are processes and documentation that let us fix problems at 3AM or diagnose it well enough to call the right person to fix it. Performance is checked regularly. Is it time for new hardware? Is it time for a rethink of the architecture to support faster performance or higher scale?

The two forces will be vector-summed. It's important to get it all pointed in a direction and magnitude that leads to success.

As a sysadmin, I used to think in the two goal paradigm. I actually don't any more; if your company has two groups with different goals, particularly in the early days, you'll waste time with friction.

My goal as a sysadmin is to help the company succeed. I try (and sometimes fail) to assume the same positive intent of my peers in engineering. If we seem to be working in different directions, it's often because there's something one or the other of us doesn't understand.

Concrete example: I have often found that my tolerance for midnight crashes is lower than that of the engineer who wrote the code. There appears to be a misalignment. But on further examination, the problem is that the engineer doesn't have the same information I do: she doesn't realize on a gut level the cost of being woken up at 2 AM.

Solution: add engineers to the first level on-call rotation. Information is added; goals become more in alignment.

As an engineer, this made me chuckle.

Mission accomplished. It's kind of a trick -- I am way in favor of letting developers look at running production servers, because how better to understand a production environment? But a lot of sysadmins are wary of that, so you have to frame it in a way that benefits tech ops as well.

I don't think that these two goals are at-odds. In many companies, the problem is just that neither group, developers or sysadmins, really seems to understand the other.

But both want the same thing -- to ship a working product.

Developers really don't want growth at the expense of software that doesn't work. But many developers come from a computer science background, and I've never seen a computer science curriculum teach students about logging, testing, runtime instrumentation, monitoring theory, security, or deployment.

In university, you write code in a very unrealistic fashion. Lots of individual coding, some teamwork, but your assignments nominally have well-defined inputs and outputs, and students effectively never have to carry projects across classes and semesters, the way that you would with a production application.

Sysadmins, having been exposed to this, assume that developers are out to kill everything with a BIOS, and as a result tend to lock systems down, or implement weird hacks to get around the lack of logging or instrumentation.

In companies I've consulted with, a bit of education goes a long way towards fixing this. I usually rotate a dev at a time through the sysadmin team, where they can both help in debugging problems, as well as tackling all the interrupts that would otherwise be divided evenly among all the engineers.

The developers learn about why logging is important, gain a better understanding of the deployment process and environment, and the sysadmins gain a greater understanding of what the devs are trying to accomplish.

It's amazingly effective in fixing the false dev/ops split, and in getting the two teams working together effectively.

The article makes an interesting testable prediction: in an innovative company, an employee is likely to work on a single project but use many different skills, while in a stagnant company an employee will use few skills but work on multiple projects. Does that sound true?

Excellent summation. In my experience (mainly Fortune 100's), the latter holds true - many silos, little to no innovation.

Working for BigCo here in Brazil, and the latter holds true for me and lots of other people in our company. Seems like a great way to put it.

The whole "Slack" book by Tom DeMarco is more or less about this. If you optimize the steady state, you cut the slack that is needed for the organization to adapt. Also, the throughput-latency tradeoff might bite you.

i only wish i had learned to spot the difference earlier in my career.

Reply on news.ycombinator.com