Process is Poison | Hacker News Reader

Process is Poison

blog.gosquadron.com

20 pointsredprophet11 comments

TL;DR — Process bad, automation good.

I agree with half of that.

The problem in the article is that the example team didn't "pop the why stack" enough. "Engineer screws up a deploy. Why? <insert wrong command, lazy practice, knowledge gap, etc here> Ok, let's automate all the things!"

Automation is great—its not an incorrect response to reduce the surface area for what can result in an error. but automation in turn depends on another process. Continuous deployment? Great idea—but it depends on your tests. How do we know your tests are good? Code review! Another great idea—but also its a process.

Process isn't bad. Being a slave to process is bad. So is being a slave to automation. What happens to your continuous deployment when the new guy throws up a server to spike something (creating a snowflake) and it accidentally gets put into rotation for deployment as an app server, which now fails on it?

process is fine. it allows automation. use both intelligently.

Also, this article is limited to web-deployed applications - "deployment happens with the latest good build every so often" - which is the simplest possible deployment scenario. When you're not in the web-based world, your customers decide when and if to upgrade on their own schedules; there are beta test cycles; there are maintenance releases and hotfixes, each with its own branch in version control. Management is involved in deciding whether Release 23.07 should be delayed because big customer X found a bug. If you don't have a process to keep this stuff flowing smoothly, it just doesn't work. Sure, large parts of it can be automated, but the rest relies on lots of big and little decisions made by people.

I agree that the author has a webcentric view or experience gained primarily from web development. For most web apps the cost of an individual failure is essentially zero. In that sort of scenario tolerance of bugs is much higher so you can get away with playing fast and loose with process. If you look at industries where the result of a software failure is someone getting killed[1] then you will find lots of process. This process isn't there just for the sake of having process. It saves lives.

1. http://en.wikipedia.org/wiki/Therac-25

You don't even need to go as far as software that can kill people to find examples where bugs can be very expensive. A failure (wrong data, system down, etc.) in any software that a company depends on to operate its business can be very expensive to the company (e.g., a hotel can't take reservations for an extended period of time during a busy week, or a supplier sends a million-dollar piece of equipment to the wrong country). And it can also be very expensive for the software provider if their customer ditches them for a competitor's product.

Squadron, you can make a much better case for your product than resorting to such vague and meaningless pablum.

Yes process has overhead, can have bugs, or can be incorrectly or incompletely executed. Yes, it's nice to make it easy and reliable, automation is part of that. No, it's not poison.

An error in engineering is costly, people and dogs can die.

Yes, you're right. But if you want to use death as rhetoric, you really should have something against process beyond an explanation that process can be deficient. Go to a company where user death is an actual risk and tell them that that process is poison and you'll get laughed out of the room.

My favorite thing is when there is lots of automation, but nobody knows how the hell it works (because it's been around for 3 years chugging along obediently) and nobody's wanted to look at it (because it's an ugly horrible mess of 'just ship' code) and it just happens to be the single automation that controls whether or not critical customer updates get verified before they ship (wait for it) and now we have to ship (of course) and now it's broken (of course) and nobody has a clue how to fix it (of course) because process is kind of annoying.

If your automation works perfectly, you still need process to make sure your automation works perfectly, and fix it when it doesn't.

That may work for software people. If you to have a production line running, you have more problems: documentation may be there, but the personnel might be unaware/unable to understand it. Some things might be impossible or very difficult to automate. If you train them, they might forget the next day. I'm still looking for good ways to implement constantly changing manufacturing requirements that need to be understood by real people.

The error in saying process is bad, is simply that its also subject to human error.

But process is refined over time. It may have a wrong step, but then you fix that and now it doesn't. That's the point: you write down what you should do so you don't do the wrong thing again.

I'll go further: automation is bad because a human may have made an error in writing the automation code. See what I did there?

Process is fine when used intelligently. One of the things it allows you to do is standardize across large teams how things are done. You can also predict what people are going to do if there's a process (ok, bob opened a bug against that other group, I know they have a next-day triage process to look at it so let's sit and wait to see what they say).

Automation is a by product of Process. Automation hides (and outsources to a machine) the complexities of certain Process(es). When one decides to automate the Process is a question of debate and usually revolves around some sort of Opportunity costs. In the end Process is Great since it sets boundaries (rules) and leads to Automation.

know places where even poison taking follows strict process.

Reply on news.ycombinator.com