What are the problems with process gates?
rubick.com
rubick.com
Process gates create Waterfall. Every process gate is a new step in the waterfall. If one wanted to create a Waterfall methodology, process gates are literally the thing you would use to accomplish that. Which isn't always a bad thing!
For example, we often forgot to backport fixes. So I added a question to the PR template: "Does this PR need to be backported?"
I thought if people saw that question they would stop and think about it, and wouldn't forget backporting bugfixes.
It turns out they just always typed "No".
I just added friction and the problem persisted. When people don't care, you can't fix the problem with process, because they'll just mindlessly follow the process and make the same mistakes that the process was supposed to prevent in the first place.
Then if they don't you can ding them for not following the process. You can fix problems with process, but all that does is set a minimum floor of quality, it doesn't guarantee anything higher than that.
In your example, I'd have standards for coding practices, and reviews to be sure that they're met, and merges aren't allowed unless the reviewers all approve.
What is working, I think, is to have everybody agreed about 1) what actually needs to be done - a good understanding of details of the project - and 2) how it needs to be done - on high level, and on the low level some variations shouldn't matter too much.
Of course this doesn’t always work for every problem, just an idea that may work for some instances of this kind.
One-shot checks are a thing and have a place, but that's not what the article is about.
Contrasted with Facebook where you were expected to commit code to production your first week. I couldn't believe how different the eng cultures were.
(Each has their pros and cons, not saying one's perfect and the other terrible.)
Management has learned too much and not enough from Lean Manufacturing. I worked in a manufacturing plant. I know why some of these rules exist. We aren't using them correctly in software development.
Quality is about improving process, not just inspection. Inspection is required, but it is treated as an input to process improvement. Are your PRs an input to process improvement, or an excuse to yell at junior devs?
Management techniques in Lean Manufacturing are all about producing consistency and efficiency in human effort. Software Development is not really like this. When you understand an IT process well enough, you automate it, and the software product that performs such automation should be very consistent and efficient.
We want that effort to create automation products to be more efficient and consistent. But that effort is not at all like manufacturing. It is not repeatable. If it was repeatable, someone would have written a tool for it already. And thus, if we are doing our jobs, we only have hard problems.
Managing software developers is more like managing other managers than it is like managing workers on an assembly line.
If you are an executive who oversees managers who are making changes to processes, you need to provide contingent guidance and goals, not dictates. Software developers are making changes to processes.
This is essentially what Team Software Process aims to do. It applies a feedback model to the development process to continuously improve predictability and quality. However, it's time consuming and for it to work, you have to capture a lot of process artifacts like time spent on a task, defects found at every stage of development on that task. They basically try to apply something similar to statistical process control to software. It works (quality definitely improves), but it can be a PITA.
Like any intervention that is a "change in process", they should have clear dates and/or criteria under which such a gate either straightforwardly expires, or is at least expected to undergo review to confirm it is still needed, works as intended, and isn't causing more trouble than it is solving.
This isn't just a "project manager" thing. It's a very common mentality found in most professions. Doctors, for instance, are notorious for being very eager to "start" drugs, but very reluctant to "stop" them; especially when "someone else" started it. This leads to an endless cycle of people ending up on bucketfuls of pills every day, many of which interact with each other in adverse ways, and some of which are only there to counter side-effects from drugs that could have simply been stopped a long time ago.
When all of these benefit massively is when there's large information/knowledge asymmetry, but the right questions and review usually doesn't get to the right person. E.g. junior devs architecting things poorly and then wasting time to implement a bad idea. Devs pretending to be designers and coding a UI that's just bad without consulting the designer when they should have.
Does just telling them "you devs to consult the designer more" work? I haven't seen it work. With the right steps to review design, architecture and so you could easily save more time later than you invest upfront.
Not immediately, but it does. The problem with process gates is that they don't fix the problem, they just hide it. Even worse, they treat problem as unfixable by nature.
If your juniors are architecting, something is wrong with your structure. Juniors should typically execute somebody else's designs.
If your devs don't have a good sense of UX, they should not have the power to make the decision to ship the product where UX is crucial. OTOH if they ship internal dashboards for one user (or for themselves), they don't need a designer approval.
There are checks everywhere. Pre-checks at the beginning of the supply process, checks that the supplies are valid, checks during manufacturing, checks during assembly, checks post-production. The value chain must continue to flow freely and continuously, and any bottleneck must be found and dealt with immediately, anywhere in the flow.
During the entire lifecycle of producing the car, there are gates for both quality and waste. You can't get quality if you have waste, because waste leads to defects, which you have to fix, which takes more time and money, and exponentially destroys your productivity & product.
Process gates are necessary to eliminate waste and increase quality. You have to immediately fix the things the gates catch, and continuously improve the process of fixing them, until identifying and fixing problems becomes fast and natural.
So process gates are a good thing - IF you actually follow Lean, and are actively working to improve your process. Most people know jack shit about Lean because the company didn't give a shit about training their employees, so their employees suck, and don't improve the process. Toyota, however, does train their employees, and so the employees follow this process, and the result are the most reliable cars at competitive prices.
If you are resolved to just have shitty people and not train them, then sure you can remove process gates. And you will churn out cars faster. But those cars will be shitty, just like American cars from the 50s-90s. You'll need to spend a lot more to fix all the defects, and customers will be unhappy. If that's fine with you, cool! You're pushing out cars! Progress! But let's not pretend process gates are evil just because we don't want to learn to use them properly.
Of course besides gates (Andon cord, kanban) they have other processes: autonomation, poka-yoke, production leveling, kaizen, 6S, etc. But gates keep production in check, and must be efficient, and part of a continuous improvement cycle.
If you don't have process gates, your end result probably sells more due to marketing and hype (looking at you, Ford and GM muscle cars and pickups).
Toyota ships same cars over and over. You can't copy the output of a factory worker, but you can copy the output of a software engineer.
Even if you ship CRUDs all day long, it's still very different from a factory worker.
Elsewhere in the thread I reference Team Software Process (TSP) which is a development methodology that uses feedback to continuously improve the development process in a way that's very similar to manufacturing methods. I've done it; it works.
And using feedback to improve process is not specific to any industry. You don't need to learn from Toyota to understand that feedback is valuable.
Well, no, they don't, and this is the point: they are constantly refining every car model. They are always fixing bugs in the active production line, working on new features, and every year ship new "major and minor versions" of each model.
Toyota USA ships these models: Corolla, Camry, Avalon, Mirai, GR86, Supra, Sienna, Tacoma, Tundra, Sequoia, Highlander, Venza, C-HR, RAV4, Corolla Cross, 4Runner, bZ4X. For each of those there are 2-6 different trims. Assuming an average of 4 trims, that is 2,380 different configurations of vehicles. Each trim can also have optional components added or removed to customize each vehicle, increasing combinations further. A US Toyota factory worker may support tens of thousands of ways to assemble a car, and those ways change every day.
> Even if you ship CRUDs all day long, it's still very different from a factory worker.
Sure, but even a factory worker is different from a factory worker. The demands are different, the customers, compliance, tolerances. You can't treat them all the same. But some basic principles of managing companies, people, work, etc, can be applied to all of them. Waste is bad. Quality is good. Stopping to fix something once is better than stopping to fix it 20 times.
The impact of a factory issue is often much greater, more expensive, and harder to fix, than that of a software issue. But software workers can learn a lot about quality and efficiency from how well-run factories are managed. That idea gave birth to Lean Software Development (a subset of Agile).
Do these refinements face process gates though? Because that's a better mapping to software.
And perhaps they do---but I highly doubt it's as many as the cars themselves face.
Do you really believe what I meant is that Toyota ships one model, "The Toyota"?
> They are always fixing bugs in the active production line, working on new features
And people who do that are typically not described as "factory workers".
> Sure, but even a factory worker is different from a factory worker
They are the same in a sense that you can usually add more workers when you need to ship faster. Fred Brooks wrote a book explaining why the same mindset doesn't work in software.
Programmers are not workers sitting on the belt and assembling the same software over and over, they are creative persons who have to take lots of decisions.
Nominally it makes a lot of sense to make a vertical slice of a game, it means you’ve demonstrated all the core mechanics to a polished state in a production quality part of the game. All you need to do now is go wide in content production and continue to polish the game mechanics.
In reality no one is interested in whether the game is actually ready just that it’s now the arbitrary time to review. Teams end up building the vertical slice to order so they pass the gate review. It’s full of half cobbled together systems and built with half-finished art pipelines. It passes and drags all this crap into production or it fails and you have a demoralised team. Either way you’ve just wasted months of work to fulfil some arbitrary criteria. It goes hand in hand with the seagull school of management. Fly in, shit all over everything and fly away.
It’s more macro than the articles examples so less a death from a thousand cuts but it’s easily one of the most detrimental things I’ve seen cargo culted across game teams.
The first idea the article suggests is... use a process gate.
Yes, a narrower or short-lived one, but it's quite a mixed message.
I also don't think the death spiral described is inherent to process gates.
If you're going to claim quality gates generally cause worse quality I think you better be prepared to back that up. I don't see it.
Example: several times we've had mid-level devs do work in code that could have been handled by a language function, or built out a portion of the system without thinking about how it impacts another portion, or doing work without writing tests... all things that we've attempted to stress to everyone on the team repeatedly. Other mid-level team members who are already performing reviews aren't calling out these things, so the only thing that seems to work is adding a process-gate where some of the more senior team members are required to approve.
I hate it, but nothing else has seemed to work.
However, the long-term, sustainable solution is complexity reduction.
The reason that stuff goes back and forth between engineering and QA is that it didn't work. When choosing between "releases take a little longer", and "releases have bugs", taking the extra time is a no-brainer. Engineers don't like their code getting rejected, so it also teaches them to spend a little extra time not shipping obviously buggy code. Everybody wins.
- Increasing cycle time means you'll end up jamming a larger delivery payload (or more items) into the cycle. This increases risk, which is mitigated via gate-keeping, so cycle times tend to increase even more.
- Gate-keeping is a form of fiefdom-building. Businesses usually don't consider that when implementing them at first.
- Gate-keeping for problem FOO means everyone will treat FOO as someone else's problem; they're free to ignore foo. Quality gates, in my experience, typically make for lower quality products.
Bugs might slip through even with the most severe code reviews and that's why unit tests, integration tests and end to end tests are for. :)
You are entering a realm which is unusual, maybe it's magic or contains some kind of monster... These are just examples, it could also be something much better. Prepare to enter: the process gate.
Process gates are one of the favorite tools of "successful" managers.
Imagine coming up with action items during post-mortem. Drastic measures are easy to understand and have good optics. Narrow gates look like half-measures.
As a manager, you should be proud of process steps you eliminated, not steps you added.
We could have gates that have a random bypass. Have a 10% chance that the gate must be bypassed for this iteration.
Did any of the bad things that happened cost more that the gate? No? Gate goes away.