The Tyranny of “The Plan”
infoq.com
infoq.com
In the beginning, there was a plan, And then came the assumptions, And the assumptions were without form, And the plan without substance,
And the darkness was upon the face of the workers, And they spoke among themselves saying, "It is a crock of shit and it stinks."
And the workers went unto their Supervisors and said, "It is a pile of dung, and we cannot live with the smell."
And the Supervisors went unto their Managers saying, "It is a container of excrement, and it is very strong, Such that none may abide by it."
And the Managers went unto their Directors saying, "It is a vessel of fertilizer, and none may abide by its strength."
And the Directors spoke among themselves saying to one another, "It contains that which aids plants growth, and it is very strong."
And the Directors went to the Vice Presidents saying unto them, "It promotes growth, and it is very powerful."
And the Vice Presidents went to the President, saying unto him, "This new plan will actively promote the growth and vigor Of the company With very powerful effects."
And the President looked upon the Plan And saw that it was good, And the Plan became Policy.
[0] Mary Poppendieck’s “The Tyranny of ‘The Plan'”. https://chrisgagne.com/1255/mary-poppendiecks-the-tyranny-of...
1. Mary recounts her experience of working with computerised MRP planning systems. These detailed plans were fragile and would need to be thrown out once reality began to diverge from the plan, then re-planned. Contrast with a pull-based approach (Toyota etc)
2. What did they do before computers? Anecdote/case-study of how the Empire state building was designed and delivered on schedule and under budget. Summary of success factors: teamwork of owner+architect+builder. deeply experienced builders - fixed price contract! focus on key constraint of material flow. Decoupling of workflow "pacemakers" - the building was designed to allow parallel construction of components - windows could be fitted in parallel to floors, once the steel frame was built. Cash flow thinking. Schedule was not based on design of building, the design of building and the workflow to construct it was based on the schedule and other constraints.
3. Sketch of how "pull scheduling" could work for a software product. Delivery broken into sequence of 2-3 month increments. Don't make all the decisions up front. At the end of each increment, decide what will be worked on in the next increment. Time-box not scope box. E.g. decision point after first two months to review customer interest and decide technical approach. Mary emphasised need to really understand customer problem, intention, technical approach, and have everyone in the project talking the same way ("if you don't figure this out, if you don't pass this test, you stop and fix it, and you don't go on with the rest of the schedule"). Then 3 months of working on proof of concept, then review proof of concept and decide alpha features, etc. Baseline features for production release not decided until after reviewing results of beta.
4. Discussion of reliable workflows. Inputs from suppliers, outputs to customers. Rapid feedback if inputs or outputs are not fit for purpose. Identify and fix root causes of problems.
5. "Where do plans come from?". Anecdote/case-study of the US Polaris missile system. Complex, highly political project, unconstrained budget. Use of PERT [1] to manage project publicised by program director: "a facade to keep Congress happy" - but PERT not actually used to manage the project in early years - constant requirement and scope change, PERT bypassed by technical officers and contractors. Summary of success factors: quality of leadership - one technical director had control over performance requirements and signed off on all major decisions for first 8 years, focus on development, decentralised competitive organisation (three different contractors developed competing designs for each major subsystem, then a winner was selected. each contractor had technical freedom to propose whatever they thought best), emphasis on reliability, espirit de corps.
[1] https://en.wikipedia.org/wiki/Program_evaluation_and_review_...
So I've been studying a lot about WW1 recently and one of the things that struck me was the incredible similarity between how many of these generals operated in the early years of the war. With detailed down to the minute plans that would account for everything, trying to be more and more precise, and how the bigwigs on a software project operate, with a belief that if they can just nail things down more tight and make sure they've got every dependency mapped out they'll finally make it work and it will avoid other problems.
The results are the same as the WW1 battles the first part goes well, things seem to flow along as predicted everything is great in the first day, but as things go on reality diverges more and more from the plan and the more people insist on the plan the worst things get because they haven't been able to adapt to the real world.
My point is the tighter and tighter your plans are and the more comprehensive they are and the more autonomy they take from individuals on the ground doing the actual work, the more likely it is to go poorly and the more catastrophic it will get the longer you push on.
In this world, there is no way to have a tyrannical plan.
So the planning ends up being one of the more useful things to do because it prepares you for both your intentions but also to handle things when they fail (in some fashion). The plans, themselves, are useless because they will very rarely be carried through as written. You can't just read a plan and memorize it and expect to carry it out in an adversarial situation. If you're lucky, or your adversary is severely outmatched, then this can work. However, you can't count on luck and you can't count on your adversary being unprepared or incapable or incompetent.
Planning, contrasted to plans, prepares you (or should) for when the plans fail and you have to adapt and overcome in the field.
The enemy gets a vote
This generalizes beyond what's strictly though of as a "plan" (but if you squint enough may have some plan-like elements to it).
There are many forms of documents that we engineers produce and they serve different purposes at different times. And some of their uses may not be very effective in most cases, leading to wasting too much time in over-specifying a system that has not yet been built.
Yet, writing design documents / specs may be still useful despite the output not being useful. It's the act of writing it that yields most value.
I thought I added a nuance to what Eisenhower's said (or at least to how I noticed people interpret it) but apparently I did a poor job and adding more words would just make it worse.
An aphorism by definition doesn't contain all the nuances. If we looked through Eisenhower's whole life, which is pretty well documented, I'm sure we could find lengthy explanations of his approach to war.
What I try to add here is the fact that other forms of technical writing can be seen as forms of planning and thus there is a connection between the two aphorisms
The nature of software prevents experience.
This feels like a profound statement about programming work. And at the same time, I find myself pushing back on it in that we end up encountering things that are enough like something else but can’t be encapsulated in a library that experience still counts, whether that surfaces in the form of common practices (algorithms, design patterns, programming idioms, etc.) or just plain experience (I’ve seen this king of bug before).
Out of context, I intended it to express an ideal, that we should strive to encapsulate our experience, of those similarities.
One barrier is insufficient data to formalize just what that similarity is, and how it applies across different scenarios, especially when hidden amongst other factors.
Arguably, ChatGPT is a way of capturing such similarities - though not in a library.
I agree your suggestion of standard practices is difficult to capture. They are a kind of language for communication with other programmers, so encapsulating them defeats their purpose. Debugging is another good one - if the same type of bug keeps coming up, can we automate it away? If so, is that solution "software".
Another task that is difficult to capture is defining the problem, because this involves the world, with all its detail and complexity to cut through - though some would say that "isn't software".
In the case of the widget factory, you get many attempts at building exactly the same thing. In a typical software project, there's only ever one thing, and with each change in each increment, you're constantly changing the design and construction of that single, increasingly complicated thing.
From this perspective, both the case studies of the Empire state building and the Polaris missile system are much better approximations of how incremental design and delivery work in software projects than the mental model of a factory churning out indistinguishable widgets.
One can argue most floors in the empire state building are pretty similar, so experience gained while designing and building the 3rd floor can apply to the 53rd, and to some degree that's true. But if we mistake our floor building for zipper manufacturing and conclude this means we will be able to keep adding another 25 floors per quarter and extending the height of the building indefinitely, as we gain more and more experience at building floors, boy are we are in for a big surprise. The floors aren't independent, the project is to design and deliver a single system - the building - each increment during the project modifies that single thing. We'd better have some people involved who are thinking really hard about how the entire system is going to come together, what constraints the whole system is under, and testing our design and build assumptions.
That you could also apply to all other kind of craft (music, painting, sculpture, woodworking, building, etc.): you _could_ understand it of having a limited set of tools, techniques and vocabulary, to serve about often the same kind of output.
But now, output still vary greatly to the accustomed eye/ear/body, and how you get to that output even more so.
Experience (and taste, which refines with it) will always make a difference: how the need is understood, which technique to apply, which hypothesis to form, which to explore or relegate, how to plan/distribute work (on one’s own or on a team), how it will feel working on it, how people will be satisfied, how fast, how well, how costly, etc. all the way to the finished « thing ».
Edit: a very specific/limited example: working on the very same project from the different perspectives of a large worldwide consultancy team, a local-based design/tech studio, or a two-people startup team will be a hell of a ride because from project to tech to business to people, expectations, vocabulary, behavior, speed of execution, focus, know-how, freedom to experiment, tools, so many stuff will differ.
But we are all to be « haunted » by our own experience if we are to live, and to have to know how/when not to take it into account (which in itself is some form of experience too).