Plan the Sprint, Not the Project
critter.blog
critter.blog
Developing a plan is not waterfall. Designing an Architecture is not waterfall. Agile is about adapting to change as it occurs - how do you know if there is any change if you didn't start with a plan?
Good plans and designs account for risks and unknowns and anticipate (certain types) of changes and delay locking into assumptions until necessary. But not having a plan or design is not agile - it's failure.
Easy. The designs change. The requirements change. A plan is not a design. A plan is not the requirements. A plan is a step by step description of how you are going to implement the designs and requirements, and in what order. And if the designs or requirements change enough, you now have to throw out that part of your plan.
I think you have a very narrow definition of "plan" that does not correspond to how most people use it.
More importantly, is there a definition of "plan" in the context of software development where my claim fails? The claim that if requirements or designs change enough, some parts of your plan have to be thrown out?
The meaning I used is also a primary definition and often enough the first definition you'll find in dictionaries. [1] [2] [3]
[1] https://www.wordnik.com/words/plan
[2] https://www.thefreedictionary.com/plan
[3] https://dictionary.cambridge.org/us/dictionary/english/plan
Regardless, I'm not particularly interested in the dictionary definition. This is English, and what matters in English is how people actually use the word. You are literally the first person I've ever heard of who believes that "plan" is restricted to high levels of detail.
Yes it is. Fundamentally, any project stage that you can finish without delivering a working system to the end user is waterfall.
> Agile is about adapting to change as it occurs - how do you know if there is any change if you didn't start with a plan?
Why do you care whether there is change? The whole point of agile is that it doesn't matter if the customer's requirements change, you'll deliver value to them just as effectively whether they do or don't.
Understanding this, agile works fine. But zealots without practical experience want everything to fit the agile religion. And so you get the confusion about how to plan an architecture, how to develop overall system requirements, when everything must be incremental and in sprints.
We had pretty good solutions to many of those things twenty years ago, but that is "waterfall" now, the tarnish used to denigrate everything agile acolytes don't understand.
Agile is a product now, sold by ignorant business consultants. I've never seen a more ridiculous mess than watching a non-software organization trying to transition to 'agile' in the whole organization. Sales, marketing, business process management, doing sprints, with scrums and epics and story point estimation, etc. Yes, that is what the agile industry is selling now, agile for the whole organization, in any industry. The horrors this will produce we have yet to fully enjoy.
This reminds me: I saw SAFe everywhere a few years ago and now I don't see it in the wild as often. It seemed kind of snake-oily to me. Does anybody have any idea what happened when I wasn't looking?
But it's critical to avoid waiting too long to make decisions. Because that means the decision was to make no decision, and that has only a random chance to be the right one for you.
Planning is a big part of how you have the information you need to know when to make a decision and when to defer it.
Plenty of work be organized to fit into ~two-week, focused sprints, and can be done efficiently that way. Use your planning to figure out what should and should not be tackled that way.
There's also work where it makes no sense at all to do while looking ahead at most only two weeks.
For instance, if you have a good experienced architect, a small amount of time spent on a high-level arcitecture will pay you back on even a slightly successful project. Not to mention it will increase your agility. E.g., don't design a sophisticated privilege mechanism until you know you need it. But if there's a reasonable chance a privilege mechanism will be important in the future, know where one would fit in in a way that wouldn't require a substantial rewrite. Now you'll be able to slot in a simple privilege mechanism, if/when you need one, and later replace/evolve that with a sophisticated one, if needed, all quickly and without substantial rewrites.
What I mean by that is, the project which did not decide things on time and tried to save the decision until they knew everything suffered much more than the project which decided in a timely manner and re-evaluated the decision in the rare instance that they were wrong.
It was greatly inefficient to have people sitting around not knowing if what they’re working on will even be used because they’re touching something that hadn’t been fully decided on yet, but was required for them to deliver on something else which _had_ been decided.
It is sometimes better to decide, iterate and reevaluate. Instead of attempting to defer a decision.
> For instance, if you have a good experienced architect, a small amount of time spent on a high-level arcitecture will pay you back on even a slightly successful project. Not to mention it will increase your agility. E.g., don't design a sophisticated privilege mechanism until you know you need it. But if there's a reasonable chance a privilege mechanism will be important in the future, know where one would fit in in a way that wouldn't require a substantial rewrite. Now you'll be able to slot in a simple privilege mechanism, if/when you need one, and later replace/evolve that with a sophisticated one, if needed, all quickly and without substantial rewrites.
No. This is the most seductive myth in software development, and it's exactly what Agile exists to avoid.
I've worked with experienced architects. I've worked with very intelligent architects. I've worked with extremely highly paid architects. None of them ever managed to add value to the project rather than subtracting it.
Write the simplest thing that might possibly work. Write plain code that's easy to refactor. Ensure that your test cases (or better still your type system) capture your business requirements so that you can refactor fearlessly and mercilessly. And as if by magic, retrofitting that sophisticated privilege mechanism will turn out to be easy, because you don't have a bunch of architecture getting in your way.
How do you mean?
Of course my perspective might be skewed, I have only really worked in small teams.
How would you build a rocket if you only thought about the next 2-4 weeks?
How would you scope the size of any given project if you didn't think about it in totality?
Successful development is always a balance of waterfall and sprints except maybe when you're in maintenance mode for long living legacy projects.
Waterfall = strategy
Sprint = tactics
Why do people ask for estimates or fixed bids? It's to manage risk. And how often do projects not go over estimated budget? Even fixed bids are poor at managing risk. Lots of consulting companies will walk away from a fixed bid contract after months of work at the point they realize they will lose money on it. And honestly, what's the quality of work once the contract is no longer profitable? We've all seen it. Everything is a quick hack at that point in a desperate effort to achieve the minimum that could be defensible in court. A fixed bid project means the consulting company and the client have opposing interests: the consulting company wants to get it done as cheaply as possible and will make every shortcut they can get away with. And the client wants every last bit of quality they can squeeze out of the contract for that price. No wonder it so often ends in dissatisfaction for both sides.
What are the options for the client when the consultant says "sorry we didn't actually finish and we are out of money"? They either pay more or have to be willing to sue the consulting company. Most companies suck it up and pay more. This is so endemic to the industry that most companies put extra in the budget for software project overruns no matter how tight the fixed bid contract is.
So one way to sell agile is to help clients understand how it can help manage risk better than a fixed bid contract or always inaccurate estimates. And how it can be used so that the interests of the consulting company and the client are better aligned.
Charging per sprint is only superficially different from charging hourly/per day/per week/biweekly.
Consultants will want the project to last as long as possible. Clients will want it to be over as fast as possible.
You have to impress the client with every sprint? Just like a regular contractor must impress the client with every milestone.
How do you know how many sprints you want to pay for? So you have a budget, just like any regular old project.
You can prioritize features? But why wouldn't you do that without sprints?
What am I missing?
Agreed. Agile only puts you in a more flexible relationship. It does not fix every alignment issue. For that you need to be more creative with the contract. However it does spread the risk more evenly. And by you as the client taking on more of the risk instead of pretending an estimate or contract erased it, you will do a better job of managing the risk. Every sprint you'll decide what must be built next. Every sprint you get the opportunity to change requirements. Both of those are far better ways of managing risk than fixed price contracts or estimates.
> Charging per sprint is only superficially different from charging hourly/per day/per week/biweekly.
Not if you are delivering a working product at the end of each sprint. And yes, milestones are somewhat similar to sprints. Except as a client you've now locked yourself into what gets built and if requirements change you've got to renegotiate the contract.
> You can prioritize features? But why wouldn't you do that without sprints?
Because your fixed price contract legally "guarantees" all the listed features will be included.
> Clients will want it to be over as fast as possible.
And you can have that if each sprint ends with a working product. Up to you when the product has enough features included that you are happy with it. Or that you are out of budget. And you've managed risk aligned with the consultant instead of getting an unrealistic estimate they won't meet, or having a to head to court with a broken contract.
Many enterprise scale business centric applications have tremendously complex business rules and overreach. One can not simply make a casual decision last minute, and expect perfect execution, towards an end result of increased business value.
EDIT: I see he is a Drupal consultant. There is a very explicit ceiling on the complexities of primarily CMS driven sites.
- building
- designing
Some parts of agile do apply to building. See Lean Manufacturing, parts of it are responses to maintain quality (Andons used to stop production) and parts are to change over the manufacturing floor to build something else (same plant can make many kinds of cars without a fixed production line for each, letting them meet demand).
Other parts of agile, though, are aimed more at the design portion. Sprints, for instance, are meant to give you short iterations that let you frequently evaluate both your process and your product's success/failure.
You can apply agile concepts to developing almost any kind of thing. Actual rocket scientists create simulations, scale models, full-scale models, prototypes and other things on their way to the final system. So they're already applying the same iterative approach that most agile proponents endorse. The question is, is the iteration deliberate or accidental? Or do they try to deny the iterations even exist? Once they get to building the rocket, that might be less agile depending on circumstances (is it the first one they've built? are they setting up a production line? is it a one off?).
Note that flexible factories have other limits. Honda can't make a large truck in their factory as there isn't room. I don't know if Honda regrets this cost or not.
> Fisher's Fundamental Theorem: The better adapted a system is to a particular environment, the less adaptable it is to new environments. -- Gerald Weinberg,"The Psychology of Computer Programming"
There are always trade offs. The more flexible you make a system, the less efficient it is (most likely) in achieving its (now increased) goals. But the more efficient you make it, the harder it is to adapt it to fit additional or changed goals. Which trade offs you accept depend on your objectives, and sometimes your ability to spin up a whole new factory.
At least often/commonly. If we don't always do that with software, I think we always do it at least to some extent (in an unusual way compared to physical things), and software is (often if not always suitable to designing and building.
I think "agile" as we know it (which is really short for "agile software development", that's what the agile manifesto hasin it's title!) is particular to something you are (or can profitably) design as you build.
While there are analagous things in physical products, and there is (or profitably can be) some element of design/build cycle in physical products, I think rarely to the extent it can and does happen when making software. Rockets are probably at the far end of separating design and build.
But the tldr version is probably just: The Agile Manifesto's full name is literally "Manifesto for Agile Software Development". While some of the things may happen to be useful or applicable in other fields of endeavor, others may not be. Saying that something from the Manifesto for Agile Software Development and ensuing methodology doesn't apply to a rocket is not a relevant critique of the methodology as applied to software, as intended.
I actually had something about that but cut it for some reason.
I'd like to point out that while the manifesto is about software, a lot of the ideas (beyond the manifesto itself and influencing the manifesto) come from other disciplines. Mary and Tom Poppendieck, for example, are major proponents of Lean Software Development, whose ideas are a translation from the world of physical manufacturing. Many of the DevOps ideas (not necessarily how it's played out) come from Eli Goldratt and his Theory of Constraints.
An interesting thing about both Lean and ToC is that neither is terribly prescriptive. They're actually generalized concepts that can apply to almost any kind of system and (mostly) lack prescriptive text saying, "Thou shalt do...". Compare that to much of the Big-A Agile movement which is inflexible, ironically, and often very prescriptive. Perhaps not down to the specific tool that should be used, but often to the specific methods that should be used (Scrum: Thou shalt have 1-4 weeks Sprints; Thou shalt have time boxed meetings of this nature; SAFe: Thou shalt have a massive meeting every 6 months in the format we describe in this chapter).
Ok, that's a bit unfair to Scrum, it's actually more flexible than that. But I'll happily burn my books on SAFe (picked up for a previous job).
Many of the other ideas for Agile are born out of Systems Engineering which had already dealt with Waterfall. You had the V model (an improvement, but still many of the flaws if taken too literally, over Waterfall) which makes V&V run concurrent to design and development and encourages feedback loops. In many ways this is akin to TDD. Incremental and iterative development processes, which are fundamentally no different from Scrum except Scrum pushes you to much shorter times and has a stricter structure. Evolutionary development which is really close to what many, if not most, agile proponents endorse (MVP, then add on features and modify the base as needed). None of that came from software (specifically), it came from developing physical and (in their parlance) cyber-physical systems.
But my rocket needs to be ready when the earth-venus system is in the right stage or I delay a few years. Agile cannot handle this. Agile can handle getting rid of some features to meet the deadline, but waterfall can do priorities as well. What is needed is watching the end date and assurance that you will make it, or warning to add people before the cost of adding people is greater than their future contributions.
> Agile can handle getting rid of some features to meet the deadline, but waterfall can do priorities as well.
Waterfall is not particularly good at handling priorities. It's actually worse, in my experience, than anything else. Waterfall is about the order of doing things (it is fundamentally linear), committing to a plan (lack of flexibility), and has minimal feedback loops (lack of responsiveness).
If you have a 2 year deadline (true deadline like in your example), Waterfall only helps you if you assume everything will go right or put in drop dead dates for lower priority things. But even then, you'll have wasted so much time in your planning and designing you probably won't be able to keep it on track within a reasonable budget extension.
What you really want to handle priorities is an incremental and iterative development model. You know your target, but you break it down into smaller increments and build it out over time. After each iteration you re-evaluate your ability to hit the target. Within each iteration you may use a Waterfall-styled approach (particularly if it's a portion that's well understood in your organization or sufficiently small) or you may extend the iterative approach by adopting an Agile approach (think 3-6 months of 1-2 week sprints).
Using your Earth-Venus example. You need X, Y, Z. You want A, B, C. You only have 2 years to deliver. In Waterfall you'd spend 3-6 months designing everything before building it. If you're smart you design it with a way to drop A, B, and C. But that's a lot of time spent up front on contingency planning and pushes you closer to 6 months. Now you're 18 months from the deadline before production even starts. Oh, and testing doesn't happen until pretty late in this process. So your production is constrained to no more than 12 months in order to allow sufficient testing time (6 months).
If you're actually smart, you create checkpoints/milestones and do it incrementally. You start work on X, Y, Z and as much of A, B, C as you can afford. After 3 months you see that X is progressing fine, Y is possibly going to miss the target, and Z is definitely going to. So you check and A is going to be done in a month, B turned out to be a cluster, and C may complete on time. Cut your losses, B is gone, the money and manpower are moved to the core mission.
Next iteration you reexamine, and find out that you C may still complete but Z is barely back on track. So you take some or all the funding and people from C and dedicate them to getting Z back on track. In the best case C can still be done, but it probably won't. We accept our losses and continue this process.
That is a fundamentally agile approach. It's about responding to the changes that come up. Waterfall is not, in itself, capable of this.
He also sites case examples of restaurants and house remodellers using scrum successfully.
When we talk about building an MVP (minimum viable product), the V part is very important. It does have to be useful. It does have to look great and work great. It does have to be performant. Edge cases should not get past your QA department.
So with those constraints, about the only way you can get out a prototype in a sprint or few is by ruthlessly reducing scope until you get to the core part of your product that is adding value. That process in itself results in far better products.
My goal (that I sometimes don't achieve) is that each sprint results in a product we can ship that is better than the last iteration. Run out of budget? Company priorities change? No problem. We still have a useful feature we can ship.
Per my previous comment, this is the wrong process for plenty of products. You have to know what you are building and what the risks are, what the constraints are that you are optimizing for, and the pros and cons of different types of processes before deciding which process is the best fit. If you have a strong idea of what you are building ahead of time and you have experience building those types of things and you know there is already a market demand for those types of things, then by all means come up with a solid plan ahead of time and then execute it.
I was responding in particular to "a quick prototype in front of real users gives you far better information than any amount of internal brainstorming." Maybe it's just a semantic difference but to me "quick prototype" doesn't imply look great, work great and handle all edge cases. Perhaps you meant quick prototype that looks great, works great and handles all edge cases but I think that implication is often lost on product people. They want something deployed and don't care if all of the engineering team's fussy edge cases are handled.
Yep.
> I've just been in the position many times in my own career where product managers are pushing to get something, anything out there so they can get "real user feedback"
Me too. There are a lot of bad product managers and even companies out there. But I have worked with great ones occasionally, so it's worth describing the ideal so we know what we are aiming for.
> I was responding in particular to "a quick prototype...
You're right that I need to qualify that because it more often means what you rightly assumed based on your experience. I guess I'm just so in the habit of demanding that we reduce scope that I too easily forget that "quick" more often means the compromise was quality.
I could be wrong, but I think that was less of the case in the past. Or perhaps we just didn't know better. Building big expensive things that can kill lots of people - I'm thinking infrastructure, commercial planes, etc - comes along with the idea that we should know ahead of time how we will make it safe without it failing. We can't build a prototype of a skyscraper in a city and have it fall on people. But perhaps the way we engineer those types of things could borrow a lot more from agile than I was initially claiming.
You do know what the end-state is. It's called the minimum viable product, MVP. The most successful product owners are ones who are technical and can bridge the gap between the business requirements and technical implementation. For example, there would be no SpaceX and vertical landing of rockets if it wasn't for Elon Musk, who is both an engineer, a businessman, and an entrepreneur. He gets the full picture and can talk to all divisions of the company with incredible competency.
If you have a product owner who knows nothing about building software, in charge of a software product, you'll have a much smaller chance of succeeding, than if you had someone who was both business and technically savvy.
Okay, I’ll bite...
I’ll grant you the 2-4 weeks part isn’t quite right for building a rocket; but the key insight of Agile, of developing in small steps and staying flexible, definitely applies.
You do not start by designing the entire rocket before building anything. Your rocket won’t work. You start, you must start, by building a series of prototypes and exploring the design space. As you do that, you gradually refine your ideas of what’s possible and how to do it. As far as I know, every successful rocket design has followed that path.
Sure, at some point along the path, you put together the blueprints for the final design. So there’s an element of waterfall in there. But staying agile is still critical. Unexpected stuff is sure to come up and you have to retain the flexibility to deal with it.
As others have noted, software is somewhat special in that it’s cheaper and faster to build than physical stuff, can be copied and refactored much more easily, etc. And that shifts the agile-to-waterfall balance where you can realistically talk about working entirely from short term requirements.
That's a total non sequitur. Agile is all about building the sellable product. But doing it iteratively works a lot better than doing it via a big development plan.
To clarify for your sake, once you have hardware in the loop (eg mechanical engineers designing things, manufacturing scaling for production) and a large commercial organization who needs to be ready for a product launch, you need a waterfall type plan. Agile principles (customer focus, respond to change) can apply, and you can break up the work into 2 weeks, but you do need to look past the next 2-4 weeks and know where things are planned. Commitments matter in the $millions.
Software-only SAAS has the luxury of avoiding that complexity.
We're not building rockets. We're building business software. And the requirements for that are not a hardline set of rules laid out by the laws of physics. They are soft and fungible. Beholden to the whims of a product manager, designer, or client. Generally almost nothing is truly understood about the shape and function of the resulting software at the beginning of a development process. And sticking to a "plan" that was laid out before these things have actually been discovered is a recipe for disaster. Better to build in tiny, discrete chunks of usable MVP that can be used as a guidestone to the next desired level of functionality.
If you’re actually building a rocket or any other physical thing with real world stuff (like some of us here) you need a real plan. The fits software engineers get into about agile is odd at times. I don’t really get it.
"plans are useless, but planning is indispensible".
Also not every department is agile and wants those 6 months delivery plan.
Only software is in scope
Remember that the “build” phase of software is relatively instantaneous. The design phase is the only part anyone sees anymore. (The heroes who wove the Apollo ROM were examples of actual program builders).
Making the Saturn V static test rocket was heavily design work slowed by manufacturing. Making Apollo 11 was a process of knowing mostly what was needed and building it from start to finish.
Planning is necessary and useful. The detail of that plan depends on how far out you're going. The next sprint's plan should be very detailed. The next year's plan should be a rough sketch. The next quarter's plan should be somewhere in between. And regardless of the level of detail, those plans can still change if they need to.
I've run into enough garbage code and garbage systems to see what happens when people don't plan enough up front. No thanks.
It's odd that this bit from the Agile Manifesto is used in part as justification:
> Responding to change over following a plan.
I don't read that as "don't plan"; I read that as "plan, but expect your plan to change over time".
Yikes. Dropping tickets in the backlog for ideas that pop into my head is a great way to make sure they're not forgotten! The team may decide later that it can be closed, but it opens the door for a conversation and helps collect ideas.
So far, in 25 years of software development, the best track record I've personally seen for on time and on budget was a big defense contractor that had a bunch of senior engineers and a relatively old fashioned waterfall process with some "extreme programming" and agile practices thrown in. It was documentation heavy and a hassle to think things through up front, but it reduced surprises later on, and the rigid documentation and process made it easy for different people to take over work at different stages of the workflow (requirements -> design -> coding -> testing) for each feature.
It wasn't perfect, but it was predictable, and I never saw them toss features out at the last minute or deliver a half-assed release with a bunch of minor changes.
And I'm sure the response to this will be a dozen people telling me all the companies I've worked for have done agile wrong, and then everybody will have a unique idea on how they should've done it instead...
Agile might work great for CRUD websites, but it doesn't seem to work very well for other types of projects, and choosing something else seems reasonable.
I guess my point is, even if these projects aren't "real" agile, I don't know if it's a real problem. The goal isn't to be agile, it's to get software developed.
Of course, you need to know the direction of the project. But spending time on details that you cannot directly validate with implementation, risks both wasting time because of invalid assumptions, and risks wasting time because the requirements may change when you get around to implementing it.
Remember that "planning a sprint" gets very deep very quickly. It's not just about setting direction, it's about deciding display names, storage types and UX interactions. The point of the article is that that level of planning should not be done months in advance.
Executives and clients like to see timelines and total project estimates. I wrote an article* about this. The idea is that you do some rough planning, applying the principles of Agile story point estimation, and extrapolating a rough/relative timeline.
Relative timelines with non-specific dates are useful for getting an idea of the scope and possible time of a project. Helps determine what should be built first, prerequisites, and what other teams might need to be involved (and at what stage). Also useful to get the product managers or executives thinking a bit more deeply about the project and perhaps reconsidering it, or reducing scope if it looks like it will be larger or take longer than expected.
Once a project is kicked off, after a few sprints the initial plan can be updated based on the actual velocity of the team.
* https://jonpurdy.com/pm-framework/how-to-quickly-plan-and-es...
"Responding to change over following a plan" specifically says "over", not "instead of!"
They even say it twice, to avoid misunderstanding: "That is, while there is value in the items on the right, we value the items on the left more." In a document that's only a few sentences long, to repeat a meaning is a little like shouting.
Don't value your plan more than your ability to respond to change. But do still have a plan! How will you know if you're responding to change without one?
Product strategists, though, can't afford to be so shortsighted. And most companies don't (and maybe shouldn't) have such a clear separation of roles.
I've found that, when your job involves wearing multiple hats, the trick is to pay attention to what hat you're wearing at any given moment. Don't let them all get muddled up. When it all blends together, then the smallest unit available to you to boil is the whole ocean.
People in this industry need to let the worlds "lean" and "agile" just die the death they deserve because they mean literally nothing at this point. It's nothing but lip service for SEO and junior managers.
Committing to the plan (especially a long range one) with no intent to deviate is just stupid, and is the critical flaw of Waterfall.
Agile is about responsiveness. In order to be responsive you need tight feedback loops (a reason the idea of sprints and scrum are ok). But are you responding if you don't have a plan? You aren't responding, you're reacting. Make the plan, but don't make a commitment too far out (commitments are kinds of decisions). Respond to changing circumstances and don't let yourself get caught in an action-reaction loop, because you can't react fast enough. At some point you have to think and respond.
Also, "decide as late as possible" is better read as "decide at the last responsible moment". If I know that I can get someone hired in two weeks, I can, in the first phrasing, wait until 2 weeks before someone retires to start that process. But it's not responsible. The last responsible moment is probably 2-4 months (at least) before the retirement so people can get trained up and moved around.
I always find myself curious about the author and how they arrived there. Are they junior? Have they worked on multiple teams and/or large projects before? Are their managers terrible?
I have a hard deadline external to the company by which time the project, and it is a project, needs to be done of else the company can't sign up new customers.
It would be pretty irresponsible of me to only focus on the next two weeks until it was done as I'd have no way of really knowing how close or far it is to completion, how we're tracking and if we need to make adjustments.
We did a rough plan. We worked to it. We evaluated how we were tracking and we shifted things in sprints around accordingly.
The benefit of agile is introducing short feedback loops that provide information you should be acting on.
Good project management also provides benefits.
Take the benefits from both, or leave one set of benefits on the table.
The most valuable thing is your ideas but only in the present. Ideas are living things that unsmart with time...
The more you shape things, the less freedom you will give to yourself about the future.
Ideas are moving things in the head and should be kept like this.
Stick to just one month planning ahead is largely enought.
Like best code is no code at all, over-planning is just wasting time.
Unfortunately, I'm afraid this doesn't survive the real world, unless you're a small consultancy w/ a great relationship w/ your client, or you're doing work on your own.
A company that is any larger cannot afford to rely only on collaboration, it's impossible to have entire teams collaborate w/ each other since the communication overhead explodes [1]. I don't think you can make everybody converge down the line without some planning ahead and definition of some contracts.
Another situation is if you have any kind of external deadline (e.g. regulated companies). You can't answer with "we'll figure it out, trust me", you need a plan - even if you happen to have to change it.
The point of the Agile Manifesto was responding to change OVER following a plan - not that you shouldn't have a plan in the first place.
The team doesn't necessarily require those waterfally milestones - it could be as simple as a value delivery statement ("Fastest shopping cart west of the Mississippi" or something) - it doesn't have to be a whole big deal with specifics, but without it i'm not sure how you determine the sprint goals, let alone what the specific technical tasks or stories are going to be without that shared context.
Lastly, I'd definitely recommend some sort of longer-term roadmapping exercise up front to determine what the high risk things are so you can solve them first, and test the big hypotheses quickly. That would be a nice lean thing to do.
Plan the sprint, sure. Be militant about it if you want. "Shut up and plan the sprint!" But the sprint is not the goal. Delivering increments of value is the goal. The product owner looking at the sprint goal needs to know what they're trying build, and a longer vision can help.
I'm not sure the author meant to sound so short-sighted and maybe was expressing frustration around customers getting so wrapped up in the "plan" that they end up waterfalling / micro-managing to death, so I get the point, but there's a way to do this that doesn't sacrifice the plan, too. They probably know that.
https://jeffdoolittle.com/2020/05/22/dont-disrupt-agile-drop...
If you work in a cost center, any development methodology you use will look like waterfall.
If you work on a software product, you may be able to work in a more agile way.
If you fail to look beyond that myopic vision, then when a tru need materializes, it may be too late to react.
Agile is shown as God in this post but..why is Agile so good according to the author? The main advantages is the ability to course correct fast and be reactive but if you are making good work, Agile blinds you from seeing the whole product and causes bad, segmented, low-focus mistakes. If you are TOO agile you may build a DB/API/Logic/Data structure that is too narrow and limiting of the eventual goal.
Personally the real danger of this thought process is that it limits developers from understanding the whole of what they may be building. That's downright draining, makes every sprint seem like going to a run to nowhere. This method and plan seems just like an excuse to let management do less and define and shape work to be done less so that you can avoid the dangers of 'Waterfall'.
and it sounds awfully close to being body rental, with all that it entails.
Is "we will build a CRM" a plan?
Is "we will build a mobile CRM app" a twice as detailed plan than the one before?
There are a lot of relevant opposing forces in this discussion. Reasonable people can disagree about where in the middle to meet.
If they matter, most customers will want their money back when 1.0 gets delivered, regardless of what was said during demos.
"Plans are worthless, but planning is indispensable."
I can't recall where I first encountered this gem, but it resonates.
Throwing your hands in the air because you or you're team is unskilled in the behavior is just unprofessional.
That's so few!
Most companies fold, most construction projects go over budget too. I don't see anyone saying we shouldn't try to get good at budgeting construction, or estimating & planning a project before starting.
We're not just managing the risk of failure, we're managing days & dollars! I don't understand why software engineering is the one form of engineering we don't think we need to be able to plan.
The majority of what software engineers build is new territory. Estimating and planning are an attempt to manage risk. And a really bad way of managing risk when you've never built that thing before. Actually starting to build the new thing in question will give you far more useful information about possible time and budget than any amount of planning. And you still won't get it very right. We need to stop pretending that building completly new things is actually possible to estimate. We cannot predict the future and trying to do so is unprofessional.
I think this is where we just fundamentally disagree.
I can send you research about zero defect software, or the economics of software quality, or even research of across many disciplines how delivering value earlier is not in conflict with planning. Time spent planning in all cases saves time and money. However, if you think it's all going to be invalid because of your perception of new territory, then our discussion probably won't go very far.
I encourage you to read Capers Jones and Tom DeMarco, but they're not new.
Please look up the papers on planning & quality & cost & schedule from Capers Jones. The textbook is literally titled "The Economics of Software Quality". Tom DeMarco's classic is "Controlling Software Projects".
I'm trying to give you a starting point to see that it's not even my anecdotes, but repeatable studies that show this.
Your examples are that requirements change, and it's hard. Of the few buildings I have built, the requirements changed drastically there in all cases, so I can only say that software engineering is not magically different. After decades of practice with software engineering my experience of planning lines up with the textbooks you can find on the topic, and not this blog post.
I'm not claiming anything that's not well understood: https://ieeexplore.ieee.org/author/38632866300
Planning upfront takes less time than not planning up front, in all cases. You can plan to do a prototype, you can change your plan later, the activity of planning itself produces clarity.
Reducing the argument to "planning away" is the same as me saying you're "flailing around like an amateur". It's not what you're saying though, so I wish you'd see this thread as a discussion rather than something to win with gotchas.
Exactly. You could learn about when and how to be more efficient when planning isn't called for.
> Planning upfront takes less time than not planning up front, in all cases.
It does not.
> you can change your plan later,
That costs time and money. Maintaining and changing a plan does not come for free. The more complex and further out your plan reaches, the more you'll have to change it.
> the activity of planning itself produces clarity.
Yes it does. And sometimes the activity of building a prototype produces more clarity than a plan.
> I'm not claiming anything that's not well understood. : https://ieeexplore.ieee.org/author/38632866300
I like Capers Jones too. His recommendations are not the silver bullet you are treating them as. He sells software used for estimating, measuring, and planning... so of course he's not entirely unbiased. A lot of what he does applies very well to large organizations where large and frequent pivots are nowhere near as frequent as when working for a startup while exploring the features of something that is not evolutionary, but revolutionary. Check out this article where Capers Jones shows that Agile is one of the quickest and cheapest methodologies out there.
https://www.infoq.com/articles/evaluating-agile-software-met...
And where he concludes "Overall the Agile family and the methods that emphasize speed have achieved their goal, and they are fairly quick. The methods that emphasize quality such as TSP, RUP, and CMMI 5 have also achieved their goals, and deliver very few defects. No single method appears to be a universal panacea that can be successful on every size and kind of software application."
Huh.
> I wish you'd see this thread as a discussion
You're not even willing to consider that planning too far ahead can waste time and that planning does not optimize for speed. You are giving absolutes like "in all cases". If you want a discussion then you have to be willing to learn yourself. In my experience there are times when planning is essential. And times when planning, especially out more than a few weeks, is a waste of time. If you want to trade tips, great. If you want to continue to insist planning is an absolute, no thanks.
> And where he concludes "Overall the Agile family and the methods that emphasize speed have achieved their goal, and they are fairly quick. The methods that emphasize quality such as TSP, RUP, and CMMI 5 have also achieved their goals, and deliver very few defects. No single method appears to be a universal panacea that can be successful on every size and kind of software application."
This article must be taken in the context of his research in general. If you're optimizing for speed of initial delivery, rather than TCO, then any method with less planning delivers faster and costs less. However, if you assume your software lasts longer than ~year (as pointed out in my initial reference, and his much larger work "The Economics of Software Quality") then they perform much worse. The actual payoffs in most of his research is around 8 months of development.
Being "quick" is basically a jab from Jones about agile project quality.
> You're not even willing to consider that planning too far ahead can waste time and that planning does not optimize for speed.
I'm willing to consider it, I think. My experience has shown the opposite for any project involving more than 1 person that a customer uses for more than a year. The research I can find shows me the same. Most projects I embark on tend to live in the market for longer than a year, and thus quality, TCO, and delivering on time matter.
I guess I don't see how you manage a team, a project, a customer, or investors where you need to hire people based on work you need to do, if you're only planning out a few weeks at a time. I don't see any case where you're going to get VC funding, exec sponsorship, etc (i.e. manage the risk and expectations of capital) by saying "we'll see where we are in two weeks".
This is where I start to fall into saying things about professionalism, as to me the difference between a professional and an amateur is exactly this capital and risk management.
I feel like we are getting somewhere now. For some projects speed of initial delivery is by far the most important factor. Which is why it's so important to qualify your claims instead of using absolutes like "planning always results in..."
> However, if you assume your software lasts longer than ~year (as pointed out in my initial reference, and his much larger work "The Economics of Software Quality") then they perform much worse. The actual payoffs in most of his research is around 8 months of development.
Why would you assume that? Lots of software these days lasts far less than a year, and that's intentional. This is not the 1970s when building software was an expensive and rare thing. There are literally hundreds of millions of applications out there. And billions of dollars being pumped into startups, the vast majority of which will throw out their initial idea and eventually pivot when they learn what their customers really need. Learned by building quickly and delivering something that works well enough to get it in front of people to see how they actually use it and how different that is from what you had envisioned (or planned). What comes to mind is the cardboard box that the kids play with while the new toy it came in sits ignored.
> Being "quick" is basically a jab from Jones about agile project quality.
Why would it be? Unless you are heavily biased towards software always being high quality. Which shows a lack of flexibility and a lack of professionalism about optimizing for what is important to your client or employer. If your employer needs a project to optimize for initial speed, then focusing on quality and planning is unprofessional and irresponsible and putting your own personal preferences ahead of what the company has decided it should optimize for.
> My experience has shown the opposite for any project involving more than 1 person that a customer uses for more than a year.
I'd chalk some of this up to confirmation bias. But at least this far better than "always". At least we are starting to add qualifiers like "uses for more than a year"
> I guess I don't see how you manage a team, a project, a customer, or investors where you need to hire people based on work you need to do, if you're only planning out a few weeks at a time
You manage the team by focusing on building and learning from what you build. Let's say you plan for an easy build over a few months and hire 1 person. They start building and as a result of building a new market opportunity is discovered so new requirements come to the surface. Some of these requirements are difficult to implement and will require 10 people over a few months to build. But that is where the company now wants to pivot. How did your plan for hiring 1 person help you here? I'd say it gave you a sense of false confidence. What would be more professional and frankly more honest is to say "we don't know yet". That can't go on forever of course. The entire point of favoring building over planning is that in the course of building you should be reporting back on new risks discovered. And new opportunities. It's not all downside. When you have a plan to take six months to build something and it only needs a month, well Parkinson's law is nothing new so we all know it will take six months.
Lacking a plan, and with a focus on urgency and speed, you are forced to ruthlessly eliminate requirements. Since you will get something built in a few weeks the chances of you getting the thing you needed in a month are much higher than the six month plan. The exercise of delivering working software every couple of weeks gives you laser focus on what your core offering is.
> This is where I start to fall into saying things about professionalism, as to me the difference between a professional and an amateur is exactly this capital and risk management.
If your only tool for managing risk is extensive and detailed plans that must be constantly maintained and that do a poor job of optimizing speed over quality, then you are not being as professional as you could possibly be. There are more tools available for managing capital and risk than "let's make a detailed plan".
If you are employed by a company where the culture is all about using plans to manage risk, then you've found a great fit for yourself. But you shouldn't look down your nose at alternative ways of managing risk and call them amateur. There are different types of risk and there is no silver bullet in managing them.
One alternative for managing risk is admitting that you don't know and just building the damn thing and learning from it. This method of managing risk is especially important for really new things that have never been done before. You want to fail early and often and that is the most professional thing you could possibly do on those types of projects. At least until someone comes up with something better.
I'll just end by saying thank you for making the effort to steer this to a higher quality and more nuanced conversation. I enjoyed talking about this with you, and now I have to move onto other things.
This is the main discussion of his research, to show that quality and defect removal actually lower cost and increase speed. The tradeoff between time & quality is one he regularly shows is not a tradeoff at all, even for very short timelines.
> I enjoyed talking about this with you, and now I have to move onto other things.
Ok!
TLDR: methodology won't fix your project.