How would you build a rocket if you only thought about the next 2-4 weeks?
How would you scope the size of any given project if you didn't think about it in totality?
Successful development is always a balance of waterfall and sprints except maybe when you're in maintenance mode for long living legacy projects.
Waterfall = strategy
Sprint = tactics
We're not building rockets. We're building business software. And the requirements for that are not a hardline set of rules laid out by the laws of physics. They are soft and fungible. Beholden to the whims of a product manager, designer, or client. Generally almost nothing is truly understood about the shape and function of the resulting software at the beginning of a development process. And sticking to a "plan" that was laid out before these things have actually been discovered is a recipe for disaster. Better to build in tiny, discrete chunks of usable MVP that can be used as a guidestone to the next desired level of functionality.
"plans are useless, but planning is indispensible".
If you’re actually building a rocket or any other physical thing with real world stuff (like some of us here) you need a real plan. The fits software engineers get into about agile is odd at times. I don’t really get it.
Also not every department is agile and wants those 6 months delivery plan.
You do know what the end-state is. It's called the minimum viable product, MVP. The most successful product owners are ones who are technical and can bridge the gap between the business requirements and technical implementation. For example, there would be no SpaceX and vertical landing of rockets if it wasn't for Elon Musk, who is both an engineer, a businessman, and an entrepreneur. He gets the full picture and can talk to all divisions of the company with incredible competency.
If you have a product owner who knows nothing about building software, in charge of a software product, you'll have a much smaller chance of succeeding, than if you had someone who was both business and technically savvy.
Okay, I’ll bite...
I’ll grant you the 2-4 weeks part isn’t quite right for building a rocket; but the key insight of Agile, of developing in small steps and staying flexible, definitely applies.
You do not start by designing the entire rocket before building anything. Your rocket won’t work. You start, you must start, by building a series of prototypes and exploring the design space. As you do that, you gradually refine your ideas of what’s possible and how to do it. As far as I know, every successful rocket design has followed that path.
Sure, at some point along the path, you put together the blueprints for the final design. So there’s an element of waterfall in there. But staying agile is still critical. Unexpected stuff is sure to come up and you have to retain the flexibility to deal with it.
As others have noted, software is somewhat special in that it’s cheaper and faster to build than physical stuff, can be copied and refactored much more easily, etc. And that shifts the agile-to-waterfall balance where you can realistically talk about working entirely from short term requirements.
That's a total non sequitur. Agile is all about building the sellable product. But doing it iteratively works a lot better than doing it via a big development plan.
To clarify for your sake, once you have hardware in the loop (eg mechanical engineers designing things, manufacturing scaling for production) and a large commercial organization who needs to be ready for a product launch, you need a waterfall type plan. Agile principles (customer focus, respond to change) can apply, and you can break up the work into 2 weeks, but you do need to look past the next 2-4 weeks and know where things are planned. Commitments matter in the $millions.
Software-only SAAS has the luxury of avoiding that complexity.
- building
- designing
Some parts of agile do apply to building. See Lean Manufacturing, parts of it are responses to maintain quality (Andons used to stop production) and parts are to change over the manufacturing floor to build something else (same plant can make many kinds of cars without a fixed production line for each, letting them meet demand).
Other parts of agile, though, are aimed more at the design portion. Sprints, for instance, are meant to give you short iterations that let you frequently evaluate both your process and your product's success/failure.
You can apply agile concepts to developing almost any kind of thing. Actual rocket scientists create simulations, scale models, full-scale models, prototypes and other things on their way to the final system. So they're already applying the same iterative approach that most agile proponents endorse. The question is, is the iteration deliberate or accidental? Or do they try to deny the iterations even exist? Once they get to building the rocket, that might be less agile depending on circumstances (is it the first one they've built? are they setting up a production line? is it a one off?).
Note that flexible factories have other limits. Honda can't make a large truck in their factory as there isn't room. I don't know if Honda regrets this cost or not.
> Fisher's Fundamental Theorem: The better adapted a system is to a particular environment, the less adaptable it is to new environments. -- Gerald Weinberg,"The Psychology of Computer Programming"
There are always trade offs. The more flexible you make a system, the less efficient it is (most likely) in achieving its (now increased) goals. But the more efficient you make it, the harder it is to adapt it to fit additional or changed goals. Which trade offs you accept depend on your objectives, and sometimes your ability to spin up a whole new factory.
At least often/commonly. If we don't always do that with software, I think we always do it at least to some extent (in an unusual way compared to physical things), and software is (often if not always suitable to designing and building.
I think "agile" as we know it (which is really short for "agile software development", that's what the agile manifesto hasin it's title!) is particular to something you are (or can profitably) design as you build.
While there are analagous things in physical products, and there is (or profitably can be) some element of design/build cycle in physical products, I think rarely to the extent it can and does happen when making software. Rockets are probably at the far end of separating design and build.
But the tldr version is probably just: The Agile Manifesto's full name is literally "Manifesto for Agile Software Development". While some of the things may happen to be useful or applicable in other fields of endeavor, others may not be. Saying that something from the Manifesto for Agile Software Development and ensuing methodology doesn't apply to a rocket is not a relevant critique of the methodology as applied to software, as intended.
I actually had something about that but cut it for some reason.
I'd like to point out that while the manifesto is about software, a lot of the ideas (beyond the manifesto itself and influencing the manifesto) come from other disciplines. Mary and Tom Poppendieck, for example, are major proponents of Lean Software Development, whose ideas are a translation from the world of physical manufacturing. Many of the DevOps ideas (not necessarily how it's played out) come from Eli Goldratt and his Theory of Constraints.
An interesting thing about both Lean and ToC is that neither is terribly prescriptive. They're actually generalized concepts that can apply to almost any kind of system and (mostly) lack prescriptive text saying, "Thou shalt do...". Compare that to much of the Big-A Agile movement which is inflexible, ironically, and often very prescriptive. Perhaps not down to the specific tool that should be used, but often to the specific methods that should be used (Scrum: Thou shalt have 1-4 weeks Sprints; Thou shalt have time boxed meetings of this nature; SAFe: Thou shalt have a massive meeting every 6 months in the format we describe in this chapter).
Ok, that's a bit unfair to Scrum, it's actually more flexible than that. But I'll happily burn my books on SAFe (picked up for a previous job).
Many of the other ideas for Agile are born out of Systems Engineering which had already dealt with Waterfall. You had the V model (an improvement, but still many of the flaws if taken too literally, over Waterfall) which makes V&V run concurrent to design and development and encourages feedback loops. In many ways this is akin to TDD. Incremental and iterative development processes, which are fundamentally no different from Scrum except Scrum pushes you to much shorter times and has a stricter structure. Evolutionary development which is really close to what many, if not most, agile proponents endorse (MVP, then add on features and modify the base as needed). None of that came from software (specifically), it came from developing physical and (in their parlance) cyber-physical systems.
But my rocket needs to be ready when the earth-venus system is in the right stage or I delay a few years. Agile cannot handle this. Agile can handle getting rid of some features to meet the deadline, but waterfall can do priorities as well. What is needed is watching the end date and assurance that you will make it, or warning to add people before the cost of adding people is greater than their future contributions.
> Agile can handle getting rid of some features to meet the deadline, but waterfall can do priorities as well.
Waterfall is not particularly good at handling priorities. It's actually worse, in my experience, than anything else. Waterfall is about the order of doing things (it is fundamentally linear), committing to a plan (lack of flexibility), and has minimal feedback loops (lack of responsiveness).
If you have a 2 year deadline (true deadline like in your example), Waterfall only helps you if you assume everything will go right or put in drop dead dates for lower priority things. But even then, you'll have wasted so much time in your planning and designing you probably won't be able to keep it on track within a reasonable budget extension.
What you really want to handle priorities is an incremental and iterative development model. You know your target, but you break it down into smaller increments and build it out over time. After each iteration you re-evaluate your ability to hit the target. Within each iteration you may use a Waterfall-styled approach (particularly if it's a portion that's well understood in your organization or sufficiently small) or you may extend the iterative approach by adopting an Agile approach (think 3-6 months of 1-2 week sprints).
Using your Earth-Venus example. You need X, Y, Z. You want A, B, C. You only have 2 years to deliver. In Waterfall you'd spend 3-6 months designing everything before building it. If you're smart you design it with a way to drop A, B, and C. But that's a lot of time spent up front on contingency planning and pushes you closer to 6 months. Now you're 18 months from the deadline before production even starts. Oh, and testing doesn't happen until pretty late in this process. So your production is constrained to no more than 12 months in order to allow sufficient testing time (6 months).
If you're actually smart, you create checkpoints/milestones and do it incrementally. You start work on X, Y, Z and as much of A, B, C as you can afford. After 3 months you see that X is progressing fine, Y is possibly going to miss the target, and Z is definitely going to. So you check and A is going to be done in a month, B turned out to be a cluster, and C may complete on time. Cut your losses, B is gone, the money and manpower are moved to the core mission.
Next iteration you reexamine, and find out that you C may still complete but Z is barely back on track. So you take some or all the funding and people from C and dedicate them to getting Z back on track. In the best case C can still be done, but it probably won't. We accept our losses and continue this process.
That is a fundamentally agile approach. It's about responding to the changes that come up. Waterfall is not, in itself, capable of this.
He also sites case examples of restaurants and house remodellers using scrum successfully.
Many enterprise scale business centric applications have tremendously complex business rules and overreach. One can not simply make a casual decision last minute, and expect perfect execution, towards an end result of increased business value.
EDIT: I see he is a Drupal consultant. There is a very explicit ceiling on the complexities of primarily CMS driven sites.
Why do people ask for estimates or fixed bids? It's to manage risk. And how often do projects not go over estimated budget? Even fixed bids are poor at managing risk. Lots of consulting companies will walk away from a fixed bid contract after months of work at the point they realize they will lose money on it. And honestly, what's the quality of work once the contract is no longer profitable? We've all seen it. Everything is a quick hack at that point in a desperate effort to achieve the minimum that could be defensible in court. A fixed bid project means the consulting company and the client have opposing interests: the consulting company wants to get it done as cheaply as possible and will make every shortcut they can get away with. And the client wants every last bit of quality they can squeeze out of the contract for that price. No wonder it so often ends in dissatisfaction for both sides.
What are the options for the client when the consultant says "sorry we didn't actually finish and we are out of money"? They either pay more or have to be willing to sue the consulting company. Most companies suck it up and pay more. This is so endemic to the industry that most companies put extra in the budget for software project overruns no matter how tight the fixed bid contract is.
So one way to sell agile is to help clients understand how it can help manage risk better than a fixed bid contract or always inaccurate estimates. And how it can be used so that the interests of the consulting company and the client are better aligned.
Charging per sprint is only superficially different from charging hourly/per day/per week/biweekly.
Consultants will want the project to last as long as possible. Clients will want it to be over as fast as possible.
You have to impress the client with every sprint? Just like a regular contractor must impress the client with every milestone.
How do you know how many sprints you want to pay for? So you have a budget, just like any regular old project.
You can prioritize features? But why wouldn't you do that without sprints?
What am I missing?
Agreed. Agile only puts you in a more flexible relationship. It does not fix every alignment issue. For that you need to be more creative with the contract. However it does spread the risk more evenly. And by you as the client taking on more of the risk instead of pretending an estimate or contract erased it, you will do a better job of managing the risk. Every sprint you'll decide what must be built next. Every sprint you get the opportunity to change requirements. Both of those are far better ways of managing risk than fixed price contracts or estimates.
> Charging per sprint is only superficially different from charging hourly/per day/per week/biweekly.
Not if you are delivering a working product at the end of each sprint. And yes, milestones are somewhat similar to sprints. Except as a client you've now locked yourself into what gets built and if requirements change you've got to renegotiate the contract.
> You can prioritize features? But why wouldn't you do that without sprints?
Because your fixed price contract legally "guarantees" all the listed features will be included.
> Clients will want it to be over as fast as possible.
And you can have that if each sprint ends with a working product. Up to you when the product has enough features included that you are happy with it. Or that you are out of budget. And you've managed risk aligned with the consultant instead of getting an unrealistic estimate they won't meet, or having a to head to court with a broken contract.
When we talk about building an MVP (minimum viable product), the V part is very important. It does have to be useful. It does have to look great and work great. It does have to be performant. Edge cases should not get past your QA department.
So with those constraints, about the only way you can get out a prototype in a sprint or few is by ruthlessly reducing scope until you get to the core part of your product that is adding value. That process in itself results in far better products.
My goal (that I sometimes don't achieve) is that each sprint results in a product we can ship that is better than the last iteration. Run out of budget? Company priorities change? No problem. We still have a useful feature we can ship.
Per my previous comment, this is the wrong process for plenty of products. You have to know what you are building and what the risks are, what the constraints are that you are optimizing for, and the pros and cons of different types of processes before deciding which process is the best fit. If you have a strong idea of what you are building ahead of time and you have experience building those types of things and you know there is already a market demand for those types of things, then by all means come up with a solid plan ahead of time and then execute it.
I was responding in particular to "a quick prototype in front of real users gives you far better information than any amount of internal brainstorming." Maybe it's just a semantic difference but to me "quick prototype" doesn't imply look great, work great and handle all edge cases. Perhaps you meant quick prototype that looks great, works great and handles all edge cases but I think that implication is often lost on product people. They want something deployed and don't care if all of the engineering team's fussy edge cases are handled.
Yep.
> I've just been in the position many times in my own career where product managers are pushing to get something, anything out there so they can get "real user feedback"
Me too. There are a lot of bad product managers and even companies out there. But I have worked with great ones occasionally, so it's worth describing the ideal so we know what we are aiming for.
> I was responding in particular to "a quick prototype...
You're right that I need to qualify that because it more often means what you rightly assumed based on your experience. I guess I'm just so in the habit of demanding that we reduce scope that I too easily forget that "quick" more often means the compromise was quality.
I could be wrong, but I think that was less of the case in the past. Or perhaps we just didn't know better. Building big expensive things that can kill lots of people - I'm thinking infrastructure, commercial planes, etc - comes along with the idea that we should know ahead of time how we will make it safe without it failing. We can't build a prototype of a skyscraper in a city and have it fall on people. But perhaps the way we engineer those types of things could borrow a lot more from agile than I was initially claiming.
Remember that the “build” phase of software is relatively instantaneous. The design phase is the only part anyone sees anymore. (The heroes who wove the Apollo ROM were examples of actual program builders).
Making the Saturn V static test rocket was heavily design work slowed by manufacturing. Making Apollo 11 was a process of knowing mostly what was needed and building it from start to finish.
Only software is in scope